SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yan-Shuo Liang and Wu-Jun Li*
National Key Laboratory for Novel Software Technology,
Department of Computer Science and Technology, Nanjing University, P. R. China
liangys@smail.nju.edu.cn, liwujun@nju.edu.cn
Continual learning requires the model to learn multiple tasks in a sequential order. To perform continual learning, the model must possess the abilities to maintain performance on old tasks (stability) and adapt itself to learn new tasks (plasticity). Task-agnostic problem in continual learning is a challenging problem, in which task identities are not available in the inference stage and hence the model must learn to distinguish all the classes in all the tasks. In task-agnostic problem, the model needs to learn two new objectives for learning a new task, including distinguishing new classes from old classes and distinguishing between different new classes. For task-agnostic problem, replay-based methods are commonly used. These methods update the model with both saved old samples and new samples for continual learning. Most existing replay-based methods mix the two objectives in task-agnostic problem together, inhibiting the models from achieving a good trade-off between stability and plasticity. In this paper, we propose a simple yet effective method, called loss decoupling (LODE), for task-agnostic continual learning. LODE separates the two objectives for the new task by decoupling the loss of the new task. As a result, LODE can assign different weights for different objectives, which provides a way to obtain a better trade-off between stability and plasticity than those methods with coupled loss. Experiments show that LODE can outperform existing state-of-the-art replay-based methods on multiple continual learning datasets.
Continual learning requires the model to learn multiple tasks in a sequential order [31]. However, neural network models suffer from a phenomenon called catastrophic forgetting (CF) [12], in which the performance of the network on the old tasks degrades significantly after it learns a new task. To address this challenge and enable continual learning, the model must possess the abilities to maintain performance on old tasks (stability) and adapt itself to learn new tasks (plasticity). Nevertheless, an excess of stability or plasticity can interfere with the other [39], and hence the model needs to make a trade-off between stability and plasticity [39, 31].
There exist two different kinds of problems, task-agnostic problem and task-aware problem, in continual learning. Task-agnostic problem [39] in continual learning is a challenging problem, in which task identities are not available in the inference stage and hence the model must learn to distinguish all the classes in all the tasks. In contrast, task-aware problem [28] in continual learning enables the model to get task identities in the inference stage. Therefore, the model for task-aware problem only needs to distinguish the classes belonging to the same task. The difference between task-agnostic problem and task-aware problem [28, 29] shows that the model in task-agnostic problem
*Wu-Jun Li is the corresponding author.
(a) Different learning objectives
Figure 1: (a) shows two different learning objectives for learning the new task. The left side of (b) shows the trade-off between stability and plasticity in most existing methods, which mix different learning objectives. The right side of (b) shows that our method separates the learning of a new task into two objectives and hence separately considers the trade-off between stability and plasticity for these two learning objectives. y-axis in (b) represents the model's abilities, including plasticity and stability.
(b) Stability-plasticity trade-off in different methods
needs to learn two objectives for learning a new task, including distinguishing new classes from old classes (called new/old class distinction) and distinguishing between different new classes (called new class distinction). Figure 1 (a) illustrates these two learning objectives.
Many methods have been proposed for continual learning, including regularization-based methods [45, 20, 2], expansion-based methods [34, 18, 23] and replay-based methods [3, 10, 4]. For task-agnostic problem, replay-based methods are commonly used. These methods use a memory buffer to maintain a small portion of samples from the old classes. When learning a new task, the model retrieves old samples from memory and updates the parameters with both new and old samples. As illustrated in Figure 1 (b) from the methodological perspective, most existing continual learning methods [9, 7, 43] mix the two objectives in task-agnostic problem together (discussed later in Section 3.1). But these two learning objectives may cause different degrees of forgetting in continual learning and thus different trade-off strategies between stability and plasticity are required for these two learning objectives. More specifically, if a new learning objective leads to more forgetting, a good continual learner should pay more attention to the model's stability. On the contrary, if a new learning objective leads to less forgetting, a good continual learner should pay more attention to the model's plasticity for this objective. However, when the model mixes different learning objectives together, adjusting one of the learning objectives may influence others, inhibiting the model from achieving a good trade-off between stability and plasticity.
In this paper, we propose a simple yet effective method, called loss decoupling (LODE), for task-agnostic continual learning. The main contributions of this paper are listed as follows:
- By deeply analyzing the impacts of new/old class distinction and new class distinction, we find that these two learning objectives cause different degrees of forgetting. Therefore, mixing these two objectives together is detrimental for the model to make a good trade-off between stability and plasticity.
- LODE separates the two objectives for the new task by decoupling the loss of the new task. As a result, LODE can assign different weights for different objectives, which provides a way to obtain a better trade-off between stability and plasticity than those methods with coupled loss.
- Experiments show that LODE can outperform existing state-of-the-art replay-based methods on multiple continual learning datasets.
Continual learning can be offline or online. In offline continual learning setting [17, 32, 44, 24], the model receives the entire dataset of a new task and updates its parameters multiple times over this dataset. In online continual learning setting [11, 4, 10], the data from each task is sequentially concatenated as a non-stationary data stream and each data of each task can only appear once in the
data stream. Hence, the model can only receive a mini-batch of samples from the data stream at a time and update its parameters based on this mini-batch.
Different types of methods have been proposed for continual learning, including regularization-based methods [45, 20, 2], expansion-based methods [34, 18, 23] and replay-based methods [3, 10, 4]. For task-agnostic problem, replay-based methods are commonly used. Some replay-based methods [40, 42] achieve replay by learning a generative model for generating old samples. However, learning a good generative model is challenging in some settings, like online continual learning [13]. Some replay-based methods like experience replay (ER) [9] maintain a memory buffer to save old samples and replay them with new samples. Some methods combine experience replay with knowledge distillation [15]. Specifically, these methods store either a old model [43] or the outputs of the old models [7]. Some methods [25] like experience replay with asymmetric cross entropy (ER-ACE) [8] improve ER by changing the loss of the new task so that the model can avoid large representation drift. Some methods like error-sensitive modulation experience replay (ERMER) [36] and complementary learning system experience replay (CLS-ER) [5] use an extra set of model's parameters to aggregate the knowledge of different tasks. Other methods try to search for valuable samples [3] or optimize the distribution of the memory [19, 41] to overcome forgetting. However, all these methods mix two new learning objectives in Figure 1 (a), inhibiting the models from achieving a good trade-off between stability and plasticity.
In this section, we first formulate the problem in continual learning. Then, we deeply analyze the problem of mixing different learning objectives together in existing methods. After that, we propose our method called loss decoupling (LODE), which can be used in both offline and online continual learning settings. Finally, we discuss the relation between LODE and existing methods.
Continual learning requires the model to learn from multiple tasks in a sequential order. We use  $\mathcal{D}_t = \{\pmb{x}_i^t, y_i^t\}^{N_t}$  to denote the dataset of the  $t$ -th task, where  $\pmb{x}_i^t$  denotes an input sample and  $y_i^t$  denotes its label.  $N_t$  denotes the number of samples for task  $t$ . The total number of tasks is denoted by  $T$ . When the model learns a new task, the model can obtain limited or no data from the old tasks, potentially leading to catastrophic forgetting. In this work, we consider a challenging continual learning problem called task-agnostic problem, in which task identities are not available in the inference stage. In the task-agnostic problem, when the model learns on a new task  $t$ , some new classes are presented to the model. The model must possess the abilities to maintain performance on old classes (stability) and adapt itself to learn new classes (plasticity).
Replay-based methods [16, 7] maintain a memory buffer  $\mathcal{M}$  with limited size to store a small portion of old samples. When receiving a mini-batch of new samples  $\mathcal{B}_t$  from a new task  $t$ , the model retrieves a mini-batch of samples  $\mathcal{B}_{\mathcal{M}}$  from  $\mathcal{M}$  and replays them with the new samples  $\mathcal{B}_t$  to achieve a trade-off between stability and plasticity. The losses used in most existing replay-based methods can be written as follows:
$$
\mathcal {L} = \frac {1}{| \mathcal {B} _ {t} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {t} |} \mathcal {L} _ {\text {n e w}} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {t}\right), y _ {i} ^ {t}\right) + \frac {1}{| \mathcal {B} _ {\mathcal {M}} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {\mathcal {M}} |} \mathcal {L} _ {\text {r e p}} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {\mathcal {M}}\right), y _ {i} ^ {\mathcal {M}}\right). \tag {1}
$$
Here,  $\mathcal{L}_{new}$  is the loss for the new task and is mainly for the plasticity of the model. In contrast,  $\mathcal{L}_{rep}$  is the replay loss and is mainly for the stability of the model. We follow most existing works [7, 43] and assume that  $\mathcal{L}_{new}$  is a cross-entropy loss.  $\mathcal{L}_{rep}$  usually varies for different methods. For example,  $\mathcal{L}_{rep}$  can be a cross-entropy loss [3, 9] or a combination of cross-entropy and regularization losses [7, 5, 36]. Based on Figure 1 (a), we can find that  $\mathcal{L}_{new}$  is not only related to new/old class distinction, but also related to new class distinction in the task-agnostic problem. Hence, existing methods with the loss in (1) mix the two different learning objectives (new/old class distinction and new class distinction) together.
Note that there exists a type of replay-based methods [43, 17, 14, 38] directly sampling training data from  $\mathcal{D}_t\cup \mathcal{M}$ , which makes their losses can not be written in the form shown in (1). In some challenging continual learning settings like online continual learning setting, the model cannot access
Number of Tasks
(a) Seq-CIFAR10
Number of Tasks
(b) Seq-CIFAR100
Loss Value
(c) Seq-CIFAR10
Loss Value
(d) Seq-CIFAR100
Figure 2: (a) and (b) show the variation of the first task's accuracy on different datasets. (c) and (d) show the distribution of different losses over different datasets before learning the second task. Here, the frequency corresponding to each loss value represents the number of samples with this loss value.
the entire dataset of a new task. At this time, it is impossible for the model to sample from  $\mathcal{D}_t\cup \mathcal{M}$ . On the contrary, loss in (1) can be adapted to online continual learning by sampling  $\mathcal{B}_t$  from the data stream  $\mathcal{S}$  and sampling  $\mathcal{B}_{\mathcal{M}}$  from  $\mathcal{M}$  separately. Hence, in this work, we focus on the methods whose loss function has the form in (1). In Section 4 and Appendix, we also compare our method with the methods that sample training data from  $\mathcal{D}_t\cup \mathcal{M}$ .
3.2 Analyzing Learning Objectives by Decoupling Loss
When learning the  $t$ -th task, we assume there are  $m$  old classes, which are denoted as  $\mathcal{C}_o = \{1,2,\dots,m\}$ . We also assume there are  $n$  new classes, which are denoted as  $\mathcal{C}_n = \{m + 1,m + 2,\dots,m + n\}$ . Then, for a given new sample  $(x,y)$  where  $y\in \mathcal{C}_n$ , we can compute its loss  $\mathcal{L}_{new}$  as
$$
\mathcal {L} _ {\text {n e w}} \left(f _ {\boldsymbol {\Theta}} (\boldsymbol {x}), y\right) = \ell_ {c e} \left(f _ {\boldsymbol {\Theta}} (\boldsymbol {x}), y\right) = - \log \left(\frac {\exp \left(o _ {y}\right)}{\sum_ {i = 1} ^ {m + n} \exp \left(o _ {i}\right)}\right). \tag {2}
$$
Here,  $\ell_{ce}(\cdot)$  denotes the cross-entropy loss.  $[o_1,o_2,\dots,o_m,o_{m + 1},\dots,o_{m + n}]$  denotes the logits output by  $f_{\Theta}(\pmb {x})$  . We decouple the loss  $\mathcal{L}_{new}(f_{\Theta}(\pmb {x}),y)$  according to the two learning objectives in Figure 1 (a):
$$
\begin{array}{l} \mathcal {L} _ {\text {n e w}} \left(f _ {\Theta} (\boldsymbol {x}), y\right) = - \log \left(\frac {\exp \left(o _ {y}\right)}{\sum_ {i = m + 1} ^ {m + n} \exp \left(o _ {i}\right)}\right) - \log \left(\frac {\sum_ {i = m + 1} ^ {m + n} \exp \left(o _ {i}\right)}{\sum_ {i = 1} ^ {m + n} \exp \left(o _ {i}\right)}\right) \tag {3} \\ = \ell_ {c e} (f _ {\Theta} (\boldsymbol {x}), y; \mathcal {C} _ {n}) + \ell_ {n} (f _ {\Theta} (\boldsymbol {x})). \\ \end{array}
$$
Here, we use  $\ell_{ce}(:,\mathcal{C}_n)$  to denote the cross-entropy loss restricted to new classes  $\mathcal{C}_n$ . Obviously,  $\ell_{ce}(f_{\Theta}(\pmb{x}),y;\mathcal{C}_n)$  is related to new class distinction.  $\ell_n(f_{\Theta}(\pmb{x}))$  is related to new/old class distinction. Note that both  $\ell_{ce}(f_{\Theta}(\pmb{x}),y;\mathcal{C}_n)$  and  $\ell_n(f_{\Theta}(\pmb{x}))$  are for the plasticity of the model and may cause catastrophic forgetting. Furthermore, loss  $\ell_{ce}(f_{\Theta}(\pmb{x}),y;\mathcal{C}_n)$  and loss  $\ell_n(f_{\Theta}(\pmb{x}))$  have the same weight in (1) due to the coupling property.
We use experience replay (ER) [9], which is one of the most popular replay-based methods and can be expressed in the form of (1), to evaluate the impact of  $\ell_{ce}(\cdot; \mathcal{C}_n)$  and  $\ell_n(\cdot)$ . Specifically, we first let the model learn on the first task through valina stochastic gradient descent [21]. Then, before learning the subsequent tasks, we remove one of the two losses in (3) from ER and analyze the forgetting of the first task. The experiments are conducted on two datasets Seq-CIFAR10 and Seq-CIFAR100, which will be introduced in Section 4. The experimental settings also follow the descriptions in Section 4.1. Figure 2 (a) and Figure 2 (b) show the accuracy of the first task when the model learns subsequent tasks. Here, 'Remove  $\ell_{ce}(\cdot; \mathcal{C}_n)$ ' means that we remove  $\ell_{ce}(\cdot; \mathcal{C}_n)$  from (1) after learning the first task. 'Remove  $\ell_n(\cdot)$ ' means that we remove  $\ell_n(\cdot)$  from (1) after learning the first task. The results show that removing  $\ell_n(\cdot)$  results in less forgetting of the first task than removing  $\ell_{ce}(\cdot; \mathcal{C}_n)$ . In other words,  $\ell_n(\cdot)$  leads to more forgetting than  $\ell_{ce}(\cdot; \mathcal{C}_n)$ .
It is intuitively reasonable that  $\ell_n(\cdot)$  leads to more forgetting than  $\ell_{ce}(\cdot; \mathcal{C}_n)$ . First, since replay-based methods usually keep limited samples in memory, when the model learns a new task, it has access to much fewer samples from the old classes than from the new classes. Therefore, utilizing loss  $\ell_n(\cdot)$  to learn to distinguish between new classes and old classes introduces a risk of biasing the model towards the new classes, potentially leading to serious catastrophic forgetting. In contrast, loss
$\ell_{ce}(\cdot ;\mathcal{C}_n)$  is independent of the old classes, thereby avoiding introducing a risk of biasing the model towards the new classes. Second, in Figure 2 (c) and Figure 2 (d), we also show the value of the losses  $\ell_{ce}(\cdot ;\mathcal{C}_n)$  and  $\ell_n(\cdot)$  before using ER to learn task 2. We can find that the value of  $\ell_n(\cdot)$  is larger than the value of  $\ell_{ce}(\cdot ;\mathcal{C}_n)$  on both Seq-CIFAR10 and Seq-CIFAR100. This finding is consistent with existing works [8, 36], which suggest that larger losses for the new task may result in larger feature drift, leading to more forgetting. In Appendix, we also provide the TSNE visualizations when either  $\ell_n(\cdot)$  or  $\ell_{ce}(\cdot ;\mathcal{C}_n)$  is removed, further confirming that  $\ell_n(\cdot)$  leads to more forgetting than  $\ell_{ce}(\cdot ;\mathcal{C}_n)$ .
Since  $\ell_n(\cdot)$  leads to more forgetting than  $\ell_{ce}(\cdot; \mathcal{C}_n)$ , treating these two losses equally is not reasonable. In particular, based on the above analysis, we can find that a good continual learner should assign a larger weight to  $\ell_{ce}(\cdot; \mathcal{C}_n)$  and a smaller weight to  $\ell_n(\cdot)$ . However, loss in (1) fails to achieve this goal due to the coupling property.
3.3 Loss Decoupling for Continual Learning
Section 3.2 has demonstrated the impact of different learning objectives on the model's forgetting and the issue of the coupling property in existing replay-based methods. To address this issue, we propose a new method called loss decoupling (LODE), which removes the coupling property present in existing methods.
Specifically, our LODE uses the following loss to perform continual learning:
$$
\mathcal {L} = \frac {1}{| \mathcal {B} _ {t} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {t} |} \left(\beta_ {1} \ell_ {c e} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {t}\right), y _ {i} ^ {t}; \mathcal {C} _ {n}\right) + \beta_ {2} \ell_ {n} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {t}\right), y _ {i} ^ {t}\right)\right) + \frac {1}{| \mathcal {B} _ {\mathcal {M}} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {\mathcal {M}} |} \mathcal {L} _ {\text {r e p}} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {\mathcal {M}}\right), y _ {i} ^ {\mathcal {M}}\right). \tag {4}
$$
Here,  $\beta_{1}$  and  $\beta_{2}$  are two coefficients that control the weight of the two different learning objectives. The finding in Section 3.2 shows that LODE should set  $\beta_{2}$  to be smaller than  $\beta_{1}$ , to make the model achieve a better trade-off between stability and plasticity than existing methods usually with coupled loss. Furthermore, since  $\ell_{n}(\cdot)$  is for new/old class distinction, LODE sets  $\beta_{2}$  proportional to the ratio  $\frac{|\mathcal{C}_n|}{|\mathcal{C}_o|}$  to make the model not bias toward old or new classes. In contrast, since  $\ell_{ce}(\cdot;\mathcal{C}_n)$  is only related to the new classes, LODE sets  $\beta_{1}$  to be a constant value. More specifically, LODE sets  $\beta_{1}$  and  $\beta_{2}$  as
$$
\beta_ {1} = C, \quad \beta_ {2} = \rho \frac {\left| \mathcal {C} _ {n} \right|}{\left| \mathcal {C} _ {o} \right|}. \tag {5}
$$
Here,  $C$  and  $\rho$  are two hyperparameters. Note that when the number of tasks increases, the number of old classes also increases. In particular, when the number of old tasks is large, the number of old classes  $|\mathcal{C}_o|$  is usually much larger than the number of new classes  $|\mathcal{C}_n|$ . At this time,  $\beta_{2}$  is much smaller than  $\beta_{1}$ . Setting  $\beta_{2}$  to be as large as  $\beta_{1}$ , or setting  $\beta_{1}$  to be as small as  $\beta_{2}$  fails to make the model achieve a good trade-off between stability and plasticity, which will be verified in the experiment.
Note that we do not specify the form of  $\mathcal{L}_{rep}$  in (4). Therefore, our method can be combined with many replay-based methods with form (1) and improve these methods. Here, we give some examples which combine LODE with different state-of-the-art continual learning methods.
Combining LODE with ER and DER++ Both experience replay (ER) [9] and dark experience replay++ (DER++) [7] can be written in the form of (1). Therefore, the combinations of our LODE with these two methods are direct. Specifically, given a new batch of samples  $\mathcal{B}_t$ , the model computes  $\ell_{ce}(\cdot; \mathcal{C}_n)$  and  $\ell_n(\cdot)$  according to (3). Then, the model computes  $\mathcal{L}_{rep}$  with the old samples  $\mathcal{B}_{\mathcal{M}}$  through the specific formulation in ER or DER++. Finally, the model can get the loss (4).
Combining LODE with ESMER A recent method called error sensitive modulation experience replay (ESMER) [36] suggests that the model should learn more from smaller losses to avoid large feature drift. The loss in ESMER is slightly different from the loss in (1). Specifically, ESMER assigns different weights to different new samples in  $\mathcal{B}_t$  according to their loss values. Although the loss in ESMER is slightly different from the loss in (1), it still mixes the two different learning objectives together. Therefore, we can combine ESMER with our LODE through a similar form
Algorithm 1 Loss Decoupling (LODE) for Continual Learning
1: Input: a sequence of tasks with datasets  $\{\mathcal{D}_1,\dots,\mathcal{D}_T\}$ , a neural network model  $f_{\Theta}(\cdot)$ .
2: Output: a learned neural network model  $f_{\Theta}(\cdot)$ .
3: while Get a mini-batch of samples  $\mathcal{B}_t$  from a task  $t$  do
4: Sample a mini-batch  $\mathcal{B}_{\mathcal{M}}$  from memory  $\mathcal{M}$
5: Specify the weights for the two different learning objectives by (5);
6: Get the losses for learning objective through (3);
7: Compute the final loss through (4).
8: Perform backward propagation and update the model  $f_{\Theta}(\cdot)$  through SGD;
9: Update memory  $\mathcal{M}$  with  $\mathcal{B}_t$  through some memory update methods;
to (4). The decoupled loss for ESMER can be written as follows:
$$
\mathcal {L} = \frac {1}{| \mathcal {B} _ {t} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {t} |} w _ {i} \left(\beta_ {1} \ell_ {c e} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {t}\right), y _ {i} ^ {t}; \mathcal {C} _ {n}\right) + \beta_ {2} \ell_ {n} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {t}\right), y _ {i} ^ {t}\right)\right) + \frac {1}{| \mathcal {B} _ {\mathcal {M}} |} \sum_ {i = 1} ^ {| \mathcal {B} _ {\mathcal {M}} |} \mathcal {L} _ {\text {r e p}} \left(f _ {\Theta} \left(\boldsymbol {x} _ {i} ^ {\mathcal {M}}\right), y _ {i} ^ {\mathcal {M}}\right). \tag {6}
$$
Here,  $w_{i}$  is the weight assigned to new sample  $i$  in ESMER. We can find that the loss in (6) not only modulates weights for different new samples but also modulates weights for the two different learning objectives.
Algorithm 1 gives the whole learning process of LODE.
3.4 Relation with Existing Methods
The loss in many existing methods has the coupling property like the loss in (1). However, some methods, from the perspective of this work, use the losses that show a certain degree of decoupling property. For example, the loss in ER-ACE [8] can be written as