Year: 2021

Abstract:
When an agent encounters a continual stream of new tasks in the lifelong learning setting, it leverages the knowledge it gained from the earlier tasks to help learn the new tasks better. In such a scenario, identifying an efficient knowledge representation becomes a challenging problem. Most research works propose to either store a subset of examples from the past tasks in a replay buffer, dedicate a separate set of parameters to each task or penalize excessive updates over parameters by introducing a regularization term. While existing methods employ the general task-agnostic stochastic gradient descent update rule, we propose a task-aware optimizer that adapts the learning rate based on the relatedness among tasks. We utilize the directions taken by the parameters during the updates by additively accumulating the gradients specific to each task. These task-based accumulated gradients act as a knowledge base that is maintained and updated throughout the stream. We empirically show that our proposed adaptive learning rate not only accounts for catastrophic forgetting but also exhibits knowledge transfer. We also show that our method performs better than several state-of-the-art methods in lifelong learning on complex datasets. Moreover, our method can also be combined with the existing methods and achieve substantial improvement in performance.

heading: 1 INTRODUCTION
text: Lifelong learning (LLL), also known as continual learning, is a setting where an agent continuously learns from data belonging to different tasks (Parisi et al., 2019). Here, the goal is to maximize performance on all the tasks arriving in a stream without replaying the entire datasets from past tasks (Riemer et al., 2018). Approaches proposed in this setting involve investigating the stability-plasticity dilemma (Mermillod et al., 2013) in different ways where stability refers to preventing the forgetting of past knowledge and plasticity refers to accumulating new knowledge by learning new tasks (Mermillod et al., 2013; De Lange et al., 2019).
Unlike human beings, who can efficiently assess the correctness and applicability of the past knowledge (Chen & Liu, 2018), neural networks and other machine learning models often face various issues in this setting. Whenever data from a new task arrives, these models often tend to forget the previously obtained knowledge due to dependency on the input data distribution, limited capacity, diversity among tasks, etc. This leads to a significant drop in performance on the previous tasks - also known as catastrophic forgetting (McCloskey & J. Cohen, 1989; Robins, 1993).
Recently there has been an ample amount of research proposed in LLL (De Lange et al., 2019). Several methods, categorized as Parameter Isolation methods, either freeze or add a set of parameters as their task knowledge when a new task arrives. Another type of methods, known as Regularization-based methods, involve an additional regularization term to tackle the stability-plasticity dilemma. There are approaches based on approximate Bayesian inference, where parameters are sampled from a distribution, that suggest
controlling the updates based on parameter uncertainty (Blundell et al., 2015; Adel et al., 2019; Ahn et al., 2019). But these approaches are computationally expensive and often depend on the choice of prior (Zenke et al., 2017; Nguyen et al., 2018).
Another class of methods, namely Replay-based methods, store a subset of examples from each task in a replay buffer. These methods apply gradient-based updates that facilitate a high-level transfer across different tasks through the examples from the past tasks that are simultaneously available while training. As a result, these methods tend to frame LLL as an i.i.d. setting. While replay-based methods are currently state-of-the-art in several LLL tasks, it is important that we explore various ways to tackle the original non-i.i.d. problem (Hadsell et al., 2020). Hence the focus of this paper is to design efficient replay-free methods for LLL.
While adaptive gradient descent based optimizers such as Adam (Kingma & Ba, 2014) have shown superior performance in the classical machine learning setup, many existing works in LLL employ the conventional stochastic gradient descent for parameter update (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Farajtabar et al., 2020). Adaptive gradient descent based optimizers accumulate gradients to regulate the magnitude and direction of updates but often struggle when the dataset arrives in a non-i.i.d. manner, from different tasks, etc. These optimizers tend to ‘over-adapt’ on the most recent batch and hence suffer from poor generalization performance (Keskar & Socher, 2017; Chen et al., 2018; Mirzadeh et al., 2020).
While exploiting the adaptive nature of the gradient descent based optimizers, we alleviate catastrophic forgetting by introducing Task-based Accumulated Gradients (TAG) that is a wrapper around existing optimizers. The key contributions of our work are as follows:
• We define a task-aware adaptive learning rate for the parameter update step that is also aware of relatedness among tasks in LLL.
• As the knowledge base, we propose to additively accumulate the directions (or gradients) that the network took while learning a specific task instead of storing past examples.
• We empirically show that our method prevents catastrophic forgetting and exhibits knowledge transfer if the tasks are related without introducing more parameters.
Our proposed method, described in Section 3, achieves state-of-the-art results and outperforms several Replay-free methods on complex datasets like Split-miniImageNet, Split-CUB, etc. For smaller episodic memory, our method also outperforms the Replay-based methods as shown in Section 4. Note that we propose a new approach to optimizing based on the past gradients and as such it could potentially be applied along with the existing LLL methods including replay-based methods. We demonstrate the effectiveness of doing the same in the experiments.

heading: 2 RELATED WORK
text: Methods proposed in LLL are broadly categorized into three classes: Regularization-based, Parameter Isolation and Replay-based methods (De Lange et al., 2019; Masana et al., 2020). Regularization-based methods prevent a drastic change in the network parameters as the new task arrives to mitigate forgetting. These methods further are classified as data-focused (Li & Hoiem, 2017; Triki et al., 2017) and prior-focused methods (Nguyen et al., 2018; Ebrahimi et al., 2019). In particular, Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), a prior-focused method, regularizes the loss function to minimize changes in the parameters important for previous tasks. Yet, when the model needs to adapt to a large number of tasks, the interference between task-based knowledge is inevitable with fixed model capacity. Parameter Isolation methods (Rusu et al., 2016; Xu & Zhu, 2018; Serra et al., 2018) such as (Aljundi et al., 2017) assign a model copy to every new task that arrives. These methods alleviate catastrophic forgetting in general, but they rely on a strong base network and work on a small number of tasks. Another closely related methods, called Expansion-based methods, handle the LLL problem by expanding the model capacity in order to
adapt to new tasks (Sodhani et al., 2018; Rao et al., 2019). Li et al. (2019) propose to learn task-specific model structures explicitly while retaining model primitives sharing, decoupling from model parameter estimation. Replay-based methods maintain an ‘episodic memory’, containing a few examples from past tasks, that is revisited while learning a new task (Riemer et al., 2018; Jin et al., 2020). For instance, Averaged Gradient Episodic Memory (A-GEM) (Chaudhry et al., 2018b), alleviating computational inefficiency of GEM (Lopez-Paz & Ranzato, 2017), uses the episodic memory to project the gradients based on hard constraints defined on the episodic memory and the current mini-batch. Experience Replay (ER) (Chaudhry et al., 2019) uses both replay memory and input mini-batches in the optimization step by averaging their gradients to mitigate forgetting.
Task-relatedness (Li & Hoiem, 2017; Jerfel et al., 2019; Shaker et al., 2020) or explicitly learning task representations (Yoon et al., 2017) is also an alternative approach studied in LLL. Efficient Lifelong Learning Algorithm (ELLA) (Ruvolo & Eaton, 2013) maintains sparsely shared basis vectors for all the tasks and refines them whenever the model sees a new task. Rao et al. (2019) perform dynamic expansion of the model while learning task-specific representation and task inference within the model. Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2020) maintains a space based on a subset of gradients from each task. As a result, OGD often faces memory issues during run-time depending upon the size of the model and the subset (Bennani & Sugiyama, 2020). Unlike OGD, we accumulate the gradients and hence alleviate the memory requirements by orders for each task.
A recent work (Mirzadeh et al., 2020) argues that tuning the hyper-parameters gives a better result than several state-of-the-art methods including A-GEM and ER. They introduce Stable SGD that involves an adjustment in the hyper-parameters like initial learning rate, learning rate decay, dropout, and batch size. They present this gain in performance on simplistic benchmarks like Permuted MNIST (Goodfellow et al., 2013), Rotated MNIST and Split-CIFAR100 (Mirzadeh et al., 2020). Another related work (Gupta et al., 2020) also employs an adaptive learning rate while requiring a small episodic memory. But it is based on a meta-learning setting and hence beyond the scope of our paper.

heading: 3 METHOD

heading: 3.1 LIFELONG LEARNING SETUP
text: In this section, we introduce the notations and the LLL setup used in the paper. We focus on the standard task-incremental learning scenario which is adopted in the numerous state-of-the-art LLL methods. It involves solving new tasks using an artificial neural network with a multi-head output where each head is associated with a unique task and the task identity is known beforehand (Lopez-Paz & Ranzato, 2017; van de Ven & Tolias, 2019). We denote the current task as t and any of the previous tasks by τ . The model receives new data of the form {X(t), D(t), Y (t)} where X(t) are the input features, D(t) is the task descriptor (that is a natural number in this work) and Y (t) is the target vector specific to the task t.
We consider the ‘single-pass per task’ setting in this work following (Lopez-Paz & Ranzato, 2017; Riemer et al., 2018; Chaudhry et al., 2019). It is more challenging than the multiple pass setting used in numerous research works (Kirkpatrick et al., 2017; Rebuffi et al., 2017). The goal is to learn a classification model f(X(t); θ), parameterized by θ ∈ RP to minimize the loss L(f(X(t); θ), Y (t)) for the current task t while preventing the loss on the past tasks from increasing. We evaluate the model on a held-out set of examples of all the tasks (≤ t) seen in the stream.

heading: 3.2 TASK-BASED ACCUMULATED GRADIENTS
text: The specific form of our proposed method depends on the underlying adaptive optimizer. For ease of exposition, we describe it as a modification of RMSProp (Tieleman & Hinton, 2012) here and call it TAG-
RMSProp. The TAG versions of other methods such as the Adagrad (Duchi et al., 2011) and Adam are available in Appendix A.1. A Naive RMSProp update, for a given learning rate η, looks like the following:
Vn = βVn−1 + (1− β)g2n; 1 ≤ n ≤ N
θn+1 = θn − η√
Vn + gn
(1)
where θn is the parameter vector at step n in the epoch, gn is the gradient of the loss, N is the total number of steps in one epoch, Vn is the moving average of the square of gradients (or the second moment), and β is the decay rate. We will use TAG-optimizers as a generic terminology for the rest of the paper.
We maintain the second moment V (t)n for each task t in the stream and store it as the knowledge base. When the model shifts from one task to another, the new loss surface may look significantly different. We argue that by using the task-based second moment to regulate the new task updates, we can reduce the interference with the previously optimized parameters in the model. We define the second moment V (t)n for task t for TAG-RMSProp as: V (t)n = β2V (t) n−1 + (1− β2)g2n where β2 is constant throughout the stream. We use V (t) n to denote a matrix that stores the second moments from all previous tasks, i.e., V(t)n = {V (1) N , ..., V (t−1) N , V (t) n } of size (t × P ). Hence, the memory required to store these task-specific accumulated gradients increases linearly as the number of tasks in the setting. Note that each V (τ)N (where τ < t) vector captures the gradient information when the model receives data from a task τ and does not change after the task τ is learned. It helps in regulating the magnitude of the update while learning the current task t. To alleviate the catastrophic forgetting problem occurring in the Naive RMSProp, we replace Vn (in Eq. 1) to a weighted sum of V(t)n . We propose a way to regulate the weights corresponding to V(t)n for each task in the next section.

heading: 3.3 ADAPTIVE LEARNING RATE
text: Next, we describe our proposed learning rate that adapts based on the relatedness among tasks. We discuss how task-based accumulated gradients can help regulate the parameter updates to minimize catastrophic forgetting and transfer knowledge.
We first define a representation for each task to enable computing correlation between different tasks. We take inspiration from a recent work (Guiroy et al., 2019) which is based on a popular meta-learning approach called Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017). Guiroy et al. (2019) suggest that with reference to given parameters θs (where s denotes shared parameters), the similarity between the adaptation trajectories (and also meta-test gradients) among the tasks can act as an indicator of good generalization. This similarity is defined by computing the inner dot product between adaptation trajectories. In the experiments, Guiroy et al. (2019) show an improvement in the overall target accuracy by adding a regularization term in the outer loop update to enhance the similarity.
In case of LLL, instead of a fixed point of reference θs, the parameters continue to update as the model adapts to a new task. Analogous to the adaptation trajectories, we essentially want to capture those task-specific gradient directions in the LLL setting. Momentum serves as a less noisy estimate for the overall gradient direction and hence approximating the adaptation trajectories. The role of momentum has been crucial in the optimization literature for gradient descent updates (Ruder, 2016; Li et al., 2017). Therefore, we introduce the task-based first moment M (t)n in order to approximate the adaptation trajectories of each task t. It is essentially the momentum maintained while learning each task t and would act as the task representation for computing the correlation.
The M (t)n is defined as: M (t) n = β1M (t) n−1 + (1− β1)gn where β1 is the constant decay rate. Intuitively, if the current task t is correlated with a previous task τ , the learning rate in the parameter update step should be higher to encourage the transfer of knowledge between task t and τ . In other words, it should allow
knowledge transfer. Whereas if the current task t is uncorrelated or negatively correlated to a previous task τ , the new updates over parameters may cause catastrophic forgetting because these updates for task t may point in the opposite direction of the previous task τ ’s updates. In such a case, the learning rate should adapt to lessen the effects of the new updates. We introduce a scalar quantity αn(t, τ) to capture the correlation that is computed using M (t)n and M (τ) N :
αn(t, τ) = exp(−b M
(t) n T M (τ) N
|M (t)n ||M (τ)N | ) (2)
where |.| is the Euclidean norm. Existing adaptive optimizers, such as Adam, tend to overfit on the most recent dataset from a task, which results in catastrophic forgetting in LLL. By using the exponential term, the resulting αn(t, τ) will attain a higher value for uncorrelated tasks and will minimize the new updates (hence prevent forgetting). Here, b is a hyperparameter that tunes the magnitude of αn(t, τ). The higher the b is, the greater is the focus on preventing catastrophic forgetting. Its value can vary for different datasets. For the current task t at step n (with θ(t)1 = θ (t−1) N+1 ), we define the TAG-RMSProp update as:
θ (t) n+1 = θ (t) n − η√ αn(t, t) V (t) n +
t−1∑ τ=1 αn(t, τ) V (τ) N + gn (3)
Hence, the role of each αn(t, τ) is to regulate the influence of corresponding task-based accumulated gradient V
(τ) N of the previous task τ . Since we propose a new way of looking at the gradients, our update rule (Eq. 3) can be applied with any kind of task-incremental learning setup. In this way, the overall structure of the algorithm for this setup remains the same.

heading: 4 EXPERIMENTS
text: We describe the experiments performed to evaluate our proposed method.1 In the first experiment, we show the gain in performance by introducing TAG update instead of naive optimizers update. We analyse how our proposed learning rate adapts and achieves a higher accuracy over the tasks in the stream. Next, we compare our proposed replay-free method with other state-of-the-art baselines and also show that TAG update (in Eq. 3) can be used along with other state-of-the-art methods to improve their results.
The experiments are performed on four benchmark datasets: Split-CIFAR100, Split-miniImageNet, SplitCUB and 5-dataset. Split-CIFAR100 and Split-miniImageNet splits the CIFAR-100 (Krizhevsky et al., 2009; Mirzadeh et al., 2020) and Mini-imagenet (Vinyals et al., 2016; Chaudhry et al., 2019) datasets into 20 disjoint 5-way classification tasks. Split-CUB splits the CUB (Wah et al., 2011) dataset into 20 disjoint tasks with 10 classes per task. 5-dataset is a sequence of five different datasets as five 10-way classification tasks. These datasets are: CIFAR-10 (Krizhevsky et al., 2009), MNIST (LeCun, 1998), SVHN (Netzer et al., 2011), notMNIST (Bulatov, 2011) and Fashion-MNIST (Xiao et al., 2017). More details about the datasets are given in Appendix A.2. For experiments with Split-CIFAR100 and Split-miniImageNet, we use a reduced ResNet18 architecture following (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019). We use the same reduced ResNet18 architecture for 5-dataset. For Split-CUB, we use a ResNet18 model which is pretrained on Imagenet dataset (Deng et al., 2009) as used in (Chaudhry et al., 2019).
We report the following metrics by evaluating the model on the held-out test set: (i) Accuracy (Lopez-Paz & Ranzato, 2017) i.e., average test accuracy when the model has been trained sequentially up to the latest task,
1Code for the experiments is submitted as supplementary material and will be released publicly upon acceptance.
(ii) Forgetting (Chaudhry et al., 2018a) i.e., decrease in performance of each task from their peak accuracy to their accuracy after training on the latest task and (iii) Learning Accuracy (LA) (Riemer et al., 2018) i.e., average accuracy for each task immediately after it is learned. The overall goal is to maximise the average test Accuracy. Further, a LLL algorithm should also achieve high LA while maintaining a low value of Forgetting because it should learn the new task better without compromising its performance on the previous tasks (see Appendix A.2).
We report the above metrics on the best hyper-parameter combination obtained from a grid-search. The overall implementation of the above setting is based on the code provided by (Mirzadeh et al., 2020). The details of the grid-search and other implementation details corresponding to all experiments described in our paper are given in Appendix A.2.1. For all the experiments described in this section, we train the model for a single epoch per task. Results for multiple epochs per task are given in Appendix A.3.4. All the performance results reported are averaged over five runs.

heading: 4.1 NAIVE OPTIMIZERS
text: We validate the improvement by our proposed setting over the gradient descent based methods and demonstrate the impact of using correlation among tasks in the TAG-optimizers. Firstly, we train the model on a stream of tasks using Naive SGD update without applying any specific LLL method. Similarly, we replace the SGD update with Naive Adagrad, Naive RMSProp, Naive Adam and their respective TAG-optimizers to compare their performances. We show the resulting Accuracy (%) (in Fig. 1a) and Forgetting (in Fig. 1b) when the model is trained in with the above-mentioned optimizers.
It is clear that TAG-optimizers outperform their naive counterparts as well as Naive SGD for all four datasets by a significant amount. There is a notable decrease in Forgetting by TAG-optimizers (in Fig. 1b) in general that eventually reflects on the gain in final test Accuracy as seen in Fig. 1a. In Split-CUB, TAG-Adam (57%) shows a remarkable improvement in accuracy when compared to Naive Adam (45%) such that it even surpasses Naive SGD (55%). Interestingly, TAG-Adam results in slightly lower accuracy as compared to TAG-Adagrad except in Split-miniImageNet. Moreover, Naive Adagrad results in a better performance than Naive RMSProp and Naive Adam for all the datasets. This observation aligns with the results by Hsu et al. (2018). Naive SGD performs almost equivalent to Naive Adagrad except in 5-dataset where it is outperformed.
Next, we analyse α(t, τ) which is the average of αn(t, τ) across all steps n for all t and τ when stream is finished i.e., α(t, τ) = 1N ∑N n=1 αn(t, τ). We show how α(t, τ) values play role in the gain in TAG-
optimizers accuracies in case of Split-CUB dataset. Each plot in Fig. 2 corresponds to test accuracies at,τ (with shaded areas indicating their standard deviations) for Naive RMSProp (blue) and TAG-RMSProp (red) for a particular τ for all tasks t in the stream (x-axis). Along with that, the grey-coloured curves are maxτ ′ αn(t, τ
′) (top, dashed line), Eτ ′ [α(t, τ ′)] (middle, solid line) and minτ ′ α(t, τ ′) (bottom, dashed line) respectively. These are shown along with the corresponding α(t, τ) to indicate the rank of α(t, τ) in the set {α(t, τ ′); τ ′ ∈ [1, t]} computed the model encounters a task t in the stream. The accuracies of TAG-RMSProp and Naive RMSProp appear to correlate for most of the stream. We mark the regions (black dashed ellipse) of subtle improvements in the accuracy by TAG-RMSProp that is later maintained throughout the stream. While observing α(t, τ) in those regions particularly, we note that the rank of α(t, τ) affects the accuracy in TAG-RMSProp as following: (i) Lower (or decrease in) rank of α(t, τ) means that there exists some correlation between the tasks t and τ . So, the model should take advantage of the current task updates and seek (or even amplify) backward transfer. Such observations can be made for the following (t, τ): (4, 2), (5, 2), (14, 2), (9, 5), (14, 6), (19, 6) etc. Our method also prevents drastic forgetting as well in few cases. For example: (7, 2), (12, 2), (10, 6). (ii) Higher (or increase in) rank of α(t, τ) results in prevention of forgetting as observed in Fig. 2. Such (t, τ) pairs are (11, 2), (7, 5), (12, 5), (12, 6), etc. It also results in backward transfer as observed in (11, 5) and (15, 5). We report the same analysis for the other three datasets and show the results in Appendix A.3.3.

heading: 4.2 COMPARED WITH OTHER BASELINES
text: In the next experiment, we show that the TAG-RMSProp results in a strong performance as compared to other LLL algorithms. In Table 1, we report the performance of TAG-RMSProp and the following state-of-the-art baselines: EWC (Kirkpatrick et al., 2017), A-GEM (Chaudhry et al., 2018b), ER (Aljundi et al., 2019) with reservoir sampling and Stable SGD (Mirzadeh et al., 2020). Apart from these baselines, we report the performance of Naive SGD and Naive RMSProp from the previous section. We also report results on multi-task learning (MTL) settings on all four datasets where the dataset from all the tasks is always available throughout the stream. Hence, the resulting accuracies of the MTL setting serve as the upper bounds for the test accuracies in LLL.
Following Mirzadeh et al. (2020), the size of the episodic memory for both A-GEM and ER is set to store 1 example per class. Since we want to evaluate TAG with all other baselines on the original non-i.i.d. problem, we keep the episodic memory size in the replay-based methods small for the comparison. We still report
A-GEM and ER results with bigger memory sizes in Appendix A.3.5. The size of the mini-batch sampled from the episodic memory is set equal to the batch-size to avoid data imbalance while training. Although we utilize a similar amount of memory as (Rusu et al., 2016), an expansion-based method, we do not make any changes to the size of the model and thus the number of parameters remains the same during the test time. Hence, we do not compare our approach with the expansion-based method in our experiments.
From the results reported in Table 1, we observe that TAG-RMSProp achieves the best performance in terms of test Accuracy as compared to other baselines for all datasets. The overall improvement is decent in Split-CIFAR100, Split-miniImageNet and Split-CUB which are 3.65%, 2.5% and 2.3% with regard to the next-best baseline. On the other hand, the improvement by TAG-RMSProp is relatively minor in 5-dataset i.e., 1% as compared to ER with the similar amount of Forgetting (0.29 and 0.28) occurring in the stream. In terms of LA, TAG-RMSProp achieves almost similar performance as Naive RMSProp in Split-CUB and 5-dataset. We also note that the LA of TAG-RMSProp is higher in Split-CIFAR100, Split-CUB and 5-dataset than ER and A-GEM. The higher LA with similar Forgetting as compared to other baselines shows that while TAG exploits the adaptive nature of existing optimizers, it also ensures minimal forgetting of the gained knowledge. The existing optimizers tend to aggressively fit the model on the most recent task at an immense cost of forgetting the earlier tasks. Hence, even if a similar (or lower) Forgetting occurs in TAG, the higher test Accuracy (with high LA) shows that TAG is capable of retaining the gained knowledge from each task. Although LA is lower in Split-miniImageNet, TAG-RMSProp manages to prevent catastrophic forgetting better than these methods and hence results in a higher test Accuracy.

heading: 4.3 COMBINED WITH OTHER BASELINES
text: Lastly, we show that the existing baselines can also benefit from our proposed method TAG-RMSProp. We replace the conventional SGD update from EWC, A-GEM and ER, and apply RMSProp update (Eq. 1) and
TAG-RMSProp update (Eq. 3) respectively. We use the same task-incremental learning setup as used in the previous sections in terms of architecture and hyper-parameters. We compare the resulting accuracies of the baselines with their RMSProp and TAGed versions in Fig. 3.
For a given dataset, we see that gain in the final accuracy in the TAGed versions is similar for the baselines described in Section 4.2. That is, TAG improves these baselines with SGD update on Split-CIFAR100, Split-miniImageNet, Split-CUB and 5-dataset by at least 8%, 4%, 4% and 9% respectively. On the other hand, TAG improves the baselines with RMSProp update on the datasets by at least 12%, 12%, 7% and 9% respectively. The improvement is also significant in A-GEM with bigger episodic memory (i.e., 10 samples per class or M = 10) but we observe relatively smaller improvement (2%) by TAGed ER (M = 10) as compared to ER (M = 10). These results show that apart from outperforming the baselines independently (with smaller episodic memory in replay-based methods), TAG can also be used as an update rule in the existing research works for improving their performances. While A-GEM and ER are strong baselines for LLL, we would like to highlight that these replay-based methods are not applicable in settings where storing examples is not an option due to privacy concerns. TAG-RMSProp would be a more appropriate solution in such settings.

heading: 5 CONCLUSION
text: We propose a new task-aware optimizer for the LLL setting that adapts the learning rate based on the relatedness among tasks. We introduce the task-based accumulated gradients that act as the representation for individual tasks for the same. We conduct experiments on complex datasets to compare TAG-RMSProp with several state-of-the-art methods. Results show that TAG-RMSProp outperforms the existing methods in terms of final accuracy with a commendable margin without storing past examples or using dynamic architectures. We also show that it results in a significant gain in performance when combined with other baselines. To the best of our knowledge, ours is the first work in the LLL literature showing that we can use an adaptive gradient method for LLL and prevent forgetting better than Naive SGD. For future work, as the memory required to store the task-specific accumulated gradients increases linearly with the tasks, reducing memory complexity without compromising the performance can be an interesting direction. This can be achieved by (i) computing correlation using a smaller quantity than the task-based first moments, and (ii) clustering the similar tasks together to reduce the number of task-based second moments (in settings with a soft margin between the tasks). Another possible direction from here can be shifting to a class-incremental scenario where the task identity is not known beforehand and is required to be inferred along the stream.

heading: A APPENDIX
text: In this document, we provide the details and results excluded from the main paper. In A.1, we describe the TAG versions of Adagrad and Adam. The implementation details are described in Section A.2. We also report results obtained by performing additional experiments in Section A.3.

heading: A.1 TAG-OPTIMIZERS
text: Similar to RMSProp (in Section 3), with the task-based first moments M (t)n , let W (t) n = {αn(t, τ); τ ∈ [1, t]} (from Eq. 2) and,
W (t)n V (t) n = V (1) n ; t = 1
αn(t, t) V (t) n + t−1∑ τ=1 αn(t, τ) V (τ) N ; t > 1
(4)
We define the TAG versions of Adagrad and Adam as following:
• TAG-Adagrad:
V (t)n = V (t) n−1 + g 2 n
θ (t) n+1 = θ (t) n − η√ W (t) n Vn + gn (5)
• TAG-Adam:
V (t)n = β2V (t) n−1 + (1− β2)g2n
θ (t) n+1 = θ (t) n −
η √ 1− βn2
(1− βn1 ) √ W (t) n Vn + M (t)n (6)
Both TAG-Adagrad (Eq. 5) and TAG-Adam (Eq. 6) result in a significant gain in Accuracy and prevent Forgetting as observed in Fig. 1a and Fig. 1b respectively in Section 4.1.
A.2 IMPLEMENTATION DETAILS
The summary of the datasets used in the experiments is shown in Table 2 and Table 3.
In 5-dataset, we convert all the monochromatic images to RGB format depending on the task dataset. All images are then resized to 3× 32× 32. The overall training and test data statistics of 5-dataset are described in Table 3.
Details about the metrics used for evaluating the model:
• Accuracy (Lopez-Paz & Ranzato, 2017): If at,τ is the accuracy on the test set of task τ when the current task is t, it is defined as, At = 1t t∑
τ=1 at,τ .
• Forgetting (Chaudhry et al., 2018a): It is the average forgetting that occurs after the model is trained on all tasks. If the latest task is t and is defined as, Ft = 1t−1 t−1∑ τ=1 maxt′∈{1,...,t−1}(at′,τ − at,τ ).
• Learning Accuracy (LA) (Riemer et al., 2018): It is the measure of learning capability when the model sees a new task. For the current task t, it is defined as, Lt = 1t t∑
τ=1 aτ,τ .
We implement the following baselines to compare with our proposed method:
• EWC: Our implementation of EWC is based on the original paper (Kirkpatrick et al., 2017).
• A-GEM: We implemented A-GEM based on the the official implementation provided by (Chaudhry et al., 2018b).
• ER (Chaudhry et al., 2019): Our implementation is based on the one provided by (Aljundi et al., 2019) with reservoir sampling except that the sampled batch does not contain examples from the current task.
• Stable SGD (Mirzadeh et al., 2020): We obtain the best hyper-parameter set by performing gridsearch over different combinations of the learning rate, learning rate decay, and dropout (see Appendix A.2.1).
OGD Farajtabar et al. (2020) requires storing N (=200 in their experiments) number of gradients per task and it is evaluated only on variants of the MNIST dataset by training a small feed-forward network. On the other hand, TAG additively accumulates the gradients and hence requires memory equal to two copies of the model as the knowledge base. This enabled us to train a reduced ResNet18 on complex datasets. Due to greater memory requirements, OGD faced memory errors in our setting. We would also like to highlight that OGD use Naive-SGD and our contribution being an adaptive learning rate based method is complementary to this approach.
We provide our code as supplementary material that contains the scripts for reproducing the results from all experiments described in this paper. In the CODE folder, we include README.MD file that contains the overall code structure, procedure for installing the required packages, links to download the datasets and steps to execute the scripts. All experiments were executed on an NVIDIA GTX 1080Ti machine with 11 GB GPU memory.

heading: A.2.1 HYPER-PARAMETER DETAILS
text: In this section, we report the grid search details for finding the best set of hyper-parameters for all datasets and baselines. We train the model with 90% of the training set and choose the best hyper-parameters based on the highest accuracy on the validation set which consists of remaining 10% for the training set. For existing baselines, we perform the grid search either suggested by the original papers or by Farajtabar et al. (2020). For all TAG-optimizers, β1 is set to 0.9. For TAG-RMSProp and TAG-Adagrad, β2 is set to 0.99 and for TAG-Adam it is 0.999. In all the experiments, the mini-batch size is fixed to 10 for Split-CIFAR100, Split-miniImageNet, Split-CUB similar to (Chaudhry et al., 2019; Mirzadeh et al., 2020). We set mini-batch size to 64 for 5-dataset following (Serra et al., 2018). This is because we wanted to highlight the role of learning rate and to show how TAG-RMSProp improves the performance while the other hyper-parameters (including batch-size) were fixed.
• Naive SGD
– Learning rate: [0.1 (Split-CIFAR100, 5-dataset), 0.05 (Split-miniImageNet), 0.01(SplitCUB), 0.001]
• Naive Adagrad
– Learning rate: [0.01, 0.005 (Split-CIFAR100, Split-miniImageNet, 5-dataset), 0.001, 0.0005 (Split-CUB), 0.0001]
• Naive RMSProp – Learning rate: [0.01, 0.005 (Split-CIFAR100), 0.001 (Split-miniImageNet, 5-dataset), 0.0005, 0.0001 (Split-CUB), 0.00005, 0.00001] • Naive Adam
– Learning rate: [0.01, 0.005 (Split-CIFAR100), 0.001 (Split-miniImageNet, 5-dataset), 0.0005, 0.0001 (Split-CUB)]
• TAG-Adagrad – Learning rate: [0.005 (Split-CIFAR100, 5-dataset), 0.001 (Split-miniImageNet), 0.0005
(Split-CUB), 0.00025, 0.0001] – b: [1, 3, 5 (Split-CIFAR100, Split-miniImageNet, Split-CUB), 7 (5-dataset)]
• TAG-RMSProp – Learning rate: [0.005, 0.001, 0.0005 (5-dataset), 0.00025 (Split-CIFAR100), 0.0001 (Split-
miniImageNet), 0.00005, 0.000025 (Split-CUB), 0.00001] – b: [1, 3, 5 (Split-CIFAR100, Split-miniImageNet, Split-CUB), 7 (5-dataset)]
• TAG-Adam – Learning rate: [0.005, 0.001 (5-dataset), 0.0005 (Split-CIFAR100), 0.00025 (Split-
miniImageNet), 0.0001 (Split-CUB)] – b: [1, 3, 5 (Split-CIFAR100, Split-miniImageNet, Split-CUB), 7 (5-dataset)]
• EWC – Learning rate: [0.1 (Split-CIFAR100, 5-dataset), 0.05 (Split-miniImageNet), 0.01(Split-
CUB), 0.001] – λ (regularization): [1 (Split-CIFAR100, Split-miniImageNet, Split-CUB), 10, 100 (5-
dataset)] • A-GEM
– Learning rate: [0.1 (Split-CIFAR100, Split-miniImageNet, 5-dataset), 0.05, 0.01(SplitCUB), 0.001]
• ER – Learning rate: [0.1 (Split-CIFAR100, 5-dataset), 0.05 (Split-miniImageNet), 0.01(SplitCUB), 0.001] • Stable SGD
– Initial learning rate: [0.1 (Split-CIFAR100, Split-miniImageNet, 5-dataset), 0.05 (SplitCUB), 0.01]
– Learning rate decay: [0.9 (Split-CIFAR100, Split-miniImageNet, Split-CUB), 0.8, 0.7 (5- dataset)]
– Dropout: [0.0 (Split-miniImageNet, Split-CUB, 5-dataset), 0.1 (Split-CIFAR100), 0.25, 0.5]
In case of TAG-RMSProp, we empirically found that the best performance of the all three benchmarks with 20 tasks occurred when hyper-parameter b = 5 and for 5-dataset, b = 7. We also found that a lower value of Learning rate in TAG-RMSProp results in a better performance. These empirical observations can reduce the search space for hyperparameter setup by a huge amount when applying TAG-RMSProp on a LLL setup.
For the experiments in Section 4.3 that require a hybrid version of these methods, we use the same hyperparameters from above except for TAGed ER in Split-CIFAR100 (Learning rate = 0.0005) and Split-CUB (Learning rate = 0.0001). We choose the learning rates of TAG-RMSProp and Naive-RMSProp over EWC, A-GEM and ER.

heading: A.3 ADDITIONAL EXPERIMENTS
text: In this section, we describe the additional experiments and analysis done in this work.

heading: A.3.1 BACKWARD TRANSFER METRIC
text: While we show the occurrence of knowledge transfer in Fig. 2, we can quantify the Backward Transfer (BWT) (Chaudhry et al., 2019) by computing the difference between the final Accuracy and LA. i.e., BWT = 1t−1 ∑t−1 τ=1 at,τ − aτ,τ . We report the BWT results for all datasets and baselines in Table 4.
While TAG-RMSProp outperforms the other baselines in terms of BWT for Split-miniImageNet, it is overall the second-best method for Split-CIFAR100 and 5-dataset. In case of Split-CUB, even if TAG-RMSProp achieves the highest Accuracy, it results in a lower BWT because of a significantly higher LA as compared to the other baselines (see Table 1).

heading: Methods Split-CIFAR100 Split-miniImageNet
text: TAG-RMSProp (Ours) 61.58 (±1.24) −9.99 (±1.62) 62.59 (±1.82) −23.49 (±1.73)
A.3.2 COMPARING TAG-RMSProp WITH OTHER BASELINES
Fig. 4 provides a detailed view of the test accuracies of individual baseline as the model encounters the new tasks throughout the LLL stream in Split-CIFAR100, Split-miniImageNet and Split-CUB. At the starting task t = 1, TAG-RMSProp beats other baselines because of lower initial learning rates (see Appendix A.2.1) and reflects the performance gain by the RMSProp over SGD optimizer. In all three datasets, the performance of TAG-RMSProp is very similar to ER specially from task t = 5 to task t = 15, but ultimately improves as observed at t = 20. These results show a decent gain in the final test Accuracy by TAG-RMSProp as compared to other baselines.
Although TAG-RMSProp is outperformed by ER from task t = 2 to t = 4, it results in the highest final accuracy as compared to other methods on 5-dataset . TAG-RMSProp also outperforms other baselines including Stable SGD and A-GEM by a large margin.
A.3.3 EVOLUTION OF α(t, τ) AND TEST ACCURACY at,τ
Next, we continue the analysis done in Section ?? for Split-CIFAR100 (in Fig. 5), Split-miniImageNet (in Fig. 6), Split-CUB (in Fig. 7) for the first 9 tasks and 5-dataset (in Fig. 8) for first 3 tasks. In SplitCIFAR100 and Split-miniImageNet, the model with Naive RMSProp tends to forget the task t by significant amount as soon as it receives the new tasks. On the other hand, TAG-RMSProp prevents catastrophic forgetting and hence results in a higher accuracy throughout the stream. We can observe that for SplitCIFAR100 and Split-miniImageNet, α(τ + 1, τ) (where τ ∈ [1, 9]) generally have a higher rank in the set
{α(t, τ ′); τ ′ ∈ [1, t]}. This is because TAG-RMSProp also recognizes an immediate change in the directions when the model receives a new task (from M (t−1)N to M (t) n ). A similar observation is made in case of Split-CUB but the visible gain in the accuracy by TAG-RMSProp does not occur instantly. Apart from that, we observe that the lower and higher rank of α(t, τ) results in backward transfer and prevents catastrophic forgetting respectively in the stream. Overall, in all datasets, we arrive at the same conclusion obtained in Section ??.

heading: A.3.4 MULTIPLE-PASS PER TASK
text: In this section, we report the performance of TAG-RMSProp and all other baselines discussed in Section 4.2 for 5 epochs per task in Table 5. Hyper-parameters for this experiment are kept the same as the single-pass per task setting. TAG-RMSProp results in high average Accuracy in all the datasets. We also observe less amount
of Forgetting in TAG-RMSProp as compared to other baselines. In terms of Learning Accuracy, TAGRMSProp is outperformed by the other baselines in Split-CIFAR100, Split-miniImageNet and 5-dataset but performs better in Split-CUB.

heading: A.3.5 BIGGER MEMORY SIZE IN REPLAY-BASED METHODS
text: We also compare the performance of A-GEM and ER with a larger number of samples per class (M) in the episodic memory for all four datasets in Table 6. With M = 10, total episodic memory size for SplitCIFAR100, Split-miniImageNet, Split-CUB and 5-dataset becomes 1000, 1000, 2000 and 500 respectively. We observe ER results in a significant gain in the performance as the episodic memory size increases. But TAG-RMSProp is able to outperform A-GEM in Split-CIFAR100, Split-miniImageNet and Split-CUB with a large margin even when M is set to 10.

Review summaries:
In this paper, the authors propose a new optimization method for continual learning. The authors propose a task-aware optimizer to adapt the learning rate for each task. The proposed method is evaluated on several datasets to show its effectiveness. 

This paper proposes TAG, a method for continual learning in the task-incremental setting. This method relies on storing and using task gradients while learning a set of supervisd tasks sequentially. The influence task-based accumulated gradients is regulated through a learning rate that is adaptive according to the relatedness of the current task with the previously observed ones. The authors report results on benchmark datasets for continual learning of up to 20 disjoint tasks. Comparisons against naive non-continual optimizers such as SGD, Adam and RMSProp are reported, along with results in the continual learning setting with some state-of-the-art methods such as EWC, A-GEM and ER. The authors report performance in terms of overall accuracy, forward transfer and learning accuracy (LA).

The authors present an approach for lifelong learning where each task is processed in a single pass. They propose to adapt the learning rate of the training algorithm depending on the current task's similarity to the previously observed tasks. The learning rate is decreased when there are many dissimilar tasks to avoid catastrophic forgetting. The paper presents an adapted RMSProp algorithm, but the procedure can be adjusted to Adagrad and Adam as well. The presented evaluation on computer vision datasets shows that the proposed modification helps to increase the final accuracy of those methods while keeping the forgetting low. When compared to baselines from literature under the conditions of the single pass setting, the proposed method achieves better final accuracy and keeps approximately the same forgetting rate. 

This paper proposes a  task-aware adaptive learning rate method, TAG, for continual learning. The optimizer TAG is to promote the learning rate if they are similar to previous tasks, while decreasing the learning rates if they are dissimilar to previous tasks without storing previous examples. The authors combined the proposed method with existing optimizers, such as Adam, SGD, etc to demonstrate the effectiveness of the proposed method. Experimental results on several datasets show the improvements over naive optimizers.