TAG: Task-based Accumulated Gradients for Lifelong learning
Authors: Pranshu Malviya
Balaraman Ravindran
Sarath Chandar
Venue: ICLR 2022 Submitted (2022)
Keywords: Lifelong Learning
Continual Learning
Catastrophic forgetting

Abstract: When an agent encounters a continual stream of new tasks in the lifelong learning setting, it leverages the knowledge it gained from the earlier tasks to help learn the new tasks better. In such a scenario, identifying an efficient knowledge representation becomes a challenging problem. Most research works propose to either store a subset of examples from the past tasks in a replay buffer, dedicate a separate set of parameters to each task or penalize excessive updates over parameters by introducing a regularization term. While existing methods employ the general task-agnostic stochastic gradient descent update rule, we propose a task-aware optimizer that adapts the learning rate based on the relatedness among tasks. We utilize the directions taken by the parameters during the updates by additively accumulating the gradients specific to each task. These task-based accumulated gradients act as a knowledge base that is maintained and updated throughout the stream. We empirically show that our proposed adaptive learning rate not only accounts for catastrophic forgetting but also exhibits knowledge transfer. We also show that our method performs better than several state-of-the-art methods in lifelong learning on complex datasets. Moreover, our method can also be combined with the existing methods and achieve substantial improvement in performance.

Under review conference paper at ICLR 2022
TAG: TASK-BASED ACCUMULATED GRADIENTS FOR LIFE- LONG LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
When an agent encounters continual stream new tasks in the lifelong learning setting, it leverages the knowledge it gained from the earlier tasks to help learn the new tasks better. In such a scenario, identifying an efficient knowledge representation becomes a challenging problem. Most research works propose to either store subset of examples from the past tasks in a replay buffer, dedicate a separate set of parameters to each task or penalize excessive updates over parameters by introducing regularization term. While existing methods employ the general task-agnostic stochastic gradient descent update rule, we propose task-aware optimizer that adapts the learning rate based on the relatedness among tasks. We utilize the directions taken by the parameters during the updates by additively accumulating the gradients specific each task. These task-based accumulated gradients act as knowledge base that is maintained and updated throughout the stream. We empirically show that our proposed adaptive learning rate not only accounts for catastrophic forgetting but also exhibits knowledge transfer. We also show that method performs better than several state-of-the-an methods in lifelong learning on complex datasets. More- over, our method can also be combined with the existing methods and achieve substantial improvement in performance.
1 INTRODUCTION
Lifelong learning (LLL), also known as continual learning, is a setting where an agent continuously learns from data belonging to different tasks (Parisi al., 2019). Here, the goal to maximize performance on all the tasks arriving in stream without replaying the entire datasets from past tasks (Riemer et al.) (2088).
Approaches proposed in this setting involve investigating the stability-plastice
dilemma (Mermillod et
2013) in different ways where stability refers to preventing the forgetting of past knowledge and plasticity
refers accumulating new knowledge by learning new tasks (Mermillod et 2013 De Lange et al. 2019).
Unlike human beings, who can efficiently assess the correctness and applicability of the past knowledge (Chen & Liu 2018), neural networks and other machine learning models often face various issues in this setting. Whenever data from new task arrives, these models often tend to forget the previously obtained knowledge due to dependency on the input data distribution, limited capacity, diversity among tasks, etc. This leads to significant drop in performance the previous tasks also known catastrophic forgetting
(McCloskey & J. Cohen 1989 Robins 1993).
Recently there has been ample amount of research proposed in LLL DDe Lange et al. 2019). Several methods, categorized as Parameter Isolation methods, either freeze or add set of parameters as their task knowledge when involve an additional regularization term to tackle the based on approximate Bayesian inference, where parameters sampled from distribution, that suggest
task arrives. Another type of methods, known Regularization-bassee methods, dilemma. There are approaches

---

Under review conference paper at ICLR 2022
controlling the updates based on parameter uncertainty (Blundell et 2015 Adel al. 2019 Ahn et al.
2019). But these approaches are computationally expensive and often depend on the choice of prior (Zenke et
2017
Nguyen et al. 2018).
Another class of methods, namely Replay-based methods, store a subset of examples from each task in a replay buffer. These methods apply gradient-based updates that facilitate a high-level transfer across different tasks through the examples from the past tasks that are simultaneously available while training. As result, these methods tend to frame I.I.I as an i.i.d. setting. While replay-based methods are currently state-of-the-art in several LLL tasks, it is important that explore various ways to tackle the original non-ii.d. problem (Hadsell et al. 2020). Hence the focus of this paper is to design efficient replay-free methods for LLL.
While adaptive gradient descent based optimizers such Adam (Kingma & Ba 2014) have shown superior performance in the classical machine learning setup, many existing works in LLL employ the conventional
stochastic gradient descent for parameter update (Lopez-Paz & Ranzato 2017: Chaudhry et al. 2019] Farajtabar et al. 2020). Adaptive gradient descent based optimizers accumulate gradients to regulate the magnitude and direction of updates but often struggle when the dataset arrives in non-i.i.d. manner, from different tasks, etc. These optimizers tend to 'over-adapt" on the most recent batch and hence suffer from poor generalization performance (Keskar & Socher 2017 Chen et al. 2018 Mirzadeh et al. 2020).
While exploiting the adaptive nature of the gradient descent based optimizers, we alleviate catastrophic forgetting by introducing Task-based Accumulated Gradients (TAG) that is a wrapper around existing optimizers. The key contributions of our work are as follows:
We define awske adaptive learning rate for the parameter update step that is also aware of relatedness among tasks in LLL.
As the knowledge base, we propose to additively accumulate the directions (or gradients) that the network took while learning specific task instead of storing past examples.
We empirically show that our method prevents catastrophic forgetting and exhibits knowledge transfer the tasks are related without introducing more parameters.
Our proposed method, described in Section 3. achieves ttate-of-the-art results and outperforms several Replay-free methods on complex datasets like Split-minilmageNett Split-CUB, etc. For smaller episodic memory, our method also outperforms the Replay- bbsee methods as shown in Section Note that we propose a new approach to optimizing based on the past gradients and as such could potentially be applied along with the existing I.I.I. methods including replay-based methods. We demonstrate the effectiveness of doing the same in the experiments.
2 RELATED WORK
Methods proposed in LLL are broadly categorized into three classes: Regularization-basee Parameter Isolation and Replay-based methods (De Lange et al. 22199; Masana et al. 2020). Regularization- based methods prevent drastic change in the network parameters as the new task arrives to mitigate forgetting.
These methods further are classified as data-focused (Li & Hoiem 2017 Triki et al. 2017) and frccu-focused methods (Nguyen et al. 2018 Ebrahimi al., 2019). In particular, Elastic Weight Consolidation (EWC) (Kirkpatrick et al.) 2017) a frouu-focused method, regularizes the loss function to minimize changes in the parameters important for previous tasks. Yet, when the model needs to adapt to a large number of tasks, the interference between task-based knowledge is inevitable with fixed model capacity. Parameter Isolation methods (Rusu et al.] 2016; Xu & Zhu| 2018 Serra et al. (018)) such as (Aljundi et al., 2017) assign a model copy to every new task that arrives. These methods alleviate catastrophic forgetting in general, but they rely on a strong base network and work on a small number of tasks. Another closely related methods, called Expansion- based methods, handle the LLL problem by expanding the model capacity in order
2

---

Under review conference paper at ICLR 2022
adapt new tasks (Sodhani al. 2018 Rao et al. 2019). Li et al. (2019) propose learn task-specific model structures explicitly while retaining model primitives sharing, decoupling from model parameter estimation. Replay-band methods maintain an 'episodic memory', containing few examples from past tasks, that is revisited while learning a new task (Riemer et al. 2018 Jin et al. (020).. For instance, Averaged Gradient Episodic Memory (A-GEM) (Chaudhry et al. 2018b). alleviating computational inefficiency of GEM (Lopez-Paz & Ranzato 2017), uses the episodic memory to project the gradients based on hard constraints defined on the episodic memory and the current mini-batch. Experience Replay (ER) (Chaudhry et al. 2019) uses both replay memory and input mini-batches in the optimization step by averaging their gradients to
mitigate forgetting.
Task-relatedness (Li & Hoiem 2017 Jerfel et al. 2019 Shaker et al., 2020) explicitly learning task representations (Yoon et al. [201] is also an alternative approach studied in LLL. Efficient Lifelong Learning Algorithm (ELLA) (Ruvolo & Eaton 2013) maintains sparsely shared basis vectors for all the tasks and refines them whenever the model sees a new task. Rao et al. (2019) perform dynamic expansion of the model while learning task-specific representation and task inference within the model. Orthogonal Gradient Descent (OGD) (Farajtabar et al. (2200) maintains a space based on a subset of gradients from each task. As a result, OGD often faces memory issues during run-time depending upon the size of the model and the subset (Bennani & Sugiyama 2020). Unlike OGD, we accumulate the gradients and hence alleviate the memory requirements by orders for each task.
A recent work (Mirzadeh et al., 2020) argues that tuning the hyper-panee parameters gives better result than several state-of-the-art methods including A-GEM and ER. They introduce Stable SGD that involves an adjustment in the prper-paraeeters like initial learning rate, learning rate decay, dropout, and batch size. They present this gain in performance on simplistic benchmarks like Permuted MNIST (Goodfellow et al. 2013), Rotated MNIST and Split-CIFAR100 (Mizaadeh et al. (220)). Another related work (Gupta et al. 2020 also employs an adaptive learning rate while requiring a small episodic memory. But it is based on a meta-learning setting and hence beyond the scope of our paper.
3 METHOD
3.1 LIFELONG LEARNING SETUP
In this section, we introduce the notations and the LLL setup used in the paper. We focus on the standard task-incremental learning scenario which is adopted in the numerous state-of-the-art LLL methods. It involves solving new tasks using an artificial neural network with a multi-head output where each head is associated with unique task and the task identity is known beforehand (Lopez-Paz Pazzz-Paz& & Ranzato 2017 van de Ven & Tolias 2019). We denote the current task as / and any of the previous tasks by T. The model receives new data of the form (x D(t), ( where (1) are the input features, D is the task descriptor (that is a natural number in this work) and (1) is the target vector specific to the task t.
We consider the "single-pass per task' setting in this work following (Lopez-Paz & Ranzato 2017: Riemer et al. 2018 Chaudhry et al.] 2019). is more challenging than the multiple pass setting used in numerous research works (Kirkpatrick et al. 2017: Rebuffi al.) 2017). The goal is to Iearn classification model ((x("); 0), parameterized by 0 E R to minimize the loss (x) 0), (+)) for the current task t while preventing the loss on the past tasks from increasing. We evaluate the model on a held-out set of examples of all the tasks (< t) seen in the stream.
3.2 TASK-BASED ACCUMULATED GRADIENTS
The specific form of our proposed method depends on the underlying adaptive optimizer. For ease of exposition, describe it as modification of RMSProp (Tieleman & Hinton! 2012) here and call it TAG-
3

---

Under review conference paper at ICLR 2022
RMSProp. The TAG versions of other methods such as the Adagrad (Duchi et al. 201 and Adam are available in Appendix A.1 A Naive RMSProp update, for a given learning rate 77 looks like the following:
V =BV (1 3) 1 isnsN
"
0n+ 0n, VVn + 9n.
where 8₂ the parameter vector at step n in the epoch, In is the gradient of the loss, N is the total number of steps in one epoch. V, is the moving average of the square of gradients (or the second moment), and , is the decay rate. We will use TAG-optimizers as a generic terminology for the rest of the paper.
We maintain the second moment vi(() for each task é in the stream and store it as the knowledge base. When the model shifts from one task to another, the new loss surface may look significantly different. We argue that by using the task-based second moment to regulate the new task updates, we can reduce the interference with the previously optimized parameters in the model. We define the second moment Vi(") for task for n-1
RMSProp as: Vi(*) = 32 3 + (1 10---p -22g9 where 32 is constant throughout the stream. We use v(t) to
denote a matrix that stores the second moments from all previous tasks, i.e., vi( in (v(") (((-1) V of size (1 x P). Hence, the memory required to store these task-specific accumulated gradients increases
linearly as the number of tasks the setting. Note that each ( (where r < t) vector captures the gradient information when the model receives data from a task and does not change after the task is learned. It helps in regulating the magnitude of the update while learning the current task t. To alleviate the catastrophic forgetting problem occurring in the Naive RMSProp, we replace V, (in Eq. 1 to a weighted sum of v((). We propose a way to regulate the weights corresponding to V(!) for each task in the next section.
3.3 ADAPTIVE LEARNING RATE
Next, we describe our proposed learning rate that adapts based on the relatedness among tasks. We discuss how task-based accumulated gradients can help regulate the parameter updates to minimize catastrophic forgetting and transfer knowledge.
We first define representation for each task to enable computing correlation between different tasks. We take inspiration from a recent work (Guiroy et al. (019) which is based on popular meta-learning approach called Model-Agnostic Meta-Learning (MAML) (Finn et al. 2017). Guiroy et al. (2019) suggest that with reference to given parameters 0* (where s denotes shared parameters), the similarity between the adaptation trajectories (and also meta-test gradients) among the tasks act as indicator of good generalization. This similarity is defined by computing the inner dot product between adaptation trajectories. In the experiments, Guiroy al. (2019) show an improvement in the overall target accuracy by adding a regularization term in the outer loop update to enhance the similarity.
In case of LLL.,, instead of fixed point of reference 08, the parameters continue update as the model adapts to a new task. Analogous to the adaptation trajectories, we essentially want to capture those task-specific gradient directions in the LLL setting. Momentum serves less noisy estimate for the overall gradient direction and hence approximating the adaptation trajectories. The role of momentum has been crucial in the optimization literature for gradient descent updates (Ruder, 2016 Li et al., 2017). Therefore, we introduce the task-based first moment M((") in order to approximate the adaptation trajectories of each task t. is essentially the momentum maintained while learning each task t and would act as the task representation for computing the correlation.
The M((") is defined as: M. Bg M the current task t is correlated with a previous task т, the learning rate in the parameter update step should be higher to encourage the transfer of knowledge between task t and In other words, it should allow
31 )r, where 31 is the constant decay rate. Intuitively, if
4

---

Under review conference paper at ICLR 2022
knowledge transfer. Whereas if the current task is uncorrelated or negatively correlated previous task T the new updates over parameters may cause catastrophic forgetting because these updates for task may point in the opposite direction of the previous task 7's updates. In such case, the learning rate should adapt to lessen the effects of the new updates. We introduce scalar quantity or,, (t) to capture the correlation that is computed using 1 and !
M M (t, =np(,))=exp(-b | IM
(2)
where 1.] is the Euclidean norm. Existing adaptive optimizers, such Adam, tend to overfit on the most recent dataset from a task, which results in catastrophic forgetting LLL. By using the exponential term, the resulting o,, (1. r) will attain a higher value for uncorrelated tasks and will minimize the new updates (hence prevent forgetring). Here, b is hyperparameter that tunes the magnitude of a (4, r). The higher the is, the greater is the focus preventing catastrophic forgetting. Its value vary for different datasets. For the
current task at step n (with g()) ANTT)). we define the TAG-RMSProp update as:
( n-
O())
9n.
Jantt (t, ( + an (t, ((r) te
Hence, the role of each or, 74, is regulate the influence of corresponding task-based accumulated gradient ((r) of the previous task r. Since we propose new way of looking at the gradients, our update rule (Eq. 3 can be applied with any kind of task-incremental learning setup. In this way, the overall structure of the
algorithm for this setup remains the same.
4 EXPERIMENTS
We describe the experiments performed evaluate our proposed method In the first experiment, we show the gain in performance by introducing TAG update instead of naive optimizers update. We analyse how our proposed learning rate adapts and achieves higher accuracy over the tasks in the stream. Next, we compare our proposed replay-free method with other state-of-the-art baselines and also show that TAG update (in Eq. 3 can used along with other state-of-the-ae methods to improve their results.
The experiments are performed on four benchmark datasets: Split-CIFARE CUB and 5-dataset. Split-CIFARIOn and malii-minilmageNet splits the CIFAR-100 (Krizhevsky et al. 2009 Mirzadeh et al. (020) and Mini-imagenet (Vinyals al. 2016 Chaudhry et al. 2019) datasets into 20 disjoint 5-way classification tasks. Split-CUB splits the CUB (Wah et al. 2011 dataset into 20 disjoint tasks with 10 classes per task. 5-dataset is sequence of five different datasets as five 10-way classification
Split-minilmageNet, Split-
tasks. These datasets are: CIFAR-10 (Krizhessky et al.) (2009), MNIST (LeCun) 1998). SVHN (Netzer et al. 2011), notMNIST (Bulatov 201 1) and MAshion-MNIST (Xiao
2017). More details about the datasets are given in Appendix A.2 For experiments with Split-CIFA R100 100 and Split-minilmageNer we
(XiSo et al.
use a reduced ResNet18 architecture following (Lopez-Paz & Ranzato 2017 Chaudhry al., 2019). We use
the same reduced ResNet 8 architecture for 5-dataset. For Split-CUB, we use a ResNet18 model which is pretrained on Imagenet dataset (Deng al. 2009) as used in (Chaudhry et al. 2019)
We report the following metrics by evaluating the model on the held-out test set: (i) Accuracy Papezz-Paz & Ranzato 2017) i.e., average test accuracy when the model has been trained sequentially up the latest task,
Cooee for the experiments submitted supplementary material and will be released publicly upon acceptance.
5

---

Under review conference paper at ICLR 2022
(ii) Forgetting (Chaudhry et al. 2018a i.e., decrease in performance of each task from their peak accuracy to their accuracy after training on the latest task and (iii) Learning Accuracy (LA) (Riemer et al. 2018) i.e., average accuracy for each task immediately after it learned. The overall goal is to maximise the average test Accuracy. Further, LLL algorithm should also achieve high LA while maintaining low value of Forgetting because it should learn the new task better without compromising its performance on the previous tasks (see Appendix/A.2) A.22)
We report the above metrics on the best hyper-paramenn combination obtained from grid-search. The overall implementation of the above setting is based on the code provided by (Mirzadeh et al. (020). The details of the grid-search and other implementation details corresponding to all experiments described in our paper are given in Appendix [A.2I For all the experiments described in this section, we train the model for single epoch per task. Results for multiple epochs per task are given in Appendix AA3.4 All the performance results reported are averaged over five runs.
4.1 NAIVE OPTIMIZERS
We validate the improvement by our proposed setting over the gradient descent based methods and demonstrate the impact of using correlation among tasks in the TAG-optimizers, Firstly, we train the model on stream of tasks using Naive SGD update without applying any specific LLL method. Similarly, we replace the SGD update with Naive Adagrad, Naive RMSProp. Naive Adam and their respective TAG-optimizers to compare their performances. We show the resulting Accuracy ( (%) ) (in Fig. Tab and Forgetting (in Fig. [bb when the model is trained in with the above-mentiones optimizers.
85-
Nahv Adagmd
TAG Adug
halve ressivo
0.3
TAG RMSSTrp
Nalie isee
Adamn
45-
75
C6R2300
52.5
cA
o
200
(a) Accuracy (%)
(b) Forgetting
Figure 1: Final average test Accuracy (%) (higher is better) and Forgetting (lower is better) obtained after the stream is finished for all four datasets. The vertical bars with hatches are the performance by TAGG-ptimiiers while others are Naive optimizers. TAG-optimizers outperforms the naive optimizers in all datasets in terms of accuracy and also results in a lower forgetting.
is clear that TAG-optimizers outperform their naive counterparts well Naive SGD for all four datasets by a significant amount. There is a notable decrease in Forgetting by TAG-optimizers (in Fig. Tb) in general that eventually reflects the gain in final test Accuracy as seen in Fig. Ta In Split-CUB, TAG-Adam (57%) shows remarkable improvement in accuracy when compared to Naive Adam (45%) such that it even surpasses Naive SGD (55%). Interestingly. TAG-Adam results in slightly lower accuracy as compared to TAG-Adagrad except in Split-minilmageNet. Moreover, Naive Adagrad results in a better performance than Naive RMSProp and Naive Adam for all the datasets. This observation aligns with the results by HHsu et al. (2018). Naive SGD performs almost equivalent to Naive Adagrad except in 5-dataset where it is outperformed.
Next, we analyse al,77) which is the average of an(1, r) across all steps n for all l and when stream is finished i.e., at, 7) & E an (t, We show how att, 7) values play role in the gain in TAG-
6

---

Under review conference paper at ICLR 2022
12
10-
tcciney
Recurncy
0.6
Naw sarovei
10:10
0.6
FBC
04
04
Take 30
(a) 2
(b) T =
(c) =
Figure Evolution of (a) (t,, and test accuracy 0r (left), (b) att, 5) and test accuracy 0r.5 (middle) and (c) atl,, 6) and test accuracy 01,6 (right) along the stream of 20 tasks in the Split-CUB dataset. The grey- coloured lines are max, Os, (1, (top, dashed line), (,,(o(t, (middle, solid line) and min,r r') (bottom, dashed line) that indicate the range of T'). Elliptical regions (black dashed) highlight subtle gain in the accuracy by RMG-RMMPrpp that are maintained throughout the stream. Observing corresponding att, r) in those regions validates hypothesis discussed from Section 3.3
optimizers accuracies in case of Split-CUB dataset. Each plot in Fig. 2 corresponds to test accuracies at,7 (with shaded areas indicating their standard deviations) for Naive RMSProp (blue) and V'GG-RMPProp (red) for particular for all tasks in the stream (x-axis). Along with that, the grey-coloured curves are max, an (t, r') (top, dashed line), lE, [ax(t, r'Il (middle, solid line) and min,' att, r') (bottom, dashed line) respectively. These are shown along with the corresponding att, {a(t, r' E [1, } computed the model encounters task in the stream.
indicate the rank of a(t, r) in the set
The accuracies of TAG-RMSPPP RMSProp and Naive RMSProp appear to correlate for most of the stream. We mark the regions (black dashed ellipse) of subtle improvements in the accuracy by TAG-Ru MAG-RMSProp that is later maintained throughout the stream. While observing att, 7) those regions particularly, we note that the rank of ex(t, r) affects the accuracy in VAG-RMSProp following: (i) Lower (or decrease in) rank of act,, r) means that there exists some correlation between the tasks and т. So, the model should take advantage of the current task updates and seek (or even amplify) backward transfer. Such observations can be made for the following T): (4, 2). (5,2). 2). (14, 2). (9, 5). (14, 6). (19, 6) etc. Our method also prevents drastic forgetting
well in few cases. For example: (7,2). 2). (12. 2), (10, 6). (ii) Higher (or increase in) rank of att, r) results
in prevention of forgetting as observed Fig. 2 Such (t, 7) pairs are (11, (7, 5), (12, 5). (12, 6). etc.
also results in backward transfer observed in [1, (1,5) 5) and (15, 5). We report the same analysis for the other
three datasets and show the results in Appendix
A.3.3
4.2 COMPARED WITH OTHER BASELINES
In the next experiment, we show that the TAG-R RAG-RMSProp results in a strong performance compared to other LLL. algorithms. In Table T. we report the performance of TAG- TAG-RMPProp and the following ofate-of-the-an baselines: EWC (Kirkpatrick et al.] (017). A-GEM (Chaudhry et al.) 2018b). ER (Aljundi et al.] 2019) with reservoir sampling and Stable SGD (Mirzadeh et al.J (0220). Apart from these baselines, we report the performance of Naive SGD and Naive RMSProp from the previous section. We also report results on multi-task learning (MTL) settings on all four datasets where the dataset from all the tasks is always available throughout the stream. Hence, the resulting accuracies of the MTL setting serve as the upper bounds for the test accuracies in L.I...
Following Mirzadeh et al. 2020), the size of the episodic memory for both A-GEM and ER is set to store 1 example per class. Since we want to evaluate TAG with all other baselines on the original non-i.i.d, problem, we keep the episodic memory size in the replay-base based methods small for the comparison. We still report
7

---

Under review conference paper at ICLR 2022
Table 1:
Comparing performance in terms of final average test Accuracy (%) (higher is better), Forgetting (lower is better) and Learning Accuracy (LA (%) (higher is better) of the existing baselines with TAG- RMSProp on all four datasets. All metrics are averaged across 5 runs. Overall, TAG-RMSProp outperforms all other methods in terms of Accuracy. *M77. assumes that the whole dataset from all tasks is always available during training, hence it is different setting and its accuracy acts as an upper bound.
Split-IFFAR00
SplitmminIlmageNet
Methods
Accuracy (%) )
Forgetting
LA (%)
Accuracy (%)
Forgetting
LA (%)
Naive SGD
51.36 (=3.21)
0.18 (10.03)
68.46 (+1.93)
48.19 (=0.79)
0.13 (10.01)
60.6 (1:0.55
Naive RMSProp
48.91 (+2.88)
0.2 (+0.03)
67.28
(+0.43)
45.06 (+0.6)
0.21 (+0.01)
64.39 (+1.02)
EWC
49.06 (13.44)
0.19 (10.04)
66.82
[11.41)
47.87 [ 2.08)
0.15 (10.02)
61.66 (11.06)
A-GEM
54.25 (+:2.0)
0.16 (+0.03)
68.98
(=1.19)
50.32 (-1.29)
0.11 (=0.02)
61.02 (+0.64)
ER
59.14 (-1.77)
0.12 (+0.02)
70.36
(+1.23)
54.67 (-071)
(+0.01)
64.06 (+0.41)
Stable SGD
57.04 (=1.07)
0.09 (+0.0)
64.62
(0.91)
51.81 (-166)
0.09 (=0.01)
59.99
(0.94)
TAG-RMSProp (Ours)
62.79 (=0.2
0.1 (=0.01)
72.06 (=.01)
57.
(+13
0.06 (=0.02) 62.73
-0.61
MTL*
67.7 (+0.58)
66.14 (+10
Split-CUB
5-dataset
Methods
Accuracy (%)
Forgetting
LA (%)
Accuracy (%)
Forgetting
L. (%)
Naive SGD
54.88 (11.83)
0.12 (10.01)
65.97 (10.59)
46.48 (36.62)
0.48 (110.05)
84.55 (11.06)
Noive RMSProp
49.4 (+1.77)
0.24 (+0.01)
71.76 (=0.94)
45.49 (=1899
0.5 (=0.03)
85.58 (+1.21)
EWC
55.66 (=0.97)
0.12 (10.01)
66.36 (=0.71)
48.58 (=1.47)
(-000)))
79,56 [#3.18)
A-GEM
56.91 1=37)
0.1 (10001)
65.6 (1:0.73)
55.9 (12.58)
0.34 (1:0.04)
82.61
(
ER
59.25 (=0.82)
0.1 (=0.01)
66.17 (=0.42)
61.58 (=2.65)
0.28 (=0.04)
84.3 (+1.08)
Stable SGD
53.76 (-2.14)
0.11 (10.01)
62.15 (11.12)
46.51 2-775
0.46 (10.03)
83.3 (11.44)
TAG-RMSProp (Ours)
61.58 (=1224)
0.11 (10.01)
71.56 (00.74)
62.59 1=.82)
0.29 (20.02)
86.08 (=0.55)
MTL
71.65 (=0.7))
70.0 (:4.44)
A-GEM and ER results with bigger memory sizes in Appendix A.3.3
The size of the mini-batch sampled from the episodic memory is set equal to the batch-size to avoid data imbalance while training. Although we utilize a similar amount of memory as (Rusu et al., (016), an expansion-
method, we do not make any changes to the size of the model and thus the number of parameters remains the same during the test time. Hence, we do not compare our approach with the expansion-bass -baed method in our experiments.
From the results reported in Table D we observe that TAG-RMSProp achieves the best performance in terms of test Accuracy as compared to other basclines for all datasets. The overall improvement is decent in Aplit-CIFAR100,
and Split-CUB which are 3.65%, 2.5% and 2.3% with regard to the next-best baseline. On the other hand, the improvement by TAG-RMSProp is relatively minor in 5-dataset i.e., 1% compared to ER with the similar amount of Forgetting (0.29 and 0.28) occurring in the stream. In terms of LA, TAG- RMSProp achieves almost similar performance as Naive RMSProp in Split-CUB and 5-dataset. We also note that the LA of TAG-RMSProp is higher in Split-CIFAR10N Split-CUB and 5-dataset than ER and A-GEM. The higher LA with similar Forgetting as compared to other baselines shows that while TAG exploits the adaptive nature of existing optimizers, it also ensures minimal forgetting of the gained knowledge. The existing optimizers tend to aggressively fit the model on the most recent task at an immense cost of forgetting the earlier tasks. Hence, even a similar (or lower) Forgetting occurs in TAG, the higher test Accuracy (with high LA) shows that TAG is capable of retaining the gained knowledge from TAG-RMSPPP manages to prevent catastrophic
each task. Although LA is lower in Split-minilmageNet. forgetting better than these methods and hence results in higher test Accuracy.
4.3 COMBINED WITH OTHER BASELINES
Lastly, we show that the existing baselines can also benefit from our proposed method TAG-RMSProp. We replace the conventional SGD update from EWC, A-GEM and ER, and apply RMSProp update (Eq. [ and
8

---

Under review conference paper at ICLR 2022
RoG update (Eq. B respectively. We use the same task-incremental learning setup used in the previous sections in terms of architecture and syper-parametent We compare the resulting accuracies of the baselines with their RMSProp and TAGed versions in Fig. 3
Accuracy (%)
70-
Fa Nishinp CAC
ACEE (am
60-
Rar Shopp GDC (twd)
He Sirpp GaM tueo2
50
HM Ariimpp govd)
arttin (owtr) Tictett pharte
Split CIFAR100
Split minilmageNeet
Split CUB
5 dataset
Figure 3: Comparing performance for different existing methods with their RMSProp and TAGed versions on all four datasets in terms of final average test Accuracy (%) along with A-GEM and ER for different samples per class (M) in the episodic memory. The vertical bars with hatches are the performance by T'Ged versions of the baselines. All results are averaged across 5 runs. All TAGed versions results in similar gain in the accuracy over baselines with both SGD and RMSProp update.
For a given dataset, we see that gain in the final accuracy in the TAGed versions is similar for the baselines described in Section 4.2 That is, TAG improves these baselines with SGD update on Split-CIARR100 Split-minilmageNet Split-CUB and 5-dataset by at least 8%, 4%, 4% and 9% respectively. On the other hand, TAG improves the baselines with RMSProp update on the datasets by at least 12% 12%. 7% and 9% respectively. The improvement is also significant in A-GEM with bigger episodic memory (i.e.. 10 samples per class or MI 10) but we observe relatively smaller improvement (2%) ) by TAGed ER (M = 10) as compared to ER (M 10). These results show that apart from outperforming the baselines independently (with smaller episodic memory in replay-based methods). TAG can also be used as an update rule the existing research works for improving their performances. While A-GEM and ER are strong baselines for LLL, we would like to highlight that these replay-based methods are not applicable settings where storing examples is not option due to privacy concerns. I'A-RRMPrrop would be more appropriate solution in such settings.
5 CONCLUSION
We propose task-aware optimizer for the LLL setting that adapts the learning rate based the relatedness among tasks. We introduce the task-based accumulated gradients that act as the representation for individual tasks for the same. We conduct experiments on complex datasets to compare TAG-RMSProp with several sate-offthheart methods. Results show that TAG-RMSProp outperforms the existing methods in terms of final accuracy with commendable margin without storing past examples or using dynamic architectures. We also show that it results in significant gain in performance when combined with other baselines. To the best our knowledge, ours is the first work in the LLL literature showing that we can use an adaptive gradient method for LLL and prevent forgetting better than Naive SGD. For future work, the memory required to store the task-specific accumulated gradients increases linearly with the tasks, reducing memory complexity without compromising the performance (i) computing correlation using a smaller quantity than the task-based first moments, and (ii) clustering the similar tasks together reduce the number of task based second moments (in settings with a soft margin between the tasks). Another possible direction from here can be shifting to iaass-nncrementll scenario where the task identity is not known beforehand and is required be inferred along the stream.
be an interesting direction. This be achieved by
9

---

Under review conference paper at ICLR 2022
REFERENCES
Tameem Adel, Han Zhao, and Richard E Turner. Continual learning with adaptive weights (claw). arXiv preprini urXiv:/911.0955 2019.
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-bast continual learning with adaptive regularization. In Advances in Neural Information Processing Systems, pp. 4392-44022 2019.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
Lifelong learning with network
2017.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11849-18860, 2019.
Mehdi Abbana Bennani and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006. 1942, 2020.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXhv:/505.0542) 2015.
Yaroslav Bulatov. Notmnist dataset. Technical report, Google (Books/OCR), 2011. URL http://
yaros lavvb. www.llavvvlloo.oo.... wtlog/po...t/22111/9//11/.
data-dttttt html
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk
for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:18/2.00420, 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 102. 2019.
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv: 1rXiv:1806.000 0188000066633 2018.
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):9-1,, 2018
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv: (tt::/999.08833 2(6), 2019.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guiia continual learning with bayesian neural networks. arXiv preprint 1906.0990000225 2019.
10

---

Under review conference paper at ICLR 2022
Mchrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773. PMLR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint
2017.
lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based based neural networks. arXiv preprint arXiv:13/2.6211,, 2013.
Simon Guiroy, Vikas Verma, and Christopher Pal. Towards undcrstanding generalization gradient- beasdd meta-learning. arXiv preprint arXiv:1907.072877 2019.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: Look-ahead meta learning for continual learning.
arXiv preprint arXi:: 13904, 2020.
Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028 1040, 2020. ISSN 1364-6613, doi: https://doi.org/10.. 1016 tics. 2020 090.009000.. 9.044. URL http:/ wwww sww.ncediieeiccccccccc rect www.sccimccc.o.cc
atticcleep11/9134441113333
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong basclines. arXiv preprint arXiv:1810.12488, 2018.
Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In Advances in Neural Information Processing Systems, pp- 9132-9133, 2019.
Xisen Jin, Junyi Du, and Xiang Ren. Gradient based memory editing for task-free continual learning. arXiv preprint arXiv:2006.1523 15294, 2020.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. CoRR, abs/1712.000 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3521-3326 2017,
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits, 1998. URL http:h/t lecun. com/ excbb/
mn i
Qunwei Li, Yi Zhou, Yingbin Liang, and Pramod K Varshney. Convergence analysis of proximal gradient with momentum for nonconvex optimization. arXiv preprint DrXiv:1705.04923. 2017.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. arXiv preprint a9Xiv: 0900400310, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-29 29477 2017.
11

---

Under review conference paper at ICLR 2022
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pp. 6467-6476, 2017.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation. arXiv preprint arXiv:2010.15277
2020.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume of Psychology of Learning and Motivation, pp. 109 165. Academic Press, 1989. doi:
URLhtttp //www /www.wwww.....oo sci encedirect
com /ttp://ccaoooeec/cdee/e//////122
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. arXiv preprint arXiv:2006.0699 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Reprexentations, 2018.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 2019.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, pp. 7647- 7657, 2019.
Sylvestre-Alvin Alvise Rebuffi, Alexander Kolesnikov. Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: /810. /910, 2018.
A. Robins. Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In Proceedings 1993 The First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems, pp. 65-68, 1993, doi: 10. 19/1109//NEE...99.3..
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint 17X4771609.04777, 2016.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 160iv:1606.044 04671, 2016.
Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In International Conference on Machine Learning, pp. 507-515, 2013.
12

---

Under review conference paper at ICLR 2022
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with agemention to the task. arXiv preprint arXiv:/801.0142) 2018.
Ammar Shaker, Shujian Yu, and Francesco Alesiani. Modular-relatedness for continual learning. arXiv preprint 1..iv22011.0122., 2020.
Shagun Sodhani, Sarath Chandar, and Yoshua Bengio. On training recurrent neural networks for lifelong learning. CoRR, abs/1811.07017, 2018.
T. Tieleman and G. Hinton. Lecture 6.5 RmsPrpp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Amal Rannen Triki, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In ICCV, 2017.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734.00 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-201. dataset. 2011.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07743: 2017.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pp. 899-908, 2018
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 1708.00.77 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Pro- ceedings of machine learning research, 70:3987, 2017.
A APPENDIX
this document, we provide the details and results excluded from the main paper. An
we describe the
TAG versions of Adagrad and Adam. The implementation details are described in Section A.2 We also report results obtained by performing additional experiments in Section A.3
A.1 TAG-OPTIMIZERS
Similar R.SProp (in Section 3). with the task- first moments ! let w!!) (l,(1, E (from Fq. 2 and,
( [1,4)
vilu;
t=1
w (1)
(4)
anlt,t) Vi(1) + a
t>
We define the TAG versions of Adagrad and Adam following:
13

---

Under review conference paper at ICLR 2022
TAG-Adagrad:
V(O v((1)
n n-1 + g
g(t) 8(0)
7
(5)
2+1
9n
V Wnn V +e
TAG-Adam:
(((1) = 32V n- (1 - B2)9
( n+1 = k
mI 15
(9
(6)
(1 BT) W Vn
+
Both TAG-Adagrad (Eq. 5) and TAG-Adam (Eq. 6 result in a significant gain in Accuracy and prevent
Forgetting observed in Fig. Ta and Fig. 1b respectively in Section 4.1
A.2 IMPLEMENTATION DETAILS
The summary of the datasets used in the experiments is shown in Table and Table 3
Table 2: Dataset Statistics
Input size
Training samples per task
Test samples per task
Split-CIFAR100 Split- Split-CUB
3 x 32 x 32
2500
500
3 x x 84
2400
600
3 x 224 x 224
300
290
5-dataset, we convert all the monochromatic images to RGB format depending on the task dataset. All images are then resized to 3 x 32 x 32. The overall training and test data statisties of 5-dataset are described Table 3
Table 3: 5-dataset statistics.
Training samples
Test samples
CIFAR-10
50000
10000
MNIST
60000
10000
SVHN
73257
26032
notMNIST
16853
1873
Fashion-MNIST
60000
10000
Details about the metrics used for evaluating the model:
Accuracy (Lopez-Paz & Ranzato 2017): If ar,r is the accuracy on the test set of task 7 when the
current task is 1, it is defined as, A, Σ dt.
Forgetting (Chaudhry et al. 2018a): It is the average forgetting that occurs after the model is trained
on all tasks. If the latest task is f and is defined as, F,
+ Σ maxpe()....t1))00e -t.7).
14

---

Under review conference paper at ICLR 2022
Learning Accuracy (LA) Riemer et al. 2018): the measure of learning capability when the model sees new task. For the current task t, it is defined as, L
Σ ar.
We implement the following baselines to compare with our proposed method:
FWC:
Our implementation of EWC is based on the original paper (Kirkpatrick et al. 2017
A-GEM: We implemented A-GEM based on the the official implementation provided by (Chaudhry et al.
2018b).
ER (Chaudhry et al. 2019): Our implementation is based on the one provided by (Aljundi et al. 2019) with reservoir sampling except that the sampled batch does not contain examples from the current task.
We obtain the best hyper-parameter set by performing grid- search over different combinations of the learning rate, learning rate decay, and dropout (see Appendix A.2.1).
Stable SGD (Mirzadeh et al. 2020):
OGD Farajtabar et al. (020)) requires storing N (=200 in their experiments) number of gradients per task and it is evaluated only on variants of the MNIST dataset by training a small feed-forward network. On the other hand, TAG additively accumulates the gradients and hence requires memory equal to two copies of the model as the knowledge base. This enabled us to train reduced ResNet18 on complex datasets. Due to greater memory requirements, OGD faced memory errors in our setting. We would also like to highlight that OGD use Naive-SGD and our contribution being an adaptive learning rate based method is complementary to this approach.
We provide our code as supplementary material that contains the scripts for reproducing the results from all experiments described in this paper. In the CODE folder, include README.MD file that contains the overall code structure, procedure for installing the required packages, links to download the datasets and steps to execute the scripts. All experiments executed NVIDIA GTX 1080Ti machine with 11 GB GPU memory.
A.2.1 HYPER-PARAMETER PARAMETER DETAILS
this section, we report the grid search details for finding the best set of hyper- hepee-yparaeeeee parameters for all datasets and baselines. We train the model with 90% of the training set and choose the best byper-parameters based on the highest accuracy on the validation set which consists of remaining 10% for the training set. For existing baselines, perform the grid search either suggested by the original papers by barajjaaa et al. (2020). For all TAG-optimizers, 31 is set to 0.9. For TAG-RMSProp and TAG-Adagrad, B2 is set to 0.99 and for TAG-Adam it 0.999. In all the experiments, the mini-batch size is fixed to 10 for Split-CIFAR Split-minilmageNet. Split-CUB similar to (Chaudhry et al. [019 Mirzadeh et al.) (2200). We set mini-batch size 64 for 5-dataset following (Serra et 2018). This because we wanted to highlight the role of learning rate and to show how TAG-RMSProp improves the performance while the other hyper-parameters (including batch-size) were fixed.
Naive SGD
Learning rate: [0.1 (Split-CFFAR100, 5-dataset), 0.05 (Split-minilmageNee)), 0.01 CUB), 0.001]
Naive Adagrad
Learning rate: [0.01, 0.005 (Split-CIFAR100 Split-minilmageNet, (Split-CUB), 0.0001]
http::CKAK......piiiiiiiinnnnnn 5-dataset), 0.001, 0.0005
15

---

Under review conference paper at ICLR 2022
Naive RMSProp
Learning rate: [0.01, 0.005 (Split-CIFAR100), 0.001 (Spiit-minilmageNe,, 5-dataset),
0.0005, 0.0001 (Split-CUB), 0.00005, 0.00001]
Naive Adam
Learning rate: [0.01, 0.005 (Split-CFFAR100, 0.001 (Split-minilmageNee, 5-dataset), 0.0005, 0.0001 (Split-CUB))
TAG-Adagrad
Learning rate: [0.005 (Split-CIFAR100, 5-dataset), 0.001 (SpittmminilmageNee)), 0.0005 (Split-CUB), 0.00025, 0.0001] [1, 3, Split-CIFAR1O0. Split-minilmageNet, Split-CUB). (5-dataset)
TAG-RMSProp
Learning rate: [0.005, 0.001, 0.0005 (5-dataset), 0.00025 (Split-CIFAR10), 0.0001 (Split-
minilmageNNt),, 0.00005, 0.000025 (Split-CUB), 0.00001] b: [1, (Spii-CCCTA,,,,,, Split-minilmageNet, Split-CUB), 7 (5-dataset)
TAG-Adam
Learning rate: [0.005, 0.001 (5-dataset), 0.0005 (Split-CIFAR100) 0.00025 (Split-
minilmageNet), 0.0001 (Split-CUB)]
b: [1, (Split-CIFAR100, Split-minilmageNet, Split-CUB), (5-dataset))
EWC
- -Learning rate: [0.1 (Split-CIFAR100, 5-dataset), 0.05 (Split-minilmageNee)), O.01(Split-
CUB), 0.001]
(regularization): [1 (Split-CIFAR100, Split-minilmageNet, Split-CUB), 10, 100 (5- dataset)]
A-GEM
LLearning rate: [0.1 (Split-CIFAR100, Split-minilmageNet, 5-dataset), 0.05, 0.01 CUB), 0.001]
EER
-Leaming rate: [0.1 (Split-CIFAR100, 5-dataset), 0.05 (Split-iinilmageNee), 0.01 CUB), 0.001]
Stable SGD
Initial learning rate: [0.1 ( Split-minilmageNet. 5-dataset), 0.05 (Split- CUB), 0.01]
Learning decay: [0.9 (Split-CIFAR100, Split-minilmageNet,, Split-CUB), 0.8, 0.7 (5- dataset)]
Dropout: [0.0 (Split-minilmageNet, Split-CUB, 5-dataset), 0.1 (Split-CIFAR100, 0.25, 0.5]
case TAG-RMSProp. we empirically found that the best performance of the all three benchmarks with 20 tasks occurred when hyper-parameter b = 5 and for 5-dataset, b = 7. We also found that a lower value of Learning rate TAG-R GGG-MMPPrp results in better performance. These empirical observations can reduce the search space for hyperparameter setup by huge amount when applying TAG- TAG-RMSProp
LLL setup.
For the experiments Scction 4.3 that require a hybrid version of these methods, we use the same hyper- parameters from above except for TAGed ER in Split-CIFAR100 (Learning rate 0.0005) and Split-CUB (Learning rate = 0.0001). We choose the learning rates of TAG-RMSProg and Naive-RMSProp over EWC, A- A-GEM and ER.
16

---

Under review conference paper at ICLR 2022
A.3 ADDITIONAL EXPERIMENTS
In this section, we describe the additional experiments and analysis done in this work.
A.3.1 BACKWARD TRANSFER METRIC
While we show the occurrence of knowledge transfer in Fig. we can quantify the Backward Transfer (BWT) (Chaudhry al. 2019) by computing the difference between the final Accuracy and LA. i.e. Ot,r Gr,7- We report the BWT results for all datasets and baselines in Table 4
BWT = -
While TAG-RM TAG-RMSProp outperforms the other baselines in terms of BWT for Split-minilmageNer it is overall the second-bes best method for Split-CIARR100 and 5-dataset. In case of Split-CUB, even TAG-RMSProp achieves the highest Accuracy, it results in lower BWT because of significantly higher LA compared to the other baselines (see Table
Table Comparing performance in terms of final average test Accuracy (%) (higher is better) and BWT (higher is better) with the standard deviation values for different existing methods with TAG-RM TAG-RMSProp for all four datasets. All metries are averaged across runs.
Split-CIFAR100
Split-minilmageNet
Methods
Accuracy (%)
BWT (%)
Accuracy (%)
BWT (%)
Naive SGD
51.36 (+3.21)
-17.1
(+2.64)
48.19 (+0.79)
-13.83
(+1.97)
Naive RMSProp
48.91 (+2.88)
-18.37
(+2.71)
45.06 (+0.6)
-19.32
(+1339)
EWC
49.06 (13.44)
-17.76
(+3.35)
47.87
(#2.08)
-13.79
(+2.26)
A-GEM
54.25 (+2.0)
-14.73
(+2.48)
50.32
(#1.29)
-10.69
(+1.77
ER
59.14 (+1.77)
-11.22
(+2.19)
54.67
(+0.711)
-9.99
(+0.64)
Stable SGD
57.04 (+1.07)
-7.59
(+0.36)
51.81
(=166
-8.18
(+1.1))
TAG-RMSProp (Ours)
62.79 (=0.29
9.27
(11.16
57.2
(1137
5.52 11171
Split-CUB
5-dataset
Methods
Accuracy (%)
BWT (%)
Accuracy (%)
BWT (%)
Naive SGD
54.88 (+1.83)
-11.99 (=1443
46.48
(#3.62)
-38.06 (#3.69)
Naive RMSProp
49.4 (#1.77)
-2.36 (0.95)
45.49
(1.9)
-40.09 (+2.6)
EWC
55.66 (+0.97)
-10.7
(++.39)
48.58
(#1.47)
330.8
(+3.34)
A-GEM
56.91 (+1337)
-8.69 (+0.93)
55.9
(+2.58)
-26.71
(+3.6)
ER
59.25 (+0)82)
-6.93 (+0.92)
61.58
(+2.65)
-22.72
(=3.08)
Stable SGD
53.76
(2214)
-8339 (+126))
46.51
(+277
-36.79
(2211
TAG-RMSProp (Ours)
61.58
11224)
9.99
(162))
62.59
(1.2)
-23.49
(+177
A.3.2 COMPARING TAG-RMSProp WITH OTHER BASELINES
Fig. 4 provides detailed view of the test accuracies of individual baseline as the model encounters the new tasks throughout the LLL stream in Split-CIFARII
Split-minilmageNee and Split-CUB. At the starting task t 1, TAG-RM TAG-RMSProp beats other baselines because of lower initial learning rates (see Appendix 1 and reflects the performance gain by the RMSProp over SGD optimizer. In all three datasets, the performance of TAG-RMSProp is very similar ER specially from task 5 to task t = 15, but ultimately improves as observed at = 20. These results show a decent gain in the final test Accuracy by TAG- GMM-oo as compared to other baselines.
17

---

Under review conference paper at ICLR 2022
Although TAG-RMSProp outperformed by ER from task to 4, it results in the highest final accuracy as compared to other methods on 5-dataset. TAG-RMSProp also outperforms other baselines including Stable SGD and A-GEM by large margin.
60
Naize SGB
RMSProp
EWC
55
A-GEM
Accuracy (%)
60
Stable
Accuracy (%6
TAC- HMMPPap
55
40
Naive SGD
RMSProp
35
A-GEM
TAG AMSProp
25
10
15
20
10
15
20
Tasks
Tasks
(a) Split-CIFAR100
(b) SpeetmminnlmggeeNtt
80
Naive RMSProp
EWC
65
ER
60
70
Stable SGD
Accuracy (%))
55
60
Neive RMSProp
45
EWC
A-GEM
40
50
Stable SGI
TAG RM5Prop
35
10
20
Tasks
Tasks
(c) Split-CUB
(d) Split-CUB
Figure 4: Evolution of average test Accuracy (%) A, for different existing methods and TAG-RMSProp throughout the stream on datasets. results are averaged across 5 runs and the shaded area represent standard deviation. Performing similar ER for major part of the stream, TAG-RMSProp always results in the highest final accuracy as compared to other methods with low standard deviation.
A.3.3 EVOLUTION a(t, r) AND TEST ACCURACY at.r
Next, we continue analysis done Section ?? for Split-CIFAR100 (in Fig. 5), Split-minilmageNet (in Fig. (). Split-CUB (in Fig. 7 for the first tasks and 5-dataset (in Fig. 8) for first tasks. In Split- CIFARI00 and Split-minilaggeNee the model with Naive RMSProp tends to forget the task by significant amount
it receives the tasks. On the other hand, TAG-RMSProp prevents catastrophic forgetting and hence results a higher accuracy throughout the stream. We can observe that for Split- CIFAR100 and Split-minilmageNer a(T + 1, r) (where [1.9) 11. generally have higher rank in the set
18

---

Under review conference paper at ICLR 2022
1.4
14
1.2
Ti vithcs
sarangg
12-
12-
TCl
1.0
1.0
0.-
17.11
00
Foe 41:12
0.85
a3.0
0.6
0.6
Accunace
0.6
04-
04
0.4
02
23
24
20
20 25
(a) -=
(b) 7 = 2
(c) 3
2.6
3.6
1.4
2.4
3.2
1.2
12-
11
1.0
10-
1at,7
088
0.
0.6
Uo-
0.
0.4
20
22
(d)
7
T
1.6
P. 1.6
1.5
8 3.4
1.2
1.2
12
1.0
x7-11
1.0
ALO
10-
0.8
088
0.8
Ter
0.6
0.6
0.6
0.4
6.1-
a3.
14
14
26 16
(g) 7=)
(h) 8
9
Figure 5: Evolution of att, T) and test accuracy orr where T E [1, 9] along the stream of 20 tasks in the Split-CIFAR100 dataset. The grey-coloured lines max, a, (t, r') (top, dashed line), [a(tt 7')] (middle, solid line) and min, a(t, r') (bottom, dashed line) that indicate the range of ott,,'').
(a)(1,"');'' E [1, This because TAG-RMSProp also recognizes an immediate change in the directions when the model receives new task (from M M&-11 to Mr("). A similar observation is made in case of Split-CUB but the visible gain in the accuracy by TAG-RMSProp does not occur instantly. Apart from that, we observe that the lower and higher rank of (<(4, 7) results in backward transfer and prevents catastrophic forgetting respectively in the stream. Overall, in all datasets, we arrive at the same conclusion obtained in Section ??.
A.3.4 MULTIPLE-PAR -AASS PER TASK
In this section, report the performance of TAG-RMMPProp and all other baselines discussed in Section 42 for epochs per task in Table Haper-parameters for this experiment are kept the same as the single-pase per task setting. TAG-RMSProp results in high average Accuracy in all the datasets. We also observe less amount
19

---

Under review conference paper at ICLR 2022
1.6
1.6
16-
1.4
1.4
14
1.2
vithcs sarangg
1.2
12
RASPRR
1.0
0.8
20.11
0.8
0.6-
Accuncey
0.6
0.6
0.4
04
0.4
0.2
02:
62
33
24
21
21
25
(a) 1=
(b) 7 2
( 3
1.6
1.6
1.-
1.4
1.4
1.2
1/2
12
1.0
69.41
1.0
I lel.11 If
0.8
13:11
0.8
0.8
0.6
0.6
0.
0.4
0.4
0.4
02
20
20
(d)
5
T== 6
2.6
3.6
1.4
1.4
1.4
1.2
1.2
Saaan SONANY
12
1.0
x7-11
(02-10
1.0
#USS
1.0
0.0
0.8
0.8
0.6
0.4
04
6A
a3.0
14
26 16
7=)
(h) 8
T== 9
Figure 6: Evolution of act, r) and test accuracy at, where 7 E [1, 9] along the stream of 20 tasks in the Split-minilmageNel dataset. The grey-coloured lines max, a. (t, 7') (top, dashed line), E, [a(tt 7']] (middle, solid line) and min, att, r') (bottom, dashed line) that indicate the range of ott,=').
of Forgetting in AGGRMMProp as compared to other baselines. In terms of Learning Accuracy, TAG- RMSProp is outperformed by the other baselines in Split-CIFAR100, Split-minilmageNNe and 5-dataset but performs better in Split-CUB.
A.3.5 BIGGER MEMORY SIZE IN REPLAY-BASED METHODS
We also compare the performance of A-GEM and ER with a larger number of samples per class (M) in the episodic memory for all four datasets in Table 6 With MI = 10, total episodic memory size for Split- CIFARI00, mppit-minilmageNNe.. Split-CUB and 5-dataset becomes 1000, 1000, 2000 and 500 respectively. We observe ER results in significant gain in the performance as the episodic memory size increases. But TAG- -RMSPPpp is able to outperform A-GEM in SpIAt-CIFAR100, Split-minilmageNet and Split-CUB with a large margin even when M is set to 10.
20

---

Under review conference paper at ICLR 2022
1.0
uo sanance
Wo serenoo
0.6
nan at.
ma aL
0.2
(a) T
(b) 3
(c)==4
3.2
3.2
1.0
3.0
10
0.8
6.8
Aaraiin
0.6
0.6
0.6
D91-10 of
0.4
at.:
0.4
64
+0.1
+0.11
17 24
06 16
Td)==
(e) T =
T 9
Figure 7: Evolution of ax(4, 7) and test accuracy dt,r where E (1,3,4,7,8,9) along the stream of 20 tasks in the Split-CUB dataset. The grey-coloured lines are max, (1, (top, dashed line), E (a(,77)))) (middle, solid line) and min, att, r²) (bottom, dashed line) that indicate the range of a(t,T').
25
2.5
23
2.0
1.5
3.5
15
1.0
1.0
Nav tithee
1.0
05.
0.5
apo a1.
Tpx
0.0
Taa
Tabs
Tasla
(a)
(b) 2
(c)
Figure 8: Evolution of c<(t, 7) and test accuracy dr,7 where T E 21,2,3, along the stream of 5 tasks in the 5-dataset dataset. The grey-coloured lines are max, (1, 7') (top, dashed line), [a(((((, (middle, solid line) and min,' act, r') (bottom, dashed line) that indicate the range of at, r').
21

---

Under review as conference paper at ICLR 2022
Table 5: Comparing performance in terms of Final average test Accuracy (%) (higher is better), Forgetting (lower is better) and Learning Aceuracy (LA ( (%)) (higher is better) with the standard deviation values for different existing methods with TAG-RMSProp running for epochs per task for all four datasets (see Section 4 All metrics are averaged across 5 runs. *MTL assumes that the dataset from all tasks are always available during training, hence is a different setting and its accuracy acts as an upper bound.
Split-CIFARI00
Split-minllngeNett
Methods
Accuracy (%)
Forgetting
LA (%)
Accuracy (%)
Forgetting
LA (%)
Naive SGD
52.26 (10.65)
0.28 (10.01)
78.45 (10.41)
46.9 (11.18)
0.25 (10.02)
70.82 (.006)
Naive RMSProp
46.12 (=2.33)
0.32 (10.02)
76,29 (20.53)
41.07 (=0.66)
0.32 (10.01)
71.43 (40.42)
EWC
51.7 (#1.71)
0.27 (10.02)
77.72
(=0884
48.17 (=0.8]
0.25 (=0.01)
71.87
(=0.26)
A-GEM
54.24 (=1.44)
0.25 (10.01)
78.38 (1:0.39)
49.08 0.52)
0.23 (1.0.1)
70.49 (10.4)
ER
60.03 (=0.96)
0.19 (+0.01)
78.15 (+007)
54.01 (=0.6))
0.19 (=0.01)
71.77
(+0.58)
Stable SGD
58.92 (=0.73)
0.19 (10.01)
76.91 (=0.72)
51:23 (0888)
0.22 (10.01)
71.77
(10.56)
TAG-RMSProp (Ours)
60.64
(=1.3% .38)
0.17 (10.01)
77.12 (00.6)
58.0 (1.11
0.11 (20002)
68.14 (10.38)
MTL
67.7 (+0.58)
66.14 (=1.0)
Split-CUB
5-dataset
Methods
Accuracy (%)
Forgetting
LA (%)
Accuracy (%)
Forgetting
LA (%)
Naive SGD
59.87 (=1.48)
0.21 (+0.02)
79.77 (=0.44)
49.95 (=2.42)
0.51 (=0.04)
90.86 (+0.0 +.)633)
Naive RMSProp
35.87 (11.14)
0.46 (10.01)
79.59 (1003)
50.47
0.99
0.51 (10.01)
90.89 (10.44)
EWC
59.73 (12.4)
0.21 (10.02)
79.8 (10.58)
52.51
7.34)
0.43 (10009)
86.8 (12.52)
A A-GEM
62.65 (+1.61)
0.17 (+0.02)
79.1 (+0.4)
62.48 (-3.16)
0.35 (+0.04)
90.53 (+0.733
ER
66.06 (=1.28)
0.14 (+0.02)
78.79 (=0.55)
62.84 (=1.58)
0.35 (=0.02)
90.52 (+0.69)
Stable SGD
58. 75 (10.96)
0.19 (10.01)
76.6 (10064)
51.95 ( 3.83)
0.48 (10.05)
90.41 (10.29)
TAG-RMSProp (Ours)
68.0 (11.01)
0.13 (10.01)
80.15 (10.22)
61.13 3.05)
0.36 (10.04)
89.9 (10.33)
MTL.
71.65 (-0.6)
70.0 (+4.44)
Table 6: Comparing performance in terms of Final average test Accuracy (%) (higher is better), Forgetting (lower is better) and Learning Accuracy (LA ( (%)) (higher is better) with the standard deviation values for A-GEM and ER for different number of samples per class (M) in the episodic memory with TAG-RM RAGPRMSProp for all four datasets (see Section 4.. All metrics are averaged across 5 runs. Overall, ER with bigger memory outperforms all other methods in terms of Accuracy.
Split-CIFARI00
Split moiimmiinilmageNet
Methods
Accuracy (%)
Forgetting
LA (%)
Accuracy (%)
Forgetting
LA (%)
A-GEM (M 1)
54.25 (122.))
0.16 (10.03)
68.98 (11.19)
50.32 ( 1.29)
0.11 (10.02)
61.02 (10064)
A-GEM (M-
55.74 (=1.14)
0.14 (+0.01)
68.97
(=0.56)
49.52 (=2.02)
0.12 (+0.02)
60.49 (+0.78)
A-GEM (M-
56.68 (=1.92)
0.13 (10.02)
68.72
(-0.96)
49.77 (=2.1)
0.12 (10.02)
60.6 (10.66)
ER (M 1)
59.14 (=1.77)
0.12 (#0.02)
70.36
(=1.23)
52.76 (=1.53)
(=0.01
61.7 (#0.74)
ER (M- 5)
65.74 (14447)
0.07 (10.01)
70.91 (11.13)
58.49 1.21)
0.05 (10001)
62.24 (10.85)
ER (M 10)
68.94 (+0.93)
0.05 (+0.01)
71.26 (=1.01)
60.06
(0063)
0.04 (+0.01)
62.21 (+1.24)
TAG-RMSProp (Ours)
62.79
(-0.29)
0.1 (+0.01)
72.06 (+1.01)
57.2
(+1)3
0.06 (+0.02)
62.73 (+0.61)
Split-CUB
5-dataset
Methods
Accuracy (%)
Forgetting
LA (%)
Accuracy (%)
Forgetting
LA (%)
A-GEM (M = 1)
56.91 (+1.37)
0.1 (+0.01)
65.6 (+0.73)
55.9 (+2.58)
0.34 (+0.04)
82.61 (+2.13)
A-GEM (M = 5)
56.4 (=1.5)
0,1 (=0.01)
65.63 (=0.64)
61.39 (#1.0)
0.28 (#0.01)
83.48 (+1.05)
A-GEM (M- = 10)
56.71 (11.6)
0.1 (10.01)
65.73 (100.)
62.43 (- 1.38)
0.26 (10.03)
83.38
(11.79)
ER (M 1)
50.25 (10.82)
(10.01)
66.17 (10.42)
61.58 2.65)
0.28 (10.04)
84.31
(11.08)
ER (M = 5)
68.89 (-0.31)
0.04 (+0.0)
66.76 (+0.73)
71.56
(-1.4)
0.16 (+0.02)
84.34
(+1.46)
ER (M 10)
70.73 (0.23)
0.03 (+0.01)
66.83
(0)
75.44
1.U7
0.12 (#0.02)
84.62
(0.89)
TAG-RMMPProp (Ours)
61.58 (=1.44
0.11 (#0.01)
71.56 (0.74
62.39
(1182)
0.3 (=0.02)
86.08
(0.55
22


Reviews:
[{"review_id": "xGZbfQI6jN1", "reviewer": "Reviewer_94tf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts of this paper are: \n- The method provides a simple yet effective way of dealing with catastrophic forgetting, which is to some extent original since it proposes to use the relatedness among tasks and use this information to control gradient updates while learning new tasks.\n- The method is technically sound, and most of the experimental evaluation is aligned with typical evaluations in the area, including datasets used and reported metrics.\n- The paper is well-written, well-structured and easy to follow.\n\nWeaknesses of this paper are: \n- My main concern is regarding memory/storage size requirements of the proposed method (during training). Although the proposed method avoids the need to keep examples of previous tasks, as compared to replay-based methods, and of expanding the network, compared to network expansion methods, it certainly adds memory/storage requirements of the gradients. Based on this, I would expect to see comparisons of the proposed method vs. replay-based methods and network expansion methods, in terms of memory usage.  I think that these concerns need to be fully addressed in this paper, to really demonstrate the advantages of the proposed approach. Furthermore, I would expect to see these comparisons for a reasonably large number of tasks, since memory/storage requirements increase along with the number of tasks. \n- My second biggest concern is where are the actual gains of the proposed approach. As I can infer from Table 1, most of the gain in final accuracy is due to a gain in LA. According to the definition of LA, this is actually the accuracy of each new task. Therefore, I do not agree with the claim in page 8: \"The higher LA with similar Forgetting as compared to other baselines shows that while TAG exploits the adaptive nature of existing optimizers, it also ensures minimal forgetting of the gained knowledge... Hence, even if a similar (or lower) Forgetting occurs in TAG, the higher test Accuracy (with high LA) shows that TAG is capable of retaining the gained knowledge from each task.\", since from Table 1 forgetting levels are actually very similar to those experienced by other methods and therefore a higher accuracy (which I agree is due to LA) comes mainly from learning new tasks better rather than encouraging \"minimal forgetting\" or \"retaining the gained knowledge\" as it is claimed. I would strongly suggest to clarify this point. \n- In the experiments, I do not see the reason for not comparing with network expansion methods. The fact that the proposed method is not making any change to the size of the model, as mentioned in page 8, does not create any limitation for comparing with these kinds of methods in the experimental setting used in the paper. \n- Finally, I see a lot of similarities of the proposed method with existing methods to control gradient updates such as OGD [1] and OWM [2]. Therefore, I would expect comparisons to these methods. \n\n[1] Farajtabar, M., Azizan, N., Mott, A., & Li, A. (2020, June). Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics (pp. 3762-3773). PMLR.\n\n[2] Zeng, G., Chen, Y., Cui, B., & Yu, S. (2019). Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8), 364-372.", "summary_of_the_paper": "This paper proposes TAG, a method for continual learning in the task-incremental setting. This method relies on storing and using task gradients while learning a set of supervisd tasks sequentially. The influence task-based accumulated gradients is regulated through a learning rate that is adaptive according to the relatedness of the current task with the previously observed ones. The authors report results on benchmark datasets for continual learning of up to 20 disjoint tasks. Comparisons against naive non-continual optimizers such as SGD, Adam and RMSProp are reported, along with results in the continual learning setting with some state-of-the-art methods such as EWC, A-GEM and ER. The authors report performance in terms of overall accuracy, forward transfer and learning accuracy (LA).", "summary_of_the_review": "Although the paper proposes a simple method for controlling catastrophic forgetting in the continual learning setting, there are major drawbacks flaws in the experimental results and the evaluation of the proposed approach. Given the nature of TAG, I think is is fundamental to measure and report memory/storage requirements in comparison to other methods that also require additional memory/storage such as replay-based methods and network expansion methods. Furthermore, claims around avoidance of catastrophic forgetting in the main results reported in Table 1 and explained in page 8 are not well-supported since it is clear that the method is strong at learning new tasks, while not necessarily much better than counter parts regarding catrastrophic forgetting. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aeE0YTrxg4", "reviewer": "Reviewer_m25m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n*  The paper is clearly written and easy to follow\n*  Experiments on different settings are considered. \n\n\nWeakness:\n*  The technical novelty of the proposed method is limited. In fact, TAG is similar to LA-MAML [1] (Gupta et al.,2020). Both the motivations and derived formula (Equation 2) of TAG are very similar to [1]. From this aspect, TAG can be seen as a memory-free version of LA-MAML. \n\n* misinterpretation of related works.  The author claims that “Another related work (Gupta et al.,2020) also employs an adaptive learning rate while requiring a small episodic memory. But it is based on a\nmeta-learning setting and hence beyond the scope of our paper”. This is an incorrect statement, LA-MAML  (Gupta et al.,2020) is to solve the continual learning problems with the tools from  meta learning. They are not in meta-learning setting, but instead in the continual learning setting.\n\n*  The proposed method seems to only work on the task-aware setting. More recent works focus on the task-free continual learning setting, where the task identity and boundary are unknown during the learning process.  It would strengthen the proposed method if  TAG can be applied in such scenarios. \n\n*  The experiments on more other architectures, such as MLP may demonstrate the effectiveness of the proposed method. \n\n\n\n\n\n", "summary_of_the_paper": "This paper proposes a  task-aware adaptive learning rate method, TAG, for continual learning. The optimizer TAG is to promote the learning rate if they are similar to previous tasks, while decreasing the learning rates if they are dissimilar to previous tasks without storing previous examples. The authors combined the proposed method with existing optimizers, such as Adam, SGD, etc to demonstrate the effectiveness of the proposed method. Experimental results on several datasets show the improvements over naive optimizers.  \n", "summary_of_the_review": "The method novelty is limited, and experiment needs to be improved.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WjOTKDcKht-", "reviewer": "Reviewer_qvJd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the idea is clear, and the experiments verify the effectiveness of the proposed method. However, I found the results of A-GEM in this paper are significantly worse than the original paper. For example, the accuracy of split-CIFAR100 is 54.25% in this submission, while in A-GEM the accuracy is ~62%. Could you provide more details?\n\nIn addition, though the submission addresses the catastrophic forgetting by proposing a new optimizer, the discussion of task correlations are very similar to GEM and its variants (e.g., A-GEM) and CLAW (CONTINUAL LEARNING WITH ADAPTIVE WEIGHTS). More analysis about why the proposed method can outperform other approaches would make the paper more convincing.\n\nMissing reference: [1] is a very relevant work. I think it might be valuable to compare and discuss with it.\n\nBtw, the authors change the margin of the ICLR format. Please kindly revise it in the next version.\n\nI am willing to adjust my score if the authors can address my concerns.\n\n[1] Guo, Yunhui, Mingrui Liu, Tianbao Yang, and Tajana Rosing. \"Improved schemes for episodic memory-based lifelong learning.\" arXiv preprint arXiv:1909.11763 (2019).\n\n\n---After rebuttal --- \n\nThe authors' response does partially address my concerns. After reading the authors' rebuttal and other reviewers' comments, I still think the contributions are not enough for publication. I will keep my score.", "summary_of_the_paper": "In this paper, the authors propose a new optimization method for continual learning. The authors propose a task-aware optimizer to adapt the learning rate for each task. The proposed method is evaluated on several datasets to show its effectiveness.\n", "summary_of_the_review": "The idea is clear and the results seem promising. However, the authors are supposed to provide more analysis to understand why the proposed method works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bpf6DqimTCo", "reviewer": "Reviewer_ZR5n", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* General approach that can be adapted to multiple optimization algorithms and used in other meta-algorithms\n* Strong performance in a single-pass scenario\n* Can achieve better learning accuracy than naive methods\n\nCons:\n* The effect of the proposed procedure on optimization algorithms is not studied enough. For example, it’s unclear how would a few similar tasks help to learn faster when there are many distinct tasks present. Additional insights, like visualizations of trajectories, will help to build an intuition and boost the method adoption in practice. \n* Effect of task order is not studied/highlighted: the method seems to be very dependent on the task order and it is unclear how the order is chosen in experiments.\n\nMinor comments/questions:\n* what is the point of having \\lambda(t,t) in eq. 3, i.e. why do you need to multiply the currently accumulated gradients by task similarity to itself?\n* In appendix 3.3, Figure 5(a) there is a noticable difference between RMSProp and TAG-RMSProp on the first task. What is the reason for this? Shouldn’t they be identical/close?", "summary_of_the_paper": "The authors present an approach for lifelong learning where each task is processed in a single pass. They propose to adapt the learning rate of the training algorithm depending on the current task's similarity to the previously observed tasks. The learning rate is decreased when there are many dissimilar tasks to avoid catastrophic forgetting. The paper presents an adapted RMSProp algorithm, but the procedure can be adjusted to Adagrad and Adam as well. The presented evaluation on computer vision datasets shows that the proposed modification helps to increase the final accuracy of those methods while keeping the forgetting low. When compared to baselines from literature under the conditions of the single pass setting, the proposed method achieves better final accuracy and keeps approximately the same forgetting rate. ", "summary_of_the_review": "The presented method is a general procedure that helps to improve performance of many methods in lifelong learning scenario. The presented experimental results are convicing. However, the paper would benefit from more research and intuition on why the approach works. I recommend to accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors develop a memory-based method for continual learning that stores gradient information from past tasks. This memory is then used by a proposed task-aware optimizer that, based on the task relatedness, aims at preserving knowledge learned in previous tasks.

The initial reviews were reasonable but indicated that this paper was not yet ready to be published. In particular, the reviewers seemed to agree on the somewhat limited methodological novelty of the paper given prior work (such as LA-MAML and OGD in terms of method and GEM in terms of task similarity comparison).

In their response, the authors do seem to agree to a certain extent with some of the criticisms, but also point to clear differences with respect to previous work (and other distinguishing aspects such as a smaller memory footprint than OGD). The authors also carefully responded to reviewer comments and provided additional results when possible.

In the end, the main criticism from the reviewers remained (Reviewer 95tf also suggests that the authors should compare their method to others in terms of memory consumption (which the authors partly did) and compare to replay-based methods) and this paper was a borderline one. Three, out of the four, reviewers suggest that it is not ready to be published. One reviewer did give it a high score (8) but also understood the limitations raised by the other reviewers. As a result, my recommendation is that this paper falls below the acceptance threshold. 

I am sorry that for this recommendation and I strongly suggest the authors consider the reviewer's suggestions in preparing the next version of this work. In particular, it seems like providing a full study of the memory usage of your approach vs. others as well as providing more insights about the "trajectory" (see the comment from ZR5n) might go a long way toward improving the paper.
Decision: Reject