Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer
Authors: Sen Lin
Li Yang
Deliang Fan
Junshan Zhang
Venue: NeurIPS 2022 Accept (2022)

Abstract: By learning a sequence of tasks continually, an agent in continual learning (CL) can improve the learning performance of both a new task and `old' tasks by leveraging the forward knowledge transfer and the backward knowledge transfer, respectively. However, most existing CL methods focus on addressing catastrophic forgetting in neural networks by minimizing the modification of the learnt model for old tasks. This inevitably limits the backward knowledge transfer from the new task to the old tasks, because judicious model updates could possibly improve the learning performance of the old tasks as well. To tackle this problem, we first theoretically analyze the conditions under which updating the learnt model of old tasks could be beneficial for CL and also lead to backward knowledge transfer, based on the gradient projection onto the input subspaces of old tasks. Building on the theoretical analysis, we next develop a ContinUal learning method with Backward knowlEdge tRansfer (CUBER), for a fixed capacity neural network without data replay. In particular, CUBER first characterizes the task correlation to identify the positively correlated old tasks in a layer-wise manner, and then selectively modifies the learnt model of the old tasks when learning the new task. Experimental studies show that CUBER can even achieve positive backward knowledge transfer on several existing CL benchmarks for the first time without data replay, where the related baselines still suffer from catastrophic forgetting (negative backward knowledge transfer). The superior performance of CUBER on the backward knowledge transfer also leads to higher accuracy accordingly.

Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer
Sen Lin School of ECEE Arizona State University slin70@asu. edu
Li Yang School of ECEE Arizona State University 1yang166@asu.. edu
Deliang Fan School of ECEE Arizona State University df dfan@asu.c edu
Junshan Zhang Department of ECE University of California, Davis jazh@ucdavis. edu
Abstract
By learning sequence of tasks continually, an agent in continual learning (CL) can improve the learning performance of both new task and 'old" tasks by leveraging the forward knowledge transfer and the backward knowledge transfer, respectively. However, most existing CL methods focus on addressing catastrophic forgetting in neural networks by minimizing the modification of the learnt model for old tasks. This inevitably limits the backward knowledge transfer from the new task to the old tasks, because judicious model updates could possibly improve the learning performance of the old tasks well. To tackle this problem, we first theoretically analyze the conditions under which updating the learnt model of old tasks could be beneficial for CL and also lead to backward knowledge transfer, based the gradient projection onto the input subspaces of old tasks. Building on the thcoretical analysis, next develop Contin Ual learning method with Backward knowlEdge tRansfer (CUBER), for fixed capacity neural network without data replay. In particular, CUBER first characterizes the task correlation to identify the positively correlated old tasks a layer-wise manner, and then selectively modifies the learnt model of the old tasks when learning the new task. Experimental studies show that CUBER can even achieve positive backward knowledge transfer on several existing CL benchmarks for the first time without data replay, where the related baselines still suffer from catastrophic forgetting (negative backward knowledge transfer). The superior performance of CUBER on the backward knowledge transfer also leads to higher accuracy accordingly.
1 Introduction
One ultimate goal of artificial intelligence is to build an agent that can continually learn a sequence of different tasks, as echo the remarkable learning capability of human beings during their lifespan. With more tasks being learnt, the agent is expected to be able to learn a new task more easily by leveraging the accumulated knowledge from old tasks (forward knowledge transfer), and also further improve the learning performance of old tasks based on the gained knowledge of related new tasks (backward knowledge transfer). Such learning paradigm is known as continual learning which has recently attracted much attention.
(CL) 5 23
Most existing CL methods focus addressing the catastrophic forgetting problem 20) i.e., the neural network may easily forget the knowledge of old tasks when learning a new task. The main
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

strategy therein is to avoid the model change of old tasks when learning the new task. For example, regularization based methods (e.g.. [
18) penalize the modification of important weights of
old tasks; parameter-isolation based methods (e.g.. 7
31 [ fix the model learnt for old tasks; 26
and memory-ban methods ge.g. 3 6 25 ) aim to update the model with minimal interference
introduced to old tasks. While effectively mitigating catastrophic forgetting, such a model-freezinn strategy inevitably limits the backward knowledge transfer, by implicitly and conservatively treating the model update of the new task interference to old tasks. Intuitively, careful modifications of the learnt model of old tasks may further improve the learning performance especially when the new task shares similar knowledge to the old tasks.
This work seeks to explore CL from new perspective by going beyond merely addressing forgetting: When and how could we improve the learnt model of old tasks to facilitate backward knowledge transfer? In particular, we consider a fixed neural network and the scenario where the data of old tasks is not accessible when learning new task, which naturally rules out the aypansion-based Clearly, achieving backward knowledge transfer is very challenging in this case as mitigating forgetting herein is already nontrivial. To tackle this challenge, note that the orthogonal-projection based CL methods (e.g., [25 [7)) which update the model with gradients orthogonal to the input subspaces of old tasks, have demonstrated the state-of-theearr performance under the setting above. Thus motivated, we propose a new orthogonal- projection based CL method to carefully modify the learnt model of old tasks for better backward knowledge transfer.
methods (e.g. [1 and the experience-replay methods (e.g.. [22 4
The main contributions of this work include 1) the quantification of the conditions under which modifying the learnt model of old tasks is beneficial and 2) a new CL method inspired by the analysis, so as to answer the question above. More specifically, we first introduce notions of 'sufficient projection' and 'positive correlation' based on the gradient projection onto the subspaces of old tasks to characterize the task correlation. We theoretically show that when the task gradients are sufficiently aligned in the old task subspace, appropriately updating the learnt model of old tasks when learning the new task could improve the learning performance and possibly lead to better model for the old tasks. Based on this analysis, we next propose an orthogonal projection CantinUal learning method with Backward knowlEdge tRansfer (CUBER), which 1) first identifies the positively correlated old tasks for the new task and 2) carefully updates the learnt model of selected old tasks along with the model learning for the new task. The experimental results on standard CL benchmarks show that CUBER can achieve the best backward knowledge transfer compared to all the considered baseline CL methods, which consequently improves the overall learning performance. In particular, to the best of our knowledge, CUBER is the first to demonstrate positive backward knowledge transfer on these benchmarks using fixed capacity network without experience replay, whereas all the compared baselines still suffer from catastrophic forgetting with negative backward knowledge transfer.
proooggcttt-oocttiionn based
2 Related work
Continual learning. Existing CL methods can be generally divided into three categories: 1)
Regularization- -baed methods (e.g., [12 [ [5] 18) add additional regularization in the loss function to penalize the model change of the old tasks when learning the new task. For example, Elastic Weight Consolidation (EWC) 12 regularizes the update on the important weights that are evaluated using the Fisher Information matrix; a recursive gradient optimization method is proposed in [18] to modify the gradient direction for the objective function regularized by the model change during CL. 2) Parameter-
isolation based methods (e.g.. 24 16] 31,
30 26 29)) allocate different model parameters to different
tasks, fix the parameters for old tasks by masking them out during the new task learning and expand
the network when needed. For example, Additive Parameter Decomposition (APD) [30] proposes a hierarchical knowledge consolidation method to separate task-specific parameters and task-shared parameters, where the task-specifi parameters are kept intact to address forgetting. 3) Memory-baser methods can be further divided into experience-repla methods and authogonal-projection based methods depending on if the data of old tasks is available for the new task. Experience-reelay methods (e.g., 22 g store and replay the old tasks data when learning the new task, while orthogonal-
projection based methods (e.g.. [32
17 update the model in the orthogonal direction of old tasks without storing the raw data of old tasks. particular, [10] proposes natural continual
25
learning based on Bayesian learning to unify weight regularization and orthogonal gradient projection.
2

---

However, [TO] leverages the knowledge of the current task in an implicit way to improve on the old tasks, whereas our method attempts more explicitly update the model of old tasks whenever relevant new information is available.
Knowledge transfer. Most of CL methods focus the catastrophic forgetting problem, and little attention has been put on the knowledge transfer across tasks in CL with neural networks. Progressive Network [24] considers the forward knowledge transfer but learns one model for each task. [17 proposes trust region gradient projection (TRGP) to facilitate forward knowledge transfer from correlated old tasks to the new task through scaling matrix, but the model is still updated along the direction orthogonal to the input subspaces of old tasks to mitigate forgetting. [II] studies the knowledge transfer a mixed sequence of similar and dissimilar tasks, but employs a complicated task similarity detection mechanism where two additional networks need to be trained first for each task before learning the new task.
When could we improve the learnt model of old tasks?
In continual learning, sequence of tasks T (1)T=1 arrives sequentially. For each task i, there is dataset D' {(a), 1 with No sample pairs, which is sampled from some unknown distribution P.. In this work, we consider fixed capacity neural network with weights w. When learning a new task L, we only have access to the dataset ID* and no datapoints of old tasks are available. We further denote L(w, ID²) Le (u(w) as the loss function for training, e.g., mean squared and cross-entropy loss, and w as the model after learning task
Intuitively, if the new task has strong similarity with some old tasks, appropriate model update of the new task would not introduce forgetting of the similar old tasks, but can lead to better model for these old tasks due to the backward knowledge transfer. To formally characterize this task similarity, we introduce the following conditions on the task gradients.
Definition 1 (Sufficient Projection). For any new task t E [1. TJ, we say it has sufficient gradient projection on the input subspace of old task j E [0, - 1] for some E (0,
[Prijs (VLi(w' M| 2 [Lr(
Here Projs; defines the projection on the input subspace S of task j: Projs: (A) = AB' (BYY for some matrix A and B is the bases for S3. This definition of sufficient projection follows the same line the trust region defined 17). which implies that task t and j may have sufficient common bases between their input subspaces and hence are strongly correlated, because the gradient lies in the span of the input [33.
Definition 2 (Positive Correlation). For any new task t [1, []. we say it has positive correlation with old task j e [0, - 1] for some 62 E (0,1)
(VL,(w"), VLi(w'-)) 2 VL, Vla
While sufficient projection suggests the possibly strong correlation between two tasks and j. Definition goes one step further by introducing the positive correlation between tasks, in the sense that the initial model gradient of task not conflicting with the model gradient when learning task In fact, it can be shown that the positive correlation implies the sufficient projection in Definition ] characterize the correlation based on the initial model w which allows the task correlation detection before learning the new task t.
for some e 2 E2. Note that both Definition I and 2
For ease of exposition, consider the scenario with a sequence of two tasks and 2. Let F(w) L(w, D2) + L(w, D2), (ww) V C(w, Dí) and (w) = V L(w, D2). Given the model learnt for task 1 as w we consider the following two model update rules for learning task 2, where only the data of task 2 is available:
(Rule #1): W6++1 = Wi a (lgg) Projs: (ga(m)))
(Rule #2): wh+11 Wk agz(ww))) (wa).
Here wo = and k E [0, K 1]. Rule #1 characterizes the model update of the orthogonal- projection based methods, where no interference is introduced to task as the model change
orthogonal to the input subspace S of task 1 [17] (and hence the learnt model of task will not
3

---

be updated). In contrast, the vanilla gradient descent Rule #2 for learning task 2 will inevitably modify the learnt model w' for task unless Projs- (gr(w))) = 0, i.e. the extreme case where both forgetting and knowledge transfer do not occur. In what follows, we evaluate the performance of these update rules.
Theorem 1. Suppose that the loss C is B-Lipschitz and 2 -mooth. Let a < min Ille (woll
H HBK
(2+72)llax(0)) for some r E (0, 1). We have the following results:
and 62 2
41]92("0all
(1) If is convex, Rule #2 task 2 converges to the optimal model w" = arg min F(w); (2) If C is nonconvex, Rule for task 2 converges to the first order stationary point, i.e.,
aK (F(we) F(w")) ]] 7 l1g. (wo))l
min [[V
Theorem | indicates that updating the model with Rule #2 will lead to the convergence to the minimizer of the joint objective function F(w) in the convex case, and the convergence to the first order stationary point in the nonconvex case, when task and 2 satisfy the positive correlation with (2+-2)]991
f2 2
That is to say, Rule #2 not only results in a good model for task 2, but can also be beneficial for the joint learning of task and 2. Note that since wo is the learnt model of task 1, in general we have l[g (wo))l ||ga (wg()))) |I. Proof of Theorem can be found in Appendix A.
4|[ga (wo) I|
Theorem 2. Suppose that the loss L is B- B-Lipschitz and 2 -smooth. We have the following results:
(1) Let w" and w" be the model parameters after applying one update some initial model w by using Rule #11 and Rule #2, respectively. Suppose a < minn
{ T7 HBK
2+all and ) J. 51 1
L+2oH
2 (2+r2)l99 for some E (0, It follows that F(w") S F(w"); 41og)(we)ll
(2) Let wh be the k-th iterute for task 2 with Rule #2. Suppose thut (g) (wo); 92(w:)) > (c2llg:(wo)) It follows that (i(w)) S L1 (ww)).
callgi (9:(())) for i [0, k- 11alaaa and a S
BBBLS
Intuitively, the first part of Theorem 2 shows that updating with Rule #2 can achieve lower loss value compared to Rule #1 after one step gradient update when task and 2 satisfy the sufficient
projection with E 2 / L+2aH and the positive correlation with 62 2 (2+>") (we)]l, If the positive 2+afff
41[g2(wx))] uxa)ll
correlation condition also holds for iterates of Rule #2 when learning the task 2, the second part of Theorem _ lindicates that updating the model indeed leads to a better model for task 1 with respect to L1 In a nutshell, when 1) the task 2 has sufficient gradient projection onto the subspace of task and 2) the projected gradient is also aligned well with the gradient of task in the subspace of task 1, updating the model along 92 will modify the learnt model w towards a favorable direction for CL and enable the backward knowledge transfer to task 1. Proof of Theorem is in Appendix B.
It is worth noting that the conditions for Thcorem I [ad 2 depend only the initial model gradient g2(w²) before learning task and the gradient gı (w') when learning task 1, which are both casily accessible and calculated. Particularly, the positive correlation can be evaluated by only storing the gradient 91 (w²) instead of the data of task 1. In stark contrast, the task gradient correlation characterization in B 22 involves the gradient evaluation of old tasks with respect to the current model weight, and hence requires the data of old tasks when learning the new task.
4 Continual learning with backward knowledge transfer
Based on the theoretical analysis above, we next propose a continual learning method with backward knowledge transfer (CUBER), by selectively updating the learnt model of old tasks when learning the new task. In particular, CUBER works in layer-wise manner: Given L-layer network, CUBER first characterizes the task correlation for each layer, and then employs different strategies learn the new task depending on the task correlation.
More specifically, denote the set of weights as w [wi)f where wi is the layer-wise weight for layer I. Given the data input of for task I, let æ be the input of layer and æf, = æç. Denote f as the operation of the network layer. The output æit,, for layer can be then computed as

---

Regime 2: forward knowledge transfer
Regime 1: forgetting
(RE((w)) vLL(())))
If not Regime
(VL 2e
Regime 3:
backward knowledge transfer
(pE((wE).E,,((EE)) (L((W(")))
salloe, (w/))ll VG
(c)
rrgdient Regime Gradient projection region for Regime Gradient needs to be oumiió of Regime 1.
Gradient projection region for Rog me Gradient needs to outride Regiiee
Imput subspccee S/ for Jock
Gradient tock
Figure 1: A simple illustration of the layer-wise task correlation detection. Given the new task t, an old task j belongs to (1) Regime 1 if the initial model gradient VL,(w) of task has small projection onto the subspace Sy of task j; (2) Regime 2 if the strong projection condition is satisfied while the projection of V ( onto S/ is not aligned well with the gradient of task j; (3) Regime 3 if both the strong projection condition and the positive correlation condition are satisfied.
E441, = f(w;, ai,)) Here we denote क as the representations of the input æf at layer l. Given new task & 2 2, we characterize its task correlation with some old task [1, t for layer I into three different regimes based on Definition and 2
Regime (no forgetting): We say that task Regi,1 for layer if the following holds:
[|rooss| (VL.(w) & V
In this case, the layer-wise input subspace S for task and St for task are treated nearly orthogonal ((a) in Fig. D.. As result, there little knowledge transfer between these two tasks, and updating the model along with (L((wr would not introduce much interference to task j. To reinforce the knowledge protection for task j. the model will be updated based on orthogonal projection:
(.m) (w)(
Proig; (W.((
Regime 2 (forward knowledge transfer): We say that task j Regi, for layer if the following holds:
[|Projs] (VL. (V (wl e [[
(V (w. - e2 VLI (w)ill C
In this case, task j and task can be strongly correlated for layer / but possibly with 'negative' corre- lation ((b) in Fig. [].. in the sense that updating the model along with V would substantially modify the learnt model Projs; (w--1) for task j in an unfavorable way and lead the forgetting of task j. As there will be better forward knowledge transfer from the old task j to the new task t for layer 1, we leverage the scaled weight projection in [17] to facilitate the forward knowledge transfer through scaling matrix Q. whereas the model is still updated using orthogonal projection protect the knowledge of old tasks:
(c(ww: - (c((w) - projg; (Wr)()))),
O - Qf" BVoL((w: - Projgy (wi) + wi (BB)).
(I)
Here By/ is the bases matrix for subspace sp. Intuitively, the scaled weight projection un B/ (B/y replaces the weight projection Proje; (mn) of task j by a scaled version, which transforms the knowledge of task to the appropriate model of the new task through the optimization of Qf
Regime (backward knowledge transfer): We say task j E Regf,3 for layer the following holds:
||rojs (VC.(w) (1)]l a [
(VL, VLi(w ea VI (tt

---

With the sufficient projection and the positive correlation, updating the model along with V (w)) could possibly lead a better model for continual learning and also improve the learning performance of the old task ((c) in Fig. D. To avoid overly-optinistic modification on the learnt model of task we further regularize the projection of the model change the subspace S/, given that the model projection is indeed frozen for task j to address forgetting with orthogonal projection, i.e., gr (w) ") = grojg (w/). This gives the following model update:
Wi wi (V( X||Proisy (wn - - -1]]-
Note that the gradient projection the subspaces of old tasks in Regime 1 and is removed from the gradient in the model update above. Besides, also learn scaling matrix Q!* for better forward
knowledge transfer from the old task j E Regi,3 to the new task t as in Eq.
However, continuous model update with task & gradient V (((())
will eventually lead to the task specific model for task L, which usually differs from the model of task j in Regime 3 (Fig. 2 To address this problem, note that the second part of Theorem 2 characterizes the condition under which updating the model with (.,(w,) will result in backward knowledge transfer. Thus motivated, for model update iterate w at k-th iteration when learning the new task t, we evaluate the following condition for task j E Regi,s
VLj
VLt
A simple illustration of the case where updating the model with VL, benefits the old task j E Regi,s at the be- ginning but eventually conflicts with VLj-
Figure 2:
(2) and degenerate task j to Regime 2, i.e., remove the gradient projec- tion on the subspace S/ from the task f gradient and stop modifying the model for task j. if the condition ( does not hold.
(V (L,(w),)) 2 (wP))l)
Bases extraction: After learning the model w² for task 1, we construct the input subspace for each layer 1 by extracting bases from its representation æi,, based singular value decomposition (SVD) [17]. More specifically, given a batch of samples and the learnt model w", the representation matrix for layer is denoted as R2 [af "[,2 _ Since the bases of the old tasks may include important bases for the new task t, we determine the bases for task t from the union of the bases of the old tasks and the newly generated bases. Towards this end, we first concatenate the bases B/ for j E [0, - 1] together in a matrix Of and remove the common bases. Then SVD is applied on the residual representation matrix i? = R² (O)). i.e., Ri (V where U/ is the left singular matrix. We construct B3 by selecting the most important bases from the pool of bases in Of and U/ depending on their eigenvalues, which yields a low rank matrix approximation of Rt
To conclude, the optimization problem for learning the new task can be summarized as follows:
min w. w.t0 ). Reaf.
+ E
Σ
[|rojj (wi wi w
JEReoL
s.t.
wi wi +
Σ
[wo [wo Bf Projsy (wi)),
, Regt,a
VL,(w)) = VL((w))
Σ
Proige (VLi(w))).
jEResf, Ueef
The key idea is that we conservatively update the model for old tasks in Regime 3 while using orthogonal projection to preserve the knowledge of other old tasks; in the meanwhile, we leverage the scaled weight projection reuse the model knowledge of old tasks both Regime and 3 to facilitate forward knowledge transfer. It is worth to note that the task correlation is determined before learning the new task I, as both the strong projection condition and the positive correlation condition only depend on the initial model gradient for the new task. And this can be achieved by a simple -awaad-backwwrd pass through the initial model with a batch of new task data. The overview of CUBER can be found in Algorithm

---

Algorithm 1 Continual learning with backward knowledge transfer (CUBER)
1: Input: task sequence T {(t)/=i
2: Learn the first task using vanilla stochastic gradient descent;
3: Extract the bases {Bi based on SVD using the learnt model
4: for each task do
5: Calculate gradient V C.
6: Evaluate the strong projection and the positive correlation conditions for layer-wise task correlation
detection to determine Regi, Regi,, and Regi,3 for k=1, 2.... do
7:
8:
Update the model and scaling matrices by solving Eq. 3 for task < f and E Regi,a do if (VLz(wf)) V. (wr, < e2 (xI Degenerate task to Regi,2;
9:
10:
then
11:
12:
end if end for
13:
14:
end for
15:
Store the gradient (.,(ww) the task memory;
16: Extract the bases {B)) } based on SVD using the learnt model w';
17:
end for
5
Experiments
Datasets. We evaluate the performance CUBER on four standard CL benchmarks. (1) Permuted MNIST: a variant of the MNIST dataset [4] where random permutations are applied to the input pixels. Following 19 25], we divide the dataset into 10 tasks with different permutations and each task includes 10 classes. (2) Split CIFAR- 100: we divide the CIFAR-100 dataset [13] into 10 different tasks, where each task is 10-way multi-class classification problem. (3) 5-Datasets: we consider a sequence of 5 datasets, i.c., CIFAR-10 10, MNIST, SVHN [21), Mot-MNIST Fashion MNIST(88) and the classification problem on each dataset is a task. (4) Split MinilmageNet: we divide the MinilmageNet dataset 27 into 20 tasks, where each task includes 5 classes
Baselines. this work, we compare CUBER with the following baselines on the benchmarks mentioned above. (1) EWC [12]: regularization-based method that leverages Fisher Information matrix for weights importance evaluation; (2) HAT [26]: learns a hard attention mask to preserve the knowledge of old task; (3) Orthogonal Weight Modulation (OWM) (32: : learns a projector matrix to project the gradient of the new task to the orthogonal direction of input subspace of old tasks; (4) Gradient Projection Memory (GPM) (25]: stores the bases of the input subspace of old tasks and then updates the model with the gradient projection orthogonal to the subspace spanned by these bases; (5) TRGP 17]: proposes scaled weight projection to facilitate the forward knowledge transfer from related old tasks to the new task while updating the model based on orthogonal gradient projection, which demonstrates the state-of sate-of-thhe-n performance for fixed capacity network; (6) Averaged GEM (A-GEM) 33: constrains the new task learning with the gradient calculated using the stored data of old tasks; (7) Experience Replay with Reservior sample (ER Res) [ uses small episodic memory to store old task samples for addressing forgetting; (8) Multitask: jointly learns all tasks once with a
single network using all datasets, which usually serves performance upper bound in CL. 25
Network and training details. For a given dataset, we study all CL methods using the same network architecture. More specifically, for Permuted MNIST, we consider 3-layer fully-connected network including 2 hidden layers with 100 units. And we train the network for epochs with a batch size of 10 for every task. For Split CIFAR-100,
a version of 5-layer AlexNet by following 23 17]. When learning each task, we train the network for a maximum of 200 epochs with early termination based the validation loss, and use batch size of 64. For 5-Datasets,
reduced ResNet-18 [9] and follow the same training strategy as in Split CIFAR- 100. For Split MinilmageNett reduced R8 is also used, and we train the network for a maximum of 100 epoches with early termination. The batch size is 64. Similar to [17,, we select at most two tasks to be in Regime 2 and 3 for each layer with the largest gradient projection norm, reduce the performance sensitivity on the choice of ej. In the experiments, we set é = 0.5.
To evaluate the learning performance, we consider the following two metrics, i.c., accuracy (ACC) which measures the final accuracy averaged over all tasks, and backward transfer (BWT) which
7

---

measures the average accuracy change of each task after learning new tasks:
ACC Σ AT. BWT = - (Ar, Ar,i)
where Ai, represents the testing accuracy of task j after learning task i.
5.1 Main results
As shown in Table DJ CUBER demonstrates the best performance of BWT on all datasets compared to the baselines. In particular, positive BWT can be obtained by CUBER on Split CIFAR-100 and Split MinilmageNet, which has not been achieved by previous works in a fixed capacity network without data-replay. For 5-Dataset, since the tasks therein are less related to each other as in GPM (25). we do not expect much knowledge transfer across tasks (but knowledge transfer still exists in terms of the layer- features), and there is no forgetting in CUBER. Achieving rero-forgettin is very difficult for Permuted MNIST, because all the tasks share one output layer and there is no task identifier during testing (domain-incrementa)) [25]. However, even in this case CUBER still achieves the best BWT, i.e, nearly non-forgettigg, among all methods. Clearly, the strong performance on BWT indicates that CUBER can effectively facilitate the backward knowledge transfer by wisely modifying the learnt model of old tasks.
Benefiting from the superior performance on the backward knowledge transfer. CUBER also achieves the best or comparable performance on the averaged accuracy. More specifically, CUBER improves around 1% in ACC over the best prior results on Split CIFAR-I00, Split MinilmageNet and Permuted MNIST, while showing the comparable performance with the offe-of-the-ar method TRGP even on 5-datasets where tasks are less correlated. Moreover, CUBER performs better than Multitask on both 5-Dataset and Permuted MNIST, which implies the importance of studying knowledge transfer in CL: Both TRGP and CUBER outperform Multitask on 5-Dataset because of the scaled weight projection to facilitate forward knowledge transfer, whereas by facilitating backward knowledge CUBER becomes the only method that outperforms Multitask on Permuted MNIST.
Table 1:
The ACC and BWT with the standard deviation values over 5 different runs on different datasets. Here for Split CIFAR-100 Split MinilmageNer and 5-Dataset we use multi-head network, while a single-head network is used for Permuted MNIST. Moreover, 62 = 0.0.
Multi-head
Method
CIFAR 100
Spli MinilmageNNt
D-Dtaeet
Penmuned MNIST
ACC(S)
BWT(S)
ACCOS)
BWT(S)
ACCY%)
BWT(%)
ACC(S)
BWT(S)
Multitak
79.58
0.54
69.46 0.62
91.
54 : 0.28
96.70 + 0.02
OWM
50.04
0.60
90.71 + 0.11
EWC
68,80
0.38
1.
52.01 2.53
-12 1
88
0.26
-4
89.97 1 0.57
11
HAT
72.06 +: 0.50
0=:00 0
59.78 =: 0.57
-3:t0 0
81
+ 0.18
-1 -1=0
A-GEM
63.98
1:222
2:5
57:24 1 0.72
-12.1
84.04
2 0.33
-12.2
83.56 :: 0.16
-14
FR Res
71.73
0.63
-6.1
58.94
2 0.85
-7.1
68:31
2 0.22
-1=0
87-24 - 0.53
-11-1
GPM
72.48
0.40
-0.9.1:
60.41
0.661
-0.7
01.22
0.20
-1=0
93.91 1: 0.16
-3 -3.10
TRGP
74.46
0.32
-0.9 1..9+00.00 0.01
61.78
1.76=00.00 0.60
-0.6 0.6
98.56
0.10
-0.04=00m
96.34 = 0.11
-0.8 0.1
CUBER (suns)
75.54 0.22
0.18 = 0.08
62.67 = 0.74
0.23 = 0.15
98.48 = 0.10
0.00 + 0.02
97.25 + 0.00
-0.02 + 0.00
5.2 Ablation studies
Table 2: The comparison between CUBER and TRGP in OL-CIFAR100. The selected old task (*) in (b) represents the old task with the largest number of layers in Regime 3 of the new task.
(b) BWT-S the selected old tasks.
(a) The comparison of ACC and BWT.
Tack
Tack
Task
Task
Average
Methed
ACC1%)
BWT(%)
sclected cld task"
Tiak 0
Taak
Tack
Task
CUBER
76,94
0.28
DWTS (CUBER)
0.50
0.07
0.3
TRGP
74,33
41.18
BWT-S (TRCIP)
.0.20
0.10
40.0
40.20
-0.33
Backward knowledge transfer. The value of backward knowledge transfer indicates the average accuracy improvement for each old task after learning all tasks, which implies that the new task learning provides additional useful information for learning features of similar old tasks. Intuitively, this value should depend on the task similarity in CL. To better understand the advantage of CUBER
8

---

in terms of backward knowledge transfer, we further consider special setup which includes sequence of similar and dissimilar tasks. Specifically, different with Split- 100 where no tasks have overlapping classes, we split the first 50 classes in CIFAR 100 into 7 tasks (OL-CIFARIOO) Task 0-6 contain classes 0-9, 5-14, 10- 19, 20-29, 25-34, 30-39, 40-49, respectively. We compare the performance of CUBER with TRGP in Table 2 As shown in Table 2a CUBER clearly outperforms TRGP in both ACC and BWT. In particular, CUBER has a positive BWT of 0.28%, where TRGP suffers from forgetting. We further analyze the task selections in Table 2b where "selected old task" refers to the task that has the largest number of layers in Regime of the new task. For example, according to the setup of OL-CIFAR 100, the selected old task for Task should be Task with high probability as they have overlapping classes. And we denote a new metric, namely BWT-S, to evaluate the backward knowledge transfer of the selected old task after learning the new task, i... BWT-S=A,, - A where j is the "selected old task" of the new task 1. As shown in Table 2h CUBER correctly identifies the correlated tasks for most new tasks except Task 5. This is reasonable result because the task correlation detection is based on the initial model gradient which is noisy in general, and hence only serves as estimation of underlying true correlation. However, such an estimated task correlation characterization is indeed sufficient to effectively facilitate backward knowledge transfer, as corroborated by the positive BWT- and the superior performance of CUBER.
Forward knowledge transfer. We also evaluate the forward knowledge transfer (FWT) in CUBER, compared to the best two baseline methods GPM and TRGP. Here the FWT measures the gap between A,, and the accuracy of learning task i only from scratch. For simplicity, we use the FWT of GPM as baseline and evaluate the improvements of TRGP and CUBER over GPM. It can be seen from Table 3 that CUBER performs even better than TRGP although CUBER follows the same strategy, i.e., the scaled weight projection,
facilitate the forward knowledge transfer, and achieves the best FWT among the three methods in most cases. The reason behind is that the characterization of Regime 3 in CUBER not only allows the modification of the learnt model of the old tasks prompt the backward knowledge transfer, but also relaxes the constraint on the gradient update for the new task, i.e., the model can be now updated in the subspace of the selected old tasks for learning the new task. This gradient constraint relaxation consequently leads to better model learning of the new task.
Table 3: Comparison of FWT among GPM, TRGP and CUBER. The value for GPM is zero because we treat GPM the baseline and consider the relative FWT improvement over GPM.
Method
CIARRI00
Split MinilmagaNee
S-Datset
Ponmuted MNIST
GPM
0
TRGP
|
2.01
0.38
CUBER
2.79
3,13
1.96
0,8
Impact of c2. It is clear that the selection of layer-wise Regime 3 depends on the value of the threshold <2. To show the impact of e2, we evaluate the learning performance under different values of in Split CIFAR-100. As shown in Table 4 the performance three cases and the BWT decreases as the value of e2 increases. Intuitively, 62 characterizes the conservatism in selecting tasks to Regime 3 and modifying the model of selected tasks. Specifically, with a larger 62, we just consider the backward knowledge transfer to the old tasks that are strongly correlated with the new task, and only slightly modify the learnt model of these selected tasks, because the condition Eq. ( can be quickly violated with the model update and CUBER will stop changing the learnt model of the selected tasks.
ACC is comparable for all
Table 4: The impact of E2 on the performance in Split CIFAR-100.
0.0
0.2
0.5
ACCSS)
BWT(S)
ACC(S)
BWT(%)
ACC(S)
BWT(3)
75.54
0.22
75.75
0,03
75.55
0.31
6 Conclusion
In this work, we study the problem of backward knowledge transfer in CL. Different from most existing methods that generally freeze the learnt model of the old tasks so as to mitigate catastrophic forgetting, this study seeks to carefully modify the learnt model to facilitate backward knowledge transfer from the new task to the old tasks. To this end, we first introduce notions of strong projection
9

---

and positive correlation to characterize the task correlation, and show that when the task gradients are sufficiently aligned in the old task subspace, appropriate model change for the old tasks could be beneficial for CI. and result in better backward knowledge transfer. Based on the theoretical analysis, we next propose CUBER to carefully learn the model for the new task, which would carefully update the learnt model of the old tasks that are positively correlated with the new task. As shown in the experimental results, CUBER can successfully improve the backward knowledge transfer on the standard CL benchmarks in contrast to related baselines.
Impact and limitations. The mainstream strategy nowadays to address forgetting is to minimize the interference to old tasks and avoid the learnt model change, which may however conflict with the goal of CL, in the sense that the backward knowledge transfer from the new task to old tasks can be restricted without modifying the learnt model. In this work, we go beyond this strategy and shed light on the relationship between backward knowledge transfer and model modification, by characterizing the task correlations with gradient projection. We hope that this work will serve as initial steps and motivate further research in CL community on the important while less explored problem, i.e., how to design algorithms that can provide targeted treatments to achieve backward knowledge transfer. based
However, CUBER also comes with several limitations. As in recent wrthogonal-projecte CL methods, CUBER extracts the bases of the task subspaces based on SVD, which may lead to high computational cost for large dimensional data. How to reduce the complexity is an interesting direction. Another limitation is that assume that clear task boundaries exist between different tasks. In future work, it is of great interests to extend CUBER to more general CL settings.
Acknowledgement
The work of S. Lin and J. Zhang was supported in part by the U.S. National Science Foundation Grants CNS-2203239, CNS-2203412, RINGS- 214GS-2 and CCSS-2203238. The work of L. Yang and D. Fan was supported in part by the U.S. National Science Foundation Grants No. 1931871 and No. 2144751.
References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV), pages 139-154, 2018.
[2] Yaroslav Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online]. Available: http://yyrrollavvb blogspot. it/201 10901/////ommmistaauuuseee html, 2, 2011.
[3] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint
2018.
[4] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019.
[5] Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-207, 2018
[6] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762-3773 PMLR, 2020.
[7] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 4rXiv:1701.08733 2017.
[8] Yunhui Guo, Mingrui Liu, Tianbao Yang. and Tajana Rosing. Improved schemes for episodic memory based lifelong learning algorithm. In Conference on Neural Information Processing Systems, 2020.
10

---

[9] Steven CY Hung, Cheng Ghong-Hao Tu, Cheng-En Wu, Chien- Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. arXiv preprini arXiv:1910.06565 2019.
[10] Ta-Chu Kao, Kristopher Jensen, Gido de Ven, Alberto Bernacchia, and Guillaume Hen- nequin. Natural continual learning: success is a journey, not (just) a destination. Advances in Neural Information Processing Systems, 34:28067-28079, 2021.
[11] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of a mixed sequence of similar and dissimilar tasks. Advances in Neural Information Processing Systems, 13:18493-18500 2020.
[12] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Bariinka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)33521-3355 2017.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[14] Yann LeCun. The mnist database of handwritten digits. htlp://yann. lecun. com/exab/mnist// 1998.
[15] Sang- Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint arXiv:1703.08477. 2017.
[16] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pallern analysis and machine intelligence, 40(12):2935-2947, 2017.
[17] Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. Tenth International Conference on Learning Representations, ICLR 2022, 2022.
[18] Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. arXiv preprint arXiv:2201. 12522, 2022.
[19] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.
[20] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109- 165. Elsevier, 1989.
[21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[22] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint /810 2018.
[23] Mark Bishop Ring et al. Continual learning in reinforcement environments. 1994.
[24] Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu. Razvan Pascanu, and Raia Hadsell. Progressive neural networks, arXiv preprint arXiv: 1606.01606.0467 04677, 2016.
[25] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
[26] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. International Conference on Machine Learning, pages 4548 -5577 PMLR, 2018.

---

[27] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[28] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprini urXiv:/708.07745 2017.
[29] Li Yang, Sen Lin, Junshan Zhang, and Deliang Fan. Grown: Grow only when necessary for continual learning, arXiv preprint arXiv:21 10. 000900 2021.
[30] Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Eighth International Conference on Learning Representations, ICLR 2020. ICLR, 2020.
[31] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint vrXivv:708.000 01547, 2017.
[32] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364-372, 2019.
[33] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-
115, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theorem 1 and? 2.
(b) Did you include complete proofs of all theoretical results? [Yes] See the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section 5.
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See section 5.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See section 5.
(b) Did you mention the license of the assets? [Yes] See the appendix.
(c) Did you include any new assets either in the supplemental material URL? [Yes] We include code the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yxFe0mtN1NL", "reviewer": "Reviewer_nsyh", "summary": "To tackle the problem of incrementally learning a sequence of tasks with a deep neural network, this paper proposes the establishment of three different “regimes” with each their own update rule. At the start of learning a new task, each layer of the network is assigned to one of these regimes based on a carefully motivated set of decision rules and thresholds. Regime 1 (most protective of old tasks) uses an update rule similar to the rule proposed by Saha et al. (2021, ICLR), regime 2 uses an update rule similar to the rule proposed by Lin et al. (2022, ICLR) and regime 3 (least protective of old tasks) uses a newly proposed update rule that is hoped to allow for positive backward transfer. In an empirical comparison, the proposed method is shown to achieve positive backward transfer on two task-incremental learning benchmarks, while none of the compared methods do so.", "questions": "Please see the comment above for several questions.\n", "limitations": "It appears the authors have made no attempt at all to address the limitations of their work.\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " believe this is an interesting and insightful paper, and as such potentially suitable for publication at NeurIPS. However, as will become clear below, at the moment there are too many issues preventing me to support acceptance of this paper, but I would be happy to reconsider a suitably revised version.\n\nDue to insufficient experimental details, I would not be able to reproduce the experiments in this paper. For example, the following important experimental details are not reported (or at least I was not able to find them):\n- What optimizer / learning rate is used for each of the experiments, and based on what was this selected?\n- How were “method-specific hyperparameters” selected for the various compared methods (e.g., “lambda” for EWC)?\n- What was the size of replay buffer for AGEM and ER_Res? Why do the authors select this buffer size?\n\nAnother question about the experimental details: it is stated that on several of the benchmarks early stopping is used based on the validation loss. Which validation loss is used for this? (e.g., the loss on the current task? the loss on the tasks seen so far?)\n\nIt seems that most of the experimental results reported in Table 1 have been taken from (an)other paper(s) without proper acknowledgement. Many of the results are exactly the same as those reported by Lin et al. (2022, ICLR) and Saha et al., (2021, ICLR). Code also seems to be provided only for the method proposed in this paper, and not for the other methods included in the experimental comparison. If the authors did not run all of the reported results themselves, this should be clearly stated (and it should be acknowledged where the results come from) and it should be explained how it is ensured that all comparisons are still fair.\n\nI believe the most interesting method to compare against is Natural Continual Learning proposed by Kao et al. (2021, Neurips; https://arxiv.org/abs/2106.08085). This method also tackles continual learning by putting restrictive constraints on the optimization of new tasks, but, unlike the present paper, uses an approximation of the Fisher Matrix of old tasks for this. Intuitively, this should allow for more positive backward transfer than using an orthogonal projection onto the input subspaces of olds tasks. I would like to encourage the authors to compare against this paper, ideally both conceptually and empirically.\n\nA central claim of the paper relates to backward transfer: that the proposed method is the first method to achieve this on some of the considered benchmarks. The backward transfer that is achieved however appears to be very small. It would be helpful if the authors could provide an intuition as to what the reported backward transfer values mean. In particular, does the amount of achieved backward transfer have any practical significance?\n\nOther issues:\n- The authors do not discuss or reflect on the limitations of their proposed approach.\n- I got confused by the term “Single-head” in Table 1. In the literature this term is typically used as a synonym for class-incremental learning, but this does not seem to be the case here. I thought that here this term might instead be used as a description of the type of output layer, but as the method HAT is not included with this benchmark, that also does not seem to be the case. (Or at least it is not the entire explanation, as I can’t see why HAT cannot be run with a single-headed output layer?) It seems the term is used to indicate that task labels are not available at test time, but for that I recommend the authors to instead use the terms task-incremental, domain-incremental and class-incremental (e.g., as in Kao et al., 2021; NeurIPS), to avoid the ambiguity of the term single-head.\n- In L63-L64 the authors talk about “all the baseline CL methods”. This is too strong and should be moderated.\n\n---\n\nFollowing the rebuttal, I raised my \"rating\" from 3 to 4. See my response to the authors' rebuttal for more details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nPndzLes-aU", "reviewer": "Reviewer_nMSj", "summary": "The paper proposes a continual learning method that looks at tasks’ correlation (the authors define the correlation measure as well) to improve positive backward transfer across similar tasks. Towards this, first the authors define the notions of tasks’ similarity through subspace projections and positive correlations, and then theoretically analyze how if positive correlation and sufficient projection constraints are satisfied, an SGD update without any projection can improve the learning of previous task(s). Inspired by the theoretical analysis, the authors propose a method, CUBER, that stores the gradients of the previous tasks, and either make an update orthogonal to the gradients subspace of the previous tasks (if the previous task is not positively correlated) or make un (approximately) unchanged update if the previous task is positively correlated with the current task. Experiments are conducted on 4 standard CL benchmarks and it has been shown that the proposed method consistently provides a positive backward transfer for different benchmarks. \n", "questions": "Please respond to the weaknesses section above. \n", "limitations": "See the weaknesses section above. \n\n\n--------\nI have revised my score post-rebuttal. For detailed justification of this revision, see my response to the authors' rebuttal. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\n\n* The paper is written really well and the final model is well motivated. \n* The paper develops nice intuitions regarding different components of the proposed method. \n* Experiments are thoroughly conducted to establish the claims of the paper. \n\nWeaknesses\n-----------\n\n* Comparison with GEM/ A-GEM: It seems to me that at least some ideas around checking the positive correlation and making the gradient updates based on that are similar to what GEM/ A-GEM inspired. While the authors of the present manuscript mention A-GEM and state that ‘... In stark contrast, the task gradient correlation characterization in [AGEM] involves the gradient evaluation of old tasks with respect to the current model weight, and hence requires the data of old tasks when learning the new task’, I feel the difference is superficial. My understanding of GEM/ A-GEM is that the authors chose to keep the data of previous tasks and compute the gradients of previous tasks at each new step of the current task, because storing previous data is cheap in terms of memory compared to storing the gradients. One could equivalently store the gradients in GEM/ A-GEM and their gradient projection step will not change. I feel that the proposed CUBER is an inefficient version of GEM/ A-GEM (in terms of memory). It is not clear to me what was the replay buffer size that the authors of the present manuscript used with A-GEM and ER. For a fair comparison, I would fix the memory size in MBytes and use equivalent memory for CUBER and A-GEM/ ER and then compare their numbers in Table 1. \n\n* First method to achieve positive forward transfer with a fixed capacity network: This claim is wrong! As stated above, since the update procedure of facilitating the positive backward transfer is very similar to GEM, GEM would also achieve the positive backward transfer. In fact, GEM indeed achieves the positive backward transfer on MNIST permutation and CIFAR100 (cf. Fig. 1 in https://arxiv.org/pdf/1706.08840.pdf). \n\n* Memory/ time comparison: I would like to see the memory and time comparison with other baselines. \n\nWhile I appreciate the mathematical elegance of the paper, it feels a bit distracting. For example, it is intuitively very easy to see when the learning of a new task could help in learning the previous task (in fact this is precisely the GEM’s update rule). It is not clear to me if Sec 3 is needed in all its glory. I also feel the learnings of Sec 3, for example the question about when Rule #2 can safely be used to update the model, are not applied in the final model – in regime 3 the authors still don’t use the rule #2 even when there is a positive correlation among the present and previous tasks. Given this, I would appreciate it if the authors could focus more on the intuition of the method itself – why the scaled projection would lead to forward transfer, how their method is different from GEM/ A-GEM etc. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gW-v6RQddAs", "reviewer": "Reviewer_gpWA", "summary": "This paper investigates the gradient space for backward knowledge transfer. Starting from the question of how can improve the previously learned model using new incoming data, it studies the criteria for backward transfer. It proposed a novel continual learning method with backward knowledge transfer. The experiments showed the effectiveness of the method. ", "questions": "Could you explain the storage consumption to save gradient information of the previous model?  ", "limitations": "large storage consumption. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- strong theoretical analysis of backward knowledge transfer. \n\nWeaknesses: \n- large storage consumption for saving the network gradients for backward transfer. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-NFF8DZNKx5", "reviewer": "Reviewer_B6XQ", "summary": "The authors propose a continual learning model in which the model focuses on backward knowledge transfer and learns new tasks while, when relevant, improving the learning of old tasks. The latter relies on identifying the similarities between new tasks and old tasks by analyzing the gradient projections. The model is applied to a fixed capacity network without replay. The controls and optimizations are applied to each layer separately. \n\nThis is an interesting work and, as the authors have mentioned, focuses on an area of continual learning which is less explored compared with several existing methods for forward learning. The model has been applied to several imaging benchmarks, and the performance of the model is compared with several baseline models.\n\n", "questions": "Is there any difference between the gradient project analysis in this work and the technique that is proposed in trust region gradient projection (TRGP) (Lin et al., 2022)? – ref [16] in the paper. \n\nWhat is HBK in the denominator of the equation shown in Theorem 1? \n\nFor an L-layer network, CUBER characterizes the task correlation for each layer and then uses different strategies to decide how to learn a new task. Is there a way to quantify the increased complexity when the number of tasks grows? One may assume that if there are several tasks and if there are several layers in the network and then a subset of these layers become specifically relevant to one task, then this should require an increasing number of parameters to be stored; would it be possible to elaborate on or quantify the complexity? \n", "limitations": "A limitation of the work could be the complexity of calculating SVD and assuming clear boundaries between the tasks. It would have been helpful to include a brief section to discuss these and other limitations of the work. \n\nThe societal or impact are not discussed explicitly in the paper; however, the gain in this model comes are at a relatively significant level of complexity; it would be helpful to explore what real-world cases would benefit from this additional complexity and what level of compute/memory will require to train a model with several tasks. \n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths\n\n\nThe idea of backward learning and improving (the generalisability) of the old tasks is interesting, and the evaluations show that the model achieves better results than the existing solutions. \n\nThe assumptions, experimental settings, and the results are well presented and discussed. \n\nThe paper is well written and easy to follow. The difference between the current approach and existing solutions and related work are explained well. \n\n\n*Weaknesses  \n\nThis model is largely designed based on the assumption that at least some of the tasks are similar or very similar, which makes sense; otherwise, there won’t be much backward knowledge transfer. However, this seems to also depend on having clear boundaries between the tasks. In other words, it is unclear if the model would perform well in the drifts and task changes that happen gradually. With imaging benchmarks often, it is not easy to evaluate this use case. \n\nIn construction, the base extraction an SVD technique is applied; for wide network and high dimensional data, this could be computationally very expensive to calculate this for the test of inputs; considering some of the improvements are incremental; what would be the trade-off between this high complexity and the overall improvement? \n\nThe authors are right that the backward learning models are relatively less explored; however, it would be helpful if they could compare their work with an existing model that explicitly addresses backward learning and task similarity in continual learning (e.g. Zixuan et al., 2022) – ref [10] in the paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept