TRGP: Trust Region Gradient Projection for Continual Learning
Authors: Sen Lin
Li Yang
Deliang Fan
Junshan Zhang
Venue: ICLR 2022 Spotlight (2022)
Keywords: trust region
gradient projection
scaled weight projection
continual learning
forward knowledge transfer
task correlation

Abstract: Catastrophic forgetting is one of the major challenges in continual learning. To address this issue, some existing methods put restrictive constraints on the optimization space of the new task for minimizing the interference to old tasks. However, this may lead to unsatisfactory performance for the new task, especially when the new task is strongly correlated with old tasks. To tackle this challenge, we propose Trust Region Gradient Projection (TRGP) for continual learning to facilitate the forward knowledge transfer based on an efficient characterization of task correlation. Particularly, we introduce a notion of 'trust region' to select the most related old tasks for the new task in a layer-wise and single-shot manner, using the norm of gradient projection onto the subspace spanned by task inputs. Then, a scaled weight projection is proposed to cleverly reuse the frozen weights of the selected old tasks in the trust region through a layer-wise scaling matrix. By jointly optimizing the scaling matrices and the model, where the model is updated along the directions orthogonal to the subspaces of old tasks,  TRGP can effectively prompt knowledge transfer without forgetting. Extensive experiments show that our approach achieves significant improvement over related state-of-the-art methods.

Published as a conference paper at ICLR 2022
TRGP: TRUST REGION GRADIENT PROJECTION FOR CONTINUAL LEARNING
Sen Lin', Li Yang', Deliang Fan', Junshan Zhang! 'School of ECEE, Arizona State University, 'Department of ECE, University of California, Davis {slin70, 1yang166, dfan) @asu.edu, jazh edu
ABSTRACT
Catastrophic forgetting is one of the major challenges in continual learning. To ad- dress this issue, some existing methods put restrictive constraints the optimiza- tion space of the new task for minimizing the interference to old tasks. However, this may lead unsatisfactory performance for the new task, especially when the new task is strongly correlated with old tasks. To tackle this challenge, we propose Trust Region Gradient Projection (TRGP) for continual learning facilitate the forward knowledge transfer based on an efficient characterization of task correla- tion. Particularly, we introduce notion of 'trust region' to select the most related old tasks for the new task in a layer-wist wise and single-shot manner, using the norm of gradient projection onto the subspace spanned by task inputs. Then, a scaled weight projection is proposed to cleverly reuse the frozen weights of the selected old tasks in the trust region through a layer-wise scaling matrix. By jointly opti- mizing the scaling matrices and the model, where the model is updated along the directions orthogonal to the subspaces of old tasks, TRGP can effectively prompt knowledge transfer without forgetting. Extensive experiments show that ap- proach achieves significant improvement over related state-of-the-an methods.
INTRODUCTION
Human beings can continuously learn different new tasks without forgetting the learnt knowledge of old tasks in their lifespan. Aiming to achieve this remarkable capability for the deep neural networks (DNNs), continual learning (CI.) (Chen & Liu. 2018) has garnered much attention in recent years. Nevertheless, many existing CL methods still leave the DNN vulnerable to forget the knowledge of old tasks when learning new tasks. Such a phenomenon is known as 'Catastrophic Forgetting' (McCloskey & Cohen 1989), which has become one of the major challenges for CL.
Many approaches (e.g., (Rusu al. 2016 Li & Hoiem 2017 Dhar et al., 2019 Guo et al., 2020)
Zeng al., 2019)) have been proposed to address the forgetting issue, which be generally divided into two classes depending on the network architecture, i.c., expansion methods and non- expansion methods. In order to understand the fundamental limit of fixed capacity neural network, we focus on non-expansion methods in this work. The basic idea for non-expansion methods is constrain the gradient update either explicitly or implicitly when learning the new task, so to minimize the introduced interference to old tasks. For example, the regularization-base (e.g.. (Kirkpatrick et al., 2017 Serra et al. 2018)) penalize the modification on the most important weights of old tasks through model regularizations; experience-replay based methods (e.g.. (Shin al. 2017 Chaudhry et al., 2019) constrain the gradient directions by replaying the data of old tasks during learning of new tasks, in the format of either real data synthetic data from generative models; and orthogonal-projecteo
methods
based methods (e.g., (Farajtabar et al. 2020) [Saha et al.) 202T) update the model with gradients in the orthogonal directions of old tasks, without the access to old task data. particular, the recently proposed Gradient Projection Memory (GPM) (Saha et al. 2021 has demonstrated superior performance compared to other approaches.
To sufficiently minimize the interference to old tasks, most existing non-expansion methods (partic- based methods), often put restrictive constraints the optimization
ularly the space of the new task, which may throttle the learning performance for the new task. A plausible conjecture is that such scenario is likely to oceur when the new task is strongly correlated with

---

Published as a conference paper at ICLR 2022
old tasks, and in this study we provide evidence to support this conjecture. The underlying rationale is as follows: The weights that are important to the new task are also important to the old tasks strongly correlated with the new task, which are often frozen to address the forgetting in the existing methods; however, they should be updated in the learning of the new task.
To tackle this challenge, a key insight is that for a new task that is strongly correlated with old tasks, although the model optimization space could be more restrictive, there should be better forward knowledge transfer from the correlated old tasks to the new task. With this insight, we propose an innovate continual learning approach to facilitate the forward knowledge transfer without forgetting. The main contributions can be summarized as follows:
(I) Inspired by (Schulman et al., 2015). we introduce novel notion of 'trust region' based on the norm of gradient projection onto the subspace spanned by task inputs, which selects the old tasks strongly correlated to the new task in a layer-wise wise and single-shot manner. Intuitively, the new task and the selected old tasks in the trust region have similar input features for the corresponding layer.
(2) We propose a novel approach for the new task to leverage the knowledge of the strongly corre- lated old tasks in the trust region through a scaled weight projection. Particularly, a scaling matrix is learnt in each layer for the new task to scale the weight projection onto the subspace of old tasks in the trust region, in order to reuse the frozen weights of old tasks without modifying the model.
(3) Building on the introduced trust region, scaled weight projection, and module to construct task input subspace, we develop a continual learning approach, trust region gradient projection (TRGP), that jointly optimizes the scaling matrices and the model for the new task. To mitigate the forgetting issue further, the model is updated along the directions orthogonal to the subspaces of old tasks.
(4) We evaluate TRGP on standard CL benchmarks using various network architectures. Compared to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all benchmarks, and demonstrates universal improvement on all tasks. The superior performance indi- cates that TRGP can effectively promote the forward knowledge transfer while alleviating forgetting.
2 RELATED WORK
Expansion- based methods. Expansion-basedd methods (e.g.. g.. (Rusu et al.
2016 Li & Hoiem 2017
Rosenteld & Tootoos 2010) 2018] Hung et al. [201 Yoon et al.] 2017 Li et al., 2019 Veniat et al.) 2020
dynamically expand the network capacity to reduce the interference between the new tasks and the old ones. Progressive Neural Network (PNN) (Rusu et al. 2016) expands the network architecture for new tasks and preserves the weights of old tasks. Learning Without Forgetting (LWF) (Li & Hoiem] (017) splits the model layers into two parts, i.e., the shared part co-used by all tasks, and the task-specific part which grows for new tasks. Dynamic-Expansion Net (DEN) (Yoon et al., 2017) and Compacting-Pickke Picking-Grooiing (CPG) (Hung et al. 2019) combine the strategies of model compression/pruning, weight selection and model expansion. In order to find the optimal structure for each of the sequential tasks, Reinforced Continual Learning (RCL) (Xu & Zhu, 2018 leverages reinforcement learning and (Li et al. 2019) adapts architecture search. APD (Yoon et al.) 2020) adds additional task-specific parameters for each task and selectively learns the task-shared parameters.
Regularization-bee methods. This category of methods (e.g.. (Kirkpatrick et al.
2017 Lee et al.,
2017 Chaudhry et al. [2018a] Dhar et al. [219 Ritter et al., 20188) Schwarz et al. [0188] Zenke al.
2017 protect the old tasks by adding regularization terms in the loss function to penalize the model change on their important weights. Notably, determine the weight importance, Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) leverages Fisher information matrix, HAT (Serra) al. 2018) learns hard attention masks. MAS (Aljundi et al., 2018) evaluates the model outputs sensitivity to the inputs in an unsupervised manner.
Memory-based methods. Depending on if data of old tasks utilized when learning new tasks, memory-based methods can be further divided into the following two categories. /) Experience- replay based methods. This class of methods replays the old tasks data along with the current task data to mitigate catastrophic forgetting. Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato 2017) and Averaged GEM (A-GEM) (Chaudhry et al.) 2018b) alter the current gradient based on the gradient computed with data in the memory. A unified view of episodic memory based methods is proposed in (Guo et al. 2020), based on new approaches are developed to balance between old tasks and the new task. Tiny episodic memory is considered in (Chaudhry et al. 2019) and meta-
2

---

Published as a conference paper at ICLR 2022
learning is leveraged in (Riemer al. 2018). 2) Orthogoonl-projectinn based method. To eliminate
the need of storing data of old tasks, recently a series work (Zeng et al. 2019; Farajtabar et al.
2020 Saha et al. 2021) updates the model in the orthogonal direction of old tasks, and has shown remarkable performance. Particularly, Orthogonal Weight Modulation (OWM) (Zeng et al., 2019) lears a projector matrix multiply with the new gradients. Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2020) stores the gradient directions of old tasks and projects the new gradients on the directions orthogonal the subspace spanned by the old gradients. Gradient Projection Memory (GPM) (Saha et al., 2021) stores the bases of the subspaces spanned by old task data and projects the new gradients on the directions orthogonal to these subspaces.
3 PROBLEM FORMULATION
Continual learning. Consider the setting where sequence of tasks T (f)T= arrives sequentially- Each task has dataset ID, {arg,, )r,s),² with N, sample pairs, where æt, is the input vector and yt.s is the label vector. Consider a fixed capacity neural network with L layers, and the set of weights is denoted as W = (W")_, where W' is the layer-wise weight for layer l. Given the data input i for task L, denote ær{, as the input of layer and æi,; = Zt,i. The output æ; 1+1 for layer / is computed as 2,, = f(W', ==,,), where f is the operation of the network layer. Following (Saha et al. 2021), we denote .ri,, as the representations of Zt, at layer 1. When learning task l, we only have access to dataset D, Let L(W, ( )t.,))) ) = Le (W) denote the loss function for training, e.g., mean squared and cross-entropy loss, and W/ denote the model after learning task t.
Orthogonal-projectinn based methods. To minimize the interference to old tasks, recently series of studies (Zeng et al., [219 Farajtabar ct al. 2020 Saha et al. (021) has been carried out to update the model for the new task in the direction orthogonal to the subspace spanned by inputs of old tasks. In what follows, we briefly introduce the main ideas through a basic case with two tasks 1 and 2.
Denote the subspace spanned by the inputs of task for layer as S and the learnt model for task as W, {- It is clear that ar St. When learning task 2, the model Wi will be modified in the direction orthogonal to St. by either multiplying the gradient W with a projector matrix 2019). ) or projecting the gradient V Wi onto the orthogonal direction to S (e.g.,
(e.g, (Zeng al.
Saha et al. 2021 )). Let A W denote the model change after learning task 2. It follows immediately that AW wii 0, and the model W for task 2 is W W + #WW. Therefore, for task 1:
W = ( + A ai) = W/ + AW'a',, = Wiai,; which indicates that no interference is introduced to task I after learning task 2, thereby addressing the forgetting issue. Such analysis can be generalized to a sequence of tasks.
(1)
When would orthogonal projection hinder the learning of a new task? Orthogonal projection provides a promising solution to address the forgetting in continual learning. However, by modifying the model only in the orthogonal direction to the input space of old tasks, the optimization space of learning the new task could be more restrictive, resulting in compromised performance of the new task. To get a more concrete sense, consider the following basic examples with two tasks and 2.
(Toy example 1) Suppose task has dataset ID: {(ar, y:) and task has dataset ID₂ [(=x, y where only the sign is changed for the input vectors. Consider the case where two tasks share the same classifier (Saha et al.) 2021). It is clear that for the 1-th layer, the subspace spanned by { of task 1 is same with the subspace spanned by ) of task 2, i.e., St = =S,. given the learnt model W for task 1. Based on the fact that stochastic gradient descent updates lie in the subspace spanned by the data imput (Zhang et al. 202T Saha et al. 2021). it follows that the gradient V w L2 E Sy, such that V Ww:L2 E Sq. Therefore, the projection of V WwiCz onto the orthogona direction to S is O, which means that the model W will not be updated when learning task 2, i.e., W W/. However, the optimal model for task 2 should be W! WW, because W/n! achieves the minimum loss for the label y, after learning task 1.
(Toy example 2) Suppose the input subspace of task is orthogonal to that of task 2, i.e S 1 Sy. It follows that the projection of V W: La onto the orthogonal direction to S is indeed equal to V Wi Consequently, updating the model for task based on orthogonal projection will not only introduce no interference to task 1, but also move along the direction of steepest descent for task 2.
3

---

Published as a conference paper at ICLR 2022
sutopace for task
subspace Sf for task
gradient Vyyla for task projection of vunle anto St
projection Fyily orthogonal to S?
weak correlation
strong
Figure 1: Layer-wise task correlation for the case where the subspace spanned by the representations two-dimensional plane. The subspaces are weakly correlated if they are nearly orthogonal and strongly correlated if they are nearly parallel.
Motivated by these examples, plausible conjecture is that naive orthogonal projection could possi- bly compromise the learning performance of the new task that is strongly correlated with old tasks, especially when the correlation is "negative" characterize the task correlation through the correlation between the input subspaces for two tasks. As illustrated in Fig. when the subspace is 2-dimensional, two tasks weakly correlated if their input subspaces are nearly orthogonal, and strongly correlated if their subspaces are nearly parallel.
in the toy example 1. In this study, we advocate to
4 TRUST REGION GRADIENT PROJECTION FOR CONTINUAL LEARNING
To tackle these challenges, key insight is that for a new task that is strongly correlated with old tasks, although the model optimization space could be more restrictive, there should be better for- ward knowledge transfer from the correlated old tasks the new task. With this insight, we propose a novel approach to prompt forward knowledge transfer without forgetting, by 1) introducing a novel notion of trust region to select the most related old tasks in single-shot manner and 2) cleverly reusing the frozen weights of the selected tasks in the trust region with a scaled weight projection.
4.1 TRUST REGION
To facilitate forward knowledge transfer from the correlated old tasks to the new task, the first question is how efficiently select the most correlated old tasks. Towards this end, we characterize the correlation between the input subspaces for two tasks, through the lens gradient projection.
} as the subspace spanned by the task j data for layer where M,, is the bases for 5) (totally M₁,1 bases extracted from the input). For any matrix A with suitable dimension, denote its projection onto the subspace S) as:
Specifically, denote s span {
B) = (u),..-.., w...
Proisy (A) (B))
where (.Y is the matrix transpose. We next define a layer-wise trust region for a new task as a set of its most related old tasks, based on the norm of projected gradient onto the subspaces of old tasks.
Definition (Layer-Wo (Layer-Wise Trust Region). For any new task t 2 2 and layer we define layer-wise trust region TR, = {i} for E [1,t-1], 11, where for any task j TR, the following holds: [[Prois (V Ww
(3)
where el [0, and W,-1 the model after learning task t 1.
Intuitively, for the new task the norm of its gradient projection onto the sub- sur-
Threshots of.
> dh 50 7x)
for
space of an old task serves rogate for characterizing the correlation between input subspaces for these two tasks, due to the fact that the gradient lies in the span of its input. When the condition Eq. (3) is satisfied, the gradi- ent W L. (W.1) has a large projec- tion onto the subspace of an old task j. which implies that the subspace St for task t and the subspace S for task j may have sufficient common bases for layer 1. In this case, we trust that the old task
Prot,
Tir4, for taske
projection of Wiik of
projection of y
Proi,
orthogonal to
Figure 2: Trust region for 2-dimensional subspace can be interpreted as: if the angle 9 between V VWiLi and Projs: (V (Vw:L,) is less than 6), (larger projection on $)), old task j is selected to TRI; otherwise not. 0% can be set as a large value, and we can pick tasks with top-K smallest TR.
4

---

Published as a conference paper at ICLR 2022
j is strongly correlated with the new task t in layer L, and put it into task i's trust region TR₁. A simple illustration of trust region is shown in Figure 2 Note that the notion of trust region can also be generalized to task-wise definition, where the most correlated old tasks will be selected based on the projection of the entire gradient (W However, the layer-wise trust region could select different tasks for different layers, which provides more fine-resolution characterization of task correlations in terms of layer-level features.
Practical implementation. Besides the valuable functionality provided by the trust region for se- lecting most correlated old tasks, another significant benefit is the simplicity of its practical imple- mentation. Consider the implementation for learning a new task
(1) Single-hoot manner. Given the learnt model Wi-1, we select sample batch from dataset iD and compute the gradient w: C (W+-1) in one nerward-bacwwar pass for all layers at once, Given the subspace 5 for an old task j. the condition Eq. (3) can be immediately evaluated for all old tasks.
(2) Top-K correlated tasks. It is clear that the choice of e' has nontrivial impact the selection of the most correlated old tasks. To reduce the sensitivity of the performance on d' we can set relatively small value of c', and pick the top-K old tasks with the largest gradient projection norm [|Proist (V (WW from the tasks satisfying Eq. (3). As demonstrated later in our experiments, setting K 1 is enough to achieve significant performance improvement.
4.2 SCALED WEIGHT PROJECTION
Given the layer-wise trust region TR, for the new task t, the next key question is how efficiently leverage the knowledge of the most correlated old tasks in TR, for learning task To this end, we propose novel approach to reuse the frozen weights of the selected old tasks in TR,, through a scaled weight projection with a scaling matrix.
At the outset, it is of interest to understand what knowledge is preserved for old tasks during con- tinual learning with orthogonal projection. Based on Eq. [ for the simple case with two learning tasks and 2 as mentioned carlier, it can be shown that
Projs: (W2) Proje; (Wi + AW1) = Prois; (W3) Prois; (AW)) = Proisy (W))
(4)
where the last equation holds because the model W is updated in the direction orthogonal to S when learning task 2. By generalizing Eq. (4) to the case with sequence of tasks, we can have that
for the model W after learning task 1 and any old task j < t: (W)).
Prois;
=
(S)
which indicates that the model weight projection on the subspace of old tasks is actually "frozen during continual learning so as to overcome forgetting of the old tasks.
On the other hand, because the trust region TR,, is constructed in a way that the subspace S of task t is strongly correlated with the subspace S for any old task j E TRI, the bases B) of S is very likely to contain important bases for task t. As a result, the weight projection Projs: (W(_)) is important for the new task t and should be modified accordingly in order to guar- antee the learning performance of task t, which however has to be frozen to protect task j. To find an efficient way to lever- age Proj.s (WL1) without modifying it, note that the projec- tion Prois; (W!_)) is indeed linear combination of the pro- jection of W!', onto each basis in B', and every point in S can be obtained by scaling the coordinates of Prois; (W!L.) Figure 3 shows simple example for two-dimensional sub- space. Therefore, we propose a scaled weight projection to find the best point for task t in S by leveraging the projection Projsy (W!1) through a square scaling matrix Q/;;:
Prol,
2
(Wi
d/2
s,c/2
%/1
4/a1
s2c/2
(ro)
Figure 3: [m/, n1,22) is the bases of subspace S). and (c),1: is the coordinate of Proj.! (WL_)). Any point Projs; (W') in S) can be obtained by scaling the coor- dinate (c), c/,2l with some scalar 81 and 82.
Proig (Wi-1) w!-, B'Q,,
(6)
5

---

Published as a conference paper at ICLR 2022
The dimension of Q' depends on the number of bases in B' (dimension of Sy), which is usually small for each task. In this way, we explicitly transfer the knowledge of the selected old tasks in the trust region TR, to the new task t through a scaling matrix Q'
4.3 TASK SUBSPACE CONSTRUCTION
To successfully leverage the trust region, missing ingredient is the construction of input subspaces of old tasks. We next show how the subspace 5 can be constructed for task at layer I.
For task = 1. is in (Saha et al. 2021), we obtain the bases Bi after learning task using Singular Value Decomposition (SVD) on the representations. Specifically, given the model W1 after learning task 1, we construct representation matrix R² i........ ........ " xi,0] Rm with samples, where each . IR", is the representation layer by forwarding the sample 21,; through the network. Then, we apply SVD to the matrix Ry i.e., Ri = Uj={ (VTY, where U| = [u],1.-.. ...., "m,m E Rmxm is an orthogonal matrix with left singular vector u E m V [of.1..... an orthogonal matrix with right singular vector vi,4 IR", and Σ R is a rectangular
vf,nl R
diagonal matrix with non-negative singular values { (miu((.)) on the diagonal in descending order. To obtain the bases for subspace Si, we use kf -rankk matrix approximation to pick the first kj left singular vectors in U/, such that the following condition is satisfied for a threshold v/a (0, 1):
[I Day 11 2 4/ R
(7)
where (Bi)4 E uf, (ff,)) is a ki -rank (kr S r) approximation of the representation
matrix Ry with rank I min(m. n), and || ||r is the Frobenius norm. Then the bases B for subspace 5 can be constructed as Bi- Iu.,,.... u 1,64
For task) j E []. We construct the bases B) after learning task j given the learnt model Wj- A representation matrix Ri will be first obtained in the same manner as R² Note that the bases [ learnt for old tasks may include important bases for task j. Therefore, we learn the bases BJ by selecting the most important bases from both bases of old tasks and newly constructed bases. Specifically, (1) (old bases) we first concatenate the bases |= of old tasks together in M', and climinate the common bases. For each basis / E M). we compute the corresponding eigenvalue of R (R)) .ie., 8f = (u.) R. Y which is the square of the singular value of RJ with respect to w (2) (new bases) We perform SVD on ft. = R² R², MJ. i.e., R (V))) with singular values ( hn- (3) (select the most important bases from both old and new bases) Next we concatenate { a and ((d),,))h together in a vector 8, and sort them in descending order. We perform k--rank matrix approximation of R2,, such that the summation of the first k, elements in ô greater than et, [ || Then BJ, can be constructed by selecting the bases corresponding to the first kj elements in 8.
to generate new bases beyond
4.4 CONTINUAL LEARNING WITH TRUST REGION GRADIENT PROJECTION
Building on the three modules proposed earlier, i.e., task subspace construction, trust region and scaled weight projection, we next present our approach TRGP for continual learning that efficiently facilitate forward knowledge transfer without forgetting the old tasks.
Learning task 1. The first task is learnt using standard gradient descent. The subspace { is constructed by following Section 4.3
Learning task 2, .... T. For task correlated old tasks selected for layer 1. The optimization problem for task is as follows:
TJ. we first determine the trust region TRÉ with top-K
min
L( C(WWersh,Do))
(8)
(WI) ). Q SERTRR 8.t
W = w' Σ JETRR [Projg (W') Proisy (W')),
(9)
where the gradient for updating W V of all old tasks in Section 4.3 The subspace ( 1 is obtained by following Section 4.3
L WwiL= V WwiL C- (V (V M (M²)' and M, is the bases

---

Published as a conference paper at ICLR 2022
Table 1: The averaged accuracy (ACC) and backward transfer (BWT) over all the tasks on different datasets. Note that, Multitask jointly learns all tasks only once single network by using the whole dataset, which does not adhere to CL setup.
PMNIST
CIFAR-100 Split
5-Dataset
MinilmageNet
Mothod
ACC(S)
BWT(S)
ACC(%)
BWT(S)
ACC(S)
BWT(S)
ACC(%)
BWT(%)
Multiaak
96.70
79.58
91.54
69.46
OWM
90.71
50,94
-30
EWC
89.97
68.80
88.64
$2.01
-12
HAT
72.06
91.32
59.78
-3
A-GEM
83.56
-14
63.98
-15
84.04
-12
57.24
-12
87.24
58.94
-7
GPM
93.91
72.48
-0.9
91.22
-1
60.41
-0.7
Ours (TRGP)
96.34
-0.8
74.46
-0.9
93.56
-0.04
61,78
-0.5
EXPERIMENTAL RESULTS
5.1 EXPERIMENTAL SETUP
Datasets and training details. We evaluate our method on multiple datasets against state-of-the-ar CL methods. 1) PMNIST. Following (Lopez-Paz & Ranzato, 2017, Saha et al. 2021). we create 10 sequential tasks using different permutations where each task has 1O classes. We use 3-layer fully-connected network. 2) CIFAR-100 Split. We split the classes of CIFAR-100 (Krizhevsky) ctt al. 2009) into group, and consider 10-way multi-class classification each group as a single task. Similar with (Serra al., 2018: Saha et al. 202I),
version of 5-layer AlexNet. 3) CIFAR-100 Sup. We divide the CIFAR- 100 dataset into 20 tasks where each task has 5 classes. We use modified version of LeNet-5. 4) 5-Datasets. We use sequence of 5-Datasets which includes CIFAR-10, MNIST, SVHN (Netzer et al. (011), not-MNIST (Bulatov) (011) and Fashion MNIST Xiao et al. 2017), where each dataset set to be task. We adapt reduced ResNet18 network that is used in (Lopez-Paz & Ranzato 2017). 5) MinilmageNNet Split. We split the 100 classes MinilmageNet (Vinyals al. 2016 into 20 sequential tasks where each task has 5 classes, and consider reduced ResNet18 network. In addition, for all the experiments, the threshold c' is set to 0.5, and we select top-2 tasks that satisfy condition Eq. (3). We use the same threshold d/, as GPM (Saha al. 2021) for subspace construction. More details are in the appendix.
test the efficacy of our method, we compare it with state-of-the-ar approaches in three categories: 1) Memory-based methods. We compare with Experience Replay with reservoir sampling (ER_ Res) (Chaudhry et al. 2019), Averaged GEM (A-GEM) (Chaudbry ctt al. 2018b), Orthogonal Weight Modulation (OWM) (Zeng et al. 2019) and Gradient Projection Memory (GPM) (Saha et al. 2021). 2) Regularization-basee methods. We compare with state-of- the-art HAT (Serra al. 2018)) and Elastic Weight Consolidation (EWC) (Kirkpatrick al.) 2017).
Methods for comparison.
3) Expansion-basee methods. We further compare with Progressive Neural Network (PNN) (Rusu
al. 2016), Learning Without Forgetting (LWF) (Li & Hoiem] 2017).
Expaniin Net
(DEN)
Yoon et al. 2017), and APD (Yoon et al. 2020), by using CIFAR-100 Sup dataset.
Metrics. Following GPM (Saha et al. 2021), two metrics are used to evaluate the performance:
Accuracy (ACC), the average final accuracy over all tasks, and Backward Transfer (BWT), which measures the forgetting of old tasks when learning new tasks, ACC and BWT are defined as:
ACC T E Ar, BWT - Σ Ar,- Ar
(10)
where 7 is the number of tasks, AT, is the accuracy of the model on i-th task after learning the T-th task sequentially.
5.2
MAIN RESULTS
ACC and BWT comparison. As shown in Table I TRGP achieves significantly accuracy improve- ment compared with prior works on all datasets. For example, in contrast to the best prior results, TRGP achieve the accuracy gain of 2.43%, 1.98% and 1.37% over GPM on PMNIST, CIFAR-100 Split and MinilmageNet, respectively, and 2.34% over HAT on 5-Dataset. Surprisingly, we could even achieve better accuracy than Multitask on 5-Datasets, which usually serves as an upper bound for CL benchmarks. This superior performance of TRGP clearly shows its capability to effectively facilitate forward knowledge transfer. In addition, TRGP also demonstrates strong performance with the lowest BWT, reducing 0.2% than OWM and 0.6% than GPM, even with 5.63% and 2.34%

---

Published as a conference paper at ICLR 2022
accuracy
Final CIFAR-100
100
accuracy 5-Datasets
77.5
Accuracy
96.0
75.0
Taxt Accurary
80
95.5
12.5
95.0
Dur GPM
70.0
Ours GPM
Outs
GIM
94.5
Number tasks
Number tasks
Number tasks
Figure 4: The final accuracy for all tasks on three datasets (GPM VS Ours).
Table 2: The performance for CIFAR-100 Sup dataset. Note that Single-task learning (STL) trains a separate network for each task, which does not adhere to CL setup.
Methods
STL
PNN
DEN
RCL
APD
GPM
Ours (TRGP)
ACC(%)
$1.99
56.81
57.72
58.25
(5pacity(S)
2000
271
191
184
130
100
100
curacy improvement on PMNIST and 5-Dataset, respectively. Compared with HAT on CIFAR-100 Split, TRGP has marginally worse BWT, but achieves 2.4% accuracy gain.
Moreover, TRGP exhibits an universal dominance over GPM about the final accuracy of all tasks on all the three datasets. According to the apple apple comparison with GPM in Fig. 4 one interesting phenomenon is observed: TRGP has the similar accuracy on "easy" tasks, but significantly improves the accuracy the "difficult" tasks. For example, in the 5-Dataset setting, both TRGP and GPM achieve good accuracy on Task (MNIST) and 3 (Fashion MNIST), which can be easily trained well, but TRGP significantly outperforms GPM the rest three more difficult Tasks (CIFAR-10) 10, SVHN and NotMNIST). In the end, as shown Table 2 we further compare with the expansion- based methods by using CIFAR-100 Sup setting. It can be seen that TRGP outperforms all other CL. methods, with a fixed capacity network.
Discussion. We next show the accuracy evolution of specific tasks during the training of all tasks sequentially. We randomly select three tasks for each dataset to compare with GPM (we only show results on PMNIST in Fig. and relegate the rest the appendix). There are two main obervations: 1) TRGP completely outperforms GPM during training for all the sequential tasks on the three datasets; For the PMNIST and 5-Dataset settings, TRGP could significantly reduce forgetting, To understand why, consider the case where GPM and TRGP learns a new task / given the same model W,-1, and denote [M/ ) as the bases of all old tasks. Then we can have
For GPM, the effective weight for layer l is W Projmm_ (W') + Proji (W")
(11)
where the weight projection on M frozen to protect old tasks, and only the weight projection orthogonal M can be updated for learning task
For TRGP, the effective weight for layer is Wi,; Proi(s))seTR! (W') Proj (W') + Proj (W')
(12)
where only the first term, i.e., weight projection on subspaces of old tasks that are not in the trust region TRI, is frozen for task t. In contrast to GPM, an additional and also important part of weights, i.e., the scaled weight projection subspaces of related old tasks in TR,, can be learnt in a favorable way for task As a result, TRGP can achieve better forward knowledge transfer by explicitly and cleverly reusing the important knowledge of strongly correlated old tasks in the trust region. More interestingly, benefiting from the task-unique information captured by the scaled weight projection, the backward transfer also be reduced.
Accu racy envolution for Task 2
Accuracy envolution for Task
Accuracy envolution for Task 6
Test Accuracyyy (84
Ours
Test Accuracy (N))
97.0
Test Accuracy 188
$7.0
Outs
96.5
96.5
96
96.0
Ours
$6.0
6
Number of tasks
Number of tasis
Number of tasks
Figure 5: Accuracy evolution for different tasks on PMNIST setting.
8

---

Published as a conference paper at ICLR 2022
Table 3: Ablation study on CIFAR-100 100 Split and 5 5-Datasets settings.
Inpact threshold e'
Layer wise
Number of selected
Datasets
0.2
0.5
0.7
Layer-wise
Task-wise
Top-1
Top-2
CIFAR-100
74,52
74.46
74,30
74.46
73.25
74.00
74.46
5-Datasets
93.28
93.56
93.43
93.56
93.56
5.3 ABLATION STUDY AND ANALYSIS
Impact of the threshold eL. To understand the impact of the threshold eL, we evaluate the learning performance for three different values of e (i.e., 0.2, 0.5, 0.7) as shown in Table The results show that the accuracy is very stable across the three threshold values, with ignoble accuracy difference on both CIFAR-100 Split and 5-Dataset settings. The reason behind is because only select top-2 old tasks with largest gradient projection norm into the trust region, among all tasks satisfying condition Eq. (3). Therefore, for wide range of c', the selected tasks in the trust region are actually fixed. The small accuracy fluctuation is because with some possibility only one old task satisfies Eq. and is selected for some layers when e increases. Overall, TRGP very robust to the value of e!.
Layer-wise vs. Task-wise trust region. To show the efficacy of layer-wise trust region, we compare it with the task-wise variant which shares a fixed trust region across all layers for each task. First, shown in Table 3 layer-wise could achieve 1.21% accu-
Ours Layer wise
GPM
77.5
Ours Task wize
Test Accurag,
75.0
72.5
racy gain over task-wise on CIFAR- 100 Split. Furthermore, we illustrate the final accuracy of all tasks for layer-wise and task- wise of the proposed TRGP, and GPM on CIFAR-100 Split set- ting Fig. [ First, the performance of layer-wise better than comparable to task-wise for all tasks, because mayer-wise pro- vides a much finer characterization of task correlations in terms of layer-level features. Then, it is interesting ing behavior for the three cases follows the same trend. This observation further corroborates that TRGP improve the aceuracy and mitigate forgetting on both "easy" and "difficult" tasks.
70.0
6 8
Number of tasks
Figure 6: The final accuracy for all tasks of Task-wise VS Layer- wise on CIFAR-100 Split.
that the learn-
Impact of selected tasks in trust region. We first evaluate the accuracy of Top-1 and Top-2 selected tasks as shown in Table It shows that selecting the top-2 most correlated tasks could achieve better accuracy on both 100 Split and 5-Dataset settings. Note that good performance can also be achieved even with the Top-1 case. Moreover, we illustrate the detailed task selection in the trust region for both layer-wise and task-wise on 5-Dataset setting in Fig. 7 For the task-wise, current task always selects the two adjacent previous tasks for all layers. Differently. the task selection varies for layer-wise, leading to more accurate selection of related tasks for each layer. For example, the layer wise trust region for Task (Fashion MNIST) selects Task (MNIST) or Task (not-MNIST) as the most related tasks almost for all layers, over Task 0 (CIFAR-10) and Task 2 (SVHN), which clearly makes sense because Fashion MNIST shares more common features with MNIST and NOT-MNIST.
Task wise
Layer-wise for Task
Layeroyise for T
1sl tack
Seleted
2 Seleted
Task Index
Layer
Layer
Figure 7: The detailed selected tasks on 5-Datasets setting.
6 CONCLUSION
In this work, we propose trust region gradient projection for continual learning to facilitate forward knowledge transfer with forgetting, based on an efficient characterization of task correlation. Par- ticularly, our approach is built on two key blocks, i.e., the layer-wise trust region which effectively select the old tasks strongly correlated to the new task a single-shot manner, and scaled weight projection which cleverly reuses the frozen weights of old tasks in the trust region without modifying the model. Extensive experiments show that our approach significantly improves over the related ofte-offtheeart methods.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is supported in part by NSF Grants CNS-2003081, CNS-2203239, CPS-1739344, and CCSS-2121222.
REPRODUCIBILITY STATEMENT
For the experimental results presented in the main text, we include the code in the supplemental material, and specify all the training details in Section 5. and Appendix A. For the datasets used in the main text, we also give a clear explanation in Section 5.1
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- on Computer Vision (ECCV), pp. 139-154, 2018.
Yaroslav Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.(Online). Available: http://yaroslavvb. blogspot. it/201 1/09//ommiist-daaassee hmml, 2, 2011.
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arXiv:18/2.00422 20188b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019.
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-209 2018
Prithviraj Dhar, Rajat Vikram Singh, Whan-Chuan Peng. Ziyan Wu, and Rama Chellappa. Learn- ing without memorizing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5138-5146, 2019.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773.
PMLR, 2020.
Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing, Improved schemes for episodic mem- ory based lifelong learning algorithm. In Conference on Neural Information Processing Systems, 2020.
Steven CY Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. arXiv preprint 19X.::1910.06562. 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkkss-Baiikkka,.a. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11 114(13):351-33526 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Sang- Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung- Jung-Woo Ha, and Byoung- Tak Zhang Overcom- ing catastrophic forgetting by incremental moment matching. arXiv preprint ftt3::1700.80.0 2017.
10

---

Published as a conference paper at ICLR 2022
Xilai Li. Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp. 9924-3934. PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Iransactions on pattern analysis and machine intelligence, 40(1 (0:22)233552247,, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv:1810.11990 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting, arXiv preprint
2018.
Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE transac- tions on pattern analysis and machine intelligence, 42(3):651-663 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.046% 04671, 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint 0xX1v:2103.09762, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Baabska-Barwinska Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International Conference on Machine Learning, pp. 4528-4537. PMLR, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557. PMLR, 2018
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. arXiv preprint arXiv: 1etp::105.009990 2017.
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012. 12631, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29:3630-3638, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Aushion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 170.07777,77777, 2017.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. arXiv preprint
2018.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint /rXiv:/708.001 2017.

---

Published as a conference paper at ICLR 2022
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Eighth International Conference on Learning Representations, ICLR 2020. ICLR, 2020.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent pro- cessing in neural networks. Nature Machine Intelligence, 188):344372, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995 PMLR, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
12

---

Published as a conference paper at ICLR 2022
A EXPERIMENT SETUPS
Training hyper-parameters. We evaluate our method on multiple datasets against state-of-the-art continual learning methods. 1) PMNIST. We
3-layer fully-connected network. with two hidden layer of 100 units. and train the network for 5 epochs with batch size of 10 for each task. 2) CIFAR-100 Split. CIFAR-I00 (Krizhevsky et al., 2009) consists of images from 100 generic object classes. We use version of 5-layer AlexNet and train each task for maximum of 200 epochs with the early termination strategy based on the validation loss value. The batch size set to 64. 3) CIFAR- 100 Sup. We use modified version of LeNet-5 with 20-50-800-500 neurons and train 50 epochs for each task sequentially. The batch size set to 64. 5-Datasets. We train each task for maximum of 200 epochs with the early termination strategy. The batch size is set to 64. 5) MinilmageNNet Split. Following GPM (Saha et al. 2021). we use the reduced ResNet18 architecture, where the covolution with stride 2 in the first layer. We train each task for maximum of 100 epochs with the early termination strategy with 0.1 initial learning rate and batchsize. In addition, for all the experiments, the threshold è' is set to 0.5, and we select top-2 tasks that satisfy condition Eq. (3). We use the same threshold d/, as GPM (Saha et al. 2021) for subspace construction. We initialize the scaling matrix with the identity matrix and train all models with plain stochastic gradient descent.
B
MORE EXPERIMENTAL RESULTS
B.1
ACCURACY EVOLUTION
envolution for Task
Accuracy envolution Task
Accuracy envolut for Task
Test Accuracy (86)
Test Accuracy
69
Test Accuracy
-3
68
Ours GPM
24-
GPM
73
67
----
Ours
72
Number
Number tasks
Number
(a) CIFAR- 100 Split
Accuracy envolution for Task
Accuracy envolution for Task
Accuracy envol on for Task
Test Test Accuracy (%)
99.4
80
Accuracy
99.2
Test Accuracy
GPM
78
Ours
90
99.0
76
Ours
GPM
88
98.8
2
2.0
2.5
3.0
3.5
4.0
Number of tasks
Number of tasks
Number of tasks
(b) 5- 5-Datasets
Figure 8: Accuracy evolution for different tasks on CIFAR- Split and 5-Datasets settings.
B.2 STANDARD DEVIATION
We have summarized the results the standard deviation for the averaged accuracy and backward transfer over different runs on all datasets in Table
Table 4: The averaged accuracy (ACC) and backward transfer (BWT) with the standard deviation values over different runs on different datasets.
Mahua
CIBAR-100
MinilageNaa
ACCVS)
BWT
ACCYS)
BWTTO
ACC/S)
BWT(9)
ACC/9)
BWT(S)
Miltinak
96.70 0.002
79.58 0.5c
91.54 0.28
69.46 0.62
OWM
00.11
50.94
EWC
52.01
HAT
91.32
50.78
GPM
Ohex (TRGP)
96.34 :
-0.8
74.46
-0.9 0.01
93.56 0.10
-0.04 0.01
61.78
-0.5
13

---

Published as a conference paper at ICLR 2022
B.3 FORWARD TRANSFER
To evaluate the forward transfer, we follow the metric used in (Veniat et al. 2020) and consider the accuracy of the model i-th task after learning the i-th task sequentially, i.e., As, as defined in Eq. (IO). Tables summarize the comparison of As, for each task between GPM and TRGP on PMNIST. CIFAR- 1-00 Split and 5-Dataset, respectively. As the same baseline (c.g.. the accuracy of the model learnt from scratch using the task's own data) for each task will be used when evaluating the forward transfer for GPM and TRGP, we can infer that TRGP achieves the forward transfer
gain of 0.17%, 2.01 %, 2.00% and 2.36% over GPM on PMNIST, CIFAR-100 Split, 5-Datasets and MinilmageNet respectively.
Table 5: The accuracy A1, of the model on i-th task after learning the i-th task sequentially on PMNIST 10 tasks.
Methods
2
3
4
5
6
7
10
Avg
GPM
97.5
97.5
97.3
97.1
97.0
96.9
96.8
96.4
96.5
96.5
96.95
Ours (TRGP)
97.5
97.5
97.5
97.3
97.1
97.1
96.9
96.7
96.9
96.7
97.12
Table 6: The accuracy A of the model on i-th task after learning the i-th task sequentially on CIFAR-100 Split 10 tasks.
Methods
2
3
4
5
6
7
9
10
Avg
GPM
76.8
68.5
72.4
69.9
74.8
72.3
70.3
71.9
73.2
75.1
72.52
Ours (TRGP)
76.9
69.5
75.1
74.1
75.3
75.8
72.8
73.8
73.9
78.1
74.53
Table 7: The accuracy A of the model on i-th task after learning the i-th task sequentially on 5-Dataset tasks.
Methods
4
5
Avg
GPM
78.3
99.1
87.1
99.1
94.1
91.54
Ours (TRGP)
80.9
99.3
92.8
99.4
95.3
93.54
Table 8: The accuracy A., of the model on i-th task after learning the i-th task sequentially on MinilmageNet Split 20 tasks.
16
63.6
29.0
630
740
83.
60.4
396
360
568
36.7
99.3
673
752
183
618
000 014
63.
B.4 COMPUTATIONAL COMPLEXITY
Memory: In terms of the memory, the major difference between TRGP and GPM is that TRGP requires additional memory to store the scaling matrices for each task. However, since the dimension of the scaling matrix is the same with the number of the extracted bases for the input subspace, which is usually small and controllable by the matrix approximation accuracy 6th in Eq. (7), the memory increase is marginal and controllable. Particularly, the memory usage of TRGP can be further reduced by only learning the scaling matrices for the convolutional layers.
We compare the training time between TRGP and other baselines on relatively com- plex task sequences. As shown in Table 9 for CIFAR-100 Split, TRGP takes around 65% more time than GPM, is comparable with HAT and ER_ Res, and takes less time than OWM and EWC: for 5-Datasets, TRGP takes around 21% % more time than GPM. but is much faster than other baselines including EWC, HAT, A-GEM and ER Res; for MinilmageNet, TRGP tasks around 34% more time than GPM, comparable with EWC, but is much faster than A-GEM.
Training time:
14

---

Published as a conference paper at ICLR 2022
Table 9: Training time comparison on CIFAR-100 Split, 5-Datasets and MinilmageNet. Here the training time is normalized with respect to the value of GPM. Please refer (Saha et al. 2021 for more specific time.
Methods
Dataset
OWM
EWC
HAT
A-GEM
ER Res
GPM
Ours (TRGP)
CIFAR-100
2.41
1.76
1.62
3.48
1.49
1.65
5-Datasets
1.52
1.47
2.41
1.40
I
1.21
MinilmageNet
1.22
0.91
1.79
0.82
1.34
B.5 ACCURACY VS LEARNING EPOCHS
The learning dynamics for each task are shown in Figure and T0 Clearly, our approach can perform significantly better than GPM sequence. This is because in GPM, with more tasks being learnt, the optimization space for new tasks becomes more restrictive, leading limited performance for new tasks. Note that the y-axis is the validation accuracy with split validate dataset that used during training, by following the setup in (Saha et al. 2021). The validation accuracy varies because the size of the validate dataset is relatively small (See (Saha et al. (2021) for the specific size). For the testing accuracy in all the tables, we evaluate accuracy with the testing dataset after training.
tasks, especially for the tasks in the tail of the task
Task-1
Vallo Accurace 1%)
Pky
Accuracy
70
Valid Accuracy
S
20
GPM
GPM Ours
60
GPM
55
Epochs
Epochs
Number Epochs
Task
Task
Task
Valid Accuracy (%)
75
t MM
Valld Accuracy
20
Valld Accuracy
70
20
GPM
GPM Ours
60
GPM
Ours
Ours
50
20
40
60
80
100
20
40
60
80
20
40
60
100
Number of Epochs
Number of Epochs
Number of Epachs
Valld Accuracy [%]
Mym wy
Valid Accuracy (%]
65
-00
65
60-
65
35
GPM Ours
GPM Ours
60
Ours
Epochs
Epochs
Epochs
Task-10
Valld Accuracy (%6
www. And myn
75
70-
GPM
Ours
20
40
60
Number of Epochs
Figure Accuracy vs learning epochs for different tasks on CIFAR-100 Split.
15

---

Published as a conference paper at ICLR 2022
Task-1 CIFAR10
Task-2 MNIST
Task- SVHN
Valld Accuracy (8c)
99.2
92.5
Valid Accuracy
70
99.0
Valld Accuracy
90.0
98.8
87.5
GPM
98.6
GPM
85.0
GPM
98.4
20
20
10
20
Number of Epochs
Number of Epochs
Number Epochs
T Fashion-mnist
Task Notmmist
Valld Accuracy (86
99.4
Valld Accuracy (%]]
99.2
99.0
98.8
98.6
Ours
90
Ours
30
20
30
Number of Epochs
Number Epachs
Figure 10: Accuracy vs learning epochs for five tasks on 5-Dataset.
16


Reviews:
[{"review_id": "zmgicRZ2UyC", "reviewer": "Reviewer_jo3a", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths: \n1.This paper put forward the problems of the existing methods and provides a detailed discussion.  \n2.Propose a novel solution to this problem, which achieves substantial performance improvement on all benchmarks and overwhelmed its baseline method.\n\nWeakness: \n1.Toy example 1 seems not convincing. \nIn this example, the only difference between task 1 and task 2 is the sign of the input. Since each task has a separate classifier, we can suppose a simple scenario where network has only one Linear Layer followed by a classifier. Since S2=S1, W2=W1 after learning task 2. Though W1x1 = -W2x2, we can adapt the classifier of task 2 which is task-specific to ensure Wc2 = -Wc1  and  make sure task 2 can be classified correctly\nThe example seems want to illustrate that the ideal output of task 2 should be the same as task 1, i.e., W1x1 = W2x2. But since the input has changed, why cannot the output change?\n2.The way to measure the correlation between tasks.\nDenote ΔX’∈R^(m×n), X∈R^(n×b), Δ∈R^(m×b) as the gradient spanned by the input batch of task t, where  is the input dimension and  is the label dimension. It’s projection onto the subspace of task j can be written as ΔX’ BjBj’. Perform SVD on X, the gradient projection can be rewritten as Δ(U∑V’)’ BjBj’, which is δU’BjBj’. Why use the projection of a vector spanned by U to measure the correlation instead of the base U?\n3.As said in the paper, trust region aims to select strongly correlated tasks. But in Session ‘Impact of selected tasks in trust region’, it is obvious that trust region only selects most close tasks even if they are not strongly correlated, and this phenomenon can be seen in the task-wise experiment and higher layers in the layer-wise experiments, which makes me doubt the effectiveness of the trust region selection.\n4.In the Session 4.2, this paper says that if task t is strongly correlated with task j, the weight projection on subspace of Sj is important for task t. This view seems like an assumption which is not proved. It would be more than welcome if more convincing proof could be provided.", "summary_of_the_paper": "Some existing methods put restrictive constrains on the optimization space of the new task to prevent catastrophic forgetting, which may lead to unsatisfactory performance for the new tasks.\nThis paper aims to facilitate the forward knowledge transfer based on an efficient characterization of task correlation. Main contributions can be summarized as follows:\n1.Introduce a novel notion of ‘trust region’ based on the norm of gradient projection onto the subspace spanned by task inputs to measure task correlation.\n2.Proposed a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the Trust Region through a scaled weight projection.\n3.Developed a continual learning approach, trust region gradient projection(TRGP) based on the introduced Trust Region, scaled weight projection and a module to construct task input subspace.\n4.Compared to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all Benchmarks.", "summary_of_the_review": "This paper put forward the main problem of existing methods and provides a detailed discussion. The author proposes a novel solution to this problem, which achieves substantial performance improvement on all benchmarks and overwhelms its baseline method. But some of the claims seem not convincing and some seem lack illustration.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "z_fXCJPCros", "reviewer": "Reviewer_YFLo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** \nThe paper:\n* addresses an important problem with a good solution\n* is well motivated \n* is theoretically sound\n* is well written and easy to follow\n* does a good job at concisely reviewing the important recent work on the topic\n\n**Weaknesses:**\nI don't see any major weaknesses in this paper. I have a few questions that I would like the authors to address (See below). Here are a few points that the paper can improve upon:\n\n* providing the memory footprint of the algorithm\n* comparing wall-clocks of TRGP/GP\n\n**Questions for authors:** I appreciate it if the authors can clarify the following points and provide additional information.\n\n1. The scaling matrix $Q_{j,t}^l$ plays a crucial role in your approach. This matrix is balancing the freezing/unfreezing process. Why wouldn't the network always choose $Q$s such that all parameters in the Trust Region are unfrozen? This would presumably lead to the least loss for the current task (as it provides the maximum capacity for the task). If so, why should one optimize $Q$ as opposed to simply fixing it to unfreeze all parameters in the trust region? \n\n2. Could you please comment on the memory footprint of your algorithm? Also, could you please provide a head-to-head wall-clock comparison between GP and TRGP?\n\n3. Last question is also about unfreezing and matrix $Q$. If my understanding is correct, all tasks in the Trust Region (for each layer)  are currently treated as equally important. For instance, if, for a certain layer, Task A and Task C have a correlation of 1.0 (same tasks), and Task B and Task C have a correlation of 0.8,  they are both added to the trust region of Task C, but the information that Task A is more important to unfreeze is missing and is left up to the optimization on $Q$ to decide this by itself.  Do you think there is a benefit in incorporating the correlation values into your Trust Region?\n", "summary_of_the_paper": "**Summary:** The paper focuses on gradient projection (GP) for incremental learning. The authors motivate their work by stating that while approaches based on GP lead to superior performance in overcoming catastrophic forgetting, they suffer from a significant drawback. In GP, once one calculates the subspaces spanned by layerwise inputs for a task, say Task A, the network is then forced to only update the weights in an orthogonal direction to these subspaces for learning a subsequent task, say Task B (hence keeping important parameters for Task A intact and overcome catastrophic forgetting). However, suppose Task B and Task A are similar (an extreme case is when Task B is the continuation of Task A!). In that case, we know that the essential parameters for Task A are likely to be important for Task B and that the network could benefit by continuing to update the important parameters for Task A, which is not allowed in GP algorithms. This behavior has two consequences: 1) intransigence, i.e., the network won't be able to learn Task B as effectively as possible, and 2) the network will have reduced backward transfer. The backward transfer issue is apparent in the extreme case when Task B is a continuation of Task A, and the network's performance on Task A would have improved if it was able to learn Task B using the important weights for Task A!  The paper addresses this issue, with a simple, yet practical, solution. For a new task, the authors calculate the correlation between the subspaces calculated for old tasks and the new task (layerwise), keep track of the most correlated previous tasks for each layer, and denote them as (layerwise) Trust Regions. Next, the authors propose a scaled weight projection that allows for unfreezing the important parameters in the Trust Region of the new task, while learning this task, and learn the corresponding weights as part of the optimization process. Finally, the authors report results on Permuted MNIST (PMNIST), CIFAR100 Split, CIFAR100 Sup, and a sequence of 5-Datasets with 10-class classification which includes CIFAR-10, MNIST, SVHN, not-MNIST, and Fashion MNIST, in comparison with GP methods, regularization-based methods, and memory replay method. They show consistent improvement in accuracy, and more interestingly in backward transfer. \n", "summary_of_the_review": "**Overall evaluation:**  I think the paper addresses a very important problem. Generally speaking, non-memory-replay-based methods for overcoming catastrophic forgetting could suffer from intransigence, which is the inability to learn new tasks due to increased stiffness/rigidity of the network. This reduces both forward and backward transfer, especially when the tasks are similar (e.g., revisiting an old task). This paper provides a rational solution to this problem, which provides consistent numerical improvement over the state-of-the-art. From an editorial point of view, the paper is well-written, easy to follow, and it provides a good overview of the recent literature on the topic. Hence, I think this is a good paper and vote for its acceptance.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aQfcLX-cAUt", "reviewer": "Reviewer_tgxV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- This paper points out a major issue of GPM (Saha et al.): the orthogonal projection is too restrictive.\n- The proposed heuristic empirically improves the performance.\n\n## Weaknesses\n### The writing is hard to follow\nSince this paper introduces multiple new concepts, it was hard for me to understand the overall algorithm and the intuition behind it.\nI think the writing can be improved a lot.\nFor example,\n- It was hard to find out what the authors are trying to do with the trust region. I was confused if they are going to allow or restrict the model update in the trust region.\n- In Figure 2, $S_l^f$ seems to be the 2D input subspace of layer $l$ in 3D input space. But why is there $\\nabla_{W^l} \\mathcal L_t$, which should be in the parameter space? According to Eq.(2), $\\nabla_{W^l} \\mathcal L_t$ and $\\mathrm{Proj} (\\nabla_{W^l} \\mathcal L_t)$ are matrices, but why are they represented as a vector?\n\n### Some of the reasoning steps are controversial\n- Toy example 1 is not a good example. The authors claim that orthogonal projection is problematic because the optimal model for task 2 should be $W_2^l = -W_1^l$. However, that will lead to complete forgetting of the first task. In fact, this example is impossible to solve with a single linear layer.\n- Therefore, I do not agree with the conjecture motivated by this example: *naive orthogonal projection could possibly compromise the learning performance of the new task that is strongly correlated with old tasks, especially when the correlation is “negative” as in the toy example 1*. Toy example 1 is a situation where the model has to choose between two conflicting options: (i) perfectly memorize task 1 and ignore task 2, (ii) completely forget task 1 and learn task 2. Orthogonal projection is a method for option (i), which is one of the best things a linear model can do.\n- Considering my previous arguments, the claim that the weight projection to the subspace of trust region should be modified is somewhat arbitrary. I think more theoretical justification is needed.\n\n### Lack of justification and empirical evidence for the scaled weight projection\nAlthough the authors claim scaled weight projection as one of their main contributions, there is no explanation of how the scaling matrix $Q_{j,t}^l$ in Eq.(6) is initialized and trained.\nAlso, there is no ablation study to prove its effectiveness.\nWithout further information, I cannot acknowledge this component as a contribution.\n\n### Limited novelty\nIn the end, the authors perform a relaxed orthogonal projection of the gradient.\nThe proposed method is just one way of relaxing the orthogonal projection, and there is not enough justification for why this particular algorithm should be effective.\nI think there can be various ways to relax the orthogonal projection.\nIt would be better if the authors could provide a thorough empirical analysis of multiple relaxing schemes or theoretical justification for the proposed method.", "summary_of_the_paper": "This paper proposes a continual learning method based on gradient projection memory (GPM) of Saha et al., which projects the gradient of each layer to be orthogonal to the input subspace of previous tasks.\nMotivated by the fact that the orthogonal projection can harm the performance by being too restrictive, the authors propose a heuristic algorithm to reduce the restriction.\nSpecifically, the authors choose a subset of most \"correlated\" tasks and let the model change along the subspace of the correlated tasks.", "summary_of_the_review": "This paper proposes a heuristic to improve GPM (Saha et al.).\nHowever, the proposed method lacks theoretical justification, useful insights, and some essential experiments.\nAlso, the overall writing should be improved.\nTherefore, I do not think this paper meets the ICLR standards.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VNsuHoW_wZ2", "reviewer": "Reviewer_aL1A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n    1. The problem of forward knowledge transfer is an unexplored problem in the continual learning literature.\n\n    2. The paper reads well.\n\n    3. Experiments are somewhat convincing.\n\nWeaknesses:\n\n    1. Experiments are not extensive enough and important benchmarks are missing.\n\n    2. Thorough comparison with recent works is missing.\n\n", "summary_of_the_paper": "In this paper, the problem of forward knowledge transfer in continual learning settings is explored. The idea is to measure correlations between the learned tasks based on the notion of \"trust region\" which helps to identify the most similar learned tasks to the current task. The core idea is that frozen weights for similar past tasks can be relaxed to reuse them to learn the current task. Since task similarities are used for this purpose, this will not lead to catastrophic forgetting and at the same time helps to transfer knowledge. Experiments on four benchmarks are provided to demonstrate that the method is effective.", "summary_of_the_review": "Current works in continual learning focus mostly on tackling catastrophic forgetting and overlook accumulative learning. Hence, I think the authors have selected a good area and the proposed idea also seems to be sound. However, I have some reservations about this work:\n\n1. Given the expectation in the recent literature on continual learning, only relatively simple datasets are considered in the experiments. I think results on more complex datasets such as split-Sub-ImageNet and Split-ImageNet should be added. \n\n2. I was wondering why forward transfer metric is not used for comparison? I think it is as informative as backward transfer metric.\n\n3.  An area of interest is to analyze the learning curves and dynamics of learning, i.e., performance vs learning epoch. It is helpful to see if the proposed method enables better jumpstart performance when a new task is learned.\n\n4. Comparison with more works is missing. For example, EWC and HAT are both outdated for regularization-based methods and many recent follow-ups exist. For a realistic comparison against prior works, state-of-the-art methods for each group of methods should be included. Please check the recent literature.\n\n5. Computational complexity is overlooked. Whereas it is an important factor for continual learning. I think it is necessary to include how much additional computational load should be performed to benefit from TRGP.\n\n6. For a more informative comparison, the standard deviation in results should be reported. Also, BWT metric can be reported more accurately by including more decimals.\n\n In conclusion, I think this work is in a good direction but further improvement is necessary to make it suitable for a venue similar to ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission addresses the problem of whether or not to update weights for a previous task in continual learning.  The approach is to specify a trust region based on task similarity and update weights only in the direction of the tasks that are similar enough to the current one.  The paper was on the balance well received (3/4 reviewers recommended acceptance, 2 with scores of 8) and complemented for its simple but effective approach, and good discussion of related literature.  The submission attracted a reasonable amount of engagement and discussion between reviewers and authors, which should be taken into account in the final version of the paper.
Decision: Accept (Spotlight)