Continual Learning In Low-coherence Subspace: A Strategy To Mitigate Learning Capacity Degradation
Authors: Zihao Xu
Zhengyu Li
Yufei Shi
Xihao Wang
Yingjie Liu
Kexin Ke
Xian Wei
Venue: Submitted to ICLR 2023 (2023)
Keywords: oblique manifold
continual learning
learning capacity degradation
catastrophic forgetting
low-coherence
orthogonal projection

Abstract: Methods using gradient orthogonal projection, an efficient strategy in continual learning, have achieved promising success in mitigating catastrophic forgetting. However, these methods often suffer from the learning capacity degradation problem following the increasing number of tasks. To address this problem, we propose to learn new tasks in low-coherence subspaces rather than orthogonal subspaces. Specifically, we construct a unified cost function involving regular DNN parameters and gradient projections on the Oblique manifold. We finally develop a gradient descent algorithm on a smooth manifold to jointly minimize the cost function and minimize both the inter-task and the intra-task coherence. Numerical experimental results show that the proposed method has prominent advantages in maintaining the learning capacity when tasks are increased, especially on a large number of tasks, compared with baselines.

Under review as a conference paper at ICLR 2023
CONTINUAL LEARNING IN LOW-COHERENCE SUBSPACE: A STRATEGY TO MITIGATE LEARNING CAPACITY DEGRADATION
Anonymous authors Paper under double-blind review
ABSTRACT
Methods using gradient orthogonal projection, an efficient strategy in continual learning, have achieved promising success in mitigating catastrophic forgetting. However, these methods often suffer from the learning capacity degradation prob- lem following the increasing number of tasks. To address this problem, we propose to learn new tasks in low-coherence subspaces rather than orthogonal subspaces. Specifically, we construct unified cost function involving regular DNN parameters and gradient projections on the Oblique manifold. We finally develop gradient descent algorithm on smooth manifold to jointly minimize the cost function and minimize both the inter-task and the intra-task coherence. Numerical experimental results show that the proposed method has prominent advantages in maintaining the learning capacity when tasks are increased, especially on a large number of tasks, compared with baselines. Li & Lin 2018
INTRODUCTION
Deep Neural Networks (DNNs) have achieved promising performance on many tasks. However, they lack the ability for continual learning, i.e., they suffer from catastrophic forgetting [rench] (1999) when learning sequential tasks, where catastrophic forgetting is a phenomenon of new knowledge interfering with old knowledge. Research on continual learning, also known as incremental learning Aljundi al. 2018 Chaudhry et al. (201 (2018)): Chen & Liu| (2018); Alljundi et al. (2017), and sequential learning Alljundi et al. (201 8b); McCloskey & Cohen (1989), aims to find effective algorithms that enable DNNs to simultaneously achieve plasticity and stability, i.e., to achieve both high learning capacity and high memory capacity.
Various methods have been proposed to avoid or mitigate the catastrophic forgetting De Lange et al. (2019), either by replaying training samples Rolnick et al. (2019); Ayub & Wagner (2020); Saha et al. (202T. or reducing mutual interference of model parameters, features or model architectures
between different tasks Zenke et al. (2017); Mallya & Lazebnik (2018 Wang al. (2021) Among et al. (2020); Zeng et al. (2019);
these methods, Gradient Orthogonal Projection (GOP))Chaudiee
Farajtabar et al. (2020): Li et al. (202T) is an efficient continual learning strategy that advocates projecting gradients with the orthogonal projector to prevent the knowledge interference between tasks. GOP-based methods have achieved encouraging results in mitigating catastrophic forgetting. However, from Fig. D we observe that these methods suffer from the learning capacity degradation problem: their learning capacity gradually degraded the number of tasks increases and eventually becomes unlearnable. Specifically, when learning multiple tasks, e.g., more than 30 tasks in Fig. [ their performance on new tasks dramatically decreases. These results suggest that the based methods focus on the stability and somewhat ignore the plasticity. Ignoring the plasticity may limit the task learning capacity of models, i.e., the number of tasks that a model can learn without forgetting.
To address this issue, we propose to learn new tasks in low-coherence subspaces rather than orthogonal subspaces. Specifically, Low-coherence projectors are utilized for each layer to project features and gradients into low-coherence subspaces. To achieve this, we construct unified cost function to find projectors and develop a gradient descent algorithm on the Oblique manifold to jointly minimize inter-task coherence and intra-task coherence. Minimizing the task coherence can reduce the mutual interference between tasks, and minimizing the intra-task coherence can enhance the model's expressive power. Restricting projectors the Oblique manifold can avoid the scale ambiguity

---

Under review as a conference paper at ICLR 2023
Aharon et al. ) Wei et al. (2017). i.e., preventing the parameters of the projector from being extremely large or extremely small.
The main contributions of this work summarized follows. First, to address the learning capacity degradation problem of GOP, we propose a novel method, namely, Low-coherence Subspace Projection (LcSP), that replaces the orthogonal projectors with the low-coherence gradient projectors, allowing the DNN to maintain both plasticity and stability. Additionally, our work observes that the GOP models with Batch Normalization (BN) loffe Szegedy (2015) layers could cause catastrophic forgetting. This paper proposes two strategies in LcSP to solve this problem, i.e., replacing BN with Group Normalization (GN) Wu & He (0188) or learning specific BN for each task.
RELATED WORK
In this section, we briefly review some existing works of continual learning and the GOP based methods.
Replay-basee Strategy The basic idea of this type of approach is to use limited memory to store small amounts of data (c.g.. raw samples) from previous tasks, called episodic memory, and replay them when training a new task. Some of the existing works focused on selecting a subset of raw samples from the previous tasks (askickolnickk al. 2019); [ssele & Cosgun (2018); Chaudhry et al. (2019); Zhang et al.] (2020). In contrast, others concentrated on training a generative model to synthesize new data that can substitute for the old data Shin al. (2017); Van de Ven & Tolias (2018): Lavda et al. (2018) Ramapuram et al. 20200)
Regularization-basee Strategy This strategy prevents catastrophic forgetting by introducing regularization term in the loss function to penalize the changes in the network parameters. Existing works be divided into data-focused, and prior-focused methods De Lange et al. (222T) The
Data-focused methods take the previous model as the teacher and the current model as the student, transferring the knowledge from the teacher model the student model through knowledge distillation.
Typical methods include LwFFLi& & Hoiem (017). LFL. Jung et al. (016)). Rannen et al. (2017). DMC ZZang al. (2020) and GD-WILD Lee al.] (001)) The prior-focused methods estimate distribution over the model parameters, assigning an importance score to each parameter and penalizing the changes in significant parameters during learning. Relevant works include SI Zenke
et al. (2017). EWC Kirkpatrick al. (2017), RWalk Chaudhry et al. (2118)). AGS-CL Jung et al.
(2020) and IMM Lee et al. (2017).
Parameter Isolation-based Strategy This strategy considers dynamically modifying the network architecture by pruming, parameter mask, or expansion to greatly or even completely reduce catas- trophic forgetting. Existing works can be roughly divided into two categories. One is dedicated to isolating separate sub-networks for each task from a large network through pruning techniques and parameter mask, including PackNet |aaklet/Mallyaa & Lazebnik (2018). PathNet Fernando al. (2017). HAT Serra et al. (2018) and Piggyback Mallya et al. (2018). Another class of methods dynamically expands the network architecture, increasing the number neurons or sub-network branches, to break the limits of expressive capacity (Rusu et al. 2016 Aljundi et al. [017]; Xu & Zhu 2018 Rosenfeld & Tsotsos 2018). However, as the number of tasks growing, this approach also complicates the network architecture and increases the computation and memory consumption.
Gradient Orthogonal Projection-based Strategy Methods based on GOP strategies, which reduce catastrophic forgetting by projecting gradient or features with orthogonal projectors, have been shown to be effective continual learning with encouraging results Farajtabar et al. (2020); ZZng al.) (2019): (Saha et al.) (0221): Wang et al. (2021); Chaudhry et al. (2020). According to the different ways of finding the projector, we can further divide the existing works into Context Orthogonal Projection (COP) and Subspace Orthogonal Projection (SOP). Methods based on COP, such OWM Zeng et al. 2019). Adam-NSCL WWang et al. (202T), and GPM Saha et al. (202T). always rely on the context of previous tasks to build projectors. In contrast to COP, SOP-based methods such as WETHOO-SUBSPPAS (oreedhrhy et al. (2020) use hand-crafted, task-specific orthogonal projectors and yield competitive results.
2

---

Under review as a conference paper at ICLR 2023
This paper proposes novel approach to continual learning called LcSP. Compared with other methods based on GOP, LcSP trains the network on the low-coherence subspaces to balance plasticity and stability and overcomes the learning capacity degradation problem, which significantly decreases the performance of GOP methods with the increasing number of tasks.
CONTINUAL LEARNING SETUP
This work adopts the Yask-Incrementee Learning (TIL) setting, where multiple tasks are learned sequentially. Let assume that there are T tasks, denoted by Tr for .,..., I with its training data D. {( 3/, T }. Here, the data (I,, 34) E X x V. is assumed to be drawn from some independently and identically distributed random variable, and т E denotes the task identifier. In the TIL setting, the data D, be accessed if and only if task Te arrives. When episodic memory is adopted, a limited number of data samples drawn from old tasks can be stored in the replay buffer M so that D, U M can be used for training when task T, arrives.
Assuming that a network / parameterized with + = =0, consists of two parts, where denotes the parameters of the backhone network and op denotes the parameters of the classifier. Let [(;; X' x T + H. denote the backbone network parameterized with 0 {Wi)f=1, which encodes the data samples a into feature vector. Let f (x; :p) H + y denote the classifier parameterized with w which returns the classification result of the feature vector obtained by fa;;0) 0). The goal of TIL is learn T tasks sequentially with the network f and finally achieve the optimal loss on all tasks.
Evaluation Metrics Once the training all tasks is finished, evaluate the performance of algorithm by calculating the average accuracy A and forgetting F F(Chaudhry et al. (2020) of the network the T tasks -T,, Tr). Suppose all tasks come sequentially, let Ace;,j denote the test accuracy of the network on task T, after learning task Tj. where S j. The average accuracy is defined as
A
ACciTT
(1)
T
and the forgetting is defined as
max je T-11 (Acci. A.c..r).
(2)
T
CONTINUAL LEARNING IN LOW-COHERENCE SUBSPACES
In this section, we first introduce how to find task-specific, low-coherence projectors for LcSP on the Oblique manifold. We then describe how
in specific DNN architecture project features and gradients. Finally, we analyze the factors that enable LeSP to maintain plasticity and stability.
4.1 CONSTRUCTING LOW-COHERENCE PROJECTORS ON OBLIQUE MANIFOLD
Here, we first introduce the concept of coherence metric. The coherence metric is usually used in compressed sensing and sparse signal recovery to describe the correlation of the columns of measurement matrix Candes et al. 1011): Candes & Romberg 2007 Formally, the coherence of matrix M is defined as
I(M;, =
maxj<k
[M
((MN))
max;,j
(M,NN [ N
M
where M, and M, denote the column vectors of matrix M. Without causing confusion, we use (MM denote p(M, M). To measure the coherence between different projectors, introduce the

---

Under review as a conference paper at ICLR 2023
Babel function Li & Lin (2018), measuring the maximum total coherence between a fixed atom and collection of other atoms in a dictionary, which can be described as follows.
KM;, M2)
((M)
max max A.JA==M REA
[M,]] |M,ll
JEA
With the concept of a coherence metric in mind, we then introduce the main optimization objective in finding projectors. Specifically, suppose that the DNN has learned the task T1. T2, .... T in the subspace S1, S2, .... S respectively, and P2, P denote the projectors of all previous tasks. When learning task Tr, we project features and gradients into a d4-dimensional low-coherence subspace St with projector P. so that the LcSP can prevent catastrophic forgetting. The projector P. can be found by optimizing
arg min B(P)),
(5)
S.L. Pt E R=
rank de.
Two considerations need be taken in solving Eq. (5) i... considering the rank constraint and avoiding the entries Pt being extremely large or extremely small. With these considerations in mind, we can rephrase the rank and scale constrained problem manifold, more specifically on the Oblique manifold O.M(m, de). by setting P. = O,O," and normalizing the columns of O₂ i.e, diag(O, O.) In. where diag(-) represents the diagonal matrix and I is the n x n identity matrix. With these settings, the new cost function J(.) and optimization problem can be described as follows:
problem the Riemannian
t> J(O,) A ((O,0))) +Y (O,0)) (0,00))
1=1²
(6)
O arg min (Or), s.t. O, OBE, d:).
In the cost function J(O,), further divided the optimization objective into inter-task B(O,0,)) and intra-task (O,0, ), and utilize parameters À and y to provide a trade-off between them. Here, intra-task coherence is optimized to maintain the full rank of O,, Meeting the full-rank constraint helps to balance plasticity and stability in the case of increasing tasks. Relevant ablation studies and numerical analyses are given in the appendix.
Optimization on the Oblique manifold, i.c., the solution lies on the Oblique manifold, is a well- established area of research Absil al. (2009); Absil & Gallivan (2006); /eelvan al. (2012). Here, we briefly summarize the main steps of the optimization process. Formally, the Oblique manifold O.(nn.) p) is defined as
OM(n, p) A {X R"XP, diag X) = Ip)
representing the set of all n x p matrices with normalized columns. O.M can also be considered as an embedded Riemannian manifold of Roxp, endowed with the canonical inner product
(X1, X2) trace (x] X2),
(8)
where trace(-) represents the sum of the diagonal elements of the given matrix. For given point X on OM, the tangent space at X, denoted by T OM, is defined as
TxOM(n, p) = {U E gTxp, diag(X X U)= = =}.
(9)
Further, the tangent space projector Px at X which projects H E Raxp into TMM is represented
Px = X ddiag [ H)
(10)
where ddiag sets all off-diagona entries of matrix to zero. When optimizing on OM, the kth iterate X must move along descent curve on OM for the cost function, such that the next iterate X will be fixed on the manifold. This is achieved by the retraction
(1)) where normalize scales each column of the input matrix to have unit length. Finally, with this knowledge, we can extend the gradient descent algorithm to solve any unconstrained optimization problems on O.,, which can be summarized as
Rx, (U) normalize Xx + U),
U Px, EXx(Vx,) J). X Rx, (-aU),
(12)
4

---

Under review as a conference paper at ICLR 2023
where Vx, J denotes the Euclidean gradient the kth iterate and is the step size. Finally, algorithm for finding O, in OM to task Te summarized in Algorithm
Algorithm Construct the O on OM for Task Tr
Input: Ou..... O,-1
Output: O
1: Rxx (U) normalize X U))
normalize scales cach column input
2: Xo + random initiaiiaaiinn on O.M
3: k
4: Set telerance error 0 & <
5: while True do
6: G + vx(
Calculate Eaclidean Gradient Calculate Riemann Gradient
7:
U G X ddiag( X G)
8:
U| & then
9:
break
10:
end if
11:
a (.,0.5) B (0,1)
12:
Initial step Searching the slep size for calculating next slerate
13:
while JRx (-t- U}) ((x,) - a. [[0]]] ]
14:
8.
15:
end while
16: XE+1 Rx, (-t-U)
Updating the iterate Xx+1 and fixing manifold
17:
1+1
18:
end while
O, X,
20:
return
4.2 THE APPLICATION OF LOW-COHERENCE PROJECTORS IN DNNS
With the LcSP hand, the following introduces some technical details of applying LcSP in DNNs. When learning task Te. LcSP first constructs sssc-fpcc projector P. for each layer before training, and freezes them during training. These projectors are used to project the features and gradients, ensuring that the DNN learns in the low-coherence subspace. Specifically, suppose that network with linear layers used as DNN architecture, let W a/. z/, a², and P denote the model parameters, the input features, the output features, the activation function, and the introduced low- coherence projector in layer E {1,... J,, respectively, LcSP introduces P/ immediately after w such that the pre-activation features are projected into the subspace, i.e.,
2 = (ar) W)) x =+1= d' (2()).
(13)
According to the chain rule of derivation, the gradients at W! will also be multiplied with P/ in backpropagation, as follows
aC
OC
az!
(WW)
0z
((W))
(14)
OC
0zh k+l
(x)), P!
dzk
2
where (Wiue) represents the ith row of W and (x/); is the ith element of æ In Convolutional Neural Networks (CNNs), the input and the output typically represent the image features and have more than two dimensions, e.g., input channel, output channel, height, and width. this case, we z' E hoc)==-oo)xee Xowe and align the dimension of projector with the output channel so that P! E IR Xeau After the projection, we recover the shape of 2 so that it can be used as input for the next layer.
reshape z Rew x ((enn
Overcoming the Catastrophic Forgetting in BN based models BN is widely used module in DNNs to make training of DNNs faster and more stable through normalization of the layers' features by re-centering and re-scaling loffe & Szegedy (015)) However, re- re-centering and re-scaling of the layers' features changes the data distribution (e.g., the mean and the variance) of features previous tasks, which often leads to the catastrophic forgetting of LcSP. For example, when learning the new
5

---

Under review as a conference paper at ICLR 2023
task Tr W may not work for Ti due the change in data distribution caused by BN. To solve this problem, we propose two strategies in LcSP: the strategy (1) learning specific BN for each task, or the strategy (2) using GN instead of BN. We verify the effectiveness of these strategies in experiments and compare their performance in
4.3 METHOD ANALYSIS
In this section, we provide analysis on plasticity and stability of LcSP.
Stability Analysis Let 0 - ( denote the parameter set of f: A0
denote set of variation values of parameters after learning task Tr: P; = [P. denote the projectors set obtained by Algorithm 1 . and zn,, denote the input and ouput when feeding the data of task Ty (q S t) into the network f which has been optimized in learning task TT.
Lemma 1. Assume that f is fed the data of task Te (g < t), then f can effectively overcomes catastrophic forgetting if
z 4.9
gg
Vq St
(15)
holds for E (1,2,...,11 I.).
Lemma 1 suggests that f can overcome catastrophic forgetting if the output of f to previous tasks invariant. In the following, we prove that LcSP achieves approximate invariance the output of previous tasks.
Proof. Suppose q =
When 1, x.y, = zqq.q
9.9" Then
2 x + AW AW;)
= W , AW
(16)
= + x t P.
Let si denote the gradient when training the network on task Te In backpropagation, AW; P.P. Then a', = x. P: If the inter-task coherence p(P), 0 0, then P Projectors satisfying this condition can be found by Algorithum 7 We can prove that 44 ~ n. holds for all layers by repeating the above process. This proof can also be generalized to any previous task T
Plasticity Analysis Let 9. g P denote the projected gradient at W f can achieve optimal loss on task Tr (g': g > 0 holds for each {1,..., L}. where (.,)) represents inner product. Here, we prove that (g. g 0 holds for each E [1.....)).
Proof. Let g = P! denote the projected gradient, we have
(g'. g.) = g... g gi
(17)
0g.0% 0 > o.
Note that |gg0%%|| is always positive unless O is O. This result is easy to generalize to each layer.
5 EXPERIMENTS
In this section, we evaluate our approach on several popular continual learning benchmarks and com- pare LcSP with previous of-he-rf methods. The result of accuracy and forgetting demonstrate the effectiveness of our LcSP, especially when the number of tasks is large.
5.1 BENCHMARKS
Benchmarks for Learning 20 Tasks We conducted this experiment on four image classification datasets: Permuted MNIST, Rotated MNIST. Split CIFAR100 and Split minilmageNet. Permuted MNIST is constructed by randomly rearranging LeCu (1998) image pixels, using different seeds for different tasks. Rotated MNIST constructed by rotating the MNIST image certain

---

Under review as a conference paper at ICLR 2023
Table The average accuracy and forgetting results of the proposed LcSP and baselines. Memory denotes whether the method is trained using replay strategy with episodic memory.
Permuted MNIST
Rotated MNIST
Methods
Memory
(ccuracy(S))
Forgettigg
Accuracy(Si
Fargetting
EWC
Kirkpatrick al.
2017
08.4(=#0.76)
0.25(#0.01)
13.6(#0.88 81)
0.53(
AGEM
Chauddary al.
20186
78.3(+00.2))
0.15(+:.01)
60.5(+1.77)
0.36(=0.00
ER
NI-RING Chaudhry
2019
09.3(+
0.12(10.01)
70.9(10.388)
(.000)))
KCL Derakhshuni
2021
X
85.5(110.78)
(:0.00)
81.8(100.60)
0.01(110.00
OWM Zeng al
2019)
X
46.35
0.01
44.78
0.01
Adam
NSCT Wany
2021
X
26.44
0.70
42.73
0.56
GPM
Saha Saha al
2021
X
80.54
0.16
80.34
0.16
ORTHOG-SUBSPACE Chaudhry
2020
x
86.6(=0.91)
0.04(#0.01)
80. (=:005)))))
0. 0..00)))0
LeSP
92.2(=0.65)
0.05(=00011)
86.6(=00.96
0. 0.08(=0.01)
Split CIEARI00
Split minilmageeNtt
Methods
Memory
Accuracy(S)
Forgetting
Accuracy(S)
Forgetting
EWC Kirkpstrick
2017
43.2(=2.77)
0.26(+0.02)
31.8(+2.34)
0.24(=0.00)
AGEM Chaudhry
31.3(
0.18(::.08)
42.3(41.40
0.17(10.01)
Chourlrry
2019
39.6(11.19)
0.14(10.01)
49.8(12.92)
0.12(10.01)
2018
X
72.06
Derakhshuni
2021
62.7(110.89)
0.06 (..001))01)
53.3(100557)
0.04(10.00)
7eng al
2019
50.94
ORTHOG.SURRSACE
Chaudhry et
2020
64.3(+0.50)
0.07(#0.01)
51.4(#1.44)
0. 0.10(=0.01)
GPM Saha et al
72.48
60.41
Adam-NSCI Wang al.
2021
X
75.95
0.04
63.27
0.06
ORTHOG-SUBSPACE
X
29. 29.1(#1.99)
0.53(#0.03)
31.6(#11.11
0.22(=0.01)
OKTHHOG-UUBSPACE-NN
X
04.3(=
0.35(#0.01)
31.2(+1.30)
(.23(=0.02)
ORTHHOC-SUSPPACE-NN
X
40.9(+2.31)
0.11(+:.08)
38.6(11.25 23)
(00.02)
LcSP
X
24.1(10.76)
0.52(10.011
23.0(110.8))
0.50(
LeSP-GN
X
66.7 (...44)
1.000100.00)
65.0(11.14)
0.000 0.000)
LeSP-BN
X
(2.7(1
0.0.01)0.00)
67.9 (10.40))))
(.0000)..00)
angle. The rotation angle is random value between [0, π] and arbitrary selection for different tasks. In this experimental setup, both of the above MNIST datasets contain 20 tasks, each task containing 10,000 samples from 10 classes. Split CIFAR is constructed by splitting CIFAR 100 into multiple tasks, where each task contains the data pertaining to five random classes (without replacement) out of the total 100 classes. Split minilmageNet Vinyals et al. (2016) is a subset of ImageNet. In Split minilmageNet, each task contains the data from five random classes (without replacement) out of 100 classes. Both CIFAR100 and minilmageNNt contain 20 tasks, each contains 250 samples from each of the five classes.
Benchmarks for Learning 150 Tasks and 64 Tasks This experiment conducted on two image classification datascts: Permuted MNIST and Permuted CIFARIO. Both Permuted MNIST and Permuted CIFARIO are obtained by randomly rearranging image pixels. In this experimental setup, Permuted MNIST contains 150 tasks, and Permuted CIFARIO contains 64 tasks, each containing 10,000 samples from 10 classes.
5.2 BASELINES
We compare the proposed method with SOTA based on GOP. As aforementioned, we generalize GOP into COP and SOP. For methods based on the COP strategy, we compare proposed method with OWM Zeng al. 2019), Adam-NSCL Wang et al. (2021) l) and GPM /aha al. (2021). For methods based on the SOP strategy, we compare the proposed method with ORTHOG- BAAOOGG-UBSAACCE Chaudhry also compare method with HATTSSrraa al. (2018), 8), EWC Kirkpatrick
et al. (2020). Moreover,
cet al. 2017), ) ER-Rin Ri-R Chaudhry et al. (2019). AGEM AGEMCChaddrry et al. 2018b) and Kernel Continual Learning (KCL) Derakhshani al. (0021).
7

---

Under review as a conference paper at ICLR 2023
Permuted MNIST (last 10 tasks)
Permuted MNIST (last 10 tasks)
95
(cSP tina ceterrnne)
ONI
OTT SUSSPACE
GPM
Jccuracytse (88
80
LSSP (oetroponnt
70-
60
Loee jou cinemee
MAM 199
50-
Average
30
10
20
100
150
20
60
1000
150
Number of Task
Number of Task
(a)
(b)
Permuted CIFARIO (last 5 tasks)
Permuted CIFAR10 (last 5 tasks)
1:57 ON Sow ccherrrce
JPP BN (orttggons
GPM
OWM
SUBSRACE
Aram RSCL
Average Accuracyt
0.2
40-
Forgetting
1:57 ON Tow coherance)
LC57 ON to (ottogoona DWM
GPM
SLESTACE
Adam
30-
0.1
0.0
24 32 40 48
56
16
24
32
64
Number of Task
Number of Task
(C)
(d)
Figure 1: Fig. 1(a) and 1(b) show the average accuracy and forgetting of the last 10 tasks on Permuted MNIST when learning 150 tasks. Fig. I(c) and (dd show the average accuracy and forgetting of the last tasks on Permuted CIFAR1O when learning 64 tasks.
Average Accuracy on Permuted MNIST
Forgetting on Permuted MNIST
48.42
73.54
7355
73.54
72.44
72.55
0:58
6:24
3:21
0.26
9.29
0.76
325
75.22
3.35
10.24
76.42
73.52
6.12
2:23
0.0
0.2
0.21
024
Lambda
72.04
17.32
76.56
75.07
Lambda 0.6
6.51
6:21
321
0.20
021
0.21
022
17.09
76.32
75.27
321
0.20
821
0.2
321
020
7600
16.1
15.04
54
0.8
322
0.11
429
-72
49.59
15.28
24
75.35
27.53
15:24
1.0
0.41
0.22
2.22
0.20
0.20
0.11
-118
0.4
0.6
0.8
1.0
-78
0.2
0.6
0.8
1.0
-317
Gamma
Gamma
(a)
(b)
Figure 2: Average accuracy and forgetting for different À and on Permuted MNIST. A fully connected network with 2 hidden layers, each with 64 neurons, used for this experiment.
5.3 IMPLEMENTATION DETAILS
Learning Tasks For experiments on Permuted MNIST and Rotated MNIST, all methods use fully connected network with two hidden layers, each with 256 neurons, using ReLU activations. For experiments on CIFAR and minilmageNet, all methods use standard ResNet18 architecture except OWM, HAT. and GPM which use AlexNet Krizhevsky proposed LcSP makes two changes to the BN layer in standard ResNet18: (1) learning specific BN for each task and (2) replacing BN with GN. We also apply these strategies to ORTHOG-SUBSPACCE
al. (2012). As described in 4.2 the
8

---

Under review as a conference paper at ICLR 2023
for additional comparison. For experiments on MNIST, all tasks share the same classifier. For experiments on CIFAR and minilmageNet, each task requires a task-specific classifier. For all experiments, LcSP does not episodic memory to store data samples for data replay. For all methods, We use Stochastic Gradient Descent (SGD) uniformly. The learning rate is set to 0.01 for experiments on MNIST and 0.003 for experiments CIFAR and ImageNet. Both À and y in Eq. (??) are set to 1. All experiments were run five times with five different random seeds, with the batch size besing 10.
Learning 150 Tasks and 64 Tasks For experiments on Permuted CIFARIO, LcSP uses ResNetl 8 architecture and applies the strategy (1) to change BN in ResNet18. To compare the performance of different GOP strategies, we did not use episodic memory for ARTHOG-SUBSPACCE Except for these changes, other experimental settings are the same as described above.
5.4 EXPERIMENTAL RESULTS
Comparisons of Learning 20 Tasks Table compares the average accuracy and forgetting results of the proposed LeSP and its variants (LeSP-BN and LcSP-GN) with baselines on the four continual learning benchmarks. Therein, LcSP-BN and LeSP-GN adopt strategy (1) and (2) described in respectively. First, as shown in Table I the proposed methods outperform all baselines on MNIST and minilmageNNe. On Permuted MNIST and Rotated MNIST, the average accuracy of LeSP surpasses the baselines by 23.8% ~ 5.6% and 43% ~ 4.8%, respectively. On minilmageNee, the average accuracy of LcSP-BN surpasses the baselines by 44.9% 4.63%. On CIFAR 100, the proposed LcSP-BN achieved a competitive performance with the second highest average accuracy, 3.25% lower than Adam-NSCL, and a forgetting rate of O. The average accuracy of LeSP-GN also outperforms most baselines, being lower than Adam-NSCH but higher than compared methods on minilmageNNe. These results suggest that minimizing the inter-task and intra-task coherence with low-coherence projectors is effective strategy for solving catastrophic forgetting. Secondly, results on CIFAR 100 and minilmageNet also show that BN in ORTHOG-SUBSPACCECCNE
8 4.2
HAT and GPM CIFAR 100
et al. (2000) and LcSP may change previous tasks' data distribution and lead to catastrophic Torgetting. Both strategies (1) and (2) described in 8 4.2 can effectively solve this problem.
Comparisons of Learning 150 Tasks and 64 Tasks In order to demonstrate the high advantage of proposed methods learning a long sequence tasks, the following experiments compare the results with 64 tasks and 150 tasks. Note that, in Fig. [ LcSP (orthogonal) and LcSP-BN (orthogonal) use orthogonal projectors as comprisons while LcSP (low-coherence) and LcSP-BN (low-coherence) use low-coherence projectors. Fig. [(a) and [(b) report the average accuracy and forgetting of last 10 tasks, with learning 150 tasks on Permuted MNIST. Fig. [(c) and I(J] [ the average accuracy and forgetting of last tasks with learning 64 tasks on Permuted CIFARIO. 10. The average accuracy of all methods, except the proposed LcSP-BN (low-coherence), dramatically degrades or is consistently low as the number of tasks increases. Furthermore, it can be seen from I(d) tha) that all methods except ORTHOG-SUBSPACCE have almost no forgetting. This result indicates that methods using orthogonal projectors gradually lose their learning capacity with increasing number of tasks. The proposed method uses the low-coherence projector to relax the orthogonal restriction, effectively solving this problem. Fig. give the ablation study and shows the performance our method with different and When À equals y, the result of average accuracy on Permuted MNIST reached the highest. Results reach the worst when either X or 7 equals zero. These results indicate that both inter-task and intra-task coherence should be minimized to solve the plasticity and stability dilemma.
6 CONCLUSION
This paper proposed a novel gradient projection approach for continual learning to address gradient orthogonal projection's learning capacity degradation problem. Instead of learning in orthogonal subspace, we propose projecting features and gradients via low-coherence projectors to minimize inter-task and intra-task coherence. Additionally, two strategies have been proposed to mitigate the catastrophic forgetting caused by the BN layer, i.e., replacing BN with GN or learning specific BN for each task. Extensive experiments show that our approach works well in alleviating forgetting and has significant advantage in maintaining learning capacity, especially in learning long sequence tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
P-A Absil and Kyle A Gallivan. Joint diagonalization on the oblique manifold for independent component analysis. 5:V-V, 2006.
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms matrix manifolds. 2009.
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on signal processing. 54(11): 4311-4322, 2006.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. pp. 3366-3375, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. pp. 139-154, 2018a.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. arXiv preprint arXiv: 05421, 2018b.
Ali Ayub and Alan R Wagner. Storing encoded episodes as concepts for continual learning, arXiv preprint arXiv:2007.c 2000:20077.06377, 2020.
Emmanuel Candes and Justin Romberg. Sparsity and incoherence in compressive sampling. Inverse problems, 23(3):969, 2007.
Emmanuel Candes, Yonina C Eldar, Deanna Needell, and Paige Randall. Compressed sensing with coherent and redundant dictionaries. Applied and Computational Harmonic Analysis, 31(1):59-73, 2011.
Miguel Carreira- A review of dimension reduction techniques. Department of Computer Science. University Sheffield. Tech. Rep. CS-96-09, 9:1-69, 1997.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. pp. 532-547, 2018a.
Arslan Chaudhry, Marc Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv: 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Ranzato. Continual learning with tiny episodic memories. 2019.
Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33:9900-9911, 2020.
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1 2018
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis. Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv: /0t::/900 08383, 2(6), 2019.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-33 3385,,3335500 2021.
Mohammad Mahdi Derakhshani, Xiantong Zhen, Ling Shao, and Cees Snoek. Kernel continual learning. pp. 2621-2631, 2021.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. pp. 3762-3773, 2020.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha. Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 1701008734, 2017.
10

---

Under review as a conference paper at ICLR 2023
Robert M French. Catastrophie forgetting in connectionist networks. Trends in Cognitive Sciences, (4):128-1355 135, 1999.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. pp. 448-456, 2015.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. 32(1), 2018.
Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetingg learning in deep neural networks. arXiv preprint arXiv:1607.00122 2016.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. Advances Neural Information Processing Systems, 33:3647-3658, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinsker et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Frantzeska Lavda, Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Continual classification learning using generative models. arXiv preprint arXiv: 106/2, 2018.
Yann LeCun. The mnist database of handwritten digits. htlp://yann. lecun. com/exdb/nmist// 1998.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data the wild. pp. 312-321, 2019.
Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Wor Woo Ha, and Byoung-Tak Zhang. Overcom- ing catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
Huan Li and Zhouchen Lin. Construction of incoherent dictionaries via direct babel function minimization. In Asian Conference on Machine Learning, pp. 598-613, PMLR, 2018.
Yanni Li, Bing Liu, Kaicheng Yao, Xiaoli Kou, Pengfan Lv, Yueshen Xu, and Jiangtao Cui. De- feating catastrophic forgetting via enhanced orthogonal weights modification. arXiv preprint 1rXiv:2111.10077 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12))2935-2944 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. pp- 7765-7773. 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. pp. 67-82, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. 24:109-166 165, 1989.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling.
Neurocomputing, 044:3814000 2020.
Amal Rannen, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. pp. 1320-1332 1328, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Under review as a conference paper at ICLR 2023
Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE transactions on pattem analysis and machine intelligence, 42(3):651-663, 2018.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1406.0461, 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.0977 2021.
Suviseshamuthu Easter Selvan, Umberto Amato, Kyle A Gallivan, Chunhong Qi, Maria Francesca Carfora, Michele Larobina, and Bruno Alfano. Descent algorithmss on oblique manifold for source-adaptive ica contrast. IEEE Transactions on Neural Networks and Learning Systems, 23 (12):1930-1947, 2012.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. pp. 4548 4557, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Gido M Van de Ven and Andreas S Tolias. Generative replay with feedback connections general strategy for continual learning. arXiv preprint 1rXiv:1809.10633 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. pp. 193-193,00 2021.
Xian Wei, Yuanxiang Li, Hao Shen, Weidong Xiang, and Yi Lu Murphey. Joint learning sparsifying linear transformation for low-resolution image synthesis and recognition. Pattern Recognition, 66: 2017.
Yuxin Wu and Kaiming He, Group normalization. pp. 3-19, 2018.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. Advances in Neural Information Processing Systems, 31, 2018.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, (009:364-372,, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. pp. 3987-3995, 2017.
Junting Zhang. Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. ctass-incremental learning via deep model consolidation. pp. 1131-1140, 2020.
A APPENDIX
In this section, we give the implementation details about the experiments on Permuted MNIST and Permuted CIFAR10, help readers reproduce these experiments. Additionally, more ablation studies and experimental results are provided here to further support the conclusions and contributions.
A.1 IMPLEMENTATION DETAILS
The main hyperparameter settings listed in Table 2 and For the baselines, adopt the default settings provided their code to bring out the proper performance. For fair comprision, we use the uniform Batch size for all methods.
12

---

Under review as a conference paper at ICLR 2023
Table 2: Implementation details for the experiments Permuted MNIST.
Method
Optimizer
Momentum
Learning rate
Max epochs
Early stop
Batch size
OWM
SGD
0.9
2.0
100
GPM
SGD
0.9
0.01
5
10
ORTHOG-SUBSPACE
SGD
0.9
0.1
LeSP(low-coherenee)
SGD
0.1
15
10
LeSProrthogona)
SGD
0.1
15
Table 3: Implementation details for the experiments on Permuted CIFARIO.
Method
Optimizer
Momentum
Learning rate
Max epochs
Early stop
Batch size
OWM
SGD
0.9
0.02
25
64
GPM
SGD
0.9
0.01
200
64
ORTHOG-SUBSPACE
SGD
0.9
0.1
80
64
LeSP(Jow-coherenne
SGD
0.003
80
64
LeSP(orthoonna)
SGD
0.003
Permuted (last tasks)
Permuted CIFAR10 (last 20 tasks)
60
OWN
Average Accuracy(99)
880
Average Accuracy
LSSP Low cananrea LIS tatlegenas
40-
-6
Ro.
%6
10
Number of Task
Number of Task
Permuted MNIST (last 20 tasks)
Permuted CIFARID (last 20 tasks)
0.6
0.3
cortnggons
06D-06 9.855M2
6PM
GPM
0.5
LdP (atthggcnal)
O1THOS SUBSPAC2
Acem ASCL
forgetting
Forgetting
0.2
0.3
0.1
0.0
00
100
150
20
56
Number of Task
Number of Task
Figure 3: The average accuracy and forgetting of the last 20 tasks Permuted MNIST (left) and Permuted CIFARIO (right). The x-axis shows the number of tasks learned and y-axis represents the corresponding average accuracy and forgetting on the last 20 tasks.
A.2 ABLATION STUDIES AND ADDITIONAL RESULTS
Additional results on Permuted MNIST and Permuted CIFARI0 Readers may wonder whether our conclusion holds if we evaluate the average performance with more tasks (e.g., the average accuracy and forgetting on the last 20 tasks). As shown in Fig LcSP still outperforms all baselines with significant advantage. However, the phenomenon of learning capacity degradation in base- lines becomes more imperceptible, e.g., the average accuracy of OWM on Permuted CIFARIO is consistently low, rather than significantly decreasing. To further investigate the learning capacity degradation problem, we give the accuracy of baselines for the last task on Permuted MNIST and Permuted CIFAR10. As shown in Fig 7 all baselines, except LcSP, suffer from this problem with different degrees and show some decrease in accuracy compared to the initial (66.16% ~ 24.63% on Permuted MNIST and 24.8% 3.48% on Permuted CIFAR10). These results suggest that this
13

---

Under review as a conference paper at ICLR 2023
Permuted MNIST (last task)
Permuted CIFAR1O (last task]
65
90
6PM
Average Accuracy(%)
SKTMCG SUB37AC2
Acem-
50
Accuracyl
60-
CWV CMM
demico-suSRRE GUBSENEE
USS lor Brggnnn
40-
30-
30
20
150
16
24
56
Number
Number
Figure 4: The accuracy of the last task on Permuted MNIST (left) and Permuted CIFAR10 (right), respectively.
MNIST
95
Orthogonal Projector
90-
56
80
48-
Accuracy
70-
32
16
50-
Ld5P
40
Rank Projector
Number Task
(b)
Figure 5: (a). gives the ablation study for different rank of projectors MNIST. Fig.(b) shows the average rank of all projectors when the number tasks increasing. Here, the dimension m of features is 64.
problem is the critical factor that results in degrading the performance of GOP-based methods in the case of a large number of tasks.
Ablation studies and experiments for rank and scale constraints Further ablation studies and experiments are conducted to investigate the effects of the rank and scale constraints on the expressive power (plasticity) and stability of DNNs.
The result in Fig ((a) suggests that projecting features or gradicnts into subspaces with low dimensions (e.g., lower than Fig 5(a) will decrease the expressive power of DNN. GOP methods and LeSP rely on the projectors to project the features or gradients (or both) into a d-dimensional subspace, which can also be considered as form dimension reduction. The dimension reduction is motivated by consensus in the high-dimensional data analysis community, i.e., the data can be summarized in low-dimensional space embedded in high-dimensional space, such
nonlinear manifold
Levina & Bickel (004)). The dimension of this fow-dimensions space is also known as the intrinsic dimension CarrrirraPPrrpinnn 1997). the is too small. e.g., d < D, important data features will be "collapsed" onto the same dimension. From the perspective of training a DNN, if the gradients are projected into a subspace with a dimension lower than D, the DNN cannot activate sufficient parameters to learn the presentation of this task.
Due to the unknown number of tasks to be learned, the limited dimensions of the features, and the strict orthogonality constraint between projectors, the orthogonal projector cannot be constrained to a fixed-rank manifold. As shown in Fig 5(b) the rank of the orthogonal projector decreases as the number of tasks increases. Therefore, methods using orthogonal projectors usually ignore the intrinsic dimension of data (features or gradients) and finally lead to suffering from the learning capacity degradation problem. In contrast to GOP methods, LcSP relaxes the orthogonality constraint
14

---

Under review as a conference paper at ICLR 2023
Permuted MNIST tasks)
Permuted MNNIST
0.5
90
LcSP
LcSP
Average Accuracy
70-
Forgeting
0.4
0.2
10
0.0
1/100
1/60
20
100
1/100
1/60
1/20
20
100
Length Projector's Column Vector
Length Projector's Vector
Figure 6: The average accuracy (left) and forgetting (right) of the last 20 tasks with different scales of columns on Permuted MNIST.
and meets the rank constraint by optimizing intra-task coherence on the Oblique manifold and thus does not suffer from this problem.
Finally, Fig b gives an ablation study for scale constraints on the projector's columns. In Fig 6 when the columns of the projector have unit length, the average accuracy reaches the highest. Result gets worse when the length of the projector's columns too small too large.
Table 4: Efficiency analysis.
OWM
GPM
Adam-NSCL
ORTIOO-SUBSPACEE
LeSP
FLOPs
26.676.256
23,202,880
558.538,752
556,
708,864
528,965,376
Time(s)
47.614
5.174
47.685
87.623
25.496
Epochs
25
200
80
80
80
Mcan inference (nferenncettmo((ms)
1,249
1.152
10,089
3.957
5.893
Based on the RTX3060 12G, we tested the efficiency of LcSP and compared methods on Permuted CIFARTO (which contains 64 tasks, and each image's 's size is 32 x 32), using AlexNet for OWM and GPM, and resnet1 8 for the other methods. We provide 4 metrics to assess the efficiency of the method: floating point operations FLOPs, time spent per epoch during training Time (s), the number of epochs needed to train to convergence Epochs, and the average time spent on inference Mean inference. time (ms). The detailed results are listed in Table 4
Permuted CIFAR10 (last: 5 tasks)
Permuted CIFAR10 (last tasks)
LESP-GR GK
(GP BN
LeSP
LGP SM
L-5P-90
Average Accuracy(M)
40
Forgetting
30
32
40
48
56
8
16
24
32
48
56
64
Number of Task
Number of Task
Figure 7: The average accuracy (left) and forgetting (right) of the last 5 tasks on Permuted CIFARIO. All methods use the low-coherence projector.
A.3 METHOD ANALYSIS
The mechanism of low-coherence learning and the difference with orthogonal learning Recall
the Lemma described in Section 4.3. Lemma 1. Assume that I is fed the data of task T (g < (), then f can effectively overcomes catastrophic forgetting if
2yg RS 2
Vg S
15

---

Under review as a conference paper at ICLR 2023
Here, 2/, denotes the output feature of I-th layer when data from task Ty are fed into the DNN that has been trained on task T.. Let us consider the case where there are only two tasks, i... 9 = t 1. By appling the LcSP's projection strategy, have
z/,e z x AW:P; 2/.4 + P
Here, Ih. zyge and P. denote the gradient, input data, and projection matrix (which is symmetric matrix) of task Tr in I-th layer, respectively. The term ar P: Pj can be thought of as the forgetting, which changes the output of DNN to previous data. Since we cannot change z/y, and g'. the key to reduce " is to minimize P.P. If P/ is orthogonal to P the forgetting is equal to O. Let us consider reducing forgetting when P// cannot be orthogonal to P (e.g., the number of tasks large and the dimensionality of the space is limited).
optimize P Pa directly, we may get P. = 0, or the column scale of P/ is too small. To find useful P LcSP constrains the length of the column vector be 1, i... to find P. on the Oblique manifold (OM). To reduce forgetting, we optimize the inter-task coherence (a( P P)) on OM so that the maximum entry of P is minimized.
In the worst case, we may obtain P consisting of only one column vector (i.e., P with rank 1). To avoid this situation, optimize the intra-task coherence μ ((PP) along with inter-task coherence, forcing the projector to satisfy the rank constraint in order to maintain the learning capacity of DNN (as shown in Fig. ((a), the rank of projector affects the plasticity DNN).
In general, low-coherence projection, which can be seen as a relaxed orthogonal constraint, aims balance plasticity and stability better, motivated by our observation that orthogonal projections reduce the plasticity of the model and make the DNN unable to adapt to new environments.
16


Reviews:
[{"review_id": "xg1ZsfMzqPp", "reviewer": "Reviewer_8QtN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a continual learning method by constructing low-coherence subspace projector for each new task. Given a new task, the projector matrix is constructed by minimizing the coherence to the previous task projectors and within the current projector to be optimized, in the oblique manifold. The proposed approach is evaluated for task incremental learning tasks, especially when the number of tasks is large (e.g., 64, 150). The results show improvements over the baseline methods, including, EWC, OWM, GPM, etc. ", "summary_of_the_review": "My major concerns on this work are some unclear details, e.g., the question (1) in weakness, and insufficient comparisons in experiments. Moreover, there are some unclear points in the experiment results, especially on the effect of normalization techniques in the results.  ", "strength_and_weaknesses": "Strength:\n\n(1) The proposed approach tackles the challenging continual learning task, and proposed a gradient projection method, based on low-coherence, instead of orthogonality. \n\n(2) The experiments on Permuted MNIST, Rotated MNIST, Split CIFAR100 and Split miniImageNet show the improved results compared with some baseline methods, e.g., EWC, GPM, OWM, etc. \n\nWeakness:\n\n(1)  The proposed method is based on construction of the projection matrix by minimizing the coherence of the projection matrix with respect to the projection matrices of previous task, and also within the current task projection matrix.  Based on the sect. 4.1, it is unclear how to generate the projection matrix for the first task. Moreover, for the projection matrix formulation in eq. (6), it seems that the current O_t is purely determined based on the previous task O_i (i = 1, ..., t-1). It seems that the task data (or their features) are not utilized in the determination of O_t. How to guarantee that that learned O_t is optimal without seeing the task data or their features?\n\n(2) According to the results in Table I, the LcSP achieves significantly lower results than the compared methods, on Split CIFAR100, Split miniImageNet. But LcSP-GN and LcSP-BN are better than the compared methods. How to explain this results? It seems that the learning BN for each task is essential to the good results on these two tasks. Are these compared methods also use the normalization technique as the LcSP-GN and LcSP-BN? Moreover, the comparisons of LcSP, LcSP-GN and LcSP-BN should be also included for other datasets.\n\n(3) The comparisons are insufficient, e.g., the Adam-NSCL is also an typical continual learning method based on orthogonal projection, which should be also compared. The compared methods in Table 1 for different datasets are not consistent, for example, GPM and OWM are not compared for the two versions of MNIST dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "puD0BdsvFj", "reviewer": "Reviewer_em94", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn new tasks in low-coherence subspaces rather than orthogonal subspaces to mitigate catastrophic forgetting in continual learning. The authors believe that Gradient Orthogonal Projection (GOP) (though helps battle catastrophic forgetting) causes  learning capacity degradation and its learning capacity \"is gradually degraded as the number of tasks increases and eventually becomes unlearnable\". \n\nTo learn new tasks in low-coherence subspaces, the authors propose a unified cost function to seek projectors and develop a gradient descent algorithm (on the Oblique manifold) by jointly minimizing inter-task coherence and intra-task coherence. The original mutual coherence (often used in compressive sensing) of a matrix is the maximum absolute value of the cross-correlations between the matrix's columns. The authors extend the concept of coherence to two matrices to capture the maximum absolute value of the cross-correlations for the columns from two matrices. To deal with catastrophic forgetting heightened by batch normalisation (BN), the authors use two strategies: (1) learning specific BN for each task, or (2) using group normalisation (GN) instead of BN. \n\nThe authors did experiments on Permuted MNIST, Rotated MNIST, Split CIFAR100 and Split miniImageNet. For 20 tasks (a typically setting), the proposed method achieved best accuracies on Permuted MNIST and Rotated MNIST, and 3nd best forgetting on Permuted MNIST and 2rd best forgetting on Rotated MNIST. The proposed method was reported to achieve best accuracies and zero forgetting (surprising to see zero forgetting) on Split CIFAR100 and Split miniImageNet. \n\nTo show other methods would struggle with increasing tasks, the authors did experiments on 64 tasks and 150 tasks where the proposed method shows a clear advantage on accuracies, but not necessarily on forgetting (the proposed method is reported to achieve  best accuracies on both Permuted MNIST and Permuted CIFAR10, and 2nd best forgetting on Permuted MNIST and 3rd best forgetting on Permuted CIFAR10). ", "summary_of_the_review": "At this stage, I am happy for this paper to be accepted. ", "strength_and_weaknesses": "Strength: \n1) learning in low-coherence subspaces can be an alternative to GOP. \n2) the experimental results are impressive.\n\nWeakness:\n1) The authors believe that GOP causes learning capacity degradation and its learning capacity \"is gradually degraded as the number of tasks increases and eventually becomes unlearnable\". This should be more thoroughly analysed and supported by ablation study (curves on 64 tasks and 150 tasks with other methods are not sufficient to serve this purpose). \n2) The proposed method's forgetting performance on different datasets and settings vary widely from 0 forgetting to the state of being beaten by one or two competitors (e.g. see Fig. 1 (b) (d)).\n3) Coherence metric can be restrictive. Babel function which extends cross-correlation between pairs of columns to the cross-correlation from one column to a set of other columns might be a better alternative.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BB5Ms4bqES", "reviewer": "Reviewer_FMJV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Regularizing the subspace of the network representation has been used in continual learning to alleviate forgetting. To mitigate learning capacity degradation caused by orthogonal projection/regularization, this paper proposes to learn in low-coherence subspace. It learns task-specific projectors to represent each task in subspace with low coherence to others. To learn in low-coherence subspace, the projectors are learned on the Oblique manifold. The proposed method performs better than the compared methods (especially the orthogonal subspace learning method) on the task-incremental setting (with known task identifiers). ", "summary_of_the_review": "The whole paper is with a clear motivation and representation overall. The novelty and the significance of the contribution can be improved. Although the experiments can prove the method can work well and better than the orthogonal subspace learning method, the experiments are limited to a very specific and even simple setting in CL, which influences the generality and significance. \n\nAfter checking all the reviews and responses, I would like to keep my original score - below the acceptance line. \nThe concept of using low-coherence subspace projection in continual learning is interesting. However, the contribution is relatively limited, considering the relationship with the methods working on orthogonal subspace and the restricted task-incremental setting (i.e., the requirement on task id). The technical details and explanations of the experimental results (in Table 1) are unclear and confusing, as indicated by 8QtN. It is blurry what is the main role of the low-coherence projection on performance improvement, compared with the normalization trick. ", "strength_and_weaknesses": "Strength\n- This paper proposes to learn the projections in the low-coherence subspace, instead of the orthogonal subspace, which lets the model capacity be used better, while reducing the parameter interference, with less capacity degradation. This is well motivated.\n\n- The experiments prove the claim about the benefits of learning in the low-coherence subspace, compared to learning with the orthogonal subspace. \n\n- The paper is generally written clearly, with some points that can be improved, as discussed in the following. \n\nWeakness\n- The work can be seen as an extension of the orthogonal subspace based continual learning paper (Chaudhry et al. 2020). The main difference is the projectors are learned on the Oblique manifold instead of the Riemannian manifold, which enables learning in the low-coherence subspace with some overlaps between the tasks. It brings limited novelty for continual learning. \n- Intuitively, the low-coherence subspace learning can be seen as a “relaxation” of the orthogonal subspace learning. The mechanism of low-coherence learning and the difference with orthogonal learning is not clearly analyzed and discussed. \n-- Specifically, some questions can be asked and discussed. For example, comparing with the orthogonal subspace learning, how the method alleviates the forgetting issue while not totally isolating the subspaces. \n- The proposed method is limited to only the task-incremental setting where the task identifiers are required in both training and testing phases, as shown on page 3 and experiments. It further influences the significance of the work, considering the insignificant technical novelty, although this is with the same setting as (Chaudhry et al. 2020). The authors may further \n- Some detailed questions:\n-- What is the gap between the implementation and the analysis on page 6? \n-- How is the efficiency of the proposed method? How is the running time compared to other methods? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject