Iterative Relaxing Gradient Projection for Continual Learning
Authors: Zeyuan Yang
Zonghan Yang
Peng Li
Yang Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: continual learning
gradient projection methods

Abstract: A critical capability for intelligent systems is to continually learn given a sequence of tasks. An ideal continual learner should be able to avoid catastrophic forgetting and effectively leverage past learned experiences to master new knowledge. Among different continual learning algorithms, gradient projection approaches impose hard constraints on the optimization space for new tasks to minimize task interference, yet hinder forward knowledge transfer at the same time. Recent methods use expansion-based techniques to relax the constraints, but a growing network can be computationally expensive. Therefore, it remains a challenge whether we can improve forward knowledge transfer for gradient projection approaches \textit{using a fixed network architecture}. In this work, we propose the Iterative Relaxing Gradient Projection (IRGP) framework. The basic idea is to iteratively search for the parameter subspaces most related to the current task and relax these parameters, then reuse the frozen spaces to facilitate forward knowledge transfer while consolidating previous knowledge. Our framework requires neither memory buffers nor extra parameters. Extensive experiments have demonstrated the superiority of our framework over several strong baselines. We also provide theoretical guarantees for our iterative relaxing strategies. 

Under review as a conference paper at ICLR 2023
ITERATIVE RELAXING GRADIENT PROJECTION FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A critical capability for intelligent systems is to continually learn given sequence of tasks. An ideal continual learner should be able to avoid catastrophic forget- ting and effectively leverage past learned experiences to master new knowledge. Among different continual learning algorithms, gradient projection approaches impose hard constraints on the optimization space for new tasks minimize task interference, yet hinder forward knowledge transfer at the same time. Recent methods expansion- -aased techniques to relax the constraints, but growing network can be computationally expensive. Therefore, it remains a challenge whether
improve forward knowledge transfer for gradient projection ap- proaches using a fixed network architecture. In this work, we propose the Iterative Relaxing Gradient Projection (IRGP) framework. The basic idea is iteratively search for the parameter subspaces most related to the current task and relax these parameters, then reuse the frozen spaces facilitate forward knowledge transfer while consolidating previous knowledge. Our framework requires neither memory buffers nor extra parameters. Extensive experiments have demonstrated the supe- riority of our framework over several strong baselines. We also provide theoretical guarantees for our iterative relaxing strategies.
INTRODUCTION
A critical capability for intelligence systems continually learn given sequence of tasks (Thrun & Mitchell,, 1995; McCloskey & Cohen 1989). Unlike human beings, vanilla neural networks straightforwardly update parameters regarding current data distribution when learning new tasks, suffering from catastrophic forgetting (McCloskey & Cohen 1989; Ratcliff 1990, Kirkpatrick et al.) 2017). As a result, continual learning is gaining increasing attention in recent years (Kurle et al, 2019 Ehret al. 2020 Ramesh & Chaudhari] 2021: Liu & Liu 2022 Teng et al. 2022). An ideal continual learner is expected to not only avoid catastrophic forgetting but also facilitating forward knowledge transfer (Lopez-Pat Paz & Ranzato (017). which is to leverage past learning experiences to
master new knowledge efficiently and effectively (Parisi et al. 2019 Finn et al. 2019).
Several types of methods have been proposed for continual learning. Replay-based methods (Lopez- Paz & Ranzato 2017: (Shin et al. (017) alleviate catastrophic forgetting by storing some old samples the memory as they are inaccessible when new tasks come, while expansion-basee methods (Rusu et al. 2016 Yoon et al. 2017]] 2019) expand the model structure to accommodate incoming knowl- edge. However, these methods require either extra memory buffers (Parisi et al. 2019) growing network architecture as new tasks continually arrive (Kong et al.) 2022), which always results in expensive computation costs (De Lange al.) (021). In order to maintain fixed network capac- ity, regullrizatioon-base methods (Kirkpatrick et al.) 2017: Zenke et al., 2017 Alljundi et al. 2018) penalize the transformation of parameters regarding the corresponding plasticity via regularization terms. While these regularization terms are applied to individual neurons, recent gradient projection methods (Zeng et al., 2019 (Saha et al., 2021: Wang et al., 2022] modify the gradients in the feature space by constraining the directions of gradient update, which achieves outstanding performance.
However, although gradient projection methods effectively mitigate forgetting within a fixed net- work capacity (Zeng al.) 2019). the capability of learning new tasks is hindered by the limited opti- mization space, resulting in insufficient forward knowledge transfer. In other words, constraining the directions of gradient update fails the plasticity in the Hability-plasticee dilemma (French) 1997..

---

Under review as a conference paper at ICLR 2023
Figure I: Illustration of our proposed IRGP method and two baselines: GPM and TRGP. Blocks painted in different colors denote the parameters optimized after different tasks. We denote the relaxing subspace within the frozen space as the painted stripes in our IRGP pipeline.
GPM
TRGP
IRGP
After training Task I
Aftr training Task
After training on Task
After training on Task 2
After training on Taak 2
After training on Task
Task
After training on Task 3
After training on Task
Task
Trust Region Gradient Projection (Lin et al. 2022) tackles this problem by expanding the selected
subspace of old tasks as trust regions with scaled weight projection, similar to other expansion-basee
methods (Yoon et al. 2019). In spite of substantial improvement, these methods are computationally expensive as a result of growing network architecture (Wang et al. 2021). Therefore, insufficient forward knowledge transfer remains key challenge for gradient projection methods.
To address this challenge, we propose the Iterative Relaxing Gradient Projection (IRGP) framework to facilitate forward knowledge transfer within fixed network capacity. We design a simple yet effective strategy to find the critical subspace within the frozen space. During the training phase, we iteratively reuse the parameters within the selected subspace. Instead of strictly freezing those pa- rameters, our method explores a larger optimization space, which allows better forward knowledge transfer and thus achieves better performance on new tasks. The procedure of our approach il- Extensive experiments on various continual learning benchmarks demonstrate
lustrated in Figure [ that our IRGP framework promotes forward knowledge transfer and achieves better classification performance compared with related state-of oftte-of-the-are approaches. Moreover, our framework performs can also be extended as an expansion-baa -based methods by storing the parameters of the selected relax- ing subspace, universally surpassing TRGP (Lin et al. (022) and other expansion-based approaches. We also provide theoretical proof to guarantee the effeiency of our relaxing strategy.
2 RELATED WORK
In this section, we review the representative approaches for continual learning and briefly analyze their differences from our method. Conceptually, these approaches can be roughly divided into the following four categories.
These methods maintain a complementary memory for old samples, which replayed during learning novel tasks. GEM (Lopez-Paz & Ranzato 2017) constrains gradients concerning previous samples and (hhaudhyy et al. (2018) further propose to estimate with random samples to accelerate. While past samples are commonly not accessible in the real world, auxiliary deep generative models are thus deployed to synthesize pseudo data (Chenshen et al.) 2018; Cong let al. (020). Recent approaches (PourKeshavarzi et al. 202T: Choi et al.) 2021 leverage a single model for both classification and pseudo data generation. However, including extra data into the current task introduces excessive training time (De Lange et al., (02T), especially on long task sequence. Our approach requires no previous data, in other words, is replay-free method.
Replay-based methods:
Expansion- based methods dynamically allocate new parameters or modules to learn new tasks. |usu et al. (2016) propose to incrementally introduce additional sub- networks with a fixed capacity. DEN (Yoon et al. 2017) selectively retrains the frozen model and expands only with necessary neurons. Moreover, Li et al. (2019) perform an explicit network archi- tecture search to decide where to expand. APD YYonn et al. 2019) further decomposes the network and utilizes sparse task-specific parameters. However, these methods face capacity explosion in- evitably after learning a long sequence of tasks. In contrast, our approach maintains a fixed network architecture to avoid expensive model growth.
Expansion-based methods:
2

---

Under review as a conference paper at ICLR 2023
Regulariization-assee methods: Methods in this category introduce extra regularization terms to the objective function to penalize the modification of parameters. EWC (Kirkpatrick et al. 2017) first proposes to constrain the change based on the importance weight approximated by Fisher In- formation Matrix. MAS (Aljundi et al. 2018) measures the importance of the sensitivity of model outputs under an unsupervised setting. Other methods, also called parameter-isolation methods, defy catastrophic forgetting via freezing the gradient updates of particular parameters (De Lange] ctt al., 2021). PackNet (Mallya & Lazebnik] 2018) iteratively prunes and allocates parameters subset to corresponding tasks, whereas HAT (Serra et al. 2018) learns task-based hard attention to identify important parameters. Instead of restricting individual parameters with estimated importance, the main idea of our approach is constraining the direction of gradients.
Gradient projection methods: Gradient projection methods directly constrain the gradients to over- come catastrophic forgetting, and our approach belongs to this category. [Mehta et al. (202T) implic- itly expands the model with respect to the frozen space and GEM (Lopez-Paz & Ranzato 2017)
utilizes complementary memory restrict the update. While our approach requires neither storing old samples nor expanding the network. OWM (Zeng et al. 2019) first proposed to modify the gra- dients upon projector matrices. OGD (Farajtabar al. (020)) keeps the gradients orthogonal to the space spanned by previous gradients, whereas GPM (Saha et al. (021] computes the frozen space based old data. NCL. (Kao et al. [202T) combines the idea of gradient projection and Bayesian weight regularization to mitigate catastrophic forgetting. In spite of minimizing backward inter- ference, these approaches suffer poor forward knowledge transfer and lack plasticity (Kong et al.) 2022). TRGP (Lin et al. (002) expands the model with trust regions based on task relationship to achieve better performance on new tasks. In contrast, we focus on facilitating forward knowledge transfer within a fixed capacity network by iteratively relaxing frozen regions with constraints.
3 ITERATIVE RELAXING GRADIENT PROJECTION
3.1 PRELIMINARIES
In a continual learning setting, we consider tasks arriving as a sequence. The datasets are denoted as ((+)_ = [xr(0) Mi ii=,, where N₂ is the number of samples. When learning the current task, the datasets of old tasks are inaccessible. We use an L-layer neural network with fixed capacity, and parameters defined as W = {W') where W denotes the parameters in the (-th layer. The model is optimized by minimizing the objective function
(1) (t) N.
N. (T) and L, is the loss function for task t.
1
L(W, = N 1=1
Σ C.(s(a W))
(1)
Gradient projection methods mitigate catastrophic forgetting by only updating the model in the orthogonal direction to frozen spaces. [Saha et al. (202T) proposed to compute the frozen spaces based on the inputs of each layer. For task t, the frozen gradient spaces for the first t tasks are denoted as U {U/- 1 where U. is the frozen space of layer for task t. During the training phase, for each layer gradients of are constrained to be orthogonal UL_ Particularly, assuming B¹ [u/ 1,1 1/v the total N basis for U gradients Ji are modified as:
gi = g. Projer (9)) = gi (B,_)
(2)
After getting the learned model W; [W for each layer I, we record the intermediate representation hig of the j-th input and stack them to obtain the representation matrix H [M] Mr,,, Then compress the representation matrices by performing Singular Value Decom- position SVD(H)) U UH (V!)). Given the threshold </a-- select the first k vectors in H based on the following criteria:
0 All 2 4 1
(3)
to construct the significant representation space N2 spant n. Frobenius norm here. The significant representation spaces, considered as the frozen spaces for current task t, are then merged into the whole frozen gradient spaces for the first t tasks:
U }.}, where || I1 denotes
u { = {UL_ U R
(4)
3

---

Under review as a conference paper at ICLR 2023
Although freezing gradients update significantly mitigates catastrophic forgetting, limited optimiza- tion space hinders the forward knowledge transfer, compromising the performance of new tasks. TRGP (Lin et al. 2022) tackles this problem by selecting old tasks relevant to the current task and expanding the corresponding frozen spaces as the trust regions. The scaled weight projection is fur- ther designed for memory-efficiet updating and storing the parameters within the trust regions by scaling the basis, instead of directly changing the parameters. Considering that task i is selected as the trust region, the scaled weight projection shown as:
S Profy: (o)) - (B)
(5)
where S denotes the scale matrix. The parameters in the trust regions are retrained with the scaled weight projection and the learnt scale matrices are stored in the memory for the inference phase. Particularly, during the forward transfer, the parameters modified with the scale matrices
W (roj (W)) Projyy (W))
W - Projo (W?) + Projee St (W))
where (.) denotes the orthogonal complemented subspace. However, as tasks come, increasing extra parameters are introduced by storing the scaling matrices. Our experiments demonstrate that TRGP requires around 5000% amount of the parameters regarding the network architecture after learning 20 tasks on MinilmageNet, see Figure 3-(c). Therefore, we propose our Iterative Relaxing Gradient Projection framework to facilitate forward knowledge transfer while maintaining fixed network capacity by wisely reusing parameters within the frozen space.
3.2 RELAXING SUBSPACE SEARCHING
We first design a searching strategy to determine which part of the frozen space relax based on the estimated importance characterized by the angle from the representation space spanned by current gradients g/. The angle between given space and a vector is defined in definition 3.1
Definition 3.1. (Angle between vector and space) We denote the angle between two inputs as ((-)
and the inner product between two vectors as {-). The angle between a vector u E R" and a space
Unxe C RD is defined as the minimum angle between the given vector and any unit vector u E U:
(v, u)
(v. U) = arccos max- NEU |Pll
(7)
Moreover, given the threshold % we define that vector dl is relaxable when:
(d, R s/
(8)
where Rg is constructed by compressing of with Equation
3 For task f, we aim find the relax- ing subspace V S U spanned all by relaxable vectors from U Particularly, we implement with the modulus of the projection, namely:
min |Prijin (u)lEF 2 Gluleeee VEV;
(9)
max
||Proj R (w)llr < |||
WEU (-1
where ç/ = cos of is the threshold and U = V denotes the complemented subspace of V
with respect to U/-1- Above criterion guarantees max (u, R(,,) S 7/ and min (v, R2,,) > 7 vEU!i.
WEV?
However, it is hard to construct V/' directly from U/_,- Therefore, we propose simple yet efficient strategy to find the relaxing subspaces. With V initiated as 0. we select the closest vector R
within Uhe by arg min (dd, Rgy.). The selected vector d is then appended into Vt as basis if it deut..
satisfics criterion (8) We repeat this procedure until no relaxable vector left to get the target V!! The podu of our searching strategy is provided in Algorithm
Considering the scope of our procedure, we further provide theoretical analysis on the upper bound of the dimension of the selected subspace V, which is also the number of iterations. Here introduce Lemma 3.2 and Theorem 3.3 which guarantee that the dimension of V is no more than of the representation matrix. Detailed proof is provided in Appendix A.T A.2
4

---

Under review as a conference paper at ICLR 2023
Algorithm Relaxing Subspace Searching
Input: gradient {oi)t=1. frozen subspace {U1-1}/= and thresholds (ch 76
Output: relaxing subspace {V:
I: for E 1,.... L do
2: Construct the significant representation space Ry from gradients of by Equation
3:
V (00 -0
4: repeat
5:
d arg min Oni (t) de
6:
if (d, Rg () S 1/ then
7:
V + V/ Ud
U/_ UL-1\V end if
8:
9:
until (d, R) Th
11:
end for
Lemma 3.2. Denote the relaxed subspace as V = spanfoi, u2,..., u.. where uN is the last base included in V. Given representation subspace U, Ve E V, we have (v, ( U).
Theorem 3.3. Denote kp as the dimension of the representation subspace and ką as the dimension of the relaxed subspace, the upper bound of ką is kpp regardless of the frozen subspace.
Moreover, according to Theorem 3.4 we figure that our strategy guarantees to find the maximum space within the whole solution set satisfying criterion (9), which further substantiates the efficiency of our searching strategy. We also include the corresponding proof in Appendix A.3
Theorem 3.4. The relaxed subspace obtained by Algorithm 7 takes up the maximum subspace of the whole solution set.
To further validate our searching strategy, we propose IRGP-Exp. a modified version of our pro- posed IRGP, directly storing the parameters in the relaxed subspaces. For task t, retrieve the cor- responding relaxed subspaces [V: and the scale matrices {S during the inference phase similar to TRGP. The modified parameters W used for inference on task is:
W W Projv; (W") + Proj (W")
(10)
where W denotes the parameters of layer 1 of current network. Replacing the parameters in the relaxed subspaces with the parameters optimized in task the model achieves better performance.
3.3 ITERATIVE MODIFYING THE SCALE MATRIX WITH CONSTRAINTS
After getting the relaxed subspaces, we want to retrain inside parameters while consolidating previ- ous knowledge, facilitate forward knowledge transfer within fixed network capacity. One direct way is to fine-tune the parameters with regularization such as EWC (Kirkpatrick et al. 2017). How- ever, regularization terms are designed for explicit parameters, which are not applicable for implicit subspace in our framework. Therefore, we introduce the scaled weight projection (Lin et al. 2022) to modify explicit parameters instead. With sealed weight projection, we fine-tune the parameters within D by adding regularization term scaling matrices S, (S instead of direct on target parameters. Specifically, the objective function of task is:
C. L(V, D(2) + 3
(S
(11)
where 1.) denotes the identity matrix with the size of the rank of the input matrix and 3 is the weight of the regularization term for layer During back propagation, gradients within frozen space U3_, are climinated and parameters within V are modified by St. Generally, in Iterative Relaxing Gradient Projection framework, we adopt our searching strategy to determine the relaxed subspaces and modify those parameters with constraints on the scaling matrices.
However, during the training phase, the direction of gradients shifts sharply and frequently due to the steep learning scope of deep neural networks. Diverse subspaces would be seleeted in different training phases. Therefore, we iteratively execute Algorithm [ during training until no extra sub- space required. Particularly, for each task, model is optimized for limited epochs first. Then
5

---

Under review as a conference paper at ICLR 2023
search for the target relaxing subspace and examine whether there exists a new subspace within the remaining frozen space. If extra frozen subspace is released, the scale of the scaling matrix re- quires to be modified, while TRGP maintains fixed-size scaling matrix throughout training. Thus, to accommodate the increasing relaxing subspace, we propose to expand the scaling matrix with the identity matrix of the corresponding size
-8, (Shood
S S..new
(12)
(V. (V.l-eww)
where V.l.ee denotes the newly included relaxing subspace. If there is no extra relaxing subspace, optimize our model thoroughly current task. After training, the parameters within V are fur- ther consolidated by Equation (6) and the scaling matrices are emptied to be identical matrices. The pseudo-code of our Iterative Relaxing Gradient Projection framework provided in Appendix D
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets: We evaluate framework five datasets. Following Saha et al. (2021). we conduct
periments on CIFAR R-PAR-1001 Split (Krizhevsky & Hinton 2009), MinilmageNeet [Vinyals et al. 2016),
Permuted MNIST (MNIST) (Kirkpatrick al. (017) and CIFAR-100 Sup Yooon et al.
2019
Moreover, Serra et al. (2018) first propose Mixture, consists of CIFAR-10 (Krizhevsky & Hinton
2009,, MNIST CLeST al.] 1998). CIFAR- (Krizhevsky & Hinton 2009), SVHN (Netzer al.,
2011
MNIST (Xiao et al. 2017). TraffieSigns (Stallkamp et al. (011T) FaceScrub (Ng &
Winkler 2014) and NotMNIST (Bulatov 2011). Here we evaluate our framework on Mixture with
seven tasks as a sequence except TraffiSiinn Details and statisties of the datasets can be found in
Appendix B.T Moreover, we include the details of network architectures Appendix [..2]
We compare our approach with competitive and well-established approaches maintain- ing a fixed network capacity following Saha et al. (2021). We adopt ER Res (Chaudhry et al. 2019) and A-GEM (Chaudhry et al.) 2018) representative replay-based methods: the memory buffer size for PMNIST,CITAR-TO0, Split, MinilmageNet, and Mixture are 1000, 2000, 500 and 3000, respec- tively. For gradient projection approaches, we consider OWM (Zeng et al. 2019) and GPM (Saha
Baselines:
ct al. 2021). For regularization approaches, we compare against EWC (Kirkpatrick et al. 2017 and
state-of-the-arr HAT (Serra et al. 2018). We also include the "multitask" baseline jointly training all tasks in a single network, which is always considered as an upper bound for continual learning. Other implementation details listed in Appendix [B.3 We exclude expansion-based methods in the main experiments as they use continually growing architecture, which is out of the scope of our work.
Metries: We first employ two standard evaluation metries: Average Accuracy (ACC) (Mirzadeh et al. 2020) and Backward Transfer (BWT) (Lopez-Paz & Ranzato 2017). Denote As,j as the test accuracy of task j after learning task i. ACC is the average test accuracy evaluated after learning all tasks, defined as ACC = - Ar, BWT is the average accuracy decrease after learning fol- lowing tasks, defined BWT T (AT;
To evaluate forward knowledge transfer, further introduce Forward Transfer (FWT) (Lopez-Paz & Ranzato 2017) and Qnew (Kemker let al. 2018). FWT reflects the influence of the observed tasks on new tasks in a zero-shot manner, while Qnew indicates the capability of acquiring new tasks. The detailed definitions are provided in Appendix(B.l In this paper, we mainly focus on Inew among the three metrics and results on FWT are provided as well. Generally, the larger ACC, the better the approach. Forward and backward knowledge transfer evaluate the capability of learning and memorizing respectively.
4.2 MAIN RESULTS
We show the comparative results on four benchmarks in Table I The experiments on Mixture are implemented by us, while other results are reported from (Saha et al. [202T). We run each experiment five times and report the mean results. We include implementation details in Appendix B.3 and detailed results including forward transfer can be found in Appendix C.2 As shown in Table our approach obtain the best accuracy with comparable forgetting across all datasets.
Wee fail to access the TrafficSigns datasets as the links provided in (Stalkkamp al. 2011
Serra et al.
2018 Saha al. (02T) all expired

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of average accuracy and forgetting tested after learning all tasks. Multitask is under non-incremental setting. All results reported are averaged over 5 runs.
CIFAR-100 Split
MinilmageNet
PMNIST
Mixture
Method
ACC (%)
BWT (%)
ACC (%)
BWT (%)
ACC (%)
BWT (%)
ACC (%)
BWT (%)
Multitask
69.46
0.62
96.70 0.02
OWM
50,94 +: 0.60
-30+1
90.71 0.11
-1
OOM
EWC
68.80 :: 0.88
-241
52.01
= 2.53
-
89.97 = 0.57
411
69.62 = 2.69
-6
HAT
72,06 0.50
0
59.78
0.57
-3:0
77.54 = 0.18
-1
A A-GEM
63,98 1.22
15:+2 2
57.24
0.72
-12
83,56
0.16
114+11
59.86
1.01
+
ER_Res
71.73 :: 0.63
-6:1
58.94
0.85
-741
87.24 + 0.53
-11:1
75.07
= 0.55
-12
GPM
60.41
0.61
93.91 0.16
-3
-5:00
Ours (IRGP)
73.52 1: 0.45
-1::0
61.26
1.68
241
94.20
0.11
2
77.91
0.45
440
Figure 2: Results on CIFAR-100 Split setting: (a) averaged accuracy after learning each task; (b)
accuracy evolution of randomly selected task; (c) accuracy tested on task i after learning task i.
(c)
IRGP
75.0
78
URGP
Test Accuracy (%)
73.5
Tost Accuracy (%)
74.3
Test Accuracy (%)
76
74.0
74
72.5
73.5
72.0
73.0
72.0
68
71.0
Task
Task ID
Task ID
Compared with replay-based methods A-GEM and ER_Res, IRGP achieves least around 2% higher ACC with less forgetting. For regularization based methods, IRGP significantly dominates EWC across all benchmarks and outperforms HAT on MinilmageNet and PMNIST. Although HAT obtains less forgetting on the other two datasets, IRGP gains 1 better ACC on average. For gradient projection methods, we observe that IRGP achieves around 1% higher ACC on CIFAR-100 Split and MinilmageNet than GPM with comparable forgetting. On PMNIST and Mixture, IRGP improves the accuracy with less forgetting, reducing 1% BWT than GPM. The averaged accuracy after learn- ing each task on CIFAR- 100 Split exhibited in Figure 2-(a) further validates that IRGP universally outperforms GPM. We include the detailed results on other benchmarks in Appendix C.I
Moreover, compare the accuracy evolution of specific tasks during sequential tasks with GPM, which achieves the highest accuracy among selected baselines according to Table []] Here we show the results of the second task CIFAR- 100 Split in Figure 2-(b). We further present the results of three randomly selected tasks on all benchmarks in Appendix C.3 We notice that IRGP achieves better accuracy right after learning new task, in other words, gains better Pnew- which is also the purpose of our relaxing strategy. Without forward knowledge transfer, approaches may have limited performance even with less forgetting (Lopez-Paz & Ranzato 2017).
Thus, we observe the accuracy tested after learning each task. As shown in Figure 2-(c), our ap- proach achieves 2.7% better average accuracy on CIFAR- Split setting than GPM. Results on other benchmarks provided in Appendix C.2 further substantiate this phenomenon. As tasks keep coming, accumulated frozen spaces lead to decreasing optimization spaces for GPM. In the contrast, IRGP explores larger optimization spaces by relaxing previous frozen spaces. Thus, IRGP achieves better forward knowledge transfer by implicitly reusing the weights within the relaxed subspaces.
In brief, our approach universally outperforms selected baselines all datasets in fixed capacity. With comparable forgetting, IRGP achieves better forward knowledge transfer with larger optimiza- tion spaces against GPM. To validate the efficiency of our relaxing strategy, further compare IRGP-Exp with well-established and competitive expansion- based methods in the next section.
4.3 COMPARED WITH EXPANSION-BASED METHODS
The above experiments exhibit the outstanding performance of our approach maintaining a fixed network capacity. By allocating neurons modules, expansion-based methods significantly mitigate backward interference with increasing capacity. Thus, to further validate our strategy, we compare IRGP and IRGP-Exp with relative expansion-based methods in this section.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Results of ACC (%) and Capacity on CIFAR-100 Sup setting. S77. under non-incremental setting, All baselines are expansion-basee methods except GPM.
Methods
Metric
STI,
PNN
DEN
RCL
APD
GPM
IRGP
ACC (%) Capacity
61.00
50.76
51.10
$1.99
56.81
57.72
58.12
20.00
2.71
1.91
1.84
1,30
1.00
1.00
Table 3: L: compare IRGP-Exp with TRGP under an expansion setting. R: compare IRGP with TRGP within a fixed network capacity. We reports the results as (ACC/ (new) for each experimental setting. Detailed results are provided in Tabel 16 and 17
Expansion
Non-Expansion
Methods
IRGP-Exp 80%
TRGP-Reg R2 5
50%
T%
w 50
CIFAR
75.15/74.76
75.38/55.02
75.06/74.91
74.46/75.01
7188572.87
72.08/73.12
72.46/72.91
73.52/74.78
PMNIST
96.68/97.18
96.99/97.29
97.03/97.22 26
96. 96.34/97.25
73.69/95.22
71.51/95.60
72.43/95.80
94.20/96.19
Net
60.81/62.35
62.03/62.44
60.84/62.11
78/63.222
55.81/60.28
58.81/62.77 77
22.69/20.12
61.26/62.80
Mixture
82.45/84.12
83.22/84.60
83.62/83.97
83.54/84.88
73.31/84.05
74.71/83.48
17,36/722
77.91/82.21
2019). ACC results shown in Table 2 are averaged over 5 different sequence orders proposed by Yoon et al. (2017). We refer to the results of baselines from Sahaa al.. (202I). Capacity denotes the model capacity normalized with respect to the network used in GPM. Here we use the same model as GPM. According to Table IRGP outperforms all baselines including GPM with the smallest capactiy.
Following Saha al. (202
perform experiments on CIFAR-100 Sup (Yoon et al.
ein al. (2022) proposed TRGP to expand the limited optimization spaces by retraining parameters within the selected trust regions, achieving superior performance. During the inference phase, TRGP reuses the parameters in corresponding trust regions memorized after learning this task. In contrast, GPM and our IRGP only store the representation of the frozen space. Therefore, although indeed a stable network capacity is allocated for each task, the entire memory size of TRGP grows continu- ally. As shown in Figure 3-(c), after learning the last task on MinilmageNet setting, TRGP requires around 5000% extra parameters with respect to the network capacity. Results other benchmarks provided in Appendix CC5 further substantiate that TRGP introduces a significant number of extra parameters. Thus, we categorize TRGP as an expansion-bant based method here.
As mentioned in Section 3.2 we propose IRGP-Exp to further validate our searching strategy. In this setting, the main difference between IRGP-Exp and TRGP is the strategy of deciding which part of the frozen space to reuse. We conduct experiments on all four benchmarks against TRGP. The results of ACC are provided in the left of Table The percentages indicate the ratios of the rank of relaxing subspaces with respect to the corresponding frozen space. We evaluate three constant ratios and further use the ratios in TRGP, denoted as T%. According already outperforms TRGP with relaxing only 50% of the frozen spaces on CIFAR CIFARTOO-SplA T00-Split and PMNIST. As TRGP selects the top tasks as the trust regions, T% is larger than 80% most times. Moreover, our approach gains better ACC on all benchmarks with a comparable size of relaxing subspaces, which substantiate the efficiency of our subspace searching strategy.
the left of Table 3 IRGP-Exp
We further modify TRGP TRGP-Res Reg with similar regularization terms the scale matrices as our IRGP to compare the relaxing strategies. We report the results on four benchmarks with three representative regularization weights on TRGP-Reg in the right of Table 3 As shown in Table IRGP significantly outperforms TRGP-Reg, especially on PMNIST, gaining over 20% ACC improvement. Generally, IRGP achieves better comparable knew than TRGP under without the constraint of a fixed network capacity. Detailed results are included in Appendix C.7
ANALYSIS AND DISCUSSION
To gain deeper insight into IRGP, we investigate the trend of scales of the subspaces relaxed by our strategy. With the theoretical upper bound of the rank of the relaxed subspace provided in Theorem 3.3 we further inspect the ratios of the relaxed subspaces concerning corresponding frozen spaces in practical. Results of the last layer on three different settings are provided in Figure 3-(a). As shown in Figure 3-(a), relaxing ratios maintain stable trend, fluctuating smoothly in similar
8

---

Under review as a conference paper at ICLR 2023
Figure 3: (a) Relaxing ratios of the last layer on CIFAR-100 Split, MinilmageNet, and PMNIST. (b) Test accuracy of GPM and IRGP of different on CIFAR-I00 Split. The optimum value of GPM is annotated by red circle. (c) Ratios of the amount of extra parameters concerning the amount of the parameters of the network architecture on MinilmageNet.
(a)
(b)
0.16
MitllingeSet
PAINIST
73.5
ERGP
IRGP
CIFARIOD
Ratio (%)
0.12
Test Accuracy
73.0
Ratio (%)
3000
0.08
2000
72.0
71.5
1000
3
4
0.96
0.965
0.99
0.975
0.98
2
8
9
Task ID
The value of'e
Task ID
range over sequential tasks, on both benchmarks. As different tasks explore different optimization directions, ideal relaxing subspaces vary across tasks, accord with the fluctuation of our results.
The dimension of the frozen spaces keeps growing the tasks come, leading to expanding range for searching relaxing subspaces. Therefore, the computation complexity and time consumption are supposed to increase gradually. To investigate the practical efficiency of our approach, reported the time consumption of IRGP on CIFAR-100 Split and MinilmageNeet against other base- lines in Appendix C.4 According to Table 14 approach takes around 60% more time than GPM on both settings. In general, the practical efficiency of our approach is accecptable.
further
To understand our relaxing strategy better, further conduct experiments on different thresholds mentioned in Equation 3 which regulate the criterion of the frozen spaces. [Saha et al. (2021) argue that e controls the scale of the frozen space to mediate the publlity-plasticity dilemma, and thus is critical for GPM. However, IRGP enables the frozen space to be dynamically regulated regarding the current task. Therefore, plays much less important role in IRGP. We present the performance of different on CIFAR- 100 Split in Figure 3-(b). As shown in Figure 3-(b), the performance of GPM drops significantly when 2 0.97, the optimal value reported in GPM, while IRGP consistently performs well even with = 0.98. Generally, IRGP is more robust the threshold
In the contrast, IRGP mediates the stability-plasticee dilemma by controlling the dimension and flexibility of the relaxing space by Ç in Equation ( and in Equation (II) respectively. We present the results CIFAR- Split in Table 4 where Scone denotes the hyper-parameter for convolutional layers and {ʃe de- notes the hyper-parameter parameter for fully connected layers. As shown in Table a larger ( guarantees less forget- ting, as a result of smaller relaxing subspaces. Detailed
Table 4: Ablation study of cos and on CIFAR 100-Split, where y is the thresh- old for the relaxing strategy and / is the weight of the regularization terms.
Soonv
Sfe
8
ACC (%)
BWT (%)
0.20
0.20
1.0
71.97
-3.0
0.50
0.50
1.0
72.58
-2.3
0.80
0.80
1.0
73.34
-2.2
results are provided Appendix C.6 Similarly, we
0.90
0.90
1.0
73.28
-1.2
observe less forgetting on larger 3, which constrains
0.95
0.90
0.0
73.15
-2.0
the update of parameters within the relaxing subspace more strictly. However, strict constraints also lead to limited performance on new tasks as discussed Sec-
0.95
0.90
0.1
73.32
-1.3
0.95
0.90
1.0
73.52
-0.9
0.95
0.90
5.0
73.14
-0.7
tion 4 Generally, Ç and / work together to overcome catastrophic forgetting with better forward transfer.
0.95
0.90
10.0
72.81
-0.6
0.95
0.95
1.0
73.08
-0.6
CONCLUSION
In this paper, we propose a novel continual learning approach that facilitates the forward knowl- edge transfer gradient projection methods with a fixed network capacity by iteratively searching and relaxing subspaces within the frozen space expand the optimization space. Extensive ex- periments demonstrate that our IRGP framework surpasses related state-of-the-an approaches on diverse benchmarks. Moreover, we propose modified version expanding the architecture with relaxing subspaces, achieving better average accuracy than other expansion- based methods. We further provide solid proof and analysis validating the efficiency of our algorithm.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 139-154, 2018.
Bulatov. Notmnist dataset.
2011.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. CoRR, abs/ htss/1902.10488 2019.
WU Chenshen, L HERRANZ, LIU Xialei, et al. Memory replay gans: Learning to generate images from new categories without forgetting [c]. In The 32nd International Conference on Neural Information Processing Systems, Montréal, Canada, pp. 5966-5976, 2018.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Dual-teacher dlass-incrementll learning with data-free generative replay. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3543-3552, 2021.
Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin. Gan memory with no forgetting. Advances in Neural Information Processing Systems, 33:16481-16494, 2020.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366 2021.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In European Conference on Computer Vision, pp. 386-402. Springer, 2020.
Benjamin Ehret, Christian Henning. Maria R Cervera, Alexander Meulemans, Johannes Von Os- wald, and Benjamin F Grewe. Continual learning in recurrent neural networks. arXiv preprint arXiv:2006. 109, 2020.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773. PMLR, 2020.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pp. 1920-1930. PMLR, 2019.
Robert M French. Pseudo-recurrent connectionist networks: An approach to the sensiiiiit
stability'dilemma. dilemma. Connection Science, 944:353-380, 1997.
Ta-Chu Kao, Kristopher Jensen, Gido van de Ven, Alberto Bernacchia, and Guillaume Hennequin. Natural continual learning: success a journey, not (just) a destination. Advances in Neural Information Processing Systems, 34:28067-28079) 2021.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barvinska, Barwinskk, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):5521335 2017.
Yajing Kong, Liu Liu, Zhen Wang, and Dacheng Tao. Balancing stability and plasticity through advanced null space in continual leamnnng arXiv preprint arXiv:2207. 12061, 2022.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick Van Der Smagt, and Stephan Günnemann. Continual learning with bayesian neural networks for non-stationary data. In International Con- ference on Learning Representations, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
Xilai Li. Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference Machine Learning, pp. 3925-3934. PMLR, 2019.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.029. 2022.
Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. arXiv preprint 17Xiv:2201.12522 2022.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Nikhil Mehta, Kevin Liang, Vinay Kumar Verma, and Lawrence Carin. Continual learning using bayesian nonparametric dictionary of weight factors. In International Conference Artificial Intelligence and Statisties, pp. 100-108. PMLR, 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. Advances in Neural Information Pro- cessing Systems, 33:7308-7733 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Hong- Wei Ng and Stefan Winkler. data-driven approach to cleaning large face datasets. In 2014 IEEE international conference on image processing (ICIP), pp. 343-347. IEEE, 2014.
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 2019.
Mozhgan PourKeshavarzi, Guoying Zhao, and Mohammad Sabokrou. Looking back on learned experiences for class/task incremental learning. In International Conference Learning Repre- sentations, 2021.
Rahul Ramesh and Pratik Chaudhari. Model zoo: A growing brain that learns continually. In International Conference Learning Representations, 2021.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2)2285, 1990.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1006:::666.04671 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint 103. 2021.
11

---

Under review as a conference paper at ICLR 2023
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557. PMLR, 2018
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. The 201 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Yunfei Teng, Anna Choromanska, Murray Camphell, Songtao Lu, Parikshit Ram, and Lior Horesh. Overcoming catastrophic forgetting via direction-constrainee optimization. In European Confer- ence Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2022.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 1561-2)225-4, 1995.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 184-193, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint orXivv/708..055: 2017.
Jaehong Yoon, Saehoon Kim, Funho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. arXiv preprint arXiv: 1902.0942. 2019.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent pro-
cessing in neural networks. Nature Machine Intelligence, 1(8):364-372, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3999 PMLR, 2017.
A PROOF
A.1 PROOF OF LEMMA
For simplification, all vectors here are assumed to be unit vectors, namely ||a||| = 1. Depicting the angle by projection form, Lemma 3.2 can be expressed as:
Ve E V, [Projo (v)] 2 (v:)))
(13)
Denote B [ui Um] as the representation matrix of the representation subspace U Un. where u, is the i-th normalized base of U. Lemma 3.2 can be expressed as:
span U1, (ean(u1,--
Yo E V, 8 BBT 2 v/ BBT 2/
(14)
As v,s are the basis iteratively appended by Algorithm I for any i < j. we have:
BBT v 2 2 BBTT
(15)
Therefore, to prove Lemma 3.2 it suffices to prove the following lemma.
12

---

Under review as a conference paper at ICLR 2023
If there exists vector _ way; satisfying the condition v "BB" 1a BB"Bue us then
can find another vector r' = such that BBBB""' o' > u BB" Uk. which contradicts Algorithm | where (ux, U) - (v, U) for vu E span( U141
First, we consider special case where the current relaxed subspace has only base, denoted by V = span(vi}. Assume there exists v = w1 + W222 that e BB" < 0g BB 12, we have:
" BB U2 > w of BB on + 200 1220 BB 02
(16)
Construct v'= = =W2t1
W1 U2, we have:
1 BB" w T
BB" 2 + wiv, BBTu, 02 2w w2u BB" U22
> w "B" 2 + ufu BB" U2 + wp BB" 01 w (17)
= oo BBB 01
which contradicts (vi U) S ( U) for Yu span(vi,
Then we consider the general case V = spanffi,.... vt). For Ve E V, we have O(u. U) S
(vvr U). After 1; is included, we assume that there exists v E V that (v, U) (v1+1, U). Then we can find the minimum s satisfying that there exists v = -,=1w; + W441 that ( U) > O(v++1, U) and for Yv' = ] uyyu + Wr41 1141, we have ( U) S ((v+11 it is similar to the special case, the proof is omitted. Thus, consider the where 2. For simplification, we express v as v = co% + C1V, + 020++1 with do = a a where 65 and a,s coefficients. We have:
U). When 8 1,
(covo CIU + c2U++1) BBT (covo ciu, + +22041) BTTBBBu. 041
(18)
As (wiio + w2ve,
< O(v,, U) < (vt++1, U)) we have:
+ w2v TB (win0 + w22s) 2 ou BB
(19)
which is:
wing BBTT up + 2w "
BB v, 2 wff. v. 2 of
(20)
Similarly we have:
w vg FTTBI BBT in + 2w wii BBT o 2 uf BBTT.
(21)
Then we can express Equation
(8) as:
oi BB Ut+1 > (c) 2)) BBT v + v. + 20 o
(22)
As |||||| = |all|
1, cg + ct + cτ = 1. Then we have:
2 1 BB v2 of. BB" u. -e BB BB 044
(23)
Construct
02V- Va+t
we have:
BBT o'
c (C BBT U. + QUU
BBT U141
20 BBTop)
BB U. + c BBd
441 + of -c70%t , BB 041)
c + 2
1 BBT Up
(24)
13

---

Under review as a conference paper at ICLR 2023
which contradicts (12,, U) Q(v for vo span Us41,.., 0., 2+11)
Thus, for Vu V = span (i...., veh, we have (v, U) S ()),
A.2 PROOF OF THEOREM 3.3
Denote the relaxed subspace and the representation subspace as V and U = spanfu1..... Uxpp respectively. With Lemma 3.2 we have Yu V. (vv, U) & (vr, < 2. In other words,
Ye E V. v LU
(25)
For the sake of contradiction, assume the dimension of is larger than kip, namely dVm > dim(U) kp- denote U as the complemented subspace of U with respect the whole space R. Obviously, dU=) = n - kp, where n is the dimension R. Then we have:
dV nt)) = dim dim (U)) dim U
(26)
> kip + (n kp) - n = 0
Thus, there exists e' E V such that v' E U too. As U is the complemented subspace, Vu U 1 U. Then we have o' I. U, which contradicts Equation (25). Therefore, the assumption is aborted. The upper bound of the dimension of V is kip namely the dimension of the representation subspace U.
A.3 PROOF OF THEOREM 3.4
Denote the whole solution set as 5 ful (u, U) S and u E UT} where U the representation subspace, U is the frozen space and is the threshold. Theorem 3.4 can be expressed as that all subspace V C S satisfies dim(V" < dim(V), where is the relaxed subspace obtained by Algorithm I Similar to Theorem 3.3 assume there exists V" C S that di'(V") dim(V)- denote V as the complemented subspace of V with respect to the frozen space Uf, According to Algorithm for Vo E Vj, we have ( U) 7. We also have:
dim(V' n Vf)= dVm dim(Vy) dim(V' Vf) 0
(27)
Thus, there exists o' E V that o' E Uf. namely there exists v' E S that O U) ) > 7, which contradict. Therefore, the relaxed subspace obtained by Algorithm [ takes up the maximum subspace of the whole solution set.
B EXPERIMENTAL SETUP
B. DATASETS
for evaluation. 1) CIFAR-100 Split Saha et al. (2021) constructed CIFAR-100 Split, by splitting CIFARION (Krizhevsky & Hinton 2009) into 10 tasks where each task has 10 classes. 2) MinilmageNet Following [aha al. (2021) we split Minilm- ageNet (Vinyals et al. 2016) into 20 sequential tasks with 5 classes each. 3) Permuted MNIST
Here introduce the datasets
(PMNIST) PMNIST Kirkpatrick et al., 2017) is variant MNIST (LeCun et 1998 where each task has different permutation of ipputing images, consists of 10 sequential tasks with 10 classes each. 4) CIFAR-100 Sup Following Yoon et al. (2019). we adopt CIFAR-100 Sup con- sisting of 20 superclasses as sequential tasks. 5) Mixture (Seraa et al. (2018) first proposed Mix- ture consisting of 8 datasets, including CIFAR- (Krizhevsky & Hinton 2009), MNIST (LeCun) et al. 1998). CIFAR- 100 (Krizhevsky & Hinton 2009), SVHN (Netzer al. 2011). FashionM- NIST (iaoo et al. 2017). TrafficSigns (Stallkamp et al. (011)) FaceScrub (Ng & Winkler 2014), and NotMNIST (ullttov] 2011). from which Ebrahimi al. (2020) further constructed 5- 5-Datasets. Here we follow the original harder benchmark. Particularly. we consider all tasks as a sequence ex- cept TraffieSigns (Stalikamp al. 2011), which we failed to access. Among all evaluated datasets,
14

---

Under review as a conference paper at ICLR 2023
PMNIST is benchmark under the (omainn-nnceeeed scenario, while other four datasets under the dusk-incremental scenario.
Moreover, provide the statistics of selected datasets in Table 5 and Table 6 For the Mixture benchmark, the images of MNIST, FashionMNIST, and notMNIST are replicated across all RGB channels following Serra al. 2018).
Table 5: Statistics of CIFAR-100 Split, MinilmageNet, and PMNIST.
CIFAR-100 Split
CIFAR-100 Sup
MinilmageNet
PMNIST
Image Size
32 x 32
32 x 32
84 x 84
28 x 28
Channels
3
3
Classes
100
100
100
10
Tasks
10
20
20
10
Classes/task
10
5
5
10
Training Samples/task
4,750
2,375
2,375
54,000
Validation Samples/task
250
125
125
6,000
Testing Samples/task
1,000
500
500
10,000
Table 6: Statistics of Mixture benchmark.
Dataset
Classes
# Taining
# Validation
Testing
CIFAR-11 10 (Krizhevsky & Hinton
2009)
10
47,500
2,500
10,000
MNIST [eCun
1998)
10
57,000
3,000
10,000
CIFAR- 100 (Knzhevsky & Hinton
2009
100
47,500
2,500
10,000
SVHN (Netzer et al.
2011)
69,595
3,662
26,032
MNSS (Xiao et al.
2017
10
57,000
3,000
10,000
FaceScrub (gg & Winkler
2014)
100
19,570
1,030
2,289
NotMNIST (Bulatov
201
10
16,011
842
1,873
B.2 MODEL DETAILS
MLP architecture: We adopt 3-layer model including two hidden layers with 100 neurons each for the PMNIST setting, the same Lopez-Paz & Ranzato (2017). ReLU is used the activate function here and for all other architectures. Also, we use softmax with cross entropy loss on all settings.
AlexNet architecture: For CIFAR-100 Split setting, we adopt the same network as Serra et al. (2018) with batch normalization, including fully connected layers and three convolutional lay- ers. The convolutional layers have 4 x 4, 3 x 3, and 2 x kernel sizes with 64, 128, and 256 filters add batch normalization and
respectively. After each convolutional layer, Each fully connected layer has 2048 units. For the first two layers, we use the dropout of 0.2, and for the rest layers, we use the dropout of 0.5.
max-pooling.
For the CIFAR- 100 Sup setting, modified LeNet-5 architecture consisting of two convolutional layers and two fully connected layers is adopted, similar to [Saha et al. 2021). Max-pooling of 3 x 2 is used after each convolutional layer. The last two layers have 800 and 500 units respectively.
Modified LeNet-5 architecture:
Reduced ResNet-18 architecture: We adopt the same reduced ResNet- 18 architecture as Saha et al. (2021) for the MinilmageNet and Mixture settings, using 2 2 average-pooling before the classifier layer instead of the 4 x 4 average-pooling used by Lopez-Paz & Ranzato (2017). Moreover, we present the dimension of the representation space of each layer our architectures in Table
B.3 IMPLEMENTATION DETAILS
We use the official implementation of GPM (Shha et al. 2021). OWM (Zngg et al. 2019).
al. (2018) and TRGP (Lin et al. 2022 Moreover, implement A A-GEM and ER Res with the
15

---

Under review as a conference paper at ICLR 2023
Table 7: Dimension of the representation space of each layer.
Network
Depth
Dimension of the representation space
MLP
3 layers
784; 100; 100
AlexNet
layers
48; 576; 512; 1,024; 2,048
LeNet-5
4 layers
75: 500; 3,200; 800
ResNet-18
17 layers and 3 short-
27: 180; 180; 180; 180; 180; 360; 20; 360; 360; 360; 720;
cut connections
40; 720; 720; 720; 1,440; 80; 1, 1,440; 1,440
official implementation by Chaudhry et al. 2018) and implement EWC with the implementation by Serra et al. 2018). Following Saha et al. (2021) and Lin et al. (2022). we run all experiments five times on an established seed without fixing the cuda settings for fair comparison. Particularly, we use five random seeds on PMNIST where there is no diversity on a single seed. For CIFAR- 1-0---00- Sup, we use five different orders provided by] Yoon al. (2019). Following Saha et al.) (202T), we report the experimental results of replay-base methods A- A-GEM and ER Res on the Mixture dataset with the same buffer size as GPM and our IRGP, which 8.98M in term of the number of parameters for the Resnet18 8 architecture.
On CIFAR- 100 Split, MinilmageNet, and PMNIST, we follow the syper-parameters utilized by Saha et al. (2021) and Lin et al. (2022), including learning rate, batch size, and the threshold e. On Mixture, as we adopt the same network architecture Saha al. (202I) use on their 5-Dataset setting, follow the provided learning rate and batch size as well. Moreover, for the threshold in GPM, we conduct experiments with e in the range of 0.95 to 1 provided in (Saha et al. (221)) and report the best results whose is 0.955. We further use 0.96 for all layers in our IRGP.
As discussed Section 5 the threshold Ç = cos 7 controls the criterion of the relaxing subspace. 0.95 for convolutional layers and 0.9 for fully
For CIFAR100-SS0 and PMNIST, connected layers. For MinilmageNet and Mixture, we use the same Ç for all layers, 0.95 and 0.9 respectively. Furthermore, we the regularization weight as 5 for the ResNet18 8 architecture and for others. Particularly, we run all the experiments on a single NVIDIA GeForce RTX 2080 Ti GPU.
B.4 METRICS
Here we present the detailed definitions of the metrics evaluating the forward knowledge transfer. Qnew (Kemker et al. 2018). Denote b; as the test accuracy of task i at random initialization, FWT, first proposed by Lopez-Paz
Ranzato 2017))
defined as FWT = T-I 2 (Ai- - bz),
evaluating the zero-shot performance the inttralization with respect to the observed tasks. While b:), reflecting the test accuracy new tasks based on the leartt knowledge. As b, stays still across different approaches, we consider Pnew T-1 ΣT D/=24 for simplicity. For this simplified Qnews we have:
Pnew. first proposed by Kemker et al. (2018), 3 is defined Qnew = 1 DT 2
T-T
Onew T ACC - BWT T-1 A11, with the ACC and BWT defined in Section 4.1
C EXPERIMENTAL RESULTS
C.I FINAL ACCURACY
We provide the test accuracy after learning each task on other benchmarks here. As diseussed in Section our IRGP universally outperforms GPM over the task sequence on all benchmarks.
C.2 FORWARD KNOWLEDGE TRANSFER
We provide the detailed forward knowledge transfer performance on all four benchmarks here. First, we present the results of Qnew and the detailed accuracy of each task after learning in to According to Table and with GPM. For other benchmarks, IRGP improves 2new by 2.7% and 1.8% on CIFAR- 100 Split and MinilmageNet respectively,
and Table [1] IRGP achieves similar forward knowledge transfer compared
shown in Table 8 and Table 9 Moreover, we provided the detailed results including the standard deviation and other baselines m Table [2 According to Table [2 our IRGP consistently achieves the best second best forward knowledge transfer compared with all
16

---

Under review as a conference paper at ICLR 2023
Figure 4:
Average accuracy after learning each task on (a) MinilmageNet, (b) PMNIST, and (c)
Mixture.
(a)
(c) 90
TRUP GPM
IREIP GPM
GPM IRCP
Test Accuracy (%)
63
Test Accuracy (%)
Test Accuracy (%)
62
96
80
60
94
19
75
11
13 17
19
3
4
5
6
Task ID
Task ID
Task ID
baselines without extra memory buffer. For S0-pli,,- IRGP even gains 2.7% better Pnew than HAT, which is the second best baseline in this setting.
Table The accuracy tested on task after learning task and nnw on CIFAR-100 Split.
Method
5
7
9
Avg (Onew)
GPM
67.7
72.5
70.1
73.6
71.8
70.3
71.0
71.8
73.7
71.4
IRGP
68.7
73.6
72.0
76.3
74.2
73.7
74.1
74.9
79.0
74.1
Table 9: The accuracy tested on task after learning task and Inew on MinilmageNet.
Method
2
6
8
10
12
14
16
18
Avg (R..)
GPM
59.0
57.1
65.6
59.6
16.2
57.2
66.5
72.8
81.
42.0
59.6
62.1
62.3
58.9
57.2
59.0
50.7
65.7
57.3
51.6
IRGP
57.6
60.1
660
622
77.0
62.5
65.7
75.6
82.3
44.4
64.0
59.2
64.5
60.0
39.6
60.5
50.0
68.3
55.6
6288
Table 10: The accuracy tested on task i after learning task i and Nnew on PMNIST.
Method
2
3
4
5
7
8
9
Avg (Anew)
GPM
97.5
97.4
97.0
96.8
96.5
96.2
96.2
95.8
95.1
96.5
IRGP
97.4
97.2
97.0
96.5
96.3
96.1
95.7
94.9
94.8
96.2
Table 11: The accuracy tested task after learning task and Unew Mixture.
Method
2
3
4
5
Avg (Pnew)
GPM
99.0
43.7
87.3
99.1
69.9
93.4
82.1
IRGP
99.1
42.9
87.6
99.1
70.6
93.5
82.2
As mentioned in Section 4.1 final accuracy forgetting (BWT) and forward knowledge trans- fer (Snew) are jointly considered to evaluate a continual learner. Despite our relaxing strategy focus- ing on facilitating the forward knowledge transfer, IRGP gains better final accuracy with comparable forgetting over all benchmarks, according to Table | Generally speaking, IRGP achieves superior performance than previous baselines with fixed network capacity.
Table 12: Comparison of forward knowledge transfer on four benchmarks, evaluated by Qnew-
Datasets
A-GEM
ER Res
EWC
HAT
GPM
IRGP
PMNIST
97.4 + 0.0
97.4 + 0.5
93.1 + 0.7
96.6 + 0.0
96.5 + 0.1
CIFAR1ON SETAR100-Splii
77.5 + 0.4
77.1 0.2
69.9 1.1
71.5
71.5 0.5
74.1 1 0.2
MinilmageNet
67.6 1.2
69.5 0.4
63.5 +28
0.66
61.6 0.6
62.8 + 0.8
Mixture
85.9 + 0.4
86.4 0.2
74.4 1.0
79.3 0.2
82.1 0.4
82.2 + 0.4
17

---

Under review as a conference paper at ICLR 2023
Moreover, provide the result of FWT (using the definition in (Lopez- Papez-Pazz & Ranzato 2017 all benchmarks in Table 13 According to Table [3] although our method facilitates the forward knowledge transfer reflected by 1 IRGP achieves better FWT than GPM on all three task- incremental benchmarks.
Table 13: Comparison of forward knowledge transfer between GPM and IRGP, evaluated by FWT.
C-FAR-100 Split
MinilmageNet
PMNIST
Mixture
Methods
FWT (%)
STD (%)
FWT (%)
STD (%)
FWT (%)
STD (%)
FWT (%)
STD (%)
GPM
-0.65
0.18
-0.26
0.88
+0,66
1.17
-0.52
1.49
Ours (IRGP)
+0.20
0.36
+0.52
0.35
-0.63
0.97
+0.13
0.77
C.3 ACCURACY EVOLUTION
Here we present the accuracy tested on three randomly selected tasks after learning them on four benchmarks. We select the 2nd, 4th, and 6th tasks for CIFAR-100 Split, MinilmageNet, and PM- NIST. As there are only 7 tasks in Mixture, we select the 1st, 3rd, and 5th tasks. Generally, IRGP outperforms GPM on selected tasks over the sequence, We further notice that the improvement is more significant on later tasks as a result of larger relaxing subspaces, discussed in Section 5
Figure 5: Accuracy evolution of the (a) 2nd, (b) 4th, and (c) 6th task on CIFAR-100 Split.
(a)
77
(c)
GPM
Test Accuracy (%)
Test Accuracy (%)
Test Accuracy (%)
73
744
75
72
GPM ERRP
73
73
70
73
Task ID
Task ID
Task TD
Figure 6: Accuracy evolution of the (a) 2nd, (b) 4th, and (c) 6th task on MinilmageNet.
(a)
62
(c) 64
GPM ERGP
GPX IRGP
GPM IRGP
Test Accuracy (%)
Test Test Accuracy (%)
61
Test Test Accuracy (%)
58
60
60
56
58
58
52-
57
54
50
52
2
6
10
12
16
18
10
12
14
16
18
12
14
16
18
Task ID
Task ID
Task ID
Figure 7: Accuracy evolution of the (a) 2nd, (b) 4th, and (c) 6th task on PMNIST.
(b)
(c)
96.0
Test Accuracy (%)
97
Test Acquracy (%)
96
Test Accuracy (%)
96
95
95
04.5
94
94.0
90
93.5
92
92
95.0
Task
Task
Task ID
18

---

Under review as a conference paper at ICLR 2023
Figure 8: Accuracy evolution of the (a) 1st, (b) 3rd, and (c) 5th task on Mixture.
(b)
(c)
GPM ERGP
Test Accuracy (%)
99.0
ERGP
Test Accuracy (%)
$7
Test Accuracy (%)
70.5
JRGP
98.8
70.0
84
98.2
Task
Task
Task ID
C.4 TIME CONSUMPTION
We report the time consumption of IRGP two benchmarks compared with relative baselines. TRGP and IRGP are both evaluated on a single NVIDIA GeForce RTX 2080 GPU and we report the results according (Lin al. 2022). As discussed in Section 5 IRGP takes acceptable extra time compared with GPM on both datasets. For CIFAR-100 Split, IRGP tasks similar time as TRGP, which is similar _R_Res Res and HAT and much less than A-GEM and OWM. For MinilmageNet, IRGP tasks around 30% time than TRGP, and is similar to A-GEM.
Table 14: Time comparison evaluated two benchmarks. We the results reported in (Lin al. 2022 and the time normalized with respect to GPM.
Methods
Datascts
OWM
EWC
HAT
A-GEM
ER_Res
GPM
TRGP
IRGP
CIFAR-100
2.41
1.76
1.62
3.48
1.49
1.00
1.65
1.62
MinilmageNet
1.22
0.91
1.79
0.82
1.00
1.34
1.69
C.5
MEMORY USAGE
We provide a comparison between TRGP and IRGP the ratio of the amount of extra parameters concerning the amount of the parameters the initial network architecture. According to Figure 9 TRGP requires at least 200% of the number of extra parameters after learning all tasks the other three benchmarks, while IRGP only stores the representation of the frozen space, which can further be released in the inference phase.
Figure 9: Ratio of the amount of extra parameters concerning the amount of the parameters of the initial network architecture on (a) CIFAR100-SplA (b) PMNIST, and (c) Mixture.
(a)
250
(b) 4000
(c)
5000
TRGP
TRCIP
TRRP ERGP
IRGP
IRGP
3000
Ratio (%)
Ratio (%)
2000
Ratio
100
2000
J000
50
1000
8
6
Task ID
Task ID
Task ID
C.6 RELAXING RATIO
We provide the results CIFAR100-Split between the hyper-parameter and the relaxing ratio of
all five layers of the AlexNet architecture in Table 15 And 3 is set to be 1.0 for all experiments here. According to Table 15, generally, larger Ç guarantees smaller relaxing ratios. As discussed in Section forgetting is mitigated by constraining the percentage the relaxing weights, as a result of increasing Ç.
19

---

Under review as a conference paper at ICLR 2023
Table 15: The relationship between and the relaxing ratio of different layers on CIFAR-100 Split.
Scong
Sfe
Convl (%)
Conv2 (%)
Conv3 (%)
Fel (%)
Fe2 (%)
ACC
BWT
0.20
0.20
72.32
61.56
69.11
38.92
29.75
71.97%
-3.0%
0.50
0.50
50.96
45.18
56.07
27.43
19.06
72.58%
-2.3%
0.80
0.80
50.99
31.09
35.74
10.56
5.41
73.34%
-2.2%
0.90
0.90
44,91
23.05
21.90
4.82
2.00
73.28%
-1.2%
0.95
0.90
42.07
17.82
16.87
4.84
2.20
73.52%
-0.9%
0.95
0.95
42.77
17.49
16.59
2.30
0.76
73.08%
-0.6%
0.99
0.99
33.68
7.08
8.07
0.16
0.00
73.19%
-0.6%
C.7
OTHER RESULTS
We provide the test accuracy over the task sequence in Figure I0 As shown in Figure 10-(a) and Figure 10-(b), our IRGP-Exp dominates TRGP both benchmarks relaxing either 80% or T% of the frozen spaces under an expansion setting. We further compare IRGP with TRGP modified with the same regularization terms. As shown in Figure 10-(c) and Figure 10-(d), the performance of TRGP drops significantly constrained in a fixed network capacity on both benchmarks. Detailed results between IRGP and TRGP are provided in Table I6 and 17
Figure 10: Test accuracy after learning each task under an expansion setting on (a) CIFAR- 100 Split and (b) PMNIST, and within a fixed network capacity on (c) CIFAR-100 Split and (d) PMNIST.
(a)
(b)
Test Accuricy (%))
474
Test Accuracy (%))
Test Accuracy (%)
TROP
Test Accuracy
72
Task ID
Task 1D
Task
Task
Table 16: Compare IRGP-Exp with TRGP on four benchmarks under an expansion setting. All results reported are average over 5 runs.
TRGP
IRGP Exp 50%
IRGP-Fxp 80%
IRGP-Exp T%
Methods
ACC (%)
BWT (%)
ACC (%)
BWT (%)
ACC (%)
BWT (%)
ACC (%)
BWT (%)
CIFAR 74.46
0.07
75.15
0.20
75.38
0.29
75.06
0.04
PMNIST
96.34
-0.80
96.68
-0.61
96.99
-0,38
97.03
-0.28
MiniImageNer
61.78
-0.50
60.81
-0.21
62.03
60.84
Mixture 83.54
-0.80
82.45
83,22
-0.48
83.62
-0.33
Table 17: Compare IRGP with TRGP-Ra Reg four benchmarks within a fixed network capacity. All results reported are average over 5 runs.
IRGP
TRGP-Ro Reg
TRGP-Reg Reg--Regaa 1
TRGP-Rep 50
Methods
ACC (%)
BWT
ACC
BWT (%)
ACC (%)
BWT
ACC (%)
BWT
CIFAR
73.52
-0,94
-1.48
72.08
-1.51
72.46
-0.96
PMNIST
94.20
-2.44
73.69
-24.14
71.51
-26.97
72.43
-26.14
MinilmageNet
61.26
-1.72
55.81
-3.69
58.81
-2.84
22.69
0.01
Mixture
77.91
-4.45
73.31
-11.05
74.71
-9.33
17.36
-0.01
C.8 FORWARD KNOWLEDGE TRANSFER: NEWLY-ADDED RESULTS FOR TABLE 2
20

---

Under review as a conference paper at ICLR 2023
Table 18: Comparison of forward knowledge transfer on CIFAR100 evaluated by Nnew
Methods
PGN
EWC
GPM
IRGP
Snew
51.1 + 0.4
57.5 = 3.1
58.7 = 0.3 58.9 0.4
21

---

Under review as a conference paper at ICLR 2023
D ALGORITHM
We present the pseudo-code of our Iterative Relaxing Gradient Projection here.
Algorithm Iterative Relaxing Gradient Projection
1: Initiate frozen subspaces Uo {U)) 0s and optimize Wi for task
2: Compute frozen subspace 1/ with Equation
3: for E 2....., do
4: Initiate relaxing subspace {V:) as
5: Initiate scaling matrices {8 as identity matrix As
6: repeat
7:
Fine-tune for pre-defined e, epochs with objective function 11
Search additional relaxing subspace {Vl.new L. li=1 by Algorithm with current gradients
8:
{g remaining frozen subspace {U, U/\ VOL Ti=i V V U V.l.new
and thresholds {E/x 7
9:
10:
Expand S with Equation
12
until V.l.new is 1
11:
12:
Optimize W4, St with objective function
13:
Update W + W Projvy (W?) + Proj (W?)
14:
Update frozen subspace Uy with Equation
15:
end for
22


Reviews:
[{"review_id": "ef1NQVgOFAo", "reviewer": "Reviewer_tNh3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to mitigate forgetting and favor forward transfer by using a gradient projection approach with a fixed-capacity architecture. The method (Iterative Relaxing Gradient Projection (IRGP)) iteratively searches for frozen (previously trained) weights that are relevant to the current task. These weights are relaxed to consolidate knowledge of the current task, which helps with one of the problems of gradient projection methods: low flexibility when training novel tasks. In addition, the authors present a version that can be expanded (increasing the number of weights in the model), which is efficient compared to a state-of-the-art method. Results superior to various methods are presented in different scenarios of Task-Incremental Learning, in addition to some ablation studies.", "summary_of_the_review": "As the paper is, I don't think it's good to accept it. However, I think it is details that separate it from a paper that can be accepted.\n\nThe first is the definition of Forward Transfer and the second is a more complete study of how many and how the weights change when they are made flexible. If many weights are made flexible, one would expect more forgetfulness, if there are few, how do we ensure that there is indeed knowledge transfer?", "strength_and_weaknesses": "S:\n- In order for CL to work, there must be motivation to add flexibility to the weights learned through previous tasks and that this flexibility causes minimal forgetting. It not only helps to consolidate learning, but it can also help with knowledge transfer. The motivation of the authors is interesting to the community.\n- The idea of adding plasticity to the model by relaxing similar connections for both tasks (old and new) is interesting. Not only does it add plasticity to the current task, but it can also help with knowledge transfer to previous tasks. \n    - Question To the authors: Do you study how the accuracy of previous tasks is affected when weights are relaxed?\n- Table 2 provides helpful insights into how the flexibility of certain connections is used. Without making the model grow, favorable results are achieved.\n\nW:\n- The definition of Forward Transfer is not the same as the one used in the paper by Lopez-Paz and Ranzato. It is not clear to me if only the formula is incorrect or if the results are as well, but the way it is presented makes me think it is not only the formula. Forward Transfer has to do with the ability of the model to acquire knowledge that may be relevant to tasks that have not yet occurred. Here it is used for the current task.\n    - Focusing on the accuracy of the current task is crucial, and not many papers focus on it. However, it is not the same as Forward Transfer.\n- As the proposed method takes 60% more time than GPM, and only increases accuracy by 1%, it is not a favorable trade-off. It may depend on the application and scenario, but it is a big difference. However, I do appreciate that the authors mention this result.\n- In Table 4, adding more extreme values would have been helpful. In order to determine whether the assumptions mentioned in the model are met, we must observe how the model behaves.\n\nQuestions:\n- Since the community is focusing more (and for a good reason) on scenarios where the task-id is unavailable in inference. Do you think this method can be adapted to work in this type of scenario?\n- Is it known what percentage of weights are relaxed in each task? Can this be related to forgetting the method?\n- In the fifth line of the paragraph after Eq 1. Should U^l_{t-1} just be U_{t-1} or not?\n- What is defined in Eq 9?\n- In Eq 10, what is the I of W^l_{t,I}?\n- Algorithm 2 is in Appendix D, not C.5\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ReV9FIjSxmK", "reviewer": "Reviewer_AinZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Iterative Relaxing Gradient Projection (IRGP) to facilitate forward knowledge transfer within a fixed network capacity for continual learning. ", "summary_of_the_review": "The proposed method is interesting but the empirical evaluation is not convincing. \n\nUpdate. I thank the authors for the response and addressing many of my and other reviewers' comments. I have updated the score to reflect the recent changes. The reason for not providing a higher score is current missing results due to the time constraint.", "strength_and_weaknesses": "Strengths\n+ Interesting solution to the stability-plasticity dilemma for gradient projection continual learning methods.\n+ Clear mathematical exposition of the related and proposed methods.\n\nWeaknesses\n- Although IRGP was introduced to facilitate forward knowledge transfer, the metric FWT is only reported in the appendix and compared with only one method. This is clearly unacceptable. FWT should be reported in Tables 1-3.\n- The proposed method is an iterative method, yet there is no mention or analysis of computational complexity or resources are mentioned anywhere in the paper let alone compared with existing work.\n- The experiments are not consistent. It is difficult to get a clear view on how the method really compares especially versus expansion-based methods because only one benchmark is reported. Why only CIFAR-100 or PMNIST?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QX8KW9n_wy", "reviewer": "Reviewer_Vp6m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To tackle the problem of incrementally learning a sequence of tasks with a deep neural network, this paper builds upon and extends a recent line of work using gradient projection to constrain the optimization of newly encountered tasks. In particular, this paper builds upon the TRGP method proposed by Lin et al. (2022, ICLR). A disadvantage of TRGP is that, when learning new tasks, TRGP expands the model with a new trust region (and associated parameter storage) for each new task. To avoid such expansion, the IRGP method proposed in this paper instead iteratively searches for the most critical parameter subspace to maintain. This results in a method that still performs well on a specific set of benchmarks (approaching the performance of TRGP, and if also using additional parameters it can outperform it) with less rapidly increasing storage costs.", "summary_of_the_review": "Although I believe this paper probably makes a minor contribution to the literature, the restricted experimental evaluation (in particular the heavy reliance on previous papers) causes me to currently recommend rejection. I encourage the authors to test their idea in more varied settings, compare their idea to other related methods and to improve the writing of the paper to reduce the reliance on knowledge of other papers.", "strength_and_weaknesses": "I think an important limitation of this paper is that it very closely follows the experimental setup of a couple of recent papers proposing similar methods, such as Saha et al. (2021, ICLR) and Lin et al. (2022, ICLR).\n\nOne resulting issue is that without knowledge of these two previous papers, it is very challenging, if not impossible, to appreciate the contribution of the current paper. I would encourage the authors to rewrite their paper to make it possible to appreciate its contribution without knowledge of these previous papers.\n\nAnother issue resulting from this is that there is an important risk of overfitting on the specific benchmarks used. On the one hand (e.g., for reproducibility) it is great that the authors include experiments on these extensively studied benchmarks, but as for several iterations now this line of work has exclusively been evaluated on this set of benchmark, I believe there is a real risk of overfitting and I would encourage the authors to evaluate their method also on other benchmarks.\n\nAnother consequence of sticking so closely to these previous papers, is that the set of methods that is compared against is exactly the same as in those previous papers. Other recent related methods are not considered. In particular, one very related method that is not considered is Natural Continual Learning proposed by Kao et al. (2021, Neurips; https://arxiv.org/abs/2106.08085). This method also tackles continual learning by putting restrictive constraints on the optimization of new tasks, but, unlike the present paper, uses an approximation of the Fisher Matrix of old tasks for this. Intuitively, this should allow for more positive transfer than using an orthogonal projection onto the input subspaces of olds tasks. I strongly encourage the authors to compare against this paper.\n\nIn the continual learning literature, often a distinction is made between task-incremental learning, domain-incremental learning and class incremental learning (e.g., as in Kao et al., 2021, NeurIPS; or see https://arxiv.org/abs/1904.07734). The current paper does not discuss these distinctions, and it is not entirely clear to which of these three scenarios of continual learning the considered benchmarks belong. My guess would be that PMNIST follows the assumptions of domain-incremental learning, while the others follow task-incremental learning? It would help the clarity of the paper to make this clear.\n\nOther issues:\n- In section 2, why is GEM not discussed under gradient projection methods? It seems clear that this method is doing gradient projection as well.\n- Whenever experimental results are copied over from another paper, this should be clearly explained in both the text and the legend of the table/figure where those results are reported.\n- I think it is important to include the standard deviations in the tables in the main text. Currently they are only provided in the Appendix, but for a correct interpretation of the results they are essential.\n- In Table 1, it is misleading that the BWT results of “Ours (IRGP)” are printed in bold. For none of the considered benchmarks this method scores the best on this metric.\n- For AGEM and ER_Res it should be stated in the main text what memory buffer size is used.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject