Continual Learning with Recursive Gradient Optimization
Authors: Hao Liu
Huaping Liu
Venue: ICLR 2022 Spotlight (2022)
Keywords: continual learning
lifelong learning

Abstract: Learning multiple tasks sequentially without forgetting previous knowledge, called Continual Learning(CL), remains a long-standing challenge for neural networks. Most existing methods rely on additional network capacity or data replay. In contrast, we introduce a novel approach which we refer to as Recursive Gradient Optimization(RGO). RGO is composed of an iteratively updated optimizer that modifies the gradient to minimize forgetting without data replay and a virtual Feature Encoding Layer(FEL) that represents different long-term structures with only task descriptors. Experiments demonstrate that RGO has significantly better performance on popular continual classification benchmarks when compared to the baselines and achieves new state-of-the-art performance on 20-split-CIFAR100(82.22%) and 20-split-miniImageNet(72.63%). With higher average accuracy than Single-Task Learning(STL), this method is flexible and reliable to provide continual learning capabilities for learning models that rely on gradient descent.

Published as a conference paper at ICLR 2022
CONTINUAL LEARNING WITH RECURSIVE GRADIENT OPTIMIZATION
Hao Liu
Huaping Liu" Department of Computer Science Tsinghua University Beijing, China hpliušt singuuuiinnuaa.no edu en
Department of Computer Science
Tsinghua University
Beijing, China
wtt.eeel....iinai..eeeeeeee..eee tsinghua .du...aadddc....
ABSTRACT
Learning multiple tasks sequentially without forgetting previous knowledge, called Continual Learning (CL). remains long-standing challenge for neural net- works. Most existing methods rely on additional network capacity or data replay. contrast, we introduce a novel approach which we refer to as Recursive Gradient Optimization (RGO). RGO is composed of an iteratively updated optimizer that modifies the gradient to minimize forgetting without data replay and virtual Fea- ture Encoding Layer (FEL) that represents different network structures with only task descriptors. Experiments demonstrate that RGO has significantly better per- formance on popular continual classification benchmarks when compared the baselines and achieves new state-of-the-art performance on 20-split-CIFAR 100 (82.22%) and 20-split-minilmmgeNee (72.63%). With higher average accuracy than Single-Tass Task Learning (STL), this method is flexible and reliable to provide continual learning capabilities for learning models that rely on gradient descent.
INTRODUCTION
In many application scenarios, one needs to learn sequence of tasks without access to historical data, called continual learning. Although variants stochastic gradient descent (SGD) have made significant contribution to the progress made by neural networks in many fields, these optimizers require the mini-batches of data to satisfy the independent identically distributed (i.i.d.) assumption. In continual learning, the violation of this requirement leads significant degradation of perfor- mance on previous tasks, called catastrophic forgetting. Recent works attempt to tackle this issue by modifying the training process from variety of perspectives.
Memory-based approaches use extra memory to store some samples (Lopez-Paz & Ranzato 2017
Chaudhry et al., 2020), gradients (Chaudhry al. 2019al 2020 Saha al. (021). or their genera- SSenn
tive models (Shin et al. 20177] (Shen et al.) (2200) to modify future training process. The memory for replay leads to linear- incear-deeeesed space complexity with respect to the number of tasks. Expansion- based approaches select the network parameters dynamically YYoon et al. 2018 Rosenbaum et al.)
2018 Serra et al., 2018: Kaushik al. (02T), add additional components as new tasks arrive (Rusu
al. 2016 Fernando et al. 2017: Alet et al. 2018 Chang al.) 2019 Li et al.) 2019), use larger networks to generate network parameters AAljundi et al. 2017 Yoon et al. 2019 yon Oswald et al. 2019). These methods reduce interference between tasks by additional task task-specife parame- ters. Single-Tass Learning (STL) can also be regarded as an expansion based method which trains network for each task separately. Regularization-base approaches encourage important param- eters to lie in a close vicinity of previous solutions by introducing quadratic penalty term to the loss function (Kirkpatrick et al., 2017 Zenke et al., 2017] Yin al. 2020) constraining the direction of parameter update (Farajtabar et al. 2019 Chauddhry et al.) 201 9a, (Saha et al. [2021). Our method also regularization-base which combines the advantages of loss penalty and gradient constraint.
In this work, we focus on continual learning in a fixed-capacity network without data replay. We aim minimize the expected increment of the total loss of past tasks without reducing the perfor- mance of the current task. To this end, we propose an upper bound of quadratic loss estimation and
"Corresponding author.

---

Published as a conference paper at ICLR 2022
design recursive optimization procedure to modify the direction of the gradient to the optimal solu- tion under this upper bound. In addition, we introduce trace normalization process to guarantee the learning rate during the training process according to the principle of current-task task-ffisstt (CFT). This normalization process makes our approach compatible with the vast majority of existing models and learning strategies well-designed for single-task solutions. As the gradient modification process is independent of data samples and previous parameters, our optimizer can be used directly in most deep architecture networks typical single-task optimizers like SGD, Further, to reduce the inter- ference between tasks, we develop a feature encoding strategy to represent the multi-moda structure of the network without additional parameters. A virtual feature encoding layer (FEL) which ran- domly permutes the output feature maps using integer task descriptor as seed is attached after each real layer. Thus, each task obtains specific virtual structure under same network parameters. Since the parameter space of the network is not changed, such a strategy will not change the fitting ability of the neural network. Experimental validations on several continual learning benchmarks show that the proposed method has significantly less forgetting and higher accuracy than existing fixed- capacity baselines. We achieve state-of-the-art performance on 20-split-CIFAN CIFAR 100 (82.22%) and -o-splitmmnnimmagNee (72.62%). In addition to minimizing forgetting, this method has comparable or better performance than single-task learning which handles all tasks individually.
2 PRELIMINARIES
Consider K sequentially arrived supervised learning tasks (Talk [K]], where [N]
(1,2,... .N} for any positive integer N. In each task Tk. there are ng data points {(TR,s Dk.i)i E [næ]] sampled from an unknown distribution Kx. Let X, Y, W be the space of inputs, targets and model parameters. By denoting the predictor as f(x, k) X x [K] + y, the loss function of 0 E W associated with data point (I, y) and task identifier can be expressed as ((f x. k), W R, and the empirical loss function of task Tr is defined as:
T2 TEA Σ 1(5(0; 2k,)),
1
(o)
In the continual learning scenario studied in this work, the parameter space W remains a fixed size, and an integer task descriptor is provided both training and testing time. Without access to past samples, we use a second-ordes Taylor expansion to estimate the loss function of previous Let 07 be the optimal parameter of (0) generated by the gradient descent process according to Vaz F L,j 0. For new model parameter @ in the neighborhood of 0j, the loss of previous task TJ( k) can be estimated as:
L,(0)= L,())) + 1 0,) 67))
where == = (03) is the Hessian matrix.
PROBLEM FORMULATION & SOLUTION
In our fixed-capacity continual learning setting, finding an appropriate joint solution that works well across the task sequence is the core goal. To this end, we introduce novel continual learning optimization problem and corresponding iterative optimization strategy.
3.1 OPTIMIZATION PROBLEM
In this paper. we formalize forgetting as the the increment of old task losses. As mentioned in Section 2 the total loss of tasks before Te. denoted as Fs, can be estimated by:
k-1 Fv(0) Σ L)( 1=1
1-1
(1) 1 10- J=1
(3)
2

---

Published as a conference paper at ICLR 2022
As Equation 3 need the explicit value of previous model parameters, Fk is too expensive to be an optimization target in continual learning. We turn to a more concise form which we refer to as recursive least loss (RLL):
R-1 FRLL F = 2 -0-- -0-0%-1) )"
1
- 1 0--1)
(4)
In Appendix A.2 we prove that FRLL and Fl equivalent for optimization if all previous tasks are fully-trained. Based on the conclusions above, the optimization problem during task T. is for- malized as:
0g: min FRLL (0), subject (L+
(5)
FELL has the same form as the regularization term in many regularization--bsse methods. The opti- mization goal of these methods are variants of L (0) XFFLL(()), derived from Bayesian posterior approximation with Gaussian prior (Kirkpatrick et al. 2017, Nguyen et al. 2018) or approxima- tion of the KL-divergence used natural gradient descent (Amari) 1998 Ritter et al. 20188] Tseran al. 2018). The Bayesian methods try to estimate and minimize the overall loss function, while our method prioritizes the performance of the current task by ()) = 0 and minimizes the expected forgetting of the past tasks Fk FRLL((().
3.2 GRADIENT MODIFICATION
For the newest task Th. the optimal solution where Vlg = 0 should be obtained by stochastic gradient descent started from the former optimal model parameter 0% the end of task T Using subscript to represent the parameters of step i, and the initial state On = 0h-1- the single step update can be expressed as:
- 7/8 LLx
Assume that the pre-set learning rate Ti is small enough to ignore the higher order terms, the loss function after the one-step update can be expressed as:
(0,))) = Lx(0-))) - (V VL
If we hope solve the task Tk only, updating the parameter 0 according to the gradient above enough. However, as mentioned above, such a method will encourage the neural network to gradually forget the old tasks. Therefore, we modify the update direction minimize the expectation of forgetting. To this end, we introduce a new positive definite symmetric matrix P with appropriate dimensions modify the gradients (g- + Pg). The modified one-step update is:
0: = 0,-1 7/ PV Lx
(o,)0)) = L. (0,
(V
To maintain the pre-set learning rate during the continual learning problem and avoid repetitive selection of hyper-parametent we impose an additional constraint on the trace of the projection matrix and prove the corresponding convergence rate consistency theorem in Appendix A.
Theorem (convergence rate consistency). Under the constraint of trace (P)=dim(P), P), the expecta- tion of the learning rate for unknown isotropic distribution is the same as the original optimizer.
For common network structures, we provide the space complexity of matrix P and the time com- plexity of the corresponding gradient modification process in Appendix B.2 As described above, the only extra memory of our approach is the projection matrix P. which contains the information of previous tasks. This allows our approach to be a space-invariant invariant method different with typical memory-based based or expansion-bal based continual learning method. As the performance of our method identified with the choice P. the following problem is: How to find good projection matrix? We will answer this question in the following parts.
3

---

Published as a conference paper at ICLR 2022
3.3 APPROXIMATE SOLUTION
The next step is to find a solution for problem 5 When the training process on Ta is finished, the final state 0g and the residual loss be obtained from the accumulation of one-step update:
ni n
0 = 0--1
PV (0,-1)
(02) I (x
Σ th (VL=(())))) PVL+
In this way, for given sample sequence and initial value -- the result of f depends on P. The optimization problem | on 0g transformed into an optimization problem P.
However, the relationship between FRLL (0t) and P is too complicated to be used in optimization process. To tackle this problem, propose upper bound of PRLL (0%) practical optimization target.
Theorem 2 (upper bound). Denote (.(+) the symbol for maximum eigenvalue and Tm the maximum single-step learning rate, the recursive least loss has an upper bound:
F (RR) S
(8)
where H =5 Hj is defined as the sum of the Hessian matrices of all old tasks.
We prove theorem 2 in Appendix mization problem for projection matrix P:
A.3
Discarding the constant terms, we get an alternative opti-
min P
6m (PH)
P:
(9)
subject to trace(P) (P) = dim(P)
We provide detailed solution of this problem in Appendix A.4
The normalized solution is:
dim H) trace H-1) H-1
P
(10)
The normalized projection procedure can be described as: finding new gradient that has similar effects on the current task to minimize the upper bound of the old task losses. Our optimizer only modifies the direction of the gradient without reducing the search region, which will guarantee the fitting ability of the network consistent throughout the task sequence.
4 IMPLEMENTATION
The main concern of our method in section Bies in the expensive space and time cost in deep neural networks. In this section, propose two approaches reduce the time and space complexity of the algorithm for models with
propagation structures.
4.1 VIRTUAL FEATURE ENCODING LAYER
In multi-layer networks, the output of the previous layer can be regarded as set of features gener- ated by the feature extractors(for example, weight matrices, bias vectors, convolution kemels, etc). In order to make the gradients generated in back-propagation process conform to our assumption coabbbttioooThoreenn T), we propose virtual Feature Encoding Layer(FEL) to apply
of isotropic ssec-iie connections to the output of the previous layer and the input of the next layer.
Definition 4.1 (Feature Encoding Layer). A feature encoding layer applies a task-specific rearrange- ment to the input feature maps, the order of which randomly generated using the task identifier as seed.

---

in all Published as a conference paper at ICLR 2022
Note that FEL is only permutation of existing feature maps, and its order does not change during the training process. Although this feature encoding layer does not require extra space, we write in matrix form for the convenience of theoretical analysis. The permutation matrix at layer I(L 1,2, is:
((( == random permutation of I with seed(k)
where is identity matrix with dimensions equal to the number of feature maps. Considering that the order of the features is critical for the next layer to obtain interpretable information, the recognition ability of the network will degrade greatly without correct feature order. FEL provides an effective way to eliminate the interference between tasks, where features encoded by a specific task descriptor will be randomly permuted in other tasks. Therefore, the same feature extractor plays different roles in different tasks. Although gradients of different layer are strongly correlated in the current task, there is little correlation from the perspective of old tasks. That means current impact on past tasks can be regarded as the independent summation of the influence from different local feature extractors.
4.2 LOCAL-GLOOBAL EQUIVALENCE
Then, during the backward propagation process of task Tki the gradient of L% intermediate layer hy can be calculated by the chain rule:
OL. 91 Ohu
L-1 aL
Shi+1 1=1 0hg
L-1
II
gL II (b)Ds j=l
(11)
dhg
where DJ+1 is a diagonal matrix representing the derivative of the nonlinear activation function. Since previous samples cannot be accessed to recalculate the gradient, common gradient-basee
methods (Li et al., 2019; Azizan et al. 2019 Farajtabar et al. 2019) assume that the joint optimal parameter lies in the neighborhood of the previous optimal parameter and use previously calculated
gradients as an approximation, which leads (0 ar) vf((";;)) and @hh ah;
We follow this
assumption and use the proposed optimizer to ensure this assumption much as possible. Thus we have:
LL yly
JhL,T 8 OhL (Shr)²
(12)
Thu
Oh,
82 Lj th
k-1 n
Oh (T"))(())) + ali
dh.
H
=
Th,
Th,
(13)
where is penalty parameter to ensure the positive definiteness of H,. We set subsequent sections according to the trace normalization proposed in Section 3
Further, decompose the global optimization problem into independent sub-problems each layer. We introduce Theorem 3 and prove it in Appendix AA.3
Theorem 3 (Local-Global Equivalence). Under the assumption of close vicinity, the global op- timization problem is equivalent to independent local optimization problems. The local optimal projection matrix of layer I is:
dim (im(Hi Hi -1 where Hi
P
HB
H3g
trace H1 -1]
4.3 ITERATIVE UPDATE
Considering that the complexity of calculating inverse matrix with dimension of is O(n²), time-consuming to calculate the inverse Hessian matrix H-1 in practice. Instead, update
5

---

Published as a conference paper at ICLR 2022
the projection matrix Pt iteratively like recursive least square(RLS) Haykinn 2002)
algorithm at
training step(see Appendix B.I for more details). This allows RGO to be an online algorithm with linear memory complexity and single-step time complexity in the number of model parameters. We summarize the gradient modification and the iterative update of the projection matrix to Algorithm
Note that feature extractors in the same layer share a projection matrix calculated by the average of the gradients considering their linear correlation. In this way, handling multiple gradients in the same layer does not increase the complexity of updating projection. We list the memory size and the single step-time complexity for different kinds of feature extractors Appendix B.2 It is worth mentioning that, because the local optimizers of different layers are independent, after obtaining the back- proppaaaton gradients, the gradient modification process of different layers can be processed in parallel to further reduce the time required.
Algorithm Learning Algorithm of Recursive Gradient Optimization
Input: Task sequence Tk, k 2, Output: optimum parameter 0%
1: Pi( +- Jr. 0 + 0g is randomly initialized.
2: for 1.2..... do
3: for (I, () ~ DD do
4:
9i = 0L2
1 1,2, ...
Stochastic/Batch Gradient of current loss
8hi
5:
= Pt. 91 -im (ci) (tr//trac(FF) Pt)
modify origin gradients update the model parameters
6:
G +- 7
7:
end for
8:
for (I, v) ~ DD do
9:
= (f(;:xx)); y)] oh
0h1
1= 1.,, < get local gradient by back-propag
22.g/((a + of Ptgr)
Pt + Pt kig Pt
Update projection matrix
12:
end for
8% <
get the optimal model parameter
14: end for
5 EXPERIMENT SETUP
Benchmarks: We evaluate the performance of our approach on four supervised continual learning benchmarks. Permuted MNIST (Goodfellow et al.] 2014 Kirkpatrick et al. 2017) and Rotated MNIST (Chaudhry et al., 2020) are variants of MNIST dataset of handwritten digits (LeCun] 1998)
with 20 tasks applying random permutations of the input pixels and random rotations of the original images respectively. Split-CIFAR 100 (Zenke et al. 2017) is random division of CIFAR 100 into 20 tasks, each task has 5 different classes. Split minilmageNet, introduced by (Chaudhry et al. 2020).
applics a similar division on a subset of the original ImageNet (Russakovsky et al. 2015 dataset.
Baselines: In this work, we perform experiments on the benchmarks above with the following fixed capacity methods and expansion-based method for comparison: (1) SGD which stochastic gradient descent optimizing procedure to finetune the model, (2) EWC (Kirkpatrick et al., 2017) which is one of the pioncering regularization methods using fisher information diagonals as important weights, (3) A A-GEM (Chaudhry et al., (019a) which uses loss gradients of stored previous data in an in-equality constrained optimization, (4) LOS (Chaudhry et al. 2020 which constraints gradients in low-rank orthogonal subspace, (5) ER-ring (Chaudhry et al. (019h) which utilizes
tiny ring memory to alleviate forgetting, (6) GPM (Saha et al., (021)) which trains new tasks in the
residual gradient subspace, (7) APD YYon et al. 2019) which is a strong expansion-based method decomposing the parameters of different tasks with a common basis, and (8) STL which trains a model for each single task. For the compared methods,
follow the original implementations to perform some necessary processing at the end of every task. The storage for memory- based methods set to 1 sample per class per task following Chaudhry al. (2020).

---

Published as a conference paper at ICLR 2022
We use average accuracy(ACC) and average accuracy decline, also called backward transfer(BWT) by Papez-Paz & Ranzato (0117). to evaluate the classification performance. De- note the accuracy of task k at the end of task T as RT ACC and BWT are defined as:
Metrics:
1 T ACC
T R RR.k
L T
RI.k:
Σ BWT T
Architectures and training details: We evaluate all of the continual learning methods for the same network architectures. The model is three-layer fully connected network with 256 hidden units in MNIST experiment and a standard ResNet18 (He et al. 2016) in CIFAR and ImageNet experiments. For RGO, we add virtual feature encoding layer attached each layer. MNIST variants are trained 1000 steps while CIFAR and minilmageNet are trained 2000 steps. Batchsize is set at 10 for all tasks. The task identifiers are provided for both training and testing time, All results are reported across 5 runs with different seeds. See Appendix C for more details.
RESULTS & DISCUSSIONS
Table 1: Performance of different methods Permuted MNIST/ Rotated MNIST/ Split Cifar 100/ Split mini- mini-Imagenet. The model is trained with 20 tasks for 5 different seeds and evaluated by final average accuracy with stds. (*) denotes methods with additional network capacity. RGO-2 is version without FEL for ablation experiments
Permuted MNIST
Rotated MNIST
Method
Replay N
ACCteat (%)
BWT(%)
ACC (%)
BWT(%)
RGO
91.15(=0:20)
-.00((+0.0)
91.25((0.01)
-1.59(#0.01)
RGO-2
N
87.95(=0.01)
-5.65(+0.38)
72.26(#0.955
-20.74(+0.01)
GPM
N
83.29(=0.01)
-8.45(#0.01)
70.02(+0.95)
-17.95(#0.01)
LOS"
N
86.56(+0.38)
..10((+0.33)
80.21(+1.11)
-13.44(+1.06)
ER-Ring
Y
79.84(+0.63)
-12.88(+0.65)
69.20(+0.79)
-2..93(+00.5)
AGEM
Y
(..32(=1.04)
-19.94(+1.02))
53.26(#1.00)
-1,.44(00.66)
EWC
N
67.79(#1.60)
-24.38(+1.53)
43.27(#0.66)
-5..74(10.76)
SGD
N
46.11(#3.91)
46.06((44.00)
44.82(+0.01)
-5..18(+0.01)
STL
N
91.33(10.20)
0.0
91.09(10.01)
0.0
Split CIFAR100
Split ImageNet
Method
Replay
ACCtest
BWT(%)
ACCtest (%)
BWT(%)
RGO
N
73.18(+0.51)
-1.67 (#0.29)
70.33(
-1.64 (=10.41)
RGO-2
N
62.82(10.98)
(+0.922
57.40(#1.90)
-22.40 (#1.89)
GPM
N
53.41(+2.87)
-2.998 (#3.14)
LOS"
56.20(+1.12)
-25.6
43.25(++2.2))
-34.75 (+-2.69)
ER-Ring
Y
53.74(+2.13)
-28.15 (+2.02)
45.88(+2.39)
-29.11 (+1.63)
AGEM
49.56(12.64)
-32.10 (#2.73)
34.67
-38.06 (#0.87)
EWC
N
47.71(+1.70))
-25.17 (#1.50)
32.61(13.67)
-24.95 (#3.662
SGD
37.02(+11.6))
44.34(+1.55)
37.69
-37.23(+0.72)
STL
70.73
0.0
67.76(L1.70)
0.0
The evolution of average accuracy is shown in Figure 1 and the final results with error bars of the indicated datasets at the end of training are reported in Table The proposed method shows a strong performance of average accuracy over the basclines on all benchmarks. The result of BWT shows that RGO can significantly reduce catastrophic forgetting especially on complex tasks and deep ar- chitectures. RGO improves upon strongest baseline considerably: 17.0% and 24.5% absolute gain average accuracy, 93.4% and 94.4% reduction in forgetting, on CIFAR 100 and minilmageNet, respectively. Meanwhile, exceeds the performance of STL which is often regarded as the upper bound of continual learning methods. The results of ablation experiments show that RGO maintains good performance without FEL, only slightly lower than LOS which has an additional task orthogonal mapping layer on Ro- tated MNIST. In Table 2
rotated MNIST and minilmageNet, we observe that our method even
list some results modified LeNet from APD (Yoon al. 2019

---

Published as a conference paper at ICLR 2022
64
21
Dhing
(a) Permuted MNIST
(b) Rotated MNIST
02
67
(c) Split C0
(d) Split minilmageNNe
Figure Evolution of average accuracy with the number of tasks during the continual learning process.
as comparison with expansion-base based methods. Contrary to the forgetting of other methods, RGO shows positive knowledge transfer and exceeds the thcoretical upper bound on the testing under a fixed network capacity.
Table 2: Average accuracy of 20-task Split- Aplit-CIFAR100 dataset with modified LeNet. Capacity de- notes the percentage of network capacity used with respect to the original network. Relative ACC represents the difference in accuracy between the corresponding method and STL under the same settings. (*) denotes results reported from APD.
Metric
PGN
DEN
RCL
APD
RGO +6.02+
relative ACC(%) Capacity
-1.324
-..0
9.01+
-4.19+0.33
271%
191%
184%
130%
100%
Further, we test our method with more architectures. As shown Table RGO achieves higher test accuracy than STL with only 5% capacity despite forgetting on the training set. RGO provides more robust features to reduce the accuracy gap between the training set and the testing set by 9% to 44%. Meanwhile, Split-CIFAR10U (20 tasks) and Split-minilmageNerr (20 tasks) respectively without a well-designed training schedule. Although RGO minimizes forgetting each local optimization problem, due to the layer-by-laye accumulation of errors, deeper structures lead to more forgetting. FEL uses random permutation to greatly reduce the coupling between layers, which plays an important role in alleviating this problem. In this perspective, shallow and wide structures may be beneficial to alleviate catastrophic forgetting in the field of continual learning.
report new state-of-the-art performance of 82.22% and 72.63% on
Table 3: Results of different architectures on 20-Split- Sifii-CCIARIOOO (00()) and 20- 20-Split-mine peglitminiilaaeNet((( A denotes the difference between training set accuracy and testing set accuracy. All experiments are trained 5 times with 20 epoches. Learning rate is set at 0,03 and 0.01 for CIFAR and minilmageNet respectively.
Single-Task Learning
Recursive Gradient Optimization
Architecture
ACC
ACC (%)
A(%)
ACCrrasi,
ACC
A(%)
BWT(())
LeNer.5"
100:0:10:00
75:01:10:32
25.0+10.3
99 99:35
81.03-10:11
1833:0.5
039:10.15
AlexNer-6"
99.90:10.03
81.60:10311
18.34:0.3
98.78:10.08
82.22-1:.24
16.6::0.2
-144::0.13
VGG-11"
98.30:10.55
79.51100.1
18.8:0666
95.17:10.08
79:81:1::222
15.41:0.3
4.00:10.119
VGG-13
95.83.10.83
75.64.10.49
20.1
91.26::11.22
77.43:10.34
15.8.1...
4.86.10.94
AlexNes-71
99.72-10.30
71.92:10.55
27.8-10.6
98. 10:10111111
72.63:10.45
25.54:0.5
.1.98.1..16
VGG-11
99.70:10.08
11.6740.11
28.04:.22
95 05.91
71.14:10.22
24.8:10.7
3.006:::223
V3G-13
97.62-10.44
67.22.100.6
30.4.0.5
92.48.1.100
66.24-11.04
26.2.10.77
4.79.0.48
ResNet-18"
97.04:10.46
68.78++1009
28.340088
86.79-40.85
71.00++0:1
15.8:10.3
-3.2040..2
8

---

Published as a conference paper at ICLR 2022
7 RELATED WORK
In this section, we present some discussions between the adopted technology with existing work. The starting point of our approach and many other coss-oonstrained approaches optimize current loss and the estimated past loss at the same time. Most of the commonly used regularization methods (Kirkpatrick et al., 2017 Zenke et al., 2017 Teng et al. 2020) use a hyperparameter to balance the current task and past tasks. The objective functions of these methods can be expressed L(0) + AF - dda). This type of method suffers from the trade-off between new tasks and old tasks and requires hyperparameter search to obtain better results. In contrast, follow the principle of current-task-firs discarding empirical trade-offs between tasks which means À = 0. In the solution space of 8 arg min L, change the path of the gradient descent process through the P matrix and find the one that minimizes F(0 Ola)) Thus, under the assumption of over over-parameters, RGO optimizes current performance and forgetting simultaneously.
A Although the starting point is different, our gradient modification process is closely related to gra- dient constraint methods like OWM (Zeng et al. 2019) and GPM (Saha al., 202T). OWM uses similar iteratively updated projectors derived from recursively least ( which regards each layer as an independent linear classifier and uses the output of the previous layer build the pro- jection matrix. This leads to layer-by-layee accumulation of errors in modern complex end-to-end network architectures. Using the gradients of the loss function directly, our approach is less worried about the depth of the network and more compatible with existing auto-grad frameworks. For single-layer linear classifier = Wæ, OWM and RGO are equivalent considering that By aw x. In addition, an extra normalization procedure in our method guarantees the learning rate of the current task. This brings an additional advantage that of original single-task models. GPM (Saha et al. 202T) projects the gradient of each layer into a lower-dimensional residual space of previous tasks, while the parameter space of RGO is consistent for different tasks. RGO will maintain the network's ' fitting ability as the number tasks increases. In addition, our method is not limited to the instability of SVD decomposition and does not require hyperparameter selection.
method can reuse the hyperparameters
Task encoding layer has been used to reduce interference between tasks in recent works like LOS Chauddhry et al. (220)) and HAT (Serra et al. (2018), which require additional network capacity. On the contrary, FEL is only permutation of the input corresponding to the task id. This provides an efficient task encoding and decoupling method which can be easily integrated into other continual learning methods.
LIMITATION
Like all methods based on gradient constraint, analyses in this paper are based on neighborhood assumption and wverpaarameterized assumption which may not be satisfied in some narrower net- works. When the number of tasks is close to the minimum number of channels in the network, this assumption fails. Although we have empirically proved that this effect is not obvious under common experimental settings, attention should be paid to the width of the network in applications.
CONCLUSION
In this paper, we propose new recursive gradient optimization method find the optimal parame- ters of fixed capacity networks, and a new feature encoding strategy to characterize the structure of the network. The feature encoding layer and the optimizer to minimize forgetting are both compati- ble with typical learning models, which allows our approach to be a general method to add continual learning capability into the vast majority of the existing network architectures learned by variants of gradient descent, with only constant times of memory/time cost than typical back- proppprrpaggtion algorithms. The theoretical derivation and experimental results show that RGO is currently the op- timal approach under the current-task-finn first principle and quadratic loss estimation for fixed capacity networks. Experiments demonstrate that RGO achieves significantly better performance than other -fate-of-the-arr methods on variety of benchmarks. Without restrictions the network structure and loss form, RGO has broad prospects in combination with other continuous learning methods and applications in other representation learning fields.
9

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We give the reproducible source code in the supplementary materials, and introduce the implementa- tion of the baseline method in Appendix C.1 C.1 See Appendix [o the selection of hyperparameters. In Python3. and TensorFls Lowl 4, all results can be reproduced. The theorems put forward in the main text have corresponding proof's in Appendix A
ACKNOWLEDGEMENT
The authors with the Beijing National Research Center for Information Science and Technology, Institute for Artificial Intelligence, and the Department Computer Science and Technology. Ts- inghua University, Beijing, 100084, China. This work supported by National Natural Science Fund for Key International Collaboration (62120106005).
REFERENCES
Ferran Alet, Tomás Lozano-Pérez, and Leslie P. Kaelbling. Modular meta-learning. (CoRL), 2018. wttpg///..///..ooo////../ 10166
URL
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network experts. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 7a1777aaauary712200111 2017. doi: 10.1109/CVPR.2017..55
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
Navid Azizan, Sahin Lale, and Babak Hassibi. Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization. pp. 1-35, 2019. URL htt.hht::/aaxx...oogggo abs/ /1t///9955 1906.990000 03830
Michael B. Chang, Sergey Levine, Abhishek Gupta, and Thomas L. Griffiths. Automatically com- posing representation transformations means for generalization. 7th International Conference on Learning Representations, ICLR 2019, pp. 1-23, 2019.
Arslan Chaudhry, Ranzato Mare' Aurelio, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. 7th International Conference Learning Representations, ICLR 2019, pp. 1-20, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Puneet K. Dokania, Philip H.S. Torr, Thalaiyasingam Ajanthan, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv, pp. 1-15, 2019b. ISSN 23318422.
Arslan Chaudhry, Naeemullah Khan, Puneet K. Dokania, and Philip H. S. Tom. Continual Learning in Low-rank Orthogonal Subspaces. (NeurIPS):1-12, 2020. URL WoLLhtttp://arvd.orr/////
20100.1663
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal Gradient Descent for Con- tinual Learning. 2019. URL (tRp:/ttt://arxiit
/ab//b////// 910.9.0000 07104
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution Channels Gradient Descent in Super Neural Networks, 2017. http:/:://////ao///////////
Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical in- vestigation of catastrophic forgetting in gradient- based neural networks. 2nd International Con- ference on Leurning Representations, ICLR 2014 Conference Track Proceedings. 2014.
Simon Haykin. Adaptive Filter Theory. Prentice Hall, 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 20166-ecemmhe:7770-788.. 2016. ISSN 10636919. doi: 10.1 0..1109/CVPR.2016.020
10

---

Published as a conference paper at ICLR 2022
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan Yuille. Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping. 2021.
URL http: www.vi.org///a. 11333
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka BarabskaBBarwinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114(13):32213322 2017. ISSN 10916490. doi: 10.1073/pnas. 161 1835114
Richard Larsen and Morris Marx. An introduction mathematical statistics. 2005.
Prentice Hall,
Yann LeCun. The mnist database of handwritten digits. 1998. URL httt:://yann.Lecm
com hte//excbmmmi/
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to Grow: A Con- tinual Structure Learning Framework for Overcoming Catastrophic Forgetting. 2019. URL
www.:////gbbbb/./..aa.... 003100
David Lopez-Paz and Marc Aurelio Ranzato. Gradient episodic memory for continual learning- Advances in Neural Information Processing Systems, 20T--DccmmeerViissx666-444777 2017.
ISSN 10495258.
Cuong V. Nguyen, Yingzhen Li, Thang D) Bui, and Richard E. Turner. Variational continual learn- ing. 6th International Conference on Learning Representations, ICLR 2018 Conference Track Proceedings, (Vi):1-18, 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances Neural Information Processing Systems, 2018- Decemmer::3338-7488, 2018. ISSN 10495258.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. 6th International Conference Learning Represen- tations, ICLR 2018 Conference Track Proceedings, pp. 1-16, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-25) 2015, ISSN 1473 doi:
URL
http://dss ottp:://k.co...0g/1100000000 wttp:/......11111100001111.00110
015- 011-26633--1000-1---------------
Andrei A. Rusu, Neil C. Rabinowizz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. 2016. URL
http: www.wi...rgrbbb..e hwtp:///g/aa///////o
04671
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient Projection Memory for Continual Learning. 09762
(2018):1-18, 2021. URL http:///wwwxxx..raa///oo///aoss/
Joan Serra, Dídac Suris, Marius Mirón, and Alexandras Karatzoglou. Overcoming Catastrophic forgetting with hard attention to the task. 35th International Conference on Machine Learning, ICML 2018, 10:7225-7234, 2018. ISSN 2640-3498.
Gehui Shen, Song Zhang, Xiang Chen, and Zhi Hong Deng. Generative Feature Replay with Or- thogonal Weight Modification for Continual Learning. arXiv, 2020.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in Neural Information Processing Systems, Dew.eeebeeeiiss)299991--1- 2017. ISSN 10495258.
11

---

Published as a conference paper at ICLR 2022
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pp. 1-14,20 2015.
Yunfei Teng, Anna Choromanska, and Murray Campbell. Continual learning with direction- constrained optimization. 2020. URL http:://axiiv org/ abs/aassssi 22111.258111111
Hanna Tseran, Mohammad Emtiyaz Khan, Tatsuya Harada, and Thang Bui. Natural Variational Continual Learning. Continual Learning Workshop at the 32nd Conference on Neural Information Processing Systems (NeurIPS), (NeurIPS):1-5, 2018.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Contin- ual learning with hypernetworks. pp. 1-28, 2019. URL http:::/acxw..oo////aaaaa/a obb.s/999../////19...
00695
Dong Yin, Mehrdad Farajtabar, and Ang Li. SOLA: Continual learning with second-order loss approximation. arXiv, pp. 1-18, 2020.
Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018 Conference Track Proceedings, pp. 1-11, 2018.
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and Order-robust Continual Learning with Additive Parameter Decomposition. pp. 1-15, 2019. URL. http:
//arxiv.crg/ant /1002 09432
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependen pro- cessing in neural networks. Nature Machine Intelligence, 1(8):364-372, 2019. doi: 10. 1038/ s42256-019-0080-X
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. 34th International Conference on Machine Learning, ICML. 2017, 8:6072-6082, 2017.
APPENDIX
Section A provides proofs of theorems introduced the paper. Section provides some details and approximations used in the implementation. Section C provides the details of codes, architectures, hyperparameters and resources used in the experiment.
A PROOFS
A.1 PROOF OF THEOREM
1
As we have mentioned, the distribution of elements of the gradients of future tasks are assumed be isotropic, and this assumption is guaranteed by our random encoding strategy under different tasks. For isotropic distribution, we have:
Lemma 4 (Distribution Consistency). Isotropism invariant under orthogonal transformation.
Larsen & Marx 2005)
Then, we introduce a lemma on trace of matrix which is often used in matrix analysis Horn & Johnson 2012):
Lemma (Trace Consistency). Trace of matrix is consistent under orthogonal transformation.
As described in Section 3. at i-th single train step during task k, we have:
{ 0, P,1 -..PV
(14)
Lx(0.) (+(0,-1) - (VV
12

---

Published as a conference paper at ICLR 2022
For any potential gradient, the mathematical expectation of the loss function decline is:
E - (
(15)
the optimizer degrades to original single task optimizer when / equals identify matrix. As posi- tive definite symmetric matrix can be orthogonal diagonalized by a orthogonal matrix V,
VPV = d1, ... )n) =
(16)
where } represent the eigen values of the Projection Matrix P. Mark dim(P) and (Lx with n and æ = (I1, Ø2. ... ,,,) respectively. We assume the distribution of unknown future gradients isotropic and apply Lemma to the expectation:
B(( PVL = Ex
V ApVx] = [x Aao
E
i== x7
B
[+]]
(17)
n
Σ
R [x x]
V
n
n
According to Lemma 5 the sum of the eigen values of Inverse Hessian Matrix P equals to that of A, that means:
A trace(P)
(18)
Thus if we add normalize constraint trace(P) dPm((P) the Inverse Hessian matrix P, every task in the continual learning procedure can have consistent expectation convergence rate.
A.2 PROOF OF LOSS FUNCTION EQUIVALENCE
Theorem (Loss Equivalence). If f arg ming FRLL(()), using F F.RL or Fk as the loss to optimize is equivalent, which means arg ming F FRLL = arg ming F& (0)
We first expand Fj in Equation at the initial state # during Tg:
k-1 ΣLL + 10 -0-0 - 0/-1 0 07)l
(o)
1=1
(19)
k-2 Σ 11 07)1 + 1 10 0z11)² H2 02-1) + const.
1
(0 - 1
For further analysis, we first introduce the following lemma which can be proved inductively.
Lemma 7. if (E/=11 H,) 0%-1) 0 holds for any k, then Σ (0) 07)1 0 also holds for any k.
Proof. We mark DH-H(H, 07)] D(k), then for k 1,2,3,... have
13

---

Published as a conference paper at ICLR 2022
D(k) - D(K - 1)
Σ (0% 07)1 Σ
H (0 -8)]]
k-2
(k - 01-1) +
Σ H2) -1-
j=1
k-1 (0) 0%-)
j=1
= 0
Using the fact that D(0) = 0 (k) = 0 holds for all positive integer k.
Ignoring the constant terms, the key to prove this theorem the second term of the expression of Fki
Proof. If VELLL (01) satisfies for all l E [K], then we can get
(O)
= =0
e [K]
Using the result of Lemma 7 we have
k-2
Σ 11,00011-011-000 vk e e [K]
1=1
Substituting this formula into Equation and discarding the constant terms lead to
Fe = Fk (R)) Ak E [K]
Thus, {ERLLIK Je and (FA) are equivalent loss sequences throughout the continual learning process.
Note that this equivalence is obtained in the case of second-order approximation of the loss func- tions of tasks, it also depends on the "close vicinity" hypothesis.
A.3 PROOF OF THEOREM 2
Lemma (Cauchy Inequality). For any positive integer n. positive symmetric definite matrix P and vectors {v.),, P denote the P-norm of v by ||l||e = Vou Pu, we have:
U
Denote T..()) (-) the symbol for finding maximum eigenvalue and nm the maximum single-step learning rate, the recursive least loss has an upper bound:
1 nk
Bk
F
n V (L.()))))
Σ H
n
L
(0--1)
2
(20)
nh
<
ô m
P
H
VL (L
mA
VL
1=1
According to Lemma
we have
14

---

Published as a conference paper at ICLR 2022
k-1
nR
FF FRLL((0))
6m(P(
Hj)1 Th
2
j=1
(21)
2 T1 om IP
H3)
Th PV
Considering that La (01) 2 the end training and loss for current task before training can be expressed as La (0%-1). the change of loss function described in Equation 7 satisfies the following inequality:
La (0 2
(O PV (0:1i)
(22)
Then we can complete the proof by combining 21 with 22
FRLL((A))
2 nm ā [P
(
(23)
A.4 SOLUTION OF PROBLEM 9
At fixed point of the model, total Hessian matrix H = ΣØ= Hj must be positive definite like
P. As positive definite symmetric matrix can be orthogonal diagonalized by an orthogonal matrix
(Hom & Johnson 2012), H and P can be expressed as H U Un P = VVV, while An 11, nnj and Ap = (*,,... )n are diagonal matrices.
Using the diagonalization step, the optimization problem can be expressed as:
min om (V (V"APUU
(24)
8.l. trace (P) = dim (P)
orthogonal transformation maintains eigen value (Horn & Johnson 2012), which means:
8. (V APVU
8. (UV A AR)
There two main variable to be optimized, diagonal matrix Ap and orthogonal matrix UVT, To simplify the derivation, we set UV to the simplest orthogonal matrix I. Under this assumption, the optimization problem is simplified as:
min A
max oide
(25)
s.t.
A = n
1=1
We get the optimal cigen values:
n
A
(26)
I a
Thus,
dim( 1
P= V=
VA T
(27)
ΣΣ d
trace(H-1)
15

---

Published as a conference paper at ICLR 2022
A.5
PROOF OF THEOREM 3
Denote the gradient of total parameter set and the total Hessian matrix as (of .92 ... " 8 La
and { 00 Jr,
L. Th, have:
L P.gr 96 Aa VU? A A
T Pt
A U.V," Ap, Vig
of Vi A
(28)
5m (A A A Hi
66m (A A2, VI" A... Vigr
Lz P.gr
E
of Pt
90
802 9o
Thr
IstrSL
Σ
om pA Jo (A Hr Ap, Ap,)9 V/ A -A Vigr
(29)
S max (6m (AnAH))
Σ of V. A A... ISTrEL
max: @ (Ap, Ar, 1190 Pgs
The optimization problem above has same form the global optimization problem in Section A.4 the solution can be easily got as:
dim Hi) Hi where Hi
P. trace H4 "
Σ Hzg
j=1
B IMPLEMENTATION DETAILS
B. DETAILS QUADRATIC ESTIMATION OF THE HESSIAN MATRIX
For the C-class classification problems, f(;;; 0) has C-logits associated to different classes We consider the most commonly used softmax cross entropy loss which is defined
My. (x; = Σ vylog(a,)
(30)
where a, czp(ff 0))/ Ee exp(fe(æ; 0)) as the j-th softmax output. The (i.j)-th element of the second derivative matrix of the loss function with respect to f(r; () is then calculated as
(u: fax 0)),s ajois -aaj
(31)
where Φi.j is Dirac function equal to 1 while = j else
As a symmetric diagonally dominant matrix, (y; far 0)) has its matrix root [(( fax 0))] This guarantees the correctness of our algorithm. implementation, for convenience, we only used the diagonal element corresponding to the ground truth label for an estimation.
B.2 TIME& & MEMORY COMPLEXITY OF RGO
We list the shape of projection matrix and time complexity of projection matrix update and gradient modification introduced by RGO for some typical feature extractors below:
First, according to Algorithm] I the time complexity of updating P is obviously O(dim(P))) The main concern comes from the matrix-mann matrix product in g' Pg for gs with higher dimension. However, we notice the linear correlation of the columns of g. we can avoid this matrix multipli- cation. Use fully connected layer y =W b. x E R y IR
example. Considering
16

---

Published as a conference paper at ICLR 2022
Kind
Shape
Size of P
time complexity
vector
(N1, n1)
matrix
(11.
(11. R1)
kernel
(mi, n2, ksize, ksize)
(ksizeeP1, ksizemm)
ksize mi
x"" If we calculate from left to right instead of calculating al OL x² we have g' = P BW By
9 first, we can avoid matrix multiplication and reduce the number of calculations from n1n2 + -nin to 11122 + nf. The amount of calculation beyond the original backpropagation is only nj. The calculation process for kernels the same except for a reshape process.
Considering that both the kernel size and have upper bounds in common neural network models, the time complexity of RGO remains the same as that of backpropagation.
C EXPERIMENT DETAILS
C.I BASELINE IMPLEMENTATIONS
EWC KKrrkpatrick et 2017 LOS (Chaudhry et al. 2020). A-GEM (Chaudhry al. 2019a), and
ER- ER-ring (Chaudhry et al. b019b) are implemented from adapting the code provided by Chaudhry al. (2020) under MIT License. GPM is implemented from the official implementation provided by Saha et al. (2021) under MIT License.
C.2 RESOURCES
All experiments of our method are completed in several hours with pieces of Avidia-208011 GPUs.
C.3 ARCHITECTURES
We provide details of architectures we used in the experiment section.
LeNet-5: A modified LeNet used by Yoon et al. (011)). There are two convolutional layers
with kernels size of (5,5) and channels of (20,50), followed by two hidden fully connected layer with (800,500) units.
AlexNet-6: A modified AlexNet used by Saha et al. (2021). There are three convolutional layers with kernels size of (4,3,2) and channels of (64, 22,,26,,,,), followed by two hidden fully connected layer with (2048,2048) units.
AlexNet-7: A modified AlexNet. There are four convolutional layers with kernels size of (5,4,3.3) and channels of (64,122,118,11)) followed by two hidden fully connected layer with (2048,2048) units.
VGG-11&VGG-13+ Original VGG11 and VGG13 proposed by (Simonyan & Zisserman
(0015).
ResNet- 18: A standard 18-layer ResNet proposed by He et al. (2016). For our approach, we remove all batch-norm layers because their parameters not updated by gradient descent.
LeNet-like and AlexNet-like architectures are attached a 2x2 maxpooling layer after each convolu- tional layer.
C.4 HYPERPARAMETERS
The learning rates of all baselines are generated by hyperparameter search in 0.003.001,00033.011..3...1 to achieve better results. Other hyperparameters of EWC, A-GEM. ER-Ring and LOS follows Chaudhry et al. (2020), while those of GPM and APD follows their official implementation.
Single Task Learning
learningrate: (.I(MNIS 0.03(CIFAR100, minilmageNet)
17

---

Published as a conference paper at ICLR 2022
Recursive Gradient Optimization(Ours)
learningrate: 0. 0.I(MNIST), 0.03(CIFARI00, minilmageNet 2000steps), 0.01 (minilm- ageNet 20epochs)
SGD
learningrate: 0.I(MNIST), 0.03(CIFARI00, minilmageNet)
EWC
learningrate: 0.I(MNIST), 0,03(CI0,RR00, minilmageNNe)
regularization: 10(NNIST, CIFARI00, minilmageNet)
A-GEM
learningrate: 0.I(MNIST), 0.03(CIFAR100, minilmageNet)
ER-Ring
learningrate: 0.I(MNIST), 0.03(CIFARI00, minilmageNet)
LOS
learningrate: O.I(MIIST) 0,4(CIFAR100) (.(mm)illmage)NN))))
GPM
learningrate: 0.I(MNIST), 0.03(CIFARI00, minilmageNet)
threshold: 0,95 for first layer and 0.99 for other lMyeIs(MNIST)) jincrease from 0.97 to I(CIFAR), increase from 0.985 to I (minilmageNNet
-dieension of representation matrices: 300(MNIST), 125(CIFAR), 100(minilma- geNet)
18


Reviews:
[{"review_id": "qD0uR1_N_m", "reviewer": "Reviewer_Ztre", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides an empirically very strong method for task-based continual learning, as measured by average task accuracy, alongside a solid theoretical derivation and justification for it, which will be of interest to the continual learning community and for these reasons I am recommending it for acceptance. The main weakness of the method is that it has a runtime that is quadratic in the number of parameters at each layer, as a result of the gradient projection at each update; the paper could greatly benefit from an empirical analysis of the runtime compared to competing methods in order to quantify this limitation. \n\nPositives:\n- Very strong empirical performance on standard CL benchmarks, exhibiting essentially no forgetting across 20 tasks.\n- The method is simple to implement.\n- Ensuring that the expected learning rate on the current task is preserved (via setting trace(P)=dim(P) and with the feature encoding layers) reduces the need for hyperparameter tuning, in contrast to other regularization methods that often require tuning of hyperparameters that control the balance of progress on the current task and preservation of previous task performance.\n- Despite the quadratic runtime in the number of parameters per layer, the projections of the gradients at each layer are independent of each other and so can be computed in parallel (once the gradients have been calculated via backprop).\n- While the method requires knowledge of task ids for the random permutation layers, these only require storage of a single integer seed each. It would be interesting to investigate how this method could be extended to a task-free setting.\n- Good discussion of related methods\n\nPotential areas for improvement / questions:\n- The runtime of the algorithm is quadratic in the number of parameters per layer, which could be concerning given that it is acknowledged in the paper that the method requires wide enough network layers to satisfy the overparameterisation assumption. A runtime comparison across methods should be conducted in order to quantify the differences.\n- It’s a bit difficult to understand what the method does from the introduction. It could do with a diagram.\n- In Table 1, shouldn’t the values in the BWT be negative as in Table 3?\n\nTypos:\n- Equation 4 typo: $\\theta_{k-1^*}$ -> $\\theta_{k-1}^*$\n- Section 4 title: “Implemention” -> “Implementation”\n- 4.1 Line 2 “matrixs” -> “matrices”\n- Definition 4.1 “applys” -> “applies”\n", "summary_of_the_paper": "This paper introduces a gradient-based approach to continual learning in neural networks called Recursive Gradient Optimization (RGO). RGO modifies the gradient direction at each update by multiplying it with a projection matrix P that is designed to minimize the increase in loss on previously encountered tasks. RGO is theoretically designed to prioritise performance on the current task and, among the optimal solutions for the current task, find the one that causes least interference with previous tasks - thus it assumes that the network is overparameterised. The derivation of the method starts by approximating the continual learning loss with the “recursive least loss”, which involves a sum of the Hessians of the previous tasks and assumes that tasks are fully trained and their solutions are close by to each other. It is then shown that the recursive least loss can be upper-bounded by an expression involving the projection matrix P, via which the solution for P that minimizes this upper bound is derived. It is shown that the expectation of the step size can be preserved by guaranteeing that trace(P)=dim(P) and by introducing random task-specific permutations at each layer - this preserves the “current-task-first” principle and reduces the need for hyperparameter tuning. Experiments are run on a number of standard continual learning image classification benchmarks, demonstrating an extremely strong performance of RGO in comparison to competing methods, sometimes surpassing the performance of a baseline that trains a separate model for each task.", "summary_of_the_review": "This paper provides an empirically very strong method for task-based continual learning, as measured by average task accuracy, alongside a solid theoretical derivation and justification for it, which will be of interest to the continual learning community and for these reasons I am recommending it for acceptance. The main weakness of the method is that it has a runtime that is quadratic in the number of parameters at each layer, as a result of the gradient projection at each update; the paper could greatly benefit from an empirical analysis of the runtime compared to competing methods in order to quantify this limitation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cU6BpuEnaBM", "reviewer": "Reviewer_oYYz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this is a nice idea and a good application of it. It is relevant for ICLR, and the idea is novel (specifically, as far as I am aware, no one has previously used results such as Theorem 1 and Feature Encoding Layers). It is nice to be able to modify the gradient on a new task with a projection matrix, while minimising forgetting on past tasks. The experiments seem comprehensive in range (many benchmarks, many architectures). \n\nIn general, the paper introduces the algorithm well and the order of information seems nice. I focus the rest of the review on various ways I think the paper could be improved further. \n\n1. The authors argue that \"Single-Task Learning ... can be regarded as the strongest expansion-based method\" (Introduction), and compare a lot to STL (in the abstract and experiments). I disagree that STL is the strongest baseline, as there is no potential for forward or backward transfer between tasks with STL. In fact, many algorithms can beat STL for this exact reason (STL has even sometimes been considered as a 'lower bound', see for example [1] and the Split CIFAR task they have, which is slightly different to the one here). Overall, I believe STL is a strong method, but it is not quite right to say it is the \"strongest\", and perhaps the authors should not emphasise comparing so much to it throughout the paper.\n\n2. Throughout the paper, there are missing references to many Bayesian-inspired approaches to continual learning. Examples include EWC [2], online EWC [3], VCL [4,5], Online Structured Laplace [6], FROMP [1], and many many more. These are usually regularisation-based approaches ([1] also uses memory). They are based off Bayes equation for continual learning. A key benefit is that Equations 4 and 5 naturally \"drop out\" of applying Bayes equation (with the Laplace approximation), and I am not sure the proofs (such as Appendix A.2) are required -- people should already see that Equation 5 holds from Bayes (in particular, online EWC / online structured Laplace use the same equation). A Bayesian view also justifies the use of alpha in Section 4.2, where alpha is the prior precision (or weight-decay parameter when training the 1st task). There is of course a key difference between the Bayesian-inspired approaches and this work: the Bayesian approaches minimise the overall loss together L_k + F_k^RLL, whereas this work minimises F_k^RLL subject to \\grad L_k = 0. The authors can make this clear when discussing the Bayesian approaches.\n\n3. Modifying the direction of gradients is closely related to natural-gradient descent (Amari, 1998) and mirror-descent. See for example NCL [7] where they discuss some of this.\n\n4. I believe a term is missing from Equation 4 (bottom line): $L_{k-1}(\\theta_{k-1}^*)$\n\n5. The authors re-use hyperparameters from previous work for baselines. Unfortunately baselines such as EWC are known to be sensitive to hyperparameter values (for EWC, this is the regularisation parameter lambda) when the benchmark and/or architecture changes. It therefore seems unfair to report results based off one value. It would be good if the authors ran a hyperparameter sweep for the baselines too for fairness, and reported in the text that additional computational effort was needed. I still expect RGO to significantly outperform baselines.\n\n6. In Appendix B.1, what do the authors mean by \"we only used the diagonal element corresponding to the ground truth label for estimation\". Why did the authors not use Equation 31 itself? I thought Equation 31 was cheap to calculate and use (requiring a forward pass only through the model).\n\n7. [Minor point] It was interesting for me to see that \"deeper structures lead to more forgetting\" (page 8). I would love to see further investigations / intuition as to why this is the case with RGO (or maybe in general). Perhaps this can be future work.\n\nTypos:\n- Section 2: the authors first introduce W as the space of model parameters, but for the rest of the paper use only theta.\n- Section 4.1 page 4: \"matrixs\" -> \"matrices\", \"applys\" -> \"applies\"\n- Algorithm 1 line 9: missing \"l=1,2,...,L\"\n- Table 2: please say the full names of all the metrics/methods reported in this table, along with references to the papers. Perhaps this information can go in the caption.\n\n\n\nReferences\n\n[1] Pan et al., 2020, \"Continual deep learning by functional regularisation of memorable past\"\n[2] Kirkpatrick et al., 2017\n[3] Schwarz et al., 2018, \"Progress & Compress...\"\n[4] Nguyen et al., 2018, \"Variational Continual Learning\"\n[5] Loo et al., 2021, \"Generalized Variational Continual Learning\"\n[6] Ritter et al., \"Online structured Laplace approximations...\"\n[7] Kao et al., 2021, \"Natural continual learning...\"", "summary_of_the_paper": "The paper introduces a new method, Recursive Gradient Optimization (RGO), for continual learning in the task-incremental scenario. This method modifies the direction of gradients on a new task in order to minimise forgetting on previous tasks, and unlike many previous works, does not require storing past raw data to do so. The authors introduce a Feature Encoding Layer to achieve this. The authors provide experiments on 4 benchmarks of varying size (MNIST to miniImageNet), showing good performance of their algorithm, with different architectures.", "summary_of_the_review": "Overall, I am leaning towards accept as I like the idea and the results are very strong. I have provided potential ways to improve the paper in my main review, specifically points 2 and 5.\n\n*****************\nAfter rebuttal: I have increased my score to 8, thanks to the authors rephrasing certain parts / adding certain comments/references to the paper. I believe the paper could still be increased by tuning hyperparameters of baselines, however, this does not detract from the quality of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Wlw2XBY5XCS", "reviewer": "Reviewer_CQxg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper strengths:\n* Continual learning is an important, challenging and practical problem which the authors have addressed. The problem is formulated as tackling forgetting (increment of old task losses) with the constraint of learning optimally for the new tasks.\n* The theoretical analysis of the problem (forgetting as increment of old task losses, its approximation to more tractable RLL and further to its upperbound) is done very good.\n* FEL with task specific random permutation of features is interesting and shown to be effective by decorrelating the features between different tasks but with strong correlation inside individual tasks.\n* Being a theoretically motivated work, the authors have done a very good job in clearly motivating the different steps with derivations and explanations as and when needed.\n* The experiments show the superiority of the approach very good.\n\nPaper weaknesses:\n* This is regarding eqn. (4). I think I got the major steps of this derivation. It uses the definition of $F_{k-1}(\\theta^*_{k-1})$ and the fact that Hessian is symmetric. However, I wonder if there would be an extra $L_{k-1}(\\theta^*_{k-1})$ term at the end. It would be helpful if a more detailed derivation is provided for.\n* Though appendix B.2 provides space complexity of projection matrix, I am interested on the advantage of RLL formulation in terms of the capacity of the network. $F_k$ in (4) and $F^{RLL}_k$ in (5) both need to store all $H_j$, I see. Any comment specific to network capacity in relation to this would be good to have.\n* Minor typos: First line of eqn. (4) one of the $\\theta$s have the asterisk as subscript. Page 4 last paragraph and section 5 first paragraph – ‘applys’ should ‘applies’.", "summary_of_the_paper": "The paper addresses a long-standing problem of deep neural networks where new training data for new tasks arrive continuously and data does not stay forever (continual learning). The network has to learn to cater for all the tasks without forgetting what it learnt for an older task as data from older tasks are no longer available. The authors came up with a formulation where learning will mean minimizing the (expectation of) forgetting and forgetting is formalized as increase of losses corresponding to the old tasks. The authors then move on to find a Taylor series expression of forgetting and an equivalent Recursive Least Loss (RLL) formulation of the loss (aka forgetting). Next the authors introduce a modification in the direction of gradient update of normal SGD that makes sure to minimize forgetting. The modification implied the introduction of a positive definite matrix and the optimization problem transformed into an optimization problem on finding the optimal positive definite matrix minimizing the RLL. As the relation between RLL and the positive definite matrix is not straightforward, an upper bound of RLL is proposed and the optimal solution is found under this upper bound. One more interesting proposal is a virtual feature encoding layer (FEL) that applies task-specific random rearrangement to the input feature maps. FEL eliminates possible interference between tasks by decorrelating features among old tasks without expanding the memory footprint. The experiments show great improvement over sota approaches on benchmark datasets and sometimes (rotated MNIST, Split ImageNet) even goes beyond supposed upperbound performance given by single task learning showing evidence of positive transfer.  Overall, I liked the approach, the presentation and the experimental results with only a few minor concerns detailed below.", "summary_of_the_review": "In summary, the paper presents a crisp solution to a challenging and important problem with mostly clear writing and experiments. Some of the concepts introduced here (e.g., gradient direction change and FEL) can be useful not only in continual learning research but applications where forgetting can be an issue or decorrelating features can be advantageous.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7mCFkLMN4-K", "reviewer": "Reviewer_DhW7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- I agree with the validity of the Taylor expansion in equation (2). However, I find the assumption in Eq(3), that \\theta is in the vicinity of each of \\theta_j^*, doubtful. This is as if we say that the optimal parameters for each task are in the vicinity of each other which would abolish the problem of catastrophic forgetting from the beginning.\n- The approach seems to depend on two main new components (1) The new projected matrix P, and (2) the FEL layer. I am still unsure which of the two reduces forgetting. The authors are encouraged to define two new baselines each of which incorporates one of the aforementioned components.\n- There is a discrepancy between Algorithm 1 and the previously derived quantities. Concretely, the square root in line 9, and I am also missing the inverse of the sum of Hessians, H_l, in Algorithm 1.\n- Is there a missing sum in the first equation of (14)?\n", "summary_of_the_paper": "The paper proposes a continual learning approach based on recursive gradient optimization. To this end, a projection matrix is derived for the gradient modification. This matrix, P, is computed incrementally and updated by integrating the Hessian on each task locally.\n", "summary_of_the_review": "I have two issues with (i) the assumptions made in parts of the derivations, and (ii) the missing important baselines that should be derived from your approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an innovative method for continual learning that modifies the direction of gradients on a new task to minimise forgetting on previous tasks without data replay. The method is mathematically rigorous with a strong theoretical analysis and excellent empirical results across multiple continual learning benchmarks. It is a clear accept. There was good discussion between the reviewers and authors that addressed a number of minor issues, including clarifying that the method has the same computational complexity as backpropagation. The authors are encouraged to make sure that these points are addressed in the final version of the paper.
Decision: Accept (Spotlight)