Gradient Projection Memory for Continual Learning
Authors: Gobinda Saha
Isha Garg
Kaushik Roy
Venue: ICLR 2021 Oral (2021)
Keywords: Continual Learning
Representation Learning
Computer Vision
Deep learning

Abstract: The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches. 

Published as a conference paper at ICLR 2021
GRADIENT PROJECTION MEMORY FOR CONTINUAL LEARNING
Gobinda Saha, Isha Garg & Kaushik Roy School of Electrical and Computer Engineering, Purdue University gsaha@purdue, edu, gargi @urrdue. edu, kaushi wetsshik@puuee képurdue.
ABSTRACT
The ability learn continually without forgetting the past tasks is desired at- tribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network represen- tations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Pro- jection Memory (GPM). With qualitative and quantitative analyses, such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse im- age classification datasets with short and long sequences of tasks and report better or on-par performance compared the state-of-the-ar approaches"
show that
INTRODUCTION
Humans exhibit remarkable ability in continual adaptation and learning new tasks throughout their lifetime while maintaining the knowledge gained from past experiences. In stark contrast, Artifi- cial Neural Networks (ANNs) under such Continual Learning (CL. paradigm (Ring) 1998: Thrun & Mitchell 1995 Lange et al. (02I) forget the information learned in the past tasks upon learning new ones. This phenomenon is known as 'Catastrophic Forgetting' or 'Catastrophic Interference" (Mc- closkey & Cohen 1989 Ratclifi 1990). The problem is rooted in the general optimization meth- ods (Goodfellow et al. (0166) that are being used to encode input data distribution into the parametric task, gradient- based opti-
representation of the network during training. Upon exposure to mization methods, without any constraint, change the learned encoding to minimize the objective function with respect to the current data distribution. Such parametric updates lead to forgetting.
Given a fixed capacity network, one way to address this problem is to put constraints on the gra- dient updates so that task specific knowledge can be preserved. To this end, Kirkpatrick et al. (017)) Zenke et al. (2017), Allunni et al. 2 Seerå et al. (2018) add a penalty term to the objec- tive function while optimizing for new task. Such term acts as structural regularizer and dictates the degree of cability-plasticityy of individual weights. Though these methods provide resource effi- cient solution to the catastrophic forgetting problem, their performance suffer while learning longer task sequence and when task identity is unavailable during inference.
Approaches (Lopez-Paz & Ranzato 2017 Chaudhry et al. 2019a) that store episodic memories of old data essentially solve an optimization problem with 'expiicit" constraints on the new gradient directions so that losses for the old task do not increase. Chaudhry et al. (2019b) the performance of old task is retained by taking gradient steps in the average gradient direction obtained from the new data and memory samples. To minimize interference, Farajtabar et al.] (2020) store gradient directions (instead of data) of the old tasks and optimize the network in the orthogonal directions to these gradients for the new task, whereas ZZeg et al. (2018) update gradients orthogonal to the old input directions using projector matrices calculated iteratively during training. However, these methods either compromise data privacy by storing raw data or utilize resources poorly, which limits their scalability.
Our code is available at httpa://github..... www.hwww......aaaaaa.i.i..c.ccca

---

Published as a conference paper at ICLR 2021
In this paper, address the problem of catastrophic forgetting in fixed capacity network when data from the old tasks are not available. To mitigate forgetting, our approach puts explicit constraints take. However, unlike contemporary methods,
on the gradient directions that the optimizer neither store old gradient directions nor store old examples for generating reference directions. In- stead we propose approach that, after learning each task, partitions the entire gradient space of the weights into two orthogonal subspaces: Core Gradient Space (CGS) and Residual Gradient Space (RGS) (Saha et al. 2020). Leveraging the relationship between the input and the gradient spaces, we show how learned representations (activations) form the bases of these gradient subspaces in both fully-comnected and convolutional networks. Using Singular Value Decomposition (SVD) on these activations, we show how to obtain the minimum set of bases of the CGS by which past knowledge preserved and learnability for the new tasks is ensured. We store these bases the memory which we define as Gradient Projection Memory (GPM). In our method, we propose to learn any new task by taking gradient steps in the orthogonal direction to the space (CGS) spanned by the GPM. Our analysis shows that such orthogonal gradient descent induces minimum to no interference with the old learning, and thus effective in alleviating catastrophic forgetting. We evaluate our approach in the context of image classification with minilmageNet, CIFAR-100, PMNIST and sequence of 5-Datasets on a variety of network architectures including ResNet. We compare our method with related state-of-the-art approaches and report comparable better classification performance. Over- all, we show that our method is memory efficient and scalable to complex dataset with longer task sequence while preserving data privacy.
2 RELATED WORKS
Approaches to continual learning for ANNs can be broadly divided into three categories. In this section we present detailed discussion on the representative works from each category, highlighting their contributions and differences with our approach.
Expansion-based methods: Methods in this category overcome catastrophic forgetting by dedicat- ing different subsets of network parameters to each task. With no constraint on network architec- ture, Progressive Neural Network (PGN) (Rusu et al.) (016)) preserves old knowledge by freezing the base model and adding new sub-networks with lateral connections for each new task. Dynam- ically Expandable Networks (DEN) (Yoon et al. 2018) either retrains or expands the network by splitiing/dullicating important units on new tasks, whereas Sarwar et al. (2020) grow the network to learn new tasks while sharing part of the base network. [i et al. (2019) with neural architecture search (NAS) find optimal network structures for each sequential task. RCL. (Xu & Zhu 2018) adap- tively expands the network at each layer using reinforcement learning, whereas APD (Yoon et al,) 2020) additively decomposes the parameters into shared and task specific parameters to minimize the increase in the network complexity. In contrast, our method avoids network growth or expensive NAS operations and performs sequential learning within fixed network architecture
methods: These methods attempt to overcome forgetting in fixed capacity model through structural regularization which penalizes major changes in the parameters that were important for the previous tasks. Elastic Weight Consolidation (EWC) (Kirkpatrick et al. (017) computes such importance from diagonal of Fisher information matrix after training. whereas Zenke jet al. (2017) compute them during training based on loss sensitivity with respect to the parameters. Additionally, Aljundi et al. (0118) compute importance from sensitivity of model outputs to the inputs. Other methods, such as PackNet (Mallya & Lazebnik 2018) uses iterative pruning to fully restrict gradient updates on important weights via binary mask, whereas HAT (Serrà al.) 2018) identifies important neurons by learning attention masks that control gradient propagation in the individual parameters. [Saha et al.] (2020) using a PCA based pruning on activations (Garg et al., 2020) partition the parametric space of the weights (filters) into core and residual (filter) spaces after learning each task. The past knowledge is preserved in the frozen core space, whereas the residual space is updated when learning the next task. In contrast to these methods, we do not ascribe importance to or restrict the gradients of any individual parameters or filters. Rather we put constraints on the 'direction' of gradient descent.
Regularization-
Memory-based methods: Methods under this class mitigate forgetting by either storing a subset of (raw) examples from the past tasks in the memory for rehearsal (Robins)
1995] Rebuffi et al.) 2017 Lopez-Paz & Ranzato [017] Chaudhry et al.) 2019a]b] Riemer et al.
2019 or synthesiz-
2017). For instance, ing old data from generative models to perform pseudo-rehearsal (Shin et al.
2

---

Published as a conference paper at ICLR 2021
Gradient Episodic Memory (GEM) (oopez-Pazz & Ranzato 2017) avoids interference with previous task by projecting the new gradients in the feasible region outlined by previous task gradients cal- culated from the samples of episodic memory. Averaged GEM (A-GEM) (Chaudhry al. (019a) simplified this optimization problem to projection in one direction estimated by randomly selected samples from the memory. Guo al. (2020) propose unified view of episodic memory- CL. methods, that include GEM and A-GEM and improves performance over these methods utilizing loss-balancing update rule. Additionally, Experience Replay (ER) (Chaudhry et al.] 2019b) and Meta-Exper Eete-Exxeeeeece Replay (MER) (Riemer et al., 2019) mitigate forgetting in online CL setup by jointly training on the samples from new tasks and episodic memory. All these methods, however, rely on the access to old data which might not be possible when users have concern over data pri- vacy. Like all the memory-based methods we also use storage unit which we call GPM. However, we do not save any raw data in GPM, thus satisfy data privacy criterion.
Our method is closely related to recently proposed Orthogonal Gradient Descent (OGD) (Farajtabar et al. (020) and Orthogonal Weight Modulation (OWM) (Zeng et al. 2018). OGD stores set of gradient directions in the memory for each task and minimizes catastrophic forgetting by taking gradient steps in the orthogonal directions for new tasks. In contrast to OGD, compute and store the bases of core gradient space from network representations (activations) which reduces the memory requirement by orders of magnitude. Moreover, OGD is shown to work under locality assumption for small learning rates which limits its scalability in learning longer task sequences with complex dataset. Since our method does not use gradient directions (like OGD) describe the core gradient spaces, we do not need to obey such assumptions, thus can use higher learning rates. On the other hand, OWM reduces forgetting by modifying the weights of the network in the orthogonal to the input directions of the past tasks. This is achieved by multiplying new gradients with projector matrices. These matrices are computed from the stored past projectors and the inputs with recursive least square (RLS) method at each training step. However, such iterative method not only slows down the training process but also shows limited scalability in end-to-end task learning with modern network architectures. Like OWM, we aim to encode new learning in the orthogonal to the old input directions. In contrast to iterative projector computation in OWM, we identify a low-dimensional subspace in the gradient space analyzing the learned representations with SVD in one-shot manner at the end of each task. We store the bases of these subspaces in GPM and learn new tasks in the orthogonal to these spaces to protect old knowledge. We quantitatively show that our method is memory efficient, fast and scalable to deeper networks for complex long sequence of tasks.
NOTATIONS AND BACKGROUND
In this section, we introduce the notations used throughout the paper and give brief overview of SVD for matrix approximation. In section[4 we establish the relationship between input and gradient spaces. In section 5 we show the steps of our algorithm that leverage such relationship.
Continual Learning: We consider supervised learning setup where T tasks are learned sequen- tially. Each task has a task descriptor, E 21. T} with a corresponding dataset, DD- {(T)rr Mi. )== } having n, example pairs. Let's consider an L layer neural network where at each layer network computes the following function for task r:
a |+1 (WWW .r(,))). 7,
(1)
Here, 1- 1,..,, (.) is a non-linear function and f(.--) is a linear function. We will use vector notation for input (xFrr) in fully connected layers and matrix notation for input (Xi,)) in convolu- tional layers. At the first layer, ,r,, R,,r represents the raw input data from task r, whereas in the subsequent layers we define at,-- as the representation of input Hi,r at layer 1. Set of parameters of the network is defined by, W, = ((W)) where Wo denotes set of parameters at initialization.
Matrix approximation with SVD: SVD can be used factorize rectangular matrix, A UEV E Ru into the product of three matrices, where U E Rmxm and V E RRaxn are orthogonal, and Σ contains the sorted singular values along its main diagonal (Deisenroth et al. 2020f. If the rank of the matrix is r (r < minnm, n)), A can be expressed as A = =>= u. where u, E U and v, E V are left and right singular vectors and o; E diag( ()) are singular values. Also, k-rank approximation to this matrix can be expressed as, A Σ>- o. where k < r and its value can be chosen by the smallest & that satisfies || I12 2 Eth & ||] Here, |1.11r is the Frobenius norm of the matrix and 6th. (0 < eth 1) is the threshold hyperparameter.
3

---

Published as a conference paper at ICLR 2021
(a) Forward Poss
Backward Pres
C
cxkak
C
Pdl p.l
c
W
D
hy Wa
X
p.P.
P
Input
Weight
Output
Input
Eror
Weight Gradient
Figure 1: Illustration of convolution operation in matrix multiplication format during (a) Forward
Pass and (b) Backward Pass.
INPUT AND GRADIENT SPACES
Our algorithm leverages the fact that stochastic gradient descent (SGD) updates lie in the span of input data points (Zang et al. (017). In the following subsections we will establish this relationship for both fully connected and convolutional layers. The analysis presented in this section generally applicable to any layer of the network for any task, and hence we drop the task and layer identifiers.
4.1 FULLY CONNECTED LAYER
Let's consider single layer linear neural network in supervised learning setup where each (input, label) training data pair comes from training dataset, D. Let, E IR" is the input vector, R the label vector in the dataset and W E IR "**x are the parameters (weights) of the network. The network is trained by minimizing the following mean-squared error loss function
L -
Wa yl
We can express gradient of this loss with respect to weights
V (Wa) y)æ² 6x=".
(3)
where E Rm is the error vector. Thus, the gradient update will lie in the span of input (æ), where elements in ō scale the magnitude of by different factors. Here, have considered per- example loss (batch size of 1) for simplicity. However, this relation also holds for mini-batch setting (see appendix B. The input-gradient relation in equation 3 is generically applicable any fully connected layer of neural network where æ is the input to that layer and ò is the error coming from the next layer. Moreover, this equation also holds for network with non-linear units (e.g. ReLU) and cross-entropy losses except the calculation of will be different.
4.2 CONVOLUTIONAL LAYER
Filters in a convolutional (Conv) layer operate in a different way on the inputs than the weights in fully connected (FC) layer. Let's consider Conv layer with the input tensor X' E R C x and filters W E C x xxxx xxxk Their convolution (X, W, *) produces output feature map, O RC.xh, xwa (Liu et al. 2018). Here, C; (C.) denotes the number of input (output) channels of the Conv layer, ht, w; (ho, W.) denote the height and width of the input (output) feature maps and is the kernel size of the filters. As shown in Figure I(),, if X' is reshaped into a (h, w.) (C; x k x k) matrix, X and W is reshaped into (C, x & x k) x C, matrix, W, then the convolution can be expressed as matrix multiplication between X and W as O=XW, where O E (ho xwn) XC. Each row of X contains an input patch vector, P; E R (xxxkx where 1, 2..., (n h w.))
Formulation of convolution in terms of matrix multiplication provides an intuitive picture of the gradient computation during backpropagation. Similar to the FC layer case, in Conv layer, during backward pass error matrix A of size (hg x wo) x C, (same size as O) is obtained from the next layer. As shown in Figure bb, the gradient of loss with respect to filter weights is calculated by
V
where, Vw L is of shape (C; x k k) C. (same size W). Since, columns of T the input patch vectors (p), the gradient updates of the convolutional filters will in the space spanned by these patch vectors.
4

---

Published as a conference paper at ICLR 2021
5 CONTINUAL LEARNING WITH GRADIENT PROJECTION MEMORY (GPM)
In this section, we describe our continual learning algorithm which leverages the relationship be- Iween gradient and input spaces identify the core gradient spaces of the past tasks. We show how gradient descent orthogonal to these spaces enable us to learn continually without forgetting.
Learning Task 1:
We learn the first task (T 1) using dataset, D3 without imposing any constraint on parameter updates. At the end of Task 1, we obtain a learned set of parameters W1. To preserve the knowledge of the learned task, we impose constraints on the direction of gradient updates for the next tasks. To do so, we partition the entire gradient space into two (orthogonal) subspaces: Core Gradient Space (CGS) and Residual Gradient Space (RGS), such that gradient steps along CGS induce high interference on the learned tasks whereas gradient steps along RGS have minimum to no interference. We aim to find and store the bases of the CGS and take gradient steps orthogonal to the CGS for the next task. In our formulation, each layer has its own CGS.
To find the bases, after learning Task 1 for each layer we construct a representation matrix, R² (z],, 1: 2221 ..... "I,.1l (for Conv layers Rj [(X,,) )" (X2,)²² " (X,_,) 1)²) concatenating n2 representations along the column obtained from forward pass of n, random samples from the current
training dataset through the network. Next, we perform SVD on R U/Ei (V'))" followed by its
k-rank approximation (R)), according to the following criteria for the given threshold, ch
[I(R))al1 1.
(5)
We define the space, s' = span ponnfuu,, i 22 i....... 2/ 1. spanned by the first k vectors in U as the space of significant representation for task at layer I since it contains all the directions with highest singular values the representation. For the next task, we aim to take gradient steps in a way that the correlation between this task specific significant representation and the weights in each layer is preserved. Since, inputs span the space of gradient descent (section 4 the bases of S will span a subspace in the gradient space which we define as the Core Gradient space (CGS). Thus gradient descent along CGS will cause maximum change in the input-weight correlation whereas gradient steps in the orthogonal directions to CGS (space of low representational significance) will induce very small to no interference to the old tasks. We define this subspace orthogonal to CGS as Residual Gradient space (RGS). We save the bases of the CGS in the memory, M = {(M') >= }. where M
[ui,1 22,1: "..,,,,/,. u We define this memory as Gradient Projection Memory (GPM).
Learning Task 2 to T: We learn task 2 with the examples from dataset Do only. Before taking gradient step, bases of the CGS are retrieved from GPM. New gradients (V (V I22) are first projected onto the CGS and then projected components are subtracted out from the new gradient so that remaining gradient components lie in the space orthogonal to CGS. Gradients are updated as
FC Layer: Vwg V w/ T-2 (V (2)
Conv Layer:
VwJT2 V w M
At the end of the task 2 training. we update the GPM with new task mask-specific bases (of CGS). To obtain such bases, we construct R2 = [= 2222: 21 using data from task 2 only. However, before performing SVD and subsequent k-rank approximation, from R2 we eliminate the common directions (bases) that are already present in the GPM so that newly added bases are unique and orthogonal the existing bases in the memory. To do so, we perform the following step:
R R (M (R)) = R2} R Proj
(8)
Afterwards, SVD is performed on RC (= ان (V))²) and & new orthogonal bases are chosen for minimum value of k satisfying the following criteria for the given threshold, dh.
R m 112 + |1 11 2 & 1R
GPM is updated by adding new bases M' [M 24, 2/22)- Thus after learning each new task, CGS grows and RGS becomes smaller, where maximum size of '' (hence the dimension of the gradient bases) is fixed by the choice of initial network architecture. Once the GPM update is complete we move on to the next task and repeat the same procedure that we followed for task 2. The pseudo-code of the algorithm is given in Algorithm in the appendix.
5

---

Published as a conference paper at ICLR 2021
6 EXPERIMENTAL SETUP
Datasets: We evaluate our continual learning algorithm on Permuted MNIST (PMNIST) (Le-
cun et al. 1998), 10-Split CIFAR-100 (Krizhevsky) 2009), 20-Spilt minilmageNet (Vinyals al.) 2016) and sequence of 5-Datasets (Ebrahimi et al. 2020b). The PMNIST dataset is a variant of MNIST dataset where each task is considered as a random permutation of the original MNIST pixels. For PMNIST, we create 10 sequential tasks using different permutations where each task has 10 classes (Ebrahimi et al. 2020a). The 10-Split CIFAR-100 classes of CIFAR-100 into 10 tasks with 10 classes per task. Whereas, 20-Spilt minilmageNet, used in (Chaddhry et al., 2019a). is constructed by splitting 100 classes of minilmageNeet into 20 sequen- tial tasks where each task has 5 classes. Finally, we sequence of 5-Datasets including CIFAR- 10, MNIST, SVHN (Netzer et al. 2011). notMNIST (Bulatov] 2011) and Fashion MNIST (Xiao jet al. 2017). where classification each dataset is considered task. In our experiments do not use any data augmentation. The dataset statistics are given in Table & S
constructed by splitting 100
in the appendix.
Network Architecture: We use fully-connected network with two hidden layer of 100 units each for PMNIST following Lopez-Paz & Ranzato| (2017). For experiments with split CIFAR-100 we use a 5-layer AlexNet similar to [Serrà et al. (2018). For split minilmageNet and 5-Datasets, similar to Chaudhry et al. (2019b), we use reduced R architecture. No bias units are used and batch normalization parameters are learned for the first task and shared with all the other tasks (follow-
Malla & Lazebnik (2018)) Details on architectures are given in the appendix section C.2 For permuted MNIST, we evaluate and compare our algorithm in 'single-head' setting (Hsu et al. 2018]] Farquhar& & Gal, 2018) where all tasks share the final classifier layer and inference is performed without task hint. For all other experiments, we evaluate our algorithm 'muti-head' setting, where each task has a separate classifier which no gradient constraint is imposed during learning.
We compare our method with state-of-the art approaches from both memory based and regularization based methods that consider sequential task learning in fixed network archi- tecture. From memory based approach, we compare with Experience Replay with reservoir sam- pling (ER Res) (Chaudhry et al. (019b). Gradient Episodic Memory (GEM) (Lopez-Paz & Ran- zato 2017), Averaged GEM (A-GEM) (Chaudhry et al.] 2019a), Orthogonal Gradient Descent (OGD) (Farattabar et al., 2020) and Orthogonal Weight Modulation (OWM) (Zeng et al.) 2018
Baselines:
Moreover, we compare with sate-of-thhe -f-te-r-t HAT (Serrà et al.) 2018) baseline and Elastic Weight Consolidation (EWC) (Kirkpatrick et al.) 2017) from regularization based methods. Additionally, we add 'multitask' baseline where all the tasks are learned jointly using the entire dataset at once in a single network. Multitask is not a continual learning strategy but will serve as upper bound on average accuracy on all tasks. Details on the implementation along with the hyperparameters considered for each of these baselines are provided in section C.4 and Table in the appendix.
Training Details: We train all the models with plain stochastic gradient descent (SGD). For each task in PMNIST and split minilmageNet we train the network for 5 and 10 epochs respectively with batch size of 10. In Split CIFAR- 100 and 5-Datasets experiments, we train each task for maximum of 200 and 100 epochs respectively with the early termination strategy based on the validation loss as proposed in [Serr et al. 2018). For both datasets, batch size is set to 64. For GEM, A-GEM and ER_ Res the episodic memory size is chosen to be approximately the same size as the maximum GPM size (GPM Max). Calculation of GPM size is given in Table in the appendix. Moreover, selection of the threshold values (4th) in our method is discussed in section C.5 in the appendix.
To evaluate the classification performance, we use the ACC metric, which is the average test classification accuracy of all tasks. To measure the forgetting we report backward transfer, BWT which indicates the influence of new learning on the past knowledge. For instance, negative BWT indicates (catastrophic) forgetting. Formally, ACC and BWT are defined as:
Performance Metrics:
T ACC = Σ RT, BWT T Σ
1 T-1
RT,
Ry..-
(10)
Here, T is the total number of sequential tasks and Ryg is the accuracy of the model on zth task after learning the T task sequentially (Lopez-Paz & Ranzato 2017).
7 RESULTS AND DISCUSSIONS
Single-head inference with PMNIST: First, we evaluate our algorithm in single-head setup for 10 sequential PMNIST tasks. In this setup task hint is not necessary. As HAT cannot perform infer-
6

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
(d)
(c)
102
1,00
1.00
1.0
1.0
OGD
0.75
0.75
GEM
10
0.50
0.50
0.5
0.5
ER, Res
A-GEM
0.25
0.25
Max
10
GPM
0.00
Memory
Memory
Meroory
Memory
Figure 2: (a) Memory utilization and (b) per epoch training time for PMNIST tasks for different methods. Memory utilization for different approaches for (c) CIFAR-100, (,, (d) minilmageNet and (e) 5-Datasets tasks. For memory, size of GPM and for time, method with highest complexity is used as references (value of 1). All the other methods are reported relative to these references.
Table Continual learning learning on different datasets. Methods that do not adhere to CL setup is indicated by (*). All the results are (re) produced by us and averaged over 5 runs. Standard deviations are the appendix.
reported in Table and 9
(a)
(b)
PMNIST
CIFAR-I00
minilmageNet
5-Datasets
Methods
ACC (%)
BWT
Methods
ACC (%)
BWT
ACC (%)
BWT
ACC (%)
BWT
OGD
82.56
-0.14
OWM
50.94
-0.30
OWM
90.71
-0.01
EWC
68.80
-0.02
52.01
-0.12
88.64
-..4
GEM
83.38
-0.15
HAT
72.06
-0.00
59.78
-0.03
91.32
-0.01
A-GEM
83.56
0.14
A-GEM
63.98
0.15
57.24
-0,12
84,04
-0.12
ER_Res
87.24
0.11
ER_Res
71.73
0.06
58.94
-0.77
88.31
0,04
EWC
89.97
0.04
GPM (ours)
72.48
0.00
60.41
0.000
91.22
0.01
GPM (ours)
93.91
-0.03
Multitask*
79.58
69.46
91.54
Multitask* 96.70
ence without task hint, is not included in the comparison. Since network size is very small (0.1M parameters) with 87% parameters in the first layer, we choose threshold value (Etx) of 0.95 for that layer and 0.99 for the other layers to ensure better learnability. From the results, shown in Table Ta), we observe that our method (GPM) achieves best average accuracy (93.91 + 0.16%). In addition, we achieve least amount of forgetting, except OWM, which essentially trades off accuracy to mini- mize forgetting. Figure aa)) compares the memory utilization of all the memory-based approaches. While OWM, GEM, A-GEM and ER Res use memory of size GPM _Mxx, we obtain better per- formance by using only 69% of the GPM_Max. Morcover, compared to OGD, we about 400 times lower memory and achieve ~ 10% better accuracy. In Figure b)), we compare the per epoch training time of different memory based methods and found method to be the fastest primarily due to the precomputation of the reference gradient bases (of CGS). Additionally, in single-epoch setting (Lopez-Paz & Ranzato 2017). shown Table in the appendix, we obtain best average accuracy (91.74 0.15%), which demonstrates the potential for our algorithm in online CL setup.
Split CIFAR-100: Next, we switch multi-head setup which enables baselines such as HAT. For ten split CIFAR- 100 tasks, as shown in T[b) we outperform all the memory based approaches while using 45% less memory (Figure (c)). We also outperform EWC and our accuracy is marginally better than HAT while achieving zero forgetting. Also, we obtain 20% better accuracy than OWM, which have high forgetting (BWT= 0.30) thus demonstrating its limited scalability to convolutional architectures.
compare with strong
Split minilmageNet: With this experiment, we test the scalability of our algorithm deeper net- work (ResNet18) for long task sequence from minilmageNe dataset. The average accuracies for different methods after learning 20 sequential tasks are given Table [bb. Again, this case outperform A-GEM, ER_Res and EWC using 76% of the GPM_Max (Figure d)). Also, achieve marginally better accuracy than HAT, however unlike HAT (and other methods) we com- pletely avoid forgetting (BWT=0000). Moreover, compared other methods sequential learning in our method is more stable, which means accuracy the past tasks have minimum to no degradation over the course of learning (shown for task accuracy in Figure 4 in the appendix).
5-Datasets: Next, we validate our approach on learning across diverse datasets, where classification on each dataset is treated task. Even in this challenging setting, shown in in Table Tb).

---

Published as a conference paper at ICLR 2021
Table 2: Total wall-clock training time measured on single GPU after learning all the tasks.
(a)
(b)
Training Time [s]
Training Time
Methods
PMNIST
Methods
CIFAR- 100
minilmageNet
5 Datasets
OGD
1638
OWM
1856
OWM
396
EWC
1352
4138
7613
GEM
1639
HAT
1248
3077
7246
A-GEM
445
A-GEM
2678
6069
12077
ER Res
259
ER Res
1147
2775
7015
EWC
645
GPM (ours)
770
3387
5008
GPM (ours)
245
Table 3: Continual learning of 20-task from CIFAR-100 Superclass dataset. (i) denotes the result reported from APD. (*) indicates the methods that do not adhere to CL setup. Single-task learning (STL), where separate network in trained for each task, serves upper bound accuracy.
Methods
Metric
STLi*
PGN1
DEN
RCLÍ
APD?
GPM (ours)
ACC (%)
61.00
50.76
51.10
51.99
56.81
57.72
Capacity (%)
2000
271
191
184
130
100
we achieve better accuracy (91.22 + 0. 0.20%) then A-GEM, ER_Res and EWC utilizing 78% of the GPM-Max (Figure e)). Though, HAT performs marginally better than our method, both HAT and achieve the lowest BWT (-0.01). In this experiment, each other. After learning 5 such tasks 78% of the gradient space is already constrained. Which implies, if the tasks are less non-similar, GPM will get populated faster and reach its maximum capacity after which no new learning will be possible. Since, we use a fixed capacity network and the size of GPM is determined by the network architecture, the ability of learning sequences of hundreds of such tasks with our method will be limited by the chosen network capacity.
have used tasks that are less related to
Training Time. Table 2 shows the total training time for all the sequential tasks for different al- gorithms. This includes time spent for memory management for the memory-based based methods, fisher importance calculation for EWC and learning activation masks for HAT. Details of time measure- ment are given in appendix section C.6 For PMNIST, CIFAR- 100, and 5-dataset tasks our algorithm trains faster than all the other baselines while spending only 0.2%, 3% and 6% of its total training time in GPM update (using SVD) respectively. Since each minilmageNet tasks are trained for only 10 epochs, our method have relatively higher overhead (30% of the total time) due to GPM update, thus runs bit slower than the fastest ER Res. Overall, our formulation uses GPM bases and projec- tion matrices reasonable dimensions (see appendix section C.&); precomputation of which at the start of each task leads to fast per-epoch training. This gain in time essentially compensates for the extra time required for the GPM update, which is done only once per task, enabling fast training.
Comparison with Expansion-based methods. To compare method with the state-of-the-art pansion based methods we perform experiment with 20-task CIFAR- 100 Superclass dataset (Yoon) al. 2020). In this experiment, each task contains different but semantically related classes from CIFAR- dataset. Similar to APD, here we use the LeNet-5 architecture. Details of architec- ture and training setup are given in appendix section CK3 Results are shown in Table 3 where ACC represents average accuracy over 5 different task sequences (used in APD) and Capacity denotes percentage of network capacity used with respect to the original network. We outperform all the CL methods achieving best average accuracy (57.73 + 0.37%) with BWT of -0.01 using the small- est network. For instance, we outperform RCL. and APD utilizing 84% and 30% fewer network parameters respectively, which shows that method induces more sharing between tasks.
Overall, we outperform the memory-based methods with less memory utilization, achieve better accuracy than expansion-basse -baed methods using smaller network, and obtain better or on-par per- formance compared to HAT in the given experimental setups. However, in the incremnnceeettall learning setup (Rebuffi et al. 2017), method (Kamra et al. 2017) that uses data replay achieves better performance than GPM (see experiment in appendix section D.3). In this setup, we believe a subset of old data replay either from storage via generation is inevitable for attaining better per-
8

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
tv.=0.5
75
0.5
6.y00.7
6n
Accuracy (%)
60
Er- 0.7
count
4
E.
3
tm
45
<v 0
Exy50.95
2
En
95
30
Eu- D. 0.95
15
-15
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
ACC
BWT
activation
activation
Figure 3: Histograms of interference activations as a function of threshold, (Eth) at (a) Conv layer (b) FC layer for split CIFAR- tasks. (c) Impact of éx, on ACC (%) and BWT(%). With increas- ing value of Ech. spread of interference reduces, which improves accuracy and reduces forgetting.
formance with minimal forgetting (Rajasegaran et al. 2019). In that quest, a hybrid approach such as combining GPM with small data replay would be an interesting direction for future exploration.
Finally, we discuss the factors that implicitly or explicitly control the amount of forgetting in our algorithm. As discussed in section 5 we propose to minimize interfer- ence by taking gradient steps orthogonal to the CGS, where CGS bases are computed such that space of significant representations of the past tasks can be well approximated by these bases. The degree of this approximation is controlled by the threshold hyperparameter, 6th (through equation 9.. For instance, low value of Eth. (closer to 0) would allow the optimizer to change the weights along the directions where past data has higher representational sigmificance, thereby significantly altering the past input-weight correlation inducing (catastrophic) interference. On the other hand, high value of Ge (closer to 1) would preserve such correlation, however learnability of the new task might suffer due high volume of constraints in the gradient space. Therefore, in our continual learning algorithm, εeh mediates the stability-plasticity dilemma. To show this analytically, let's consider a network after learning T sequential tasks with weights of the network at any layer, I expressed as T-1
Controlling Forgetting:
W+ = W +
Σ i= I
AW! = W + AW
(11)
Here, W is the weights after task and A AWl Weight update with our method ensures that A W! 1-+ is the change of weights from task 1 to T. 1--r lie in the orthogonal space of the data (rep-
resentations) of task 1, Linear operation at layer with data from task 1 (æ1) would produce: Wiaa = a + AWi after learning task T will be the same as the output after learning task (i.e. W/ae = W ). that means no interference for task 1. We define AWi_r as the interference activation for task at layer I (for any task, T < T: AW! AW!-T')) As discussed above, degree of such interference is dictated by Ceb- Figure 3a) (bb (and Figure 5 appendix) show histograms (distributions) of inter- ference activations at each layer of the network for split CIFAR-100 experiment. For lower value of ech, these distributions have higher variance (spread) implying high interference, whereas with increasing value of ctk. the variance reduces around the (zero) mean value. As a direct consequence, as shown in Figure (c). backward transfer reduces for increasing Eth, with improvement in accuracy.
If /WWi-pe = 0. then the output of the network for task data
CONCLUSION
In this paper we propose novel continual learning algorithm that finds important gradient sub- spaces for the past tasks and minimizes catastrophic forgetting by taking gradient steps orthogonal to these subspaces when learning a new task. We show how to analyse the network representations to obtain minimum number of bases of these subspaces by which past information is preserved and lcarnability for the new tasks is ensured, Evaluation on diverse image classification tasks with differ- ent network architectures and comparisons with mate-of-the-at algorithms show the effectiveness of our approach in achieving high classification performance while mitigating forgetting. We also show our algorithm is fast, makes efficient use of memory and is capable of learning long sequence of tasks in deeper networks preserving data privacy.
ACKNOWLEDGMENTS
This work was supported in part by the National Science Foundation, Vannevar Bush Faculty Fel- lowship, Army Research Office, MURI, and by Center for Brain Inspired Computing (C-BRIC), one of six centers in JUMP, a Semiconductor Research Corporation program sponsored by DARPA.

---

Published as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In The European Conference on Com- puter Vision (ECCV), pp. 139-154, 2018.
Mchdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for con- tinual learning with orthogonal gradient descent. ArXiv, abs/2006. 11942, 2020.
Yaroslav Bulatov. Notmmist dataset. Google (Books/OCR), Tech. Rep.[Online), 2011. URL BRLhhtp:
//yaroslavvb wlwgspot......... it 20t//201//033 www.nnottnnnttm.m dataset html
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. ArXiv, abs/1902.n 10466, 2019b.
Mare Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong. Mathematics for Machine Learning- Cambridge University Press, 2020.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guide continual learning with bayesian neural networks. In International Conference on Learning Rep- resentations, 2020a.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In The European Conference on Computer Vision (ECCV), 2020b.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. ArXiv, abs/1910.00 104, 2020.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. ArXiv, 0sss//80509933,, 2018.
Isha Garg, Priyadarshini Panda, and Kaushik Roy. low effort approach to structured cnn design using pea. IEEE Access, 8:1347-1360, 2020.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume MIT Press, 2016.
Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning. In Advances in Neural Information Processing Systems 33, 2020.
Yen- Cen-Chang Hsu, Yen- Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learn- ing scenarios: categorization and case for strong baselines. In NeurIPS Continual learning Workshop, 2018.
org abtp://///Ti/ /1810 0 12488
Nitin Kamra. Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. ArXiv, abs/1710.10368, 2017.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis. Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEFE Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2021.
10

---

Published as a conference paper at ICLR 2021
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278 2324, 1998.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
PMLR, 09-1099-110000 15 Jun 2019.
Zhenhua Liu, Jizheng Xu, Xiulian Peng, and Ruiqin Xiong. Frequency-domate dynamic pruning for convolutional neural networks. In Advances in Neural Information Processing Systems 31, pp- 1043-1053. 2018
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765- 7773, 2018.
Michael Mocloskey and Neil J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 24: 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Jathushan Rajasegaran, Munawar Hayat, Salman H Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for continual learning. In Advances in Neural Information Processing Systems, volume 32, pp. 12669-12679, 2019.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97 2:285-308, 1990.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representations, 2019.
Mark B. Ring. Child: A first step towards continual learning. In Learning Learn, 1998.
Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 7:123- 146, 1995.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, abs/1606.04671, 2016.
Gobinda Saha, Isha Garg, Aayush Ankit, and Kaushik Roy. Space: Structured compression and sharing of representational space for continual learning. ArXiv, abs/2001 086500, 2020.
Syed Shakib Sarwar, Aayush Ankit, and Kaushik Roy. Incremental learning in deep convolutional neural networks using partial network sharing. IEÉE Access, 46661546666,, 2020.
Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4548-454 PMLR, 10-15 Jul 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, volume 30, 2017.
Sebastian Thrun and Tom M. Mitchell. Lifelong robot learning. Robotics Auton, Syst., 15:25 46, 1995.
11

---

Published as a conference paper at ICLR 2021
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, vol- ume 29, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. ArXiv, abs/1708.007477, 2017.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Pro- cessing Systems, volume 31, pp. 899-908, 2018.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, 2018.
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust tinual learning with additive parameter decomposition. In International Conference on Learning Representations, 2020.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continuous learning of context-dependent processing in neural networks. ArXiv, 1is/1810.0126,, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelli- gence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995. PMLR, 06-11 Aug 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
B ALGORITHM
B.1 INPUT AND GRADIENT SPACES (CONT.)
Since, the batch loss is the summation of the losses due individual examples, the total batch loss for samples can be expressed as
12
12
Lbatch
W x; 1:
(12)
The gradient of this loss with respect to weights can be expressed as
V Lbatoh = 81 òι + 8222 +... + dn
(13)
The gradient update will remain in the subspace spanned by the n input examples.
B.2 ALGORITIM PSEUDO CODE
Algorithm Algorithm for Continual Learning with GPM
1: function TRAIN (Sw, Dtrain
2: Initialize, M < []. for all = 1,2, till L-1 if multi-hear setting
3: M < {(M') 1=1
4: W < Wo
5: for E 1,2, do
6: repeat
7: Bn Dtrain // sample a mini-batch of size from task
8: gradient, W, + SGD(Bn,Jw)
9: L,
LW,, see equation 6 W + W WV
10:
until convergence
12:
I/ Update Memory (GPM)
14:
Br, Dérain // sample mini-batch of size n, from task T
15: I/ construct representation matrices for each layer by forward pass (section 5}
16:
R. + forward B,,. fw). where R, [R.)
17:
for layer, / = 1,2, L do
18:
R. - PROJECT(FE) (R. M.) // see equation g Ú + SVD(R.)
19:
20: k criterial R RR,,, // see equation
M + [M Ú k:]l
22:
end for
end for
24: return fw.M
25: end function
C EXPERIMENTAL DETAILS
C.1 DATASET STATISTICS
Table 4 and Table show the summary of the datasets used in the experiments.
13

---

Published as a conference paper at ICLR 2021
Table 4: Dataset Statistics.
PMNIST
Split CIFAR-100
Split-minimage-Net
num. of tasks
10
20
input size
1 28 x 28
x 32 32
3 84 84
Classes/task
10
10
Training samples/tasks
54,000
4,750
2,375
# Validation Samplcs/tasks
6,000
250
125
Test samples/tasks
10,000
1,000
500
Table 5-Datasets Statisties. For the datasets with monochromatic images, we replicate the image
across all RGB channels so that size of each image becomes 3 32 32.
CIFAR-10
MNIST
SVHN
Fashion MNIST
notMNIST
Classes
10
10
# Training samples
47,500
57,000
69,595
57,000
16,011
Validation Samples
2,500
3,000
3.662
3,000
# Test samples
10,000
10,000
26,032
10,000
1,873
C.2 ARCHITECTURE DETAILS
AlexNet-like architecture: This is the same architecture used by [Serr al. (2018) with batch normalization added each layer except the classifier layer. The network consists of convolutional layers of 64, 128, and 256 filters with 4 4, 3, and kernel sizes, respectively, plus two fully connected layers of 2048 units each. Rectified linear units is used as activations, and 2 x max-pooling after the convolutional layers. Dropout of 0.2 is used for the first layers and 0.5 for the rest.
Reduced ResNet18 architecture: This is the similar architecture used by Lopez-Paz Ranzato (2017). For minilmageNet experiment, we use convolution with stride 2 in the first layer. For both minilmageNet and 5-Datasets experiments we replace the 4 x 4 seerage-pooling before classifier layer with 2 x average- pooaaggppooing.
All the networks use ReLU in the hidden units and softmax with cross entropy loss in the final layer.
C.3 CIFAR-100 SUPERCLASS EXPERIMENT
For this experiment, similar to APD Yoon al. 2020), with 20-50-800-500 neurons. All the baseline results are reported from APD. Like APD, we do not any data augmentation preprocessing. We keep 5% of training data from each task for validation. We train the network with our algorithm with batch size of 64 and initial learning rate of 0.01. We Train each task for maximum of 50 epochs with decay schedule and early termination strategy similar to Serrà et al. (2018). We use εta 0.98 for all the layers and increasing the value of eth. by 0.001 for each new tasks.
modified LeNet-5 architecture
C.4 BASELINE IMPLEMENTATIONS
GEM (Lopez-Paz Pazz& & Ranzato 2017). A-GEM (Chaudhry al. 2019a). ER_Res (Chaudhry al. 2019b) and OWM (Zeng et al. 2018) are implemented from their respective official implementa- tions. EWC and HAT implemented from the official implementation provided by Serrà et al. While, OGD is implemented from adapting the code provided by Bennani et al. (2020).
2018).
14

---

Published as a conference paper at ICLR 2021
C.5 THRESHOLD HYPERPARAMETER
As discussed in section 3 and section 7 the threshold hyperparameter, εд controls the degree of interference through the approximation of space of significant representations of the past tasks. Since in neural network, characteristics of learned representations vary for different architectures and different dataset, using the same value of εzh may not be useful in capturing the similar space of significance. In our experiments we use éth in the range of 0.95 to 1. For PMNIST experiment, as discussed section we use 6th 0.95 in the first layer and 0.99 the other layers. For split CIFAR- 100 experiment, we use = 0.97 for all the layers and increasing the value of Eta. by 0.003 for each new tasks. For split minilmageNet experiment, we use 6th 0.985 for all the layers and increasing the value of 6zh. by 0.0003 for each new tasks. For experiment with 5-Datasets, we use Gth 0.965 for all the layers across all the tasks.
C.6 TRAINING TIME MEASUREMENT
We measured per epoch training times (in Figure 2b)) for computation in NVIDIA GeForee GTX 1060 GPU. For ten sequential tasks in PMNIST experiment, we computed per epoch training time for each task and reported the average value over all the tasks.
Training time for different algorithms reported in Table Aa) for PMNIST tasks were measured on a Single NVIDIA GeForce GTX 1060 GPU. For all the other datasets, training time different algorithms reported in Table were measured Single NVIDIA GeForce GTX 1080 Ti GPU.
C.7 LIST HYPERPARAMETERS
Table 6: List of hyperparameters for the baselines and our approach. Here, "Ir' represents (initial) learning rate. In the table represent PMNIST 'perm', 10-Split CIFAR- 100 as 'cifar', Split minilmageNet as 'minlmg' and 5-Datasets as '5data'.
Methods
Hyperparameters
OGD
Ir 0.001 (perm)
# stored gradients: 200/task (perm)
OWM
Ir: 0.01 (cifar), 0.3 (perm)
GEM
1r 0.1 (perm)
memory size (samples): 1000 (perm)
memory strength, 0.5 (perm)
A-GEM
1r: 0.05 (cifar), 0.1 (perm, minImg. 5data)
memory size (samples): 1000 (perm), 2000 (cifar), 500 (minImg). 3000 (5data)
ER Res
Ir: 0.05 (cifar), 0.1 (perm, minlmg. 5data)
memory size (samples): 1000 (perm), 2000 (cifar), 500 (minlmg), 3000 (5data)
EWC
Ir: 0.03 (perm, minlmg, 5data), 0.05 (cifar)
regularization coefficient: 1000 (perm), 5000 (cifar, minlmg, 5data)
HAT
Ir: 0.03 (minlmg), 0.05 (cifar), 0.1 (5data)
8maz 400 (cifar, minImg, 5data)
e: 0.75 (cifar, minlmg, 5data)
Multitask
Ir: 0.05 (cifar), 0.1 (perm, minImg, 5data)
GPM (ours)
Ir: 0.01 (perm, cifar), 0.1 (minlmg. 5data)
n. 100 (minlmg, 5data), 125 (cifar), 300 (perm)
15

---

Published as a conference paper at ICLR 2021
C.8
GPM SIZE
As discussed section 4 the gradient update will lie in the span of input vectors (a:) fully con- nected layers, whereas the gradient updates of the convolutional filters will lie in the space spanned by the imput patch vectors (p). Therefore, each basis stored in the GPM for a particular layer, / will have the same dimension x! p' Thus, for any particular layer, GPM matrix, M' can have maximum size of size(æ') x size(x²) or size(p') X size(p'). Maximum size of the GPM, which refer as GPML Max, is computed by including GPM matrices (M') from all the layers. Thus the size of GPM Max is fixed by the choice of network architecture. In Table we show the maximum size of GPM matrix (M') for each layers for the architectures that have used in experiments along with the size of GPM_Max.
Table 7: Size of GPM matrices for each layer for the architectures used in our experiments. Maxi- mum sizes of the GPM in terms of number of parameters are also given.
Network
Size of maximum M
GPMMMa (parameters)
MLP
784 x 784, 100 100, 100 100
0.63M
(3 layers)
AlexNet
48 48, 576 x 576, 512 x 512,
5,84M
(5 layers)
1024 x 1024, 2048 x 2048
ResNet18
27 27, 180 x 180, 180 x 180, 180 180, 180 180,
(17 layers+
180 x 180, 360 x 360, 20 x 20, 360 360, 360 360,
8.98M
3 short-cut
360 x 360, 720 x 720, 40 x 40, 720 x 720, 720 X 720,
connections)
720 x 720, 1440 x 1440, 80 x 80, 1440 x 1440,
1440 x 1440
D ADDITIONAL RESULTS
D.1 RESULT TABLES
Table contains the additional results for PMNIST experiment in single-epoch setting along with the standard deviation values for the results shown in Table[[[l for multi-epoch (5 epoch) setting. Method that does not adhere CL setup indicated by (*) in the table. Results are reported from 5 different runs.
Table 8: Continual learning on PMNIST in single-epoch and multi-epoch setting.
Epoch
Epochs
Methods
ACC (%)
BWT
ACC (%)
BWT
OGD
85.18 0.29
-0.66 0.00
82.56 0.66
-0.14 0.01
OWM
90.55 + 0.15
0.01 + 0.00
90.71 + 0.11
-0.01 + 0.00
GEM
88.65 0.27
-0.77 0.00
83.38 -
-0.15 0.01
A-GEM
87.80 0.16
-..88 + 0.00
0.16
-0.14 + 0.00
ER_Res
90.63 1: 0.27
0.05 0.00
0.53
0.11 : 0.01
EWC
88.27 0.39
-..0 0.01
89.97
0.04 0.01
GPM (ours)
91.74 + 0.15
0.03 + 0.00
93.91 + 0.16
-0.03 + 0.00
Multitask*
95.21 + 0.01
96.70 + 0.02
16

---

Published as a conference paper at ICLR 2021
Table 9: Continual learning different datasets along with the standard deviation values for the results shown in (b).
CIFAR-100
minilmageNet
5-Datasets
Methods
ACC (%)
BWT
ACC (%)
BWT
ACC (%)
BWT
OWM
50.94 = 0,60
-0.30 + 0.01
EWC
68.80
-0.02 0.01
$2.01 2.53
-.12
88.64 0.26
0.04
HAT
72.06 + 0.50
0.00
59.78 +: 0.57
-0.03 0.00
91.32 0.18
0.01 + 0.00
A-GEM
63.98 = 1.22
-0.15
57.24 2
-.012
84.04 + 0.33
-0.12 = 0.01
ER Res
71.73 0.63
-0.06
58.94 0.85
-0.07 0.01
88.31 0.22
-0.44 0.00
GPM (ours)
72.48 +: 0.40
0.00
60.41 + 0.61
1.000
91.22 0.20
0.011 0.00
Multitask*
79.58+ 0.54
69.46 0.62
91.54 0.28
D.2 & VALUES
Table 10 (a) and (b) show the number of new bases added at each layer per PMNIST and 10-split
CIFAR- 100 task respectively. Total number of bases in the GPM after learning all the tasks is also given.
Table 10: Number of new bases (l)) added to the GPM at different layers after each (a) PMNIST task and (b) 10-split CIFAR-I00 task (for random seed configuration).
(a)
(b)
k
Task ID
FC1
FC2
FC3
Task II)
Conv
Conv2
Conv3
FCI
FC2
0.970
125
2
71
22
19
2
0.973
3
44
74
81
101
70
11
11
0.976
20
29
71
93
0.979
99
57
4
0.982
2
33
28
73
98
50
0.985
16
19
71
98
0.988
16
41
0
0.991
2
41
26
76
104
35
0
0.994
56
34
84
109
0.997
113
Total
543
100
88
Total
22
417
473
769
1012
Table 11: Continual learning of Digit dataset tasks in class inssemncremlntal learning setup (Kamra al.
2017). (t) denotes the result reported from DGDMN.
Methods
Metric
EWCt
DGR
DGDMNİ
GPM (ours)
ACC (%)
10.00
59.60
81.80
70.67
BWT
1.00
0..3
0.15
-0.66
D.3 CLASS-INCREMENT LEARNING
In this section, we evaluate our algorithm in class- incremenate learning setup (Rebuffi et al.) 2017). where disjoint classes are learned one by one and classification performed within all the learned classes without task hint (Kamra et al. 2017). This setup is different (Hsu et al. 2018 from the (single-head/mulle -eauu) evaluation setups used throughout this paper. Also, this scenario is very challenging and often infeasible for the regularization (HAT, EWC etc.) and expansion -baddd (DEN, APD etc.) methods which do not use old data replay. Using the experimental setting similar to DGDMN (Kamra et al. 2017), implemented the "Digit dataset' experiment where single class
17

---

Published as a conference paper at ICLR 2021
of MNIST digit is learned per task. Results listed in Table 11 where the baselines reported from DGDMN. While EWC forgets catastrophically, we perform better than DGR (Shin et al, 2017). which employs data replay through old data generation. DGDMN, method, outperforms all. In this setup, we believe a subset of old data replay either from storage or via generation is inevitable for attaining better performance with minimal forgetting (Rebuffi al.
improved data replay
2017
Rajasegaran et al. 2019
D.4
ADDITIONAL PLOTS
70
Accuracy (%)
EWC
60
HAT
A-GEM
50
ER. Res
GPM
40
0
2
4
6
8
10 Tasks
12
14
16
18
20
Figure 4: Evolution of task accuracy over the course of incremental learning of 20 sequential tasks from minilmageNet dataset. Learned accuracy in our method remains stable throughout learning.
(a)
(b)
(c)
Su- 0.5
Di
Cu- 0.5
3u 0.8
tam0 8
8.8
cu 0.95
Exye 0 95
Cus 0.95
2
-1.0
-0.5
0.0 activation
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
activation
activation
Figure 5: Illustration of how threshold hyperparameter controls the degree of interference (a) Conv layer (b) Conv layer (c) FC layer with the histogram plots of interference activations from Split CIFAR-100 experiment. With increasing eu spread of the inference activation decreases resulting in minimization of forgetting.
18


Reviews:
[{"review_id": "ll2LpqSlJK-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes one of the most scalable approaches to sequential continual learning with known task boundaries and related tasks, while taking steps towards enforcing data privacy and removing some of the task label constraints. At all levels in expressive deep models, SVD is used on learned representations to identify important bases of task gradient spaces and a memory is populated with such directions. Learning progresses only in directions orthogonal to gradient memory. Several recent evaluation methodologies are used to empirically validate the approach with significant success. \n\n\nStrong points: \n- Principled and relatively simple approach, yet scalable to interesting deep models.\n- Manageable computational overhead.\n- Memory of gradients introduces a layer of data privacy, and advantage compared to many other memory-based approaches.\n- Strong empirical results, although not apples-to-apples in all cases, see the comment posted earlier.\n- Clearly written paper.\n- Analysis of scalability in terms of memory and computation is a welcome bonus.\n\n\nWeak points:\n- The sequential task learning setup is of limited practical use and not too realistic.\n- It’s hard to say how the algorithm would be used in practical continual learning settings, e.g. reinforcement learning of single tasks without forgetting, or where clear task boundaries do not exist.\n- Little effort is made to see the approach relevance for non-similar sequential task learning, where there could be interference between learned representations in terms of negative forward transfer.\n\n\nRecommendation and Rationale:\n\nI strongly recommend acceptance because the method is simple, practical and the paper is well written both in terms of clarity and also analysis.\n\n\nFurther questions:\n- Do you see any positive forward transfer, e.g. later tasks are learned quicker due to previous tasks?\n- What are the limitations and roadblocks to extending this method to sequences of hundreds of tasks which are not necessarily related? Please discuss in the manuscript.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TjNWJeGmmOT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes a new approach for continual learning problems. The key idea is to maintain bases of subspaces using SVD in the Gradient Projection Memory (GPM), in which the update direction is orthogonal to the gradient subspaces deemed important for the past tasks. Image classification experiment was conducted to justify its better empirical performance in practice. Overall, the paper is well-written. I have the following comments. \n\n1. The contribution of this manuscript is kind of incremental compared with OGD (Farajtabar et al. 2020). From my understanding, the main improvement is using SVD to store the bases, which basically trades computational efficiency for memory. In addition, it is claimed that OGD only works for small learning rate. Why using SVD helps using a larger learning rate? It is not explained in the paper.\n2.  Is it possible to report running time of the proposed approach and compare with other approaches? SVD is very expensive in the high-dimensional setting such as deep neural networks. It might be impractical due to running time concerns.\n3. What is the value of $k$ in the experiments (in inequality (9)) for different $\\epsilon_{th}^l$? It is better to report these values. In addition, when $k$ becomes large, the update rule (6) and (7) may become computationally expensive.\n4. It is mentioned that “our proposed approach can perform inference without the task identity of test samples”, but with no explanations. It is desired to illustrate the meaning in the main paper.\n5. The literature review is not sufficient. There are several recent papers which also change the update direction using gradient projection. It is desired to see the difference between the proposed method and other relevant approaches. For example,\n\n\t[1]. Yu et al. Gradient surgery for multi-task learning. NeurIPS 2020.\n\n\t[2]. Guo et al. Improved schemes for episodic memory-based lifelong learning. NeurIPS 2020.\n\n========POST REBUTTAL=========\n\nI would like to thank the authors to answer my questions. It addressed most of my concerns. Hence I increase my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MyVwO0izm_j", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found the idea quite novel. Lately in continual learning the focus has been more on NAS type ideas and algorithms, but this work is a nice divergence from this direction. The idea of optimizing in a space orthogonal to the previous task is novel. The execution of the idea is nothing special since it's using standard linear algebra, but I gave the authors full merit to the idea itself. \n\nMy higher score is mostly due to the fact that the experiments are limited. The benchmark algorithms definitely miss some recent works from 2019 and 2020. They should be included as otherwise the superior performance of the algorithms is questionable. \n\nSee for example:\nhttps://arxiv.org/abs/2006.04027\nJu Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pages 899–908, 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "1tZSm-tMrxp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work targets learning multi-class classifiers in the continual learning setting. The key idea is to learn new tasks by taking gradient steps in directions orthogonal to the gradient subspaces marked as crucial for previous past tasks. The method employs SVD after learning each task to find the crucial subspaces (which it calls as Core Gradient Subspaces) and stores them in a memory. Reasonable quantitative and qualitative evaluation has been performed to compare the method against existing SOTA baselines.\n\n\nReview:\n\n1. The paper is overall clearly written and the method is adequately described.\n\n2. The work is relevant to audience at ICLR and provides reasonable details for reproducibility.\n\n3. Relationship to previous works has been explained and relevant literature has been cited appropriately.\n\n\nStrengths:\n\n1. GPM shows very little degradation in performance on past tasks and is very resilient to forgetting.\n\n2. It provides explicit control over forgetting via the \\eps_{th} parameter.\n\n\nWeaknesses:\n\n1. GPM performs fairly close to or marginally better than HAT and EWC on most datasets.\n\n2. GPM trades-off between ACC and BWT metrics by strictly controlling forgetting. Hence, it may not be able to identify tasks with very similar structure (or repeated tasks) and may not re-use the Core Gradient Space from previous tasks to improve performance on them (BWT). So, while it minimizes negative BWT, it also restricts positive BWT.\n\n\nPotential improvements and clarifications:\n\n1. The authors have repeatedly emphasized that their method performs task-free inference. However, doesn't this just mean that the tasks used are such that the inputs encode the task identity and do not require a separate task identifier. Further, on page 6, under Network Architectures, the authors state that apart from permuted MNIST tasks, they use the multi-head setting, i.e., each task has a separate classifier. How does this allow task-identifier free inference?\n\n2. All models have been trained using plain SGD. Would it be possible to extend the method to other optimizers, e.g., Adam?\n\n3. 3 runs are too few to average over. These are supervised learning experiments, so it should definitely be possible to use more runs (at least 5, but preferably 10).\n\n4. At first glance, it seems that the method may have scalability issues since it relies on performing SVD. However figure 2 counter-intuitively shows GPM to be both fast and memory efficient. Is this because these graphs in figure 2 are per-epoch? What about in between tasks when GPM needs to run SVD on all layers? Some more explanation and results are required to better understand how GPM achieves computational efficiency despite relying heavily on SVD per layer after every task.\n\n5. Lastly, no experiments with a single class per task have been performed. This setting is known to be quite challenging in general and induces significantly more catastrophic forgetting (see Kamra et al, 2017). In this setting it is also generally harder to just set batch-norm parameters using just the first task.\n\n[Kamra et al, 2017] Deep Generative Dual Memory Network for Continual Learning. Nitin Kamra, Umang Gupta and Yan Liu. ArXiv 2017.\n\nPlease respond to the 5 potential improvements and clarifications mentioned above. I will be happy to raise the score further if the above concerns are addressed.\n\n-------- Post-rebuttal edit -------\n\nThe authors have answered all my questions satisfactorily and provided additional experiments wherever it was required including settings where GPM may not outperform other baselines. I believe that the paper is strong and makes a significant technical contribution to the field of CL. Hence, I recommend acceptance and I am updating my score to reflect the same.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach to continual learning with known task boundaries that is scalable and highly performant, while preserving data privacy.  To mitigate forgetting the proposed approach restricts gradient updates to fall in the orthogonal direction to the gradient space that are important for the past tasks. The main novelty of the approach is to estimate these subspaces by analysing the activations for the inputs linked for each given task. 

All reviewers give accepting scores. R2, R3 and R4 strongly recommend accepting the paper, while R1 considers it borderline.

The authors provided an extensive response carefully considering all reviewers' comments. New experiments were introduced (training time analysis and comparisons with expansion-based methods), and several clarifications were added.

All reviewers agree that the paper is well written and its literature review adequate.

The main concern of R1 was the similarities with OGD (Farajtabar et al. 2020). R1 considered the authors’ response acceptable. R2, R3 and R4 consider the contribution well motivated and significant and highlight its simplicity. The AC agrees with this assessment.

The empirical evaluation covers most of the typical benchmarks in CL. Very strong results are reported on a variety of tasks both in terms of performance and memory efficiency, as agreed by R2, R3 and R4.

Overall the paper makes a strong contribution to the field of CL.

Decision: Accept (Oral)