Continual Learning with Filter Atom Swapping
Authors: Zichen Miao
Ze Wang
Wei Chen
Qiang Qiu
Venue: ICLR 2022 Spotlight (2022)
Keywords: continual learning

Abstract: Continual learning has been widely studied in recent years to resolve the catastrophic forgetting of deep neural networks. In this paper, we first enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning with filter atom swapping. In other words, we learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. By maintaining a small footprint memory of filter atoms, we can easily archive models for past tasks to avoid forgetting. The effectiveness of this simple scheme for continual learning is illustrated both empirically and theoretically. The proposed atom swapping framework further enables flexible and efficient model ensemble with members selected within a task or across tasks to improve the performance in different continual learning settings. Being validated on multiple benchmark datasets with different convolutional network structures, the proposed method outperforms the state-of-the-art methods in both accuracy and scalability.

Published as a conference paper at ICLR 2022
CONTINUAL LEARNING WITH FILTER ATOM SWAP- PING
Zichen Miao, Ze Wang, Wei Chen, Qiang Qiu Department of ECE Purdue University (miaoz, wang5026, chen2732, qqiu) ) @purdue. edu
ABSTRACT
Continual learning has been widely studied recent years to resolve the catas- forgetting of deep neural networks. In this paper, we first enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning with filter atom swapping. In other words, learn for each task new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. By maintaining small footprint memory of filter atoms, we can easily archive models for past tasks to avoid forgetting. The effec- tiveness of this simple scheme for continual learning is illustrated both empirically and theoretically. The proposed atom swapping framework further enables flexible and efficient model ensemble with members selected within task or across tasks to improve the performance in different continual learning settings. Being validated multiple benchmark datasets with different convolutional network structures, the proposed method outperforms the state-of-the-art methods in both accuracy and scalability.
INTRODUCTION
Humans keep acquiring new concepts without forgetting crucial ones in the past. To endow intelligent agents with the same ability of long-term knowledge accumulation, continual learning (CL) has been intensively studied in recent years. In continual learning, an agent learns from a sequence of tasks, with the goal of gaining knowledge of each new task while preserving the capacity for resolving the old ones, therefore to avoid catastrophic forgetting. The recent advances of CL mainly follow several directions. One popular category among them is to maintain an external memory of the original images (Robins) 1995 Rebufh et al. 2017). synthesized images (Shin et al. (017), or parameter gradients (Lopez- Paz & Ranzato 2017) for archiving the past. These memory based methods often suffer from heavy memory footprints, while still forgetting about the previous tasks to some extent.
Motivated by the literature on subspace modeling of tasks (Evgeniou & Pontil 2007 2013; Zhang & Yang 2021 Cooera--Parede
Maurer et al..
al. 2013 Kumar & Daume III 2012) this paper, we propose to learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. In other words, in CNN, we enforce a low-rank filter subspace by decomposing convolutional filters within each network layer small set of filter atoms. Then, we perform continual learning by simply swapping filter atoms for each task. The effectiveness of our approach is empirically validated and further explained thcoretically with excess risk bound analysis.
With the proposed approach, we can faithfully remember the past by only maintaining an atom memory with small footprint to archive task-specific filter atoms. Any previously learned CNN models can now be exactly recovered by multiplying the task-shared coefficients with the task-specific atoms, which can be retrieved efficiently from the atom memory. Thus, the introduced filter atom memory permits historical knowledge to be recalled with a guarantee against forgetting. Comparing with state-of-the-art memory-based CL methods, our approach requires storing for each task only some tiny size filter atoms, which in total are typically much smaller than the size of exemplars in memory-based methods (Rebuffi et al. 2017 Prabhu et al. 2020), and therefore potentially supports continual learning large scale.

---

Published as a conference paper at ICLR 2022
Tise-Speiiffic Atoms, Le,, Filter Subspuces
by Retrieving Atoms
Task-Stured Cueflicients
Specific Fillers
Truining
Testing
Atom Memory
D
W
of
X
a,
D!
tasks
tasks
Figure I: Illustration of proposed continual learning method with filter atom swapping. Within each CNN layer, we decompose filter W, Rex xkxk over filter subspace spanned by m filter atoms D, R W, D. where a; E Re are the subspace coefficients, and c' the number of input and output channels, filter subspace as filter atoms, and store those atoms, typically a few hundred parameters, a small footprint atom memory. At time T,, we can recall the past model at t (1 < ') through filter reconstruction W, = D:D with D fetched from the atom memory, to fully recover the previous model.
the spatial size each atom. With task shared coefficients, we learn for each task new
Our atom swapping continual learning framework can effectively support both inter-task and intra-task model ensemble to further enhance performance different continual learning settings: First, inter- task ensemble utilizes the relevant past knowledge to boost the present task performance. To ensure that ensemble with past models can affect the current task positively, we only select a relevant subset of past models. With life-long learning scenario in mind, we choose to assess task relevancy simply based the filter subspace distance, which can be on- on-the-fly computed here via the Grassmann distance (Absil et al.) (2004) among task-specific filter subspaces. Second, intra-task ensemble can be adopted in iaasmmnnrremntal setting help task prediction with the minimal-entropy criterion. Usually, ensemble members are instantiated as independent CNNs, and their learning and inference conducted separately for dissimilarity. However, this will lead significant increase in training and inference time and memory usage. We address this problem by creating within a task multiple virtual members in single CNN model by simply maintaining several groups of filter atoms in each layer. In this way, different intra-task members are integrated into a single network, while learning and inference can be conducted efficiently with group convolution.
validate our simple yet effective approach on several continual learning benchmarks such as MNIST, CIFAR 100, and minilmageNet under both class-incremental and task-incremental settings, and observe competitive results against state-of-the-arr methods on benchmarks with far less memory usage.
We summarize our contributions follows,
We learn for each task new filter subspace for each layer, and keep subspace coefficients shared across tasks.
We maintain a small footprint filter atom memory that can faithfully archive past knowledge with guarantee against forgetting in highly scalable way.
We adopt inter-task ensemble for the present task by recalling past models based on on-the-fly calculated task relevancy under the task-incremental setting.
We propose an intra-task ensemble for the insremmnctm setting by creating multiple virtual members in single CNN model through different groups of filter atoms per layer.
2 MOTIVATION
We are motivated by the literature on task subspace modeling (Evgeniou & Pontil 2007 Maurer al. 2013 Zhang & Yang 202T: (Romera-Paredes al., 2013 Kumar & Daume III (012). where it is commonly assumed that task parameters lie in a low dimensional subspace, so that tasks can be modeled as of latent basis tasks and their linear combinations. The latent basis tasks and
2

---

Published as a conference paper at ICLR 2022
the respective linear combinations often obtained via alternative optimization by fixing and optimizing the other (Kumar & Daume III 2012).
In our continual learning setting, we model tasks using convolutional neural networks (CNNs). Following QQuu et al. 2018), we decompose convolutional filter W, E Rexc xkxk for the i-th layer over filter atoms D; Rmxxx linearly combined by coefficient a; Rexe'xx xm where and d' are the number of input and output channels, is the spatial size of each atom. This can be written as W, ag x D Note that this decomposition distributes filter W; into two imbalanced parts: or for channel mixing with mae' parameters, and weight D; for spatial filtering with only mk2 entries. In all, we use W- = a x D to denote the filters decomposition in model with I convolutional layers, where W- (W.)_... = (ai) D (D.) indicate all filters, coefficients, and atoms respectively.
It is easy to observe that, within each CNN layer, we can borrow from the task subspace modeling methodology by creating a set of latent basis tasks through filter atoms D; and their linear combi- nations as atom coefficients Or, The linear combination coefficients o; learned on the first task jointly with the first group of atoms, and then atoms for subsequent tasks are alternatively optimized by fixing ax.
As illustrated in the subsequent sections, this seemingly under-fitting method not only enables an efficient way to faithfully archive past models with guarantee against forgetting, but also supports efficient inter-task and task model ensemble to further improve the performance.
METHODOL.OGY
We consider the problem of learning T tasks sequentially. Formally, we denote the data distribution that associates with the t-th task as D (X* )"), t E {1,
T) from which dataset D' xx.y1) is sampled for training. The goal of continual learning is to minimize the statistical risk of all seen tasks given no access to data D' from previous tasks S T (Delange et al. 2021):
T
Σ E (XF
(I)
where L denotes the risk function, (:: 0) is the model for task with parameter 8. Continual learning with a guarantee against forgetting can be achieved by storing learned parameters entirely in external memory M {0') given (:; = F(::0', thus any previous model can be completely recovered by retrieving the corresponding parameters from the memory. However, such a straightforward solution based on parameter memory suffers severely on its poor scalability due to the large size of modern deep neural networks and the potentially long task sequence. We address the scalability issue and achieve guaranteed non-forgetting by decomposing convolutional filters in CNN into task-specific filter atoms and task-shared coefficients. Then only hundreds of parameters per task need to be stored in an atom memory to guarantee non-forgetting. This proposed approach allows efficient inter-task and intra-task ensemble further boost performance.
3.1 A SCALABLE APPROACH AGAINST FORGETTING
In CNNs, catastrophic forgetting occurs when model learned from sequence of past tasks is updated in favor of the current task, resulting in significant performance degradation. A straightforward solution, mentioned before, is to archive W in external memory M (W') and the representation space of any previous model can be faithfully recalled by memory retrieval. However, as deep CNNs contain great amount of parameters in W this simple solution scales poorly with the number of tasks T. On the other hand, storing part of parameters, or small subsets of data for parameter flashback cannot avoid forgetting completely (Sarwar et al. 2019 Yoon et al. 2018).
This dilemma can be resolved with the proposed filter atom decomposition, as shown in Fig. T With the filter decomposition described in Sec. 2 by storing task-specific filter atoms D' into memory, and enforcing task-shared coefficients a, the model archives the entire knowledge for each time point. We refer to the memory for storing atoms as the atom memory, MD = (D V D' E m with each De learned in the t-th task with empirical risk minimization,
N₂
arg min D
C(F(x): a D').
(2)
3

---

Published as a conference paper at ICLR 2022
The task-sharee coefficients are learned on the first task jointly with the first group of atoms,
arg min D
In this way, we can guarantee that the statistical risk for previous task / at any time point remains the same as can recall faithfully the past model by multiplying the stored atoms with the task-shared coefficients,
E (CF E [C(F(X """"""")) D').
(4)
Atom memory scalability. The proposed atom memory stores group of atoms per task, which is scalable with increasing number tasks. Formally, consider a I-layer CNN model with associated filters W (W.) As mentioned in Sec.2/ each filter can be decomposed as W;
x D;. The group of atoms for task can then be denoted as D' (Di)= which requires size of Imk? in storing parameters per task. This typically introduces only a few hundred of parameters for each task to be stored in the atom memory, which potentially supports continual learning on a large scale. Details for scalability comparison are shown in Sec. 5.2.
Analysis of excess risk bound. With task-shared coefficients a, the model for each new task may seemingly expect some degree of underfitting. However, as demonstrated in Section []] we still observe superior results over the state-of-the-art continual learning methods all benchmarks have evaluated. To understand this, we here theoretically analyze the excess risk bound for new tasks in the continual learning setting. For each tasks & {1,2, T), model FF(;; () consists of representation function o' and prediction function w' (w" E I²*. The representation function o' maps an input x to feature space Z C Rt. For the analysis purpose, we assume ø is just one convolution layer, which can be decomposed oe = a x Dí. Therefore, using the training samples from task i, we can solve the following optimization problem from Eq. (2:
1 min 2 ly - (a x D' w'
2 ||a X D'I 2
X 1
I
(5)
2
where we use the mean square error as our loss, (+)+ is ReLU activation (z) = maz (0, z}. The
shared coefficients a are learned the first task jointly with the first group of atoms. Similar to (Du
ctt al., 2020), we also assume that there is a ground optimal representation function 6f* and prediction function w for task t.
Assumption / (subgaussian input). There exists p > such that, for all t {1, 2,,,,.., 7), the random vector ē ~ p² is p² susubaaussiinn The pf is the distribution of samples in task t.
Assumption (oracle network). Assume for task E zº is generated by an oracle network with parameters ax" D and wf Noise term 2' NO TT)).
21.2...., T} that y' (a" x D (x')) whi
Excess risk bound. We can bound the excess risk our learned model on the task t, i.e., how much our learned model (á, D', w¹) performs worse than the optimal model (a", Drv",, wfi*) on the task t as follows:
ER(â, D', w²) (áx D', w²) L D whrr)
SoR Ō Tr(E) +
Tr(E) + |E + P Ō
VN
NO
where fla"" x DI + ||| 11. E= Exup [xxT], N is the number of training samples.
Ô is the big O notation, and LD= is the expected loss with the data distribution D'. The detailed analysis is provided in the Appendix A
3.2 INTER-TASK MODEL ENSEMBLE
The atom memory not only serves as an efficient way to archive past models with a guarantee against forgetting, but also enables efficient recall of past models for model ensemble to improve the performance in present task. As in Breiman (200T); [aakshmnarryaanan et al. (2017), ensemble performance increases with the independent level of ensemble members. Motivated by this, model ensemble over time for the current task at time has natural advantage that each ensemble member
4

---

Published as a conference paper at ICLR 2022
Memory
Relevuncy Sabsgare Distance
Membeer-Speciffcc Aloms
Replicated Inputs
B
D
D
Shared Coefficient df
Most Relevant: (E,
sM
Group Convolution
II
(xr
(X
Wi
Model
Meddil
Inter-Taee Ensemble
Per-Member Reconstrucied Filters
Concatemated Fillers
Figure 2: Inter-task ensemble with E, =
Figure 3: Intra-task ensemble with E = 2.
I' (1 E r)) is learned from different data distribution D However, most of other methods are not affordable to perform model ensemble across tasks since they often lack an effective and efficient way to recall past models. As illustrated in Fig. 2 with the atom memory, faithfully and rapidly recall past models F² F(:; a x D', by simply fetching atoms D' from the atom memory; and then perform inter-task model ensemble by constructing uniformly-weighteed mixture model and combine the predictions as (Lakshminarayanan et al. 2017).
method can
ens (x) = S+
E (ESTU(r)
p=(y|x,0')
E
xx)))
(6)
E. +1
SESTU(r)
where S denotes the index set of previous tasks used for ensemble, and E, = | For classification problem, corresponds to averaging the predictive probabilities.
In continual learning, not all past models can bring positive effects on the current task. According to (Breiman 2001 Lakshminarayanan et al. 2017), only the ones that have enough strengths on the current r-th task can help enhance the performance as weak learners (Breiman 2001), which can be selected based on task relevancy. Our assumption that the more past task resembles the current the better performance will F achieves on the present task. The problem then transforms to evaluating the model similarity effectively and efficiently. Note that to ensure the proposed ensemble method to be scalable across a very long historical task sequence T, highly efficient task relevancy assessment is indispensable here.
Assessing task relevancy by filter subspace distance. Task relevancy assessments proposed in (Achille et al., 2019, Zamir et al., 2018) work the cost of heavy computation, which prevents their efficient applications in continual learning. While measuring the similarities among the learned mod- els can be straightforward proxy to the measurement of task relevancy, the widely studied methods, e.g., canonical correlation analysis (CCA) (Raghu et al., 2017 Morcos et al.) 2018) and centered kernel alignment (CKA) [Korbblihh et al. 2019), still introduce considerable computational cost while performing evaluations in the representation space. Directly performing relevancy measurements by calculating the distance of filters works at highly desirable efficiency, yet can perform poorly without the costly semantic alignments over channels (Raghu et al. 2017). In our approach, thanks show that model similarity
to the task- coefficients acting structural regularizations, measurements can now be efficiently evaluated through direct filter similarity measurements. And as model filters using task task-specific filter subspaces with coefficients shared, model similarity be further reduced to assessing filter subspace distance via the Grassmann distance (Absil et al. (004).
Formally, we characterize the filter subspace of the current model y and past model y as:
y" = Span( B, V = ( B,, V", V C CRL.
(7)
where L. = 22 is the dimension of kernel space, B {B, M are M (m M) linear independent vectors serve as the bases of the filter subspace. We obtain 13 by performing singular value decompo- sition (SVD) to atoms = UDVT E x and select the first columns from that correspond to the top-M singular values. By definition, yr and D are points of the Grassmann Manifold (Milnor
& Stasheff 2016), i.e., V", V" E Grass(M, L) A {M dimensional subspaces of R.). Then, the Grassmann distance between V" and V' is defined as
dv ,V)
Σ 03)1/2
(8)
3
5

---

Published as a conference paper at ICLR 2022
where 8g is the j-th principle angle, which can be calculated by,
0g - arrccos (c,), with (B") (B")""' -EVT, ,0 ,0,- -Eg
The proposed task relevancy measurement requires merely SVD to matrices with dimensions lower than L. In practice, k = 3 so that L = 9, indicating that computation of du is low. With the proposed efficient task relevancy measurement adopted in the last convolutional layer, we select the most E relevant models from previous r ones that support the current task. We provide the correlation analysis of task similarities measured with CCA (Raghu et al. 2017) and the ones assessed by the proposed subspace distance in Appendix. C.5
Although it is known that the ensemble result increases in the ensemble number E Lakshmi narayanan et al.) 2017). it does not hold in our setting based on our empirical observation. In fact, determining the E. is a trade-off between the amount and the relevancy of past knowledge, We thus empirically select the ensemble number Ex,, which is illustrated in Sec. 5.2. The selected members are further fine-tuned with new classification heads to the current task.
3.3 INTRA-TASK MODEL ENSEMBLE
Our filter decomposition not only allows ensemble with past members, but also permits an efficient way to create ensemble members within a task. In the regular deep ensemble scenario (Lakshmi narayanan et al.) (017), different members are instantiated as multiple CNNs, and need to be learned separately to ensure independence among members. Plus, obtaining ensemble results also requires inferences with multiple CNNs. This introduces significant cost in both time and memory, making it inappropriate in continual learning settings. The proposed atom decomposition allows new way of parameterization of ensemble members within a task to improve performance while substantially reducing the training and testing time cost. Formally, given task L, the intra-task ensemble model Fw-ens is composed by ( ...,....... Jt. where En the number of models. Rather than in- stantiating them as different CNN models, we reparameterize them with member-specific atoms (D DrFF2,, }. and member- shared coefficient a', as shown in Fig. 3 In this way. we create multiple virtual members using a single CNN model by simply maintaining different groups of filter atoms in each convolutional layer. The forward pass of F convolution,
we-ens can then be conducted by group
F.
Ft w- enn (x")
Σ
fo (x') = FC Fu-ens (x', W wenn)s)
(10)
Emg
where X² E (ex is the input repeated by E. times, W w-eess = ja x D |... a x
Dr.E.] gex(dxxE,xxxx () denotes concatenation) is the filter for group convolution that is concatenated from per-member reconstructed filters. To enforce the independence of different members, we initialize member-specific atoms separately before training.
With diverse predictions from different virtual members, our intra-task ensemble can directly boost the performance in task- iss-eincremenall settings. Furthermore, intra-task model ensemble makes the model to better distinguish data from out-of-task distribution, i.e., (p A t), and shows high entropy in its predictive distribution which is particularly useful in class-increment CI we perform task prediction first, sclect a specific group of atoms, and then perform classification within task. Task id is selected based on minimal-entrope criterion on predictive distributions. Thus, intra-task ensemble can enhance the accuracy of task prediction, and then the overall CI performance.
(CD) setting. Note that in
EXPERIMENTAL SETUP
Datasets. We first validate our method with the "Jass-Incrementall (CI) setting with CIFAR100 and ImageNet-Subset, which contains 100 classes selected from ImageNet (with random seed 1993). For each dataset, half of classes are used for model learning in the O-th task. Then the remaining classes are equally split into N tasks, where N can be 5 or 10. Note that in the CI setting, task information is not provided during testing. Then, we report the performance of our method under the lnssemnnrremetall (TI) setting. We validate our approach on 5-Split MNIST (LeCun ct al. 1998). 20-Split CIFARIO0 (Krizhevsky al. 2209), and 20-Split minilmageNer (Vinyals al. 2016). The 5-Split MNIST uniformly splits the original 10 classes of 0-9 MNIST digits into 5 sequential tasks. 20-Split CIFARI00 and 20-Split minilmageNer are both constructed by randomly splitting 100 classes into 20 tasks with 5 classes per task. Details of each dataset are provided in the Appendix

---

Published as a conference paper at ICLR 2022
Table 1: Class-incremental results CIFARI00 and ImageNet-Subset with different number of incremental tasks (N). We report average incremental accuracy for all methods.
CIFAR-100
ImageNet-Sub Acc.(N=10)
Method
Acc.(N=5)
Acc.(N=10)
Memory
Acc.(N=5)
Memory
LwF-E (Li Hoiem
2017
57.03
56.82
65.51
65.58
EWC-E Kirkpatrick er
2017a
56.28
55.41
65.22
64.13
iCaRL. [Kebuti el
2017
57.17
52.57
6.2 MB
65.04
$9.53
301.4 MB
SDC el al. 2020
$7.10
56.80
65.60
65.70
BiC el el 2019
59.36
54.20
70.07
LUCIK Hou 2019
63.12
60.14
70.47
68.09
Ours (Base)
60.23
55.54
71.74
2.7 MB
w/ E.
65.44
62.48
0.5 MB
75.85
72.11
7.7 MB
Network architectures and implementation details. For the class-incremental setting, we utilize ResNet-32 for CIFAR100 and ResNet-18 88 for ImageNet-Subset (Rebufti al. (017). For the task-incremental setting, we adopt an AlexNet-like network. Note that we substitute all convolutional layers in both models with decomposed version. Details of architecture Appendix C In terms of the proposed ensemble strategies, inter-task ensemble only deployed in the task-incremental setting task IDs are needed, while intra-task ensemble is utilized in both task- incremental and class-incremental settings. In inter-task ensemble, we set = 3 for the dimension of filter subspaces. When intra-task ensemble and inter-task ensemble are adopted together, member with the best results every task for calculating task relevancy. We provide the ablation study for the inter-class and intra-class ensemble numbers E,, and E., in Sec. 5.2.1] Training details are showed in the Appendix C33 For forward knowledge transfer, the most recent atoms are used to initialize the atoms of the current task.
shown in the
use
Evaluation metries. In the class-incremental setting, we evaluate the model's average class- incremental accuracy. In the task- dusk-incremental setting, we measure the performances with ACC as the average test accuracy across all tasks. To measure the forgetting, adopt the backward transfer, BWT, which shows how the previous tasks performance has degraded due to learning new tasks. Details of these two measurements are provided Appendix C.4
5 RESULTS AND DISCUSSION
In this section, start from the challenging class-incremental setting. Then move task- incremental setting with self-comparison experiments to validate the effectiveness of some key ingredients of our methods, and show the results of the proposed method several real-world datasets. both settings, our method achieves improvements over state-of-the-arr methods with significant less memory usage.
5.1 CLASS-INCREMENTAR EXPERIMENTS
As more challenging setting, class-incremental (CI) learning does not provide task id during testing. As mentioned in Sec. 3.3 we handle this setting by breaking it down to two-level task, task prediction based minimal-entropy criterion, and then within-task classification. In CI experiments, we select the number of atoms m = 12 and the number of members for intra-task ensemble E,, 2. We benchmark method by comparing with many existing methods that store 2000 exemplars in the external memory. As shown in Tab. [ our method with intra-task ensemble achieves the best results with order of magnitude less memory usage both CIFAR 100 and ImageNet-Subset, which validates the effectiveness and scalability of our framework in the challenging CI setting.
30.
10.2 PM
311-
+5.45
735
26.
213
27.25
242
02290
(m)
125
tack Ensemble Members (E,)
Ozr.
WLC
Figure 4: Left & Middle: Ablation study on number of atoms (m) intra-task ensemble members (E.) on 20-Split CIFAR100. Right: Parameter memory growth per-task for 20-Split CIFAR100. The proposed method shows significantly lower memory growth than other expansion methods.
7

---

Published as a conference paper at ICLR 2022
Table 2: Results on 20-Split CIPARIOO and 20-Split minilmageNet. (*) We re-implement PNN and APD with our network architecture. Analysis on memory is provided in Appendix [C.10)
CIFAR-100
minilmageNNet
Method
ACC%
BWTS
Memory (MB)
ACC%
BWT%
Memory (MB)
EWC
Kirkparrick et ml
2017a
55 60 1.11
23.53+t 1.19
36.61
28.17
HAT
76.96+ 1,23
0.01= 0.02
59.45
0.00
PNN
Rusu cl
2016
82.25 0.04
0.00= 0.00
165.3
70.96
0.00
165.3
APD'
77.03: 0.14
-0.021 0.01
60.5
61.67
0.07
60.5
iCaRL
Rebulli et 2017
58.082 1.44
24.22 1.35
28,8
173.6
A-GEM
2018c
54,38
3.81
-21.991:.005
52.43
-15.23
110.1
ER-RES
C namanny er m
20198
66.78 0.48
-15.01 1.11
16
$7.32
-11.34
110.1
GCL
TTny Matteson
20207
7..51+0099
6.54= 1.26
7.2
61.54
6.10
43.4
AC1
Ebrabamm et al.
2020
78.08 1.25
0.00:: 0.01
62.07
0.00
8.5
Ours (Buse)
79.134
0.00 0.00
0.14
66.01
w/ E. 3
79.91 0.15
0.00 0.00
0.43
66.83
0.00
0.43
80.21
0.00= 0.00
0.28
67.29
0.00
0.28
w/ both
80.75 0.18
0.00 0.00
0.88
67.84
0.00
0.86
5.2 RESULTS WASK-INCREMENTAN SETTINGS
5.2.1 SELF COMPARISONS ON 20-Split CIFARIOO
In this section, we empirically analyze inter-task ensemble, intra-task ensemble, and selection of atoms of our method in the task-incremental setting. We analyze the performance of our base model with different number of atoms m, As shown in the left of Fig. 4 m = 12 is the best choice in terms of both performance and efficiency. We then test the task relevancy assessment based on subspace distance, and its instructive effect to inter-task ensemble. As shown in Fig. 5 the past models with small Grassmann distance to the current model lead to performance improvement by model ensemble. And past models with large Grassmann distance to the current one, in fact result in degraded performances. In general, we test the ACC of model ensemble with the top-1 to top-3 relevant model as well as the most irrelevant model starting from the 5-th task. As shown in Fig. [ model ensemble with top-3 relevant models achieves the best results, and thus we set E: = 3 in subsequent experiments. We then explore intra-task ensemble with different members E.,- 2, 3, 4. As illustrated in the middle of Fig. 4 ensemble within task enhances the performance consistently, and we choose E.. 2 for the best performance efficiency trade-off. We further illustrate the scalability of the proposed method on the right of Fig. 4 Compared to PNN (Rusu et al., 2016) and APD (Yon al. 2019), the size of our method scales much slower with the number of tasks, even with ensemble adopted.
5.2.2 COMPARISONS WITH BENCHMARKS
We further report our results on 20-Split minilmageNer and 20-Split CIFAR100 in Tab. 2 Compar- ing with eegularzatioonbbsed memory-based and expansion- based models, the proposed method achieves the best results even with the base model. Adopting inter-task ensemble with the top-3 relevant past models, along with intra-task ensemble with 2 members, our method achieves further improved results. Especially in 20-Split minilmageNNe, the previous state-of-the-an method, ACL (Ebrahimi et al.] 2020) achieves ACC of 62.07 with memory in size of 8.5 MB; whereas the proposed method improves the results significantly to 67.8 84 with a merely 0,86 MB memory. When comparing with other memory-based methods besides ACL, the proposed method demonstrates superior scalability reflected by the much smaller memory size. We provide additional results on the standard 5-Split MNIST dataset with our base model. As shown in Tab. D our base method outperforms both regularization-bast based methods and memory- methods terms of ACC with
86-
10.49
86
86
0.75
Method
ACC
Avg. de
Memory (MB)
82
Base
79.13
0.14
80
-1.13
w/ Ens. top-1
79.61 (+0.48)
0.43
0.23
78
w/ Ens. lop-2
79.82 (+0.69)
0.48
0.34
76
w/ Ens. top-3
79.91 (+0.78)
0.56
0.43
76
w/ most relevant
00.74
72
W/ most imolovant
0.67
w/ Ens. last-1
78.27 (-0.86)
1.21
0.23
Inter-task Ensemble Members (Ex)
Figure 5: (Plot) Ensemble effect of the base model with the most relevant and irrelevant past model. (Table)
Ablation studies number of ensemble selections
8

---

Published as a conference paper at ICLR 2022
much smaller memory. The improvements over the -tate-of-the-art methods and the outstanding scalability validate the effectiveness of our method on solving real-world continual learning problems.
6 RELATED WORK
Continual learning. Recent advances on continual learning are driven by three main directions, regularization based. memory- -bsed and expansion-bare -ased methods. (Kirkpatrick et al. 2017b
Aljundi et al., 8018a Lee ct al. 2017] Zenke ct al. 2017b; Kolouri et al.] 2019) determine the importance of each model's parameter per task. which prevents the important parameters from being updated for new tasks. For example, (Kirkpatrick et al. 2017b) specify the performance of each weight with the Fisher information matrix. Theses methods can naturally explored from the lens of Bayesian optimization (Nguyen et al., 2018 TTisias et al. 2020) Schwarz et al. [0018) Ebrahimi et al. 2019; Ritter et al., 2018). All these methods address catastrophic forgetting by adding regularization terms. As pointed out in (De Lange et al., 2019), the penalty term proposed in such methods are unable prevent drifts in the loss landscape of previous tasks. While alleviating forgetting, the penalty also unavoidably prevents the plasticity to absorb new information from future tasks learned over long timescale (Hadsell al. 2020).
(De Lange et al,, 2019) assumes is feasible to access data from previous tasks by having a fixed-size
memory or a generative model able to produces samples from previous tasks (Lopez-Paz Paz & Ranzato 2017 Riemer et al., 2018 Rios Itti. 2018, [Shin et al., 2017). (Rebuffi et al., 2017) introduces models augmented with fixed-size memory, which accumulates samples in the proximity of class
centers. (Chaudhry et al. 2019b) proposes another memory- memory-based -based model by exploiting reservoir sampling strategy in the raw input data selection phase. Rather than storing the original samples, (Chaudhry et al. (018a) accumulates the parameter gradients during task learning. (Shin al. 2017) incorporate generative model into a continual learning model to alleviate catastrophic forgetiigg by producing samples from previous task and retraining the model using data from previous tasks and the the current one. These methods assume an extra neural network, such as a generative model or a memory. Different from replay-based methods, which benefit from memory retrain their model over previous tasks, our method requires storage of tiny atoms for each previous task only. which is more scalable and do not suffer from potential forgetting caused by the inconsistent memory reply in generative- based methods.
Li et al. 2019) allocate subset of the model parameters for each task. Model expansion can be achieved by a gating mechanism (Wortsman et al.
(Rusu et al. 2016 Yoon et al. 2018 Jerfel et al. 2019
2020
Masse et al. 2018), or by incrementally adding new parameters to the models (Rusu
al.
2016). Incrementally ly learning and pruning provides another direction (Mallya & Lazebnik
2018)
Given an over- pvee-parametrized model with the ability to learn potentially many tasks, (Mallya
& Lazebnik 2018) achieves model expansion by pruning the parameters not contributing to the performance of the current task, while keeping them avail- able for future tasks, Comparing to the aforementioned methods, the proposed method provides filter subspace view of modeling multiple tasks, which further allows two kinds of model ensemble.
Filter atom decomposition. (Qiu et al. 2018) proposes an convolutional filter decomposition as a truncated expansion with pre-fixed filter atoms. It not only reduces the number of learnable parameters, but also imposes filter regularity with the usage of Fourier-Besse basis. This work further inspires other works in domain adaptation (Wang et al. (0220)), adaptive convolution (Wang et al. 2021b 2020a), image generation (Wang et al.) 222 2019), video understanding (Miao et al. 2021). rotation equivariance (Cheng et 20.1) and graph convolution (Cheng et al. 2020).
CONCLUSION
In this paper, motivated by the task subspace modeling literature, we enforced a low-rank filter structure to each CNN layer across tasks in continual learning. By performing atom-coefficient filter decomposition, we learned for each task a new filter subspace for each layer, while keeping subspace coefficients shared across tasks. This simple method allows highly efficient model storage and retrieval using small footprint atom memory. The proposed method provided a guarantee against forgetting, and we demonstrated further performance improvements through model ensemble. The performance was evaluated on various continual learning tasks, and the effectiveness and scalability were demonstrated by the state-of-the-ue accuracy and the tiny size of atom memory.
ACKNOWLEDGEMENT
This work is supported by DARPA TAMI program under No. HR001 12190038.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Riemannian geometry of grassmann manifolds with view on algorithmic computation. Acta Applicandae Mathematica, 80(2):199-22 2200992000222 2004.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pictro Perona. Task2vee: Task embedding for meta-lcaming. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6430--439, 2019.
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: I Learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV). pp. 139-154, 2018a.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (RCCV), pp. 139-154, 2018b.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:1 1849-11860, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems, 32: 22:11816-11 (--5, 2019b.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 583-592, 2019.
Leo Breiman. Random forests. Machine learning, 45(1):5-333 2001.
Francisco M Castro, Manuel J Marín-liménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248. 2018.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 092-547, 2018b.
Arslan Chaudhrr,, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018c.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania. Philip HS Torr, and M Ranzato. Continual leamming with tiny episodic memories. 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. In Advances in Neural Information Processing Systems, 2019h.
Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. Rotdef: Decomposition of convolutional filters for rotation-equivvarint deep networks. arXiv preprint 1xXxvv8805006844 2018.
Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph convolution with low-rank Icamable local filters. arXiv preprint arXiv:2008.0/888 2020.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting classification tasks. 1909. arXiv preprint 10Xi:3190.00883, 2(6), 2019.
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng. Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5138-51466 2019.
Simon Shaolei Du. Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2022
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guie continual learning in bayesian neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 75-78, 2019.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XI 16, pp. 286-402. Springer, 2020.
An Evgeniou and Massimiliano Pontil. Multi-task feature learning. Advances in neural information processing systems, 19:41, 2007.
Robert M Gray. Toeplitz and circulant matrices: A review. 2006.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual leamnnng in deep neural networks. Trends in Cognitive Sciences, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Saihui Hou, Xinyu Pan. Chen Change Loy. Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Ghassen Jerfel, Erin Grant, Thomas I. Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barskkaaaaaiika,,,,. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-333 3526, 2017a.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Barrskkkk, et al. Overcoming catastrophic
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka forgetting in neural networks. Proceedingy of the national academy of sciences, 114(13):351133526 2017b.
Soheil Kolouri, Nicholas Ketz. Xinyun Zou, Jeffrey Krichmar, and Praveen Pilly. Attention- based structural- plasticity. arXiv preprint 1XX3:v190300600 2019.
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519-3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Abhishck Kumar and Hal Daume III. Learning task grouping and overlap multi task Icarning. International Conference on Machine Learning, 2012.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances Neurul Information Processing Systems, 30, 2017.
Yann LeCun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278- 2324, 1998.
Sang-Wou Woo Lee, Jin Hwa Kim, Jachyun Jun, Jung Jung-Woo Ha, and Byoung- Tak Zhang, Overcoming catastrophic forgetting by incremental moment matching. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 4655-4665, 2017.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp. 3925-3934. PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935 2045-29977772 2017.
David Lopez-Paz and Mare Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6477 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7765-7773, 2018.
Nicolas Y Masse, Gregory D Grant, and David Freedman. Alleviating catastrophic forgetting using context- dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44): F10467-E11475, 2018.
11

---

Published as a conference paper at ICLR 2022
Andreas Maurer, Massi Pontil, and Bernardino Romera-Paredes. Sparse coding for multitask and transfer learning. In International conference on machine learning, pp. 343-351, PMLR, 2013.
Zichen Miao, Ze Wang, Xiuyuan Cheng, and Qiang Qiu. Spatiotemporal joint filter decomposition in 3d convolutional neural networks. Advances in Neural Information Processing Systems, 34, 2021.
John Milnor and James D Stasheff. Characteristic Classes.(AM-75) Volume 76, volume 76. Princeton university press, 2016.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729, IEEE, 2008
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference computer vision, pp. 524-540. Springer, 2020.
Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. Defnet: Deep neural network with decomposed convolu- tional filters. In International Conference on Machine Learning, pp. 4198-4207. PMLR, 2018
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Dock--Dicsttei... Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretabiliity. In NIPS, 2017.
Sylvestre-AIvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp.
2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2018.
Amanda Rios and Laurent Itti. Closed-loop gan for continual learning. International Joint Conference on Artificial Intelligence 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In NewrlPS, 2018.
Authony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123-146. 1995.
Bernardino Pooneaa-Prrdess, Hane Aung. Nadia Berthhieeer and Massimiliano Pontil, Multilinear multitask learning. International Conference on Machine Learning, pp. 1444-1452, PMLR, 2013.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell, Progressive neural networks. arXiv preprint arXiv: (e06:00066677 2016.
Syed Shakib Sarwar, Aayush Ankit, and Kaushik Roy. Incremental learning in deep convolutional neural networks using partial network sharing, IEEE Access, 8:4615 4628, 2019.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barikkaka Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: scalable framework for continual learning. In International Conference on Machine Learning, pp. 4528-4537 PMLR, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. International Conference on Machine Learning, pp. 4548-4585 4557. PMLR, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 2994-3003, 2017.
Binh Tang and David S Matteson. Graph-based continual learning. In International Conference on Learning Representations, 2020.
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In ICLR, 2020.
12

---

Published as a conference paper at ICLR 2022
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for shot learning. Advances in neural information processing systems, 19:3630-3638 2016.
7e Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. arXiv preprint arXiv:1909.12286 2019.
Ze Wang, Xiuyuan Cheng. Guillermo Sapiro. and Qiang Qiu. Acde: Weight sharing in atom- nom-ccefficiiett decomposed convolution. arXiv preprint arXiv:2009.02::0 2020a.
Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. A dictionary approach to domain-invariant learning in deep networks. Advances in neural information processing systems, 33:65955-6005, 2020b.
Ze Wang, Seunghyun Hwang, Zichen Miao, and Qiang Qiu, Image generation using continuous filter atoms. Advances in Neural Information Processing Systems, 34, 2021a.
Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu, Adaptive convolutions with per-pixel dynamic filter atom. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 12302-12311, 2021b.
Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and Perona. Caltech-UCSD Birds 200.
Technical Report CNS- TR 2010-001, California Institute of Technology, 2010.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Farhadi. Supermasks superposition. Advances in Neural Information Processing Systems, 2020.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Hwang. Scalable and order-robust continual Icarning with additive parameter decomposition. International Conference on Learning Representations, 2019.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost de Weijer. Semantic drift compensation for class-ineremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6982-6991, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskon- omy: Disentangling task transfer learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995 PMLR, 2017a
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017b.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming 7hang, and C-C Jay Kuo. Class-incremennall learning via deep model consolidation. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1131-1140, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions Knowledge and Data Engineering, 2021.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A EXPECTED EXCESS RISK
Problem setup: Suppose we have T tasks sequentially. The data distribution DD (X',, 3²") associates f-th lask (& 21. T)). For each task E [T', we assume ø' X - Z representation function, which maps input images to feature space Z C RR. These representation functions are restricted in function class o such as neural networks. There also predictor w' : Z y for task L [T] mapping the feature representation labels. Our object is minimize empirical risk:
Exe y (1(F)
For analysis purpose, we consider representation function o' as single convolutional layer, which be decomposed ø² D'. With the predictor w', the loss be written as:
N Ey: (a ((xx)))+)
1
2N
where D*(x) is the convolution operation on the input images, (-)+ is ReLU activation. As the input x E gexi and atoms D E m== where h w are the size of input images, e is number input channel. and m is the number of kernels which have size x k. D (x) contains c x m convolution operations that maps input x into the latent 21 E |oomxk xw", The latent 2; then maps into feature space Z by a x 2t.
We can treat each convolution operation as matrix product by converting the convolution kernel to a doubly blocked Toeplitz matrix D E h/ Gray 2006), and converting input image x into Therefore, the convolution operation D'(x) is translormed as reshaped image x' multiple by m reformed kernels D To simplify the problem, we assume consider each channel of the input images and each base of atoms, such that c = and m 1. Consider of all N' samples X E N xhw with labels y E R for task t, Thus, the loss can be written as:
Rex
N
C(F((2
Σ (y: - (w", (a (((x))))))
2N
2N Y Xa"x'"")
The first task trains the a. D and w at the same time, but the following tasks only optimize over D' and w' and keep a fixed. With fixed a. we can represent the samples X ====="''"" 'x' By solving the object, we get (D's w²). We also assume that there is an optimal atom D of representation function o² and prediction function w for task t. We now analyze the bound for the excess risk of our learned model on the task t, iee. how much our Iearned model (D', w') performs worse than the optimal model (D':", w" on the task t:
ER(6', w = Lp (D',
LD' w²) LDD w"'))
(I)
- )- (X (X'D'TLi))
"X w
A standard lifting of neural networks can be formulated as infinite dimension linear regression (Du et al. (020). Define the infinite feature vector with the coordinates p(z), = (bxx)) + for every E 3do-1 Let 3r be signed fgdo-1 d3.). By convert input feature a to the lifted feature vector 6(x), according to (Du et al. 2020 Equation. [ can be bounded by: '' T
measure gdo-1. The inner product notation denotes integration: 9 [(x)
R( w = D
Let B[xT]] Σ, we have
5 (X' "'D''"""T'"""" 1 The _ 5T
With Assumption 1 and 2, based on the inequality in (uu et al. 2020), for fixed 8 > O, we further have
| D rt. wh. TT < e R R R² + 2. + 2o
where 2 1 alogg 1/// (z) |لأرك Tr(E), E A (loog 9ogg))))))))))))))))))))))())))) / |لكر| are estimation
VNT
eтror and 6ie,1 = 200²
/NT (v Tr(E) + log ((|) Gu1 = VNT V ||Σ||| Rlog } are intrinsic dimension concen-
Op
tration error, R Hllaa D w 1|.. Therefore, suppose we have the same number of samples
14

---

Published as a conference paper at ICLR 2022
in each task, with probability at least predictor on the task satisfics:
å over the samples, the expected excess risk of the learned atom and
E|ER(O w'Il of Ō Tr(()
|E
+ pRR R² Ō Tr(2) |ا||
V
N7
B ALGORITHM
We provide the algorithm of proposed method in Alg.
Algorithm Continual Learning with Filter Atom Swapping
Initialize MD + ]. a = faslmm + a. D (D! + DI. for task t 1, T do if = then
Optimize a, D according Update Atom Memory MD MD, D
else
if < then
Initialize D'
(D D then optimize D' according to (2
Update Atom Memory MD t [Mp,
else
Initialize D' = (D:) i=] f- D then optimize D' according to (2)
Initialize distance list d' = [0, ..., 0j, where length(d*) = 1.
for j=1:t-1 do
Calculate das (D/, D/) in the last convolutional layer according to and update d' = dM Di)
end for
Sort d' in ascending order, and select least E, indices to construct St, Finetune additional n heads current task. Update Atom Memory MD < [Mn, D"].
end if end if end for
return MD. a.
C EXPERIMENTAL DETAILS
C. DATASET STATISTICS
We provide the dataset statistics of 10-Splir CIFARI00 used ilass-incremental setting Tab. and statistics
of 20-Split CIFAR100and 20-Split minilmageNetin task-incremental setting in Tab. and Tab.
Table A: Statistics of 10-Split CIFARIO0.
10-Spli CIFARI00
# tasks
10
Img. Size
32
32
# tasks/task
10
# Training samplcs/task
4500
# Validation samples/task
500
Test samples/task
100
C.2 NETWORK ARCHITECTURE
C.2.1 ARCHITECTURE FOR CLASS- CLASS-INCRENEN INCRSMENTMMEN LEARNING
We adopt the ResNet-32 as in Rebuffi al. 2017 which includes 31 convolutional layers and fully connected
layer. We substitute every convolutional layer with the one has the proposed decomposed filters (m 12) that has the number of input/ output channels.
15

---

Published as a conference paper at ICLR 2022
Table B: Statistics of 20- 20-Split CIFAR100and 20-Split minilmageNNet.
20-Split CIFARI00
20-Split minilmageNeet
# tasks
20
Img. Size
32 x 32 x
84 x 84
x 3
# tasks/task
5
# Training samples/task
2,125
2,125
# Validation samples/task
375
375
# Test samples/task
500
500
Table C: Statistics of 5-Split MNIST.
Task
(0,1)
(2,3)
(4.5)
(6,7)
(8,9)
Training samples/task
10,766
10,276
9,574
10,356
10,030
Validation samples/task
1,899
1,813
1,689
1,827
1,770
# Test samples/task
2,115
2,042
1,874
1,986
1,983
C.2.2 ARCHITECTURE FOR TASK-INCREMENTAL LEARNING
Architecture for 5-Split MNIST. Conv(k=3, m=12. 16)-ReLU- m=12,32)-Rell (ttppooorrtt2-Maaoooolo) Dropout().)--FC((2,
m=12, 64)-ReLU- BeelL-DDooaatOO22)GAAPEE223
where GAP stands for global average poling, Conv(k, m, c) means the convolution layer with kernel size m atoms, e output channels, Dropout(p) indicates the dropout layer with probability p. and FC(c) is fully-connect layer with c output channels.
Architecture for 20-Split CIFAR100and 20-Split minilmageNet. Conv(k33, m=12, 64)- MaxPool(2)-Convkk33, m=12, 22--RRLLU-MMM/Pll222CCoo// m=12, ReL---C3LU,,,,,, m=12, 256)- ReLU-Comw(k=3, m=12, 256)-ReLL (--6)---L---Maseee B000://UUMMMaPP-2-----A-Pooopp00
C.3 TRAINING DETAILS
C.3.1 CLASS-INCREMENTAL LEARNING
For CIFAR 100, we choose SGD with batch-size of 128, learning rate of 0.01, momentum of 0.9 and weight decay le-3. The model is trained for 250 epochs, with learning rale drop by 0.1 at the 100-th and 200-th epoch. For ImageNet-Subset,
choose SGD with batch-size of 128, learning rate of 0.05, momentum of 0.9, and weight decay Ic-4. The model trained for 150 epochs, with learning rate drop by 0.1 at the 90-th and 120-th epoch. For CUBS Flowers, choose SGD with momentum 0.9, batch-size of 64, and weight decay of le-4. On CUBS, the model is finetuned for 200 epochs with learning rate of le-2 (drop by 0.1 at the 100th and 160th epoch) on the first task, and trained for 120 epochs with learning rate of 5e-3 (drop by 0.1 at the 70th epoch) on task 2-6. On Flowers. the model is finetuned for 120 epochs with learning rate of le-2 (decay by 0.1 the 50th and 90th epoch) the first task, and trained for 70 epochs with learning rate of 1c-3 (decay by 0.1 the 40th epoch) task 2-6.
C.3.2 TASK-INCREMENTAN LEARNING
We chonse SGD with batch-size of 64 for all experiments under the task-incremental setting. For 5-Split MNIST; we train the model for 20 epochs with learning rate of 0.001. For 20-Split CIFAR 100. we train the model for 100 epochs with learning rate of 0.001, which drops by 0.1 the 60-th epoch. For 20-Split manilmageNet, train the model for 200 epochs with learning rate of 0.001, which drops at the 100-th and 150tth epoch.
C.4 METRIC
ACC and BWT used in the task ins-emncremetall settings can be expressed as,
ACC - AccT,
1 T
BWT
Acer, Acc;,i
where 7' total number of tasks, Acc,, is performance j-th task after learning i-th task (Lopez-Pal Paz & Ranzato 2017
16

---

Published as a conference paper at ICLR 2022
0.96
r: 0.5951
Filter subspace similarities of every pair of tasks
0.94
0.90
0.88
0.86
0.84
0.82
0.76
0.78
0.80
0.82
0.84
0.86
0.88
SVCCA similarities of every pair of tasks
Figure A: Correlation analysis between task similarities calculated with CCA (Raghu et al. 2017 and task similarities assessed with filter subspace distance.
C.5 CORRELATION ANALYSTS BETWERN CCA AND FILTER SUBSPACE DISTANCE.
We conduct the analysis between the task similarities measured with CCA (Raghu et al. 2017 on deep feature spaces, and the proposed task similarities assessed with Grassmann distance on filter subspaces. The analysis is conducted on 20-Split CIFAR 100with the proposed model. randomly select 50 task pairs from (2) task pairs. Given a task pair, we use 500 images from each task to calculate the CCA similarity, and use the inverse of their Grassmann distance between their last layers' filter subspaces as the filter-subspace similarity. As shown in Fig. A filter-subspace similarities quite correlates to CCA similarities which we deem as the golden standard, while they are significantly more efficient calculate with our atom continual lcarning framework.
C.6 ADDITIONAL TASK- LESK-INCREMNNAL EXPERIMENTAL RESULTS
We provide additional results the standard 5-Splir MNIST dataset. Note that due to the simplicity of 5-Split MNIST, only apply it base model, which composed cross-lask coefficient sharing and atom swapping only. As shown in Tab. [ our base method outperforms both regularization- bassd methods and memory-based methods in terms of ACC. Besides, proposed method demonstrates a guarantee against forgetting reflected by 0 BWT. Note that the proposed base model is much smaller in size compared to the other methods. Moreover, comparing to LopezLopez-Paz & Ranzato (2017) and VCI / et al. (2018). our base network adopts far smaller memory for storing merely filter atoms for layers, each of them has hundred of parameters only.
Table D: Results on 5-Split MNIST.
Arch. Size (MB)
Memory (MB)
Method
ACC%
BWT%
EWC KKrrpatrick al. 2017a
95.78 0.35
-4.20= 0.21
1.1
HAT
(Serra el al. 2018)
99.59.1
0.00.L
UCB
(Ebaahii of al.
2019
99.63 0.02
0.00: 0.00
2.2
VCL
Nguyen et al
2018)
95.97 1.03
4.62 1.28
1.1
GEM (Lopez- Paz & Ranzato
2017)
94,341 0.82
-2.01 0.05
6.5
0.63
VCI-E
(Nguyen et al.
(0
93.6+ 0.2
-3.10 0.20
1.7
0.63
ACL
Ebrahim et al
2020)
99.76.t 0.03
0.01 0.01
1.6
MIR
Aljundi
2019a
87.6011 0.70
7.00 0.90
5.9
0.71
GSS
Aljundi
2019b
84.80+t 1.80
3.7
0.58
Ours (Base)
99.844 0.05
0.00 0.00
0.24
0.04
17

---

Published as a conference paper at ICLR 2022
C.7 ADDITIONAL
RESULTS
C.7.1 ADDITIONAL RESULTS ON 10-Split CIFARI00
We provide results 10-Splir CIFARIO0 under class-incremental setting ResNet-18 (He 2016
Tab. and Fig. Our method achieves the best result with large margin.
Table Mlass-inremmenta results on 10-Split CIFARIO0.
Method
Acc. (avg.)
Acc. (last)
LwF (Li & Hoiem
2017
44.5
23.9
EWC [Kirkpatrick et
2017a
36.2
16.4
S1
(Zeke et al. 2017a>
37.8
23.3
MAS [Aljundi
20188)
8b)
33.4
15.4
RWalk (Chaudhry
2018b
35.2
17.9
DMC (Zhang of al.
2020)
57.1
36.2
(Base)
40.57
w/
62.4
46.37
w/ E
62.9
46.50
KCC
Figure B: Class incremental learning results on 10-Split CIFARI00.
C.7.2 ADDITIONAL RESULTS ON 10-Split ImageNet-Subel
We further validate our method on a large-scale scale image classification dataset, ImageNet- Subset, which a subset of the ImageNet dataset that contains the first 100 classes. As in Masanaet et al. (2020), we split 100 classes into 10 tasks with 10 classes per task. We adopt the ResNet- 18 as the architecture and substitute the convolutional layers with our decomposed version. Training details provided in Appendix C.3.
Table F: Class-incremental results 10-Split ImageNet-Subset..
Method
Memory
Acc. (last)
iCaRL Rebuffi
2017
43.6
EEIL
Castro el al. 20188
36.6
301.4 MB
BiC (Wu el al 2019
45.6
11.2M (Belouadah & Popescu 2019
38.2
Ours (Base) w/ E.. =
7.3 MB
43,9
7.7 MB
46.3
We benchmark our method with existing methods that store 2000 exemplars the external memory. Their results are faithfully reproduced in Masamact et al. (2020). As shown the Tab. [F] our method with intra-task ensemble achieves the best result with significantly less memory usage.
C.7.3 ADDITIONAL RESULTS CUBS AND FLOWERS
We further validate methods CUBS Welinder al. 2010) and Flowers Nilsback & Zisserman 2008 datasets. Following Yu al. 2020) we the random seed (1993) select 100/50 classes the first task, and
18

---

Published as a conference paper at ICLR 2022
evenly split the rest 100/50 classes into tasks. We adopt ResNet-18 pretrained ImageNet with substituted decomposed convolutional layers. Training details are illustrated Appendix C.3.
Table G: Average incremental accuracy CUBS and Flowers datsets
Method
CUBS
Flowers
LwF-E"
Li Hoiem 2017
69.8
87.2
EWC-E"
Kirkpatrick ct al 2017a)
69.7
85.9
MAS-E*
(Alundi el al. 2018b
68.5
84.7
SDC Yuu èt al. 2020)
70.0
86.8
Ours (Base)
69.3
86.9
w/ Ew
72.1
89.3
Under the inssemnncremttall setting, we compare the average incremental accuracy with other benchmarks. On both datasets, as shown in the table above, our method produces the best results.
C.8 ABLATION STUDY DISTANCE METRICS IN INTER-TASK ENSEMBLE
present an ablation study of different distance metries including the Grassmann distance, 1.2 norm, and cosine distance for inter-task ensemble. As shown Tab. H inter-task ensemble with Grassmann distance achieves the best result.
Table H: Ablation study on different metrics in inter-task ensemble.
Method
CIFAR-100
minilmageNet
Ours (Base)
79.13
66.01
w/ E. 3 (L2 norm)
80.35
67.13
w/ E. (cosine)
80.11
66.82
3 (Grassmann)
80.75
67.84
C.9 INFERENCE TIME COMPARISONS
On the CIFAR-100 100 dataset, we compare inference time on all classes with other methods. All methods are tested on a single RTX 2080ti GPU under the class-incremental setting. Our method has comparable inference time to other methods, as shown in the Tab.
Table I: Inference time comparisons on CIFAR100.
Method
Time (s)
EWC-E" Kirkpatrick et al 201
9.37
LwM-E
DDar et al
2019
9.71
iCaRL.
(Rebuffi al. Wu et al. 2019
2017
10.03
BiC
9.35
Ours (Base) w/ E... 2
10.27
13.51
C.10
MEMORY CALCULATION
C.10.1
CLASS-INCREMENTAL LEARNING
We calculate the external memory usage for exemplar- based methods with 2000 exemplars, e.g., Rebuffi et al. (017). and our method with atom memory.
2000 exemplars (CIFAR-100): 2000 (32 * 32 x 3 1)/1e6 = 6.15 MB.
2000 exemplars (ImageNet-Suusse)) 2000 (224 224 1)/166 301.4 MB.
Our (Base) (ResNet-32): (31 + 12 12
5 64 64 10
+ 16 32 9 32 + 64 * 9) 9 -
Our (w/ E., 2): (31 1
64 5 2+9
+64 1
9
+ (100))4411/)))
4/1e6
0.52 MB.
19

---

Published as a conference paper at ICLR 2022
1.10.2 TASK-INCREMENTAL LEARNING
We calculate external memory usage for our method in 20-Split CIFARI00 and 20-Split minilmageNee below.
Our (Base): (5 * 12 * + 256 + 51/ + 19 = 0.14 MB
Our (w/ E. = 3): (5 12 9 25 256 + /)/16// 4 19 = 0.43 MB
Our (w/ Eu 2): + 12 + 256 5 + 19 = 0.28 MB
Our (w/ both): (5 = 12 + 9 2
256 5 3) 1c6 * 4+ 19 = 0.86 MB
20


Reviews:
[{"review_id": "qiU2YcywCF-", "reviewer": "Reviewer_GWGw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is well-written and easy to follow. \n2. Learning new tasks on low-rank filter subspace seems interesting and novel for solving continual learning problems. \n3. The proposed method achieves good performance on the benchmark dataset. \n\nWeaknesses:\n1. For the experimental results, we can see that the SOTA performance of the proposed method is based on the ensemble scheme. However, it may not be fair when adopting the model ensemble strategy when compared with other methods, especially the intra-task ensemble. The intra-task ensemble will linearly increase the memory and computation consumption with the E_w. \n\n2. The proposed low-rank filter strategy would allow highly efficient model storage while it seems that it does not reduce the computation during testing. The authors may need to compare the inference time among different methods. \n\n3. The proposed method is based on the over-parameterization of deep models. If we use a lightweight network architecture, the advantage of the proposed method compared with other expansion-based methods would narrow down.\n\n\n", "summary_of_the_paper": "This paper tackles the continual learning via enforcing a low-rank filter structure to each CNN layer.  They first perform atom-coefficient filter decomposition and then learn each task with a new filter subspace, so that the method only needs to save the new filters for each task. The contribution of this paper includes the low-rank filter scheme and the designed intra-task and inter-task model ensemble performing on the filters. The proposed method also achieves SOTA performance on several datasets with tiny size of model memory.\n", "summary_of_the_review": "This is an interesting paper by introducing the idea of subspace modeling of tasks into continual learning and further proposing two model ensemble strategies to improve the performance. The proposed method also achieve good performance on benchmark datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "90W9wKmNyAN", "reviewer": "Reviewer_WerS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROs:\n- The paper is for most parts well written. With very few exceptions, all the text is clear at a first read. Figures, mathematical notation and equations are properly used and truly help in understanding several technical aspects.\n- The method proposed is technically sound. By keeping separate atoms for every task, they do not interfere with each other and therefore do not imply forgetting of past knowledge.\n- The results presented for class-incremental learning are very encouraging (Tab. 1), especially considering the model does not involve any buffer, that is considered almost a standard for such settings. Even in its absence, the model can outperform a number of competing methods that store examples from past tasks, at a fraction of the memory overhead.\n- To my knowledge, the low-rank decomposition of filters (a standard in other fields such as neural network compression) is novel in the continual learning field. This allows the network to grow as new tasks are encountered, like in other models such as Progressive Neural Networks [1]. However, by restricting the growth to the D components only, memory is impacted in a much more limited way.\n\n---\nCONs:\n- The class-incremental experiment, the most interesting in my judgement, is only conducted on a single dataset (CIFAR-100). It would be interesting to check whether the same findings hold on miniImageNet as well. Moreover, the authors explain such a setting by stating \"does not provide task-id during training\". However, the exact definition is that it does not rely on task-id during inference[5,6]. Can the authors confirm this is the case for the results presented in Fig.4?\n- A substantial part of Sec. 3 describes several ensembling techniques that are used in different settings within experiments. Although I understand ensembling is beneficial for performances on a single task, I consider this contribution somewhat orthogonal to the main aim of the paper. Indeed, it does not involve any interaction between different tasks during training and it is therefore not related to the sequential nature of the continual learning problem. It seems the authors included that as their model is structurally more prone to ensembling (which I acknowledge it is) that regular CNNs. For this reason, it is a bit difficult to assess this contribution. Couldn't other ensembling techniques be used, both for the proposed model and for competing methods? \n- The derived excess risk bound is not commented. The authors should add text describing what the derived bound means and why it is meaningful. Moreover, it is derived under a couple of strong assumptions, namely a single linear layer and a MSE objective, that are not met in the experiments carried out. As such, it is a bit difficult to assess whether that would generalize to deep networks used in practice.\n- To my understanding, the model as is does not allow for any forward knowledge transfer, as past Ds are not used during training of the current task. This seems suboptimal, as other \"architectural methods\" such as [1,2,3,4,5] envision this possibility by allowing past filters to be employed during future tasks. Here, this possibility is only granted by the ensembling mechanism, which is however not explicitly trained for the purpose of forward transfer.\n- The authors employ the Grassman distance in ensembling, in order to retrieve past task parameters that are somewhat similar to the parameters of the current task. This choice is not validated in experiments, and other types of distances should be tested, such as L2 or cosine similarity in parameters space.\n- In authors' proposal, the matrix alpha is trained on the first task only. This raises the question whether the overall model is susceptible to the choice of the first task. A good way to check this would be to have a set of first tasks $F=\\{T_1,...,T_m\\}$ and a set of remaining tasks $R=\\{T_{m+1}, ..., T_n\\}$. If we repeat an experiment where the first task is sampled from F, and then all tasks from R are trained in a sequence, would the outcomes (i.e. performances on $R$, excluding the first warmup task) have a high variance?\n\nMinor, questions:\n- After Eq. 5, a $()_+$ sign is explained. However, that does not appear in the equation.\n- Referring to Tab. 1, are all regularization-based methods used in a task-incremental way, by excluding classes belonging to unrelated tasks from the prediction?\n- In Tab.1, why does HAT have (slight) backward transfer? To my knowledge, weights for the previous tasks are fixed and therefore it should have exactly 0 forgetting and BWT (same as PNN and the model proposed by authors.)\n\n---\nReferences:\n- [1] Rusu, Andrei A., et al. \"Progressive neural networks.\" arXiv preprint arXiv:1606.04671 (2016). \n- [2] Serra, Joan, et al. \"Overcoming catastrophic forgetting with hard attention to the task.\" International Conference on Machine Learning. PMLR, 2018. \n- [3] Mallya, Arun, and Svetlana Lazebnik. \"Packnet: Adding multiple tasks to a single network by iterative pruning.\" Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018. \n- [4] Mallya, Arun, Dillon Davis, and Svetlana Lazebnik. \"Piggyback: Adapting a single network to multiple tasks by learning to mask weights.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018. \n- [5] Abati, Davide, et al. \"Conditional channel gated networks for task-aware continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. \n- [6] Van de Ven, Gido M., and Andreas S. Tolias. \"Three scenarios for continual learning.\" arXiv preprint arXiv:1904.07734 (2019).", "summary_of_the_paper": "This paper introduces a model for continual learning based on the decomposition of linear filters into low-rank components, called atoms. Specifically, the authors decompose convolutional filters shaped (c,c',k,k) into two components: i) alpha, shaped (c,c',m) and D, shaped (m,k,k). The former is learned on the first task and then frozen, whereas for D every task has its own and they are do not conflict during optimization. On top of that, the authors envision two different ensembling schemes that improve performances. i) First, in task-incremental settings, they retrieve atoms from the task of interest and from similar tasks as well (based on SVD decomposition of D matrices and Grassman distance) and ensemble them. ii) Furthermore, in class-incremental settings, they explicitly setup multiple atoms per task, building a task ensemble. During inference, all ensembles are queried and their predictive variance is used to \"discover\" the relevant task, for which a prediction is carried out. Experiments are carried out on 3 datasets in both task-incremental and class-incremental learning settings.\n", "summary_of_the_review": "In my opinion, the paper introduces a very simple model for continual learning and showcases extremely encouraging results in class-incremental learning. The submission could further improve by including class-incremental experiments on a second dataset (miniImagenet) and clarifying the points raised above about model ensembling and risk bound.\n\n---- POST REBUTTAL COMMENT ----\nAs detailed in my answer below, I will increase my score to 8 after reading the author's feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7GS3LCu7BC1", "reviewer": "Reviewer_ymsc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) Interesting idea of considering filter subspace with the construction of filter atoms and sharing the subspace coefficient fixed while changing the small set of filter atoms corresponding to the task. \n(-) Different distance metrics other than Grassmann should be considered to see whether it leads to consistent performance. the computational required to compute the distance between filter subspaces requires calculating an SVD which is computationally heavy.  \n(-) The performance of the proposed approach will depend on the dataset at hand. For very diverse tasks, an inter-task ensemble would not be very effective. \n(-)The performance of the proposed algorithm can vary heavily on the diversity of tasks. \n(-) (Introduction) I do not agree with the first sentence of the introduction. Humans do forget concepts learned in the past. \n\n", "summary_of_the_paper": "The paper proposes a continual learning algorithm that enforces the convolutional filter in each layer to a low-rank filter subspace defined by a small set of filter atoms. For each task, each convolutional layer is defined by a new filter subspace but subspace coefficients are shared among the tasks. The algorithm is validated on multiple benchmark datasets. ", "summary_of_the_review": "The paper is written and organized adequately. The paper is based on extensive experimental results. The idea is relatively simple and performs well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5IshJu3OVt", "reviewer": "Reviewer_B5kZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the idea of the article is concise and clear. The survey of related work and experiment settings are sufficiently detailed. More importantly, there is a huge improvement in memory saving compared with other methods.\n\nHowever, there are two main problems that I am concerned about:\ni. In Sec 5.2.2, there are some recent methods that show better performance on Cifar100(20 tasks) in Paper with code. But the author doesn't consider them, such as \"Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping\", \"Compacting, Picking and Growing for Unforgetting Continual Learning\", etc.\nii. I think only the results of the two datasets are not enough to support the author's conclusion strongly. The author would better add a few more experiments, and there are plenty of datasets shown in other papers for your choices, such as CUBS, Stanford Cars and Flowers, etc.", "summary_of_the_paper": "The paper, motivated by the task subspace modeling literature, enforced a low-rank filter structure to each CNN layer across time in continual learning. It not only ensures that the knowledge of the past tasks is not lost but also saves a lot of computing memory. Meanwhile, the paper proposes novel intra-task ensembles and inter-task ensembles for class-incremental settings and task-incremental settings, respectively. ", "summary_of_the_review": "The paper idea is creative and has made a significant improvement in saving memory, but more experiments are still needed to support the conclusion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a memory-based continual learning method that decomposes the models' parameters and that shares a large number of the decomposed parameters across tasks. In other words, only a small number of parameters are task-specific and the memory usage of storing models from previous tasks is hence a fraction of the memory usage of previous approaches. The authors take advantage of their method to propose specific ensembling approaches and demonstrate the strong performance of their methods using several datasets.

In the rebuttal, the authors were very reactive and provided many useful additional results during including a comparison of the computational cost of their method vs. others, results using two new datasets (CUBS & Flowers), and additional results on mini-ImageNet. They also answered, through additional experiments, several reviewer questions including the robustness to different first tasks in the sequence.

Overall, the reviewers after the rebuttal/discussion period agree that this is a strong contribution: novel and fairly simple method with some theoretical justification, thorough empirical evaluation, well-written and easy to follow manuscript. It also opens a few interesting avenues some of which the authors have already explored in their paper (e.g., ensembling).
Decision: Accept (Spotlight)