ImpressLearn: Continual Learning via Combined Task Impressions
Authors: Dhrupad Bhardwaj
Julia Kempe
Artem M Vysogorets
Angela Teng
Evaristus Ezekwem
Venue: ICLR 2022 Submitted (2022)
Keywords: Catastrophic forgetting
Continual learning
Neural networks
Masking

Abstract: This work proposes a new method to sequentially train a deep neural network on multiple tasks without suffering catastrophic forgetting,  while endowing it with the capability to quickly adapt to unknown tasks.  Starting from existing work on network masking (Wortsman et al., 2020), we show that a simple to learn linear combination of a small number of task-specific masks (”impressions”) ona randomly initialized backbone network is sufficient to both retain accuracy on previously learned tasks, as well as achieve high accuracy on new tasks.

In contrast to previous methods, we do not require to generate dedicated masks or contexts for each new task, instead leveraging transfer learning to keep per-task parameter overhead negligible. Our work illustrates the power of linearly combining individual impressions, each of which fares poorly in isolation, to achieve performance comparable to a dedicated mask. Moreover, even repeated impressions from the same task (homogeneous masks), when combined can approach the performance of heterogeneous combinations if sufficiently many impressions are used.

Our approach  scales  more  efficiently  than  existing  methods,  requiring  orders  of magnitude fewer parameters and can function without modification even when task identity is missing.  In addition, in the setting where task labels are not given at inference, our algorithm gives an often favorable alternative to the entropy based task-inference methods proposed in (Wortsman et al., 2020).  We evaluate our method on a number of well known image classification data sets and architectures

Under review as a conference paper at ICLR 2022
IMPRESSLEARN: CONTINUAL LEARNING VIA COM- BINED TASK IMPRESSIONS
Anonymous authors Paper under double-blind review
ABSTRACT
This work proposes method to sequentially train deep neural network on multiple tasks without suffering catastrophic forgetting, while endowing it with the capability quickly adapt unknown tasks. Starting from existing work on network masking (Wortsman et al. 2020), we show that a simple to learn linear combination of a small number of task-specific masks ("impressions") on a randomly initialized backbone network is sufficient to both retain accuracy on previously learned tasks, as well as achieve high accuracy on new tasks. In contrast to previous methods, we do not require to generate dedicated masks or contexts for each new task, instead leveraging transfer learning to keep per-task parameter overhead negligible. Our work illustrates the power of linearly combining individ- ual impressions, each of which fares poorly in isolation, to achieve performance comparable to dedicated mask. Moreover, same task (homogeneous masks), when combined can approach the performance of heterogeneous combinations if sufficiently many impressions are used. Our approach scales more efficiently than existing methods, requiring orders of magnitude fewer parameters and can function without modification even when task identity is missing. In addition, in the setting where task labels are not given at inference, our algorithm gives an often favorable alternative to the entropy based task-inference methods proposed in /Wortsman et al. 2020). We evaluate our method on a number of well known image classification data sets and architectures.
repeated impressions from the
Keywords: Catastrophic forgetting, Continual learning, Neural networks, Masking
INTRODUCTION
Sequential learning without catastrophic forgetting has been area of active research in machine learning for some time (Maes et al. 1996; Thrun & Pratt] 1998; Serra et al., 2018). A precondition for achieving artificial general intelligence (AGI) is that models be able to learn and remember wide variety of tasks sequentially, without forgetting previously learned ones. In real-world scenarios, data from different tasks may not be available simultaneously, which makes it imperative both allow continued learning as well as to avoid catastrophic forgetting of a potentially unbounded number of tasks (see also The Sequential Learning Problem (McCloskey & Cohen 1989), Constraints Imposed by Learning and Forgetting Functions (Ratcliff, 1990) and Lifelong Learning Algorithms (Thrun & Prati 1998)). Recently, some successful approaches to combat this problem use task specific sub- models, which allow neural networks to context- switch between different learning tasks (Wortsman al. 20200 Mallya et al.) 22088] Mancini al., 2018). The underlying context for each task can be represented as "filters" or "masks". " altering the network connections for each task. Yet all of these approaches scale unfavorably with the number of unique tasks to be learned.
We propose a novel method which exploits transfer learning and network masking to sequentially learn a theoretically unlimited number of tasks with much lower parameter overhead than prevailing benchmarks. Our method, termed ImpressLearn, uses elements from Supermasks in Su- perposition (SupSup) by Wortsman et al. 2020). The SupSup approach leverages the observation that even within randomly weighted neural networks there exist task-specific supermasks subnetworks produced by overlaying a binary mask that selectively removes connections whichhhhs--hhhhhhhhhhhhhhh which achieve good performance the task. These supermasks can be learned from task-specific data and stored, one
Our contribution:

---

Under review as a conference paper at ICLR 2022
mask per task. At inference, the appropriate task-specific mask is applied when task identity is known. When tabe-s of a previously seen task are not known, the correct mask can be inferred via entropy considerations.
"basis" of masks: Our approach goes further, driven by two core ideas. In order to dramatically improve scaling of the parameters with the number of different tasks, reduce the number of necessary masks to a constant number, independent of the number of different tasks. Our basis-masks are constructed from a small number of initial tasks. Leveraging transfer learning, this set of learned masks, each of which can be interpreted as an "impression" of a previously seen prior task, serves as set of latent features for new learning objectives, encoding common structural information. Our second idea is to use the power of linear combinations to combine these impressions to quickly learn a real-valued mask that performs well for new tasks. Hence, other than fixed number of basis-masks, only a small number of coefficients need to be learned and stored for each subsequent task. This greatly benefits scalability, major drawback of previous approaches, as this in principle allows for unlimited number of new tasks while the parameter overhead is several orders of magnitude smaller than storing even a compressed binary network mask.
C,
CC2
a,
Bactbone Nitwort
Radl Mikudd Meak
Spaina Berayy Mork
@
Improssian Codffoont
Figure 1: Overview of the ImpressLearn method (best viewed in colour). Given a fixed-weigtt randomly initialized backbone network, we show that few binary mask "impressions" from training prior tasks (3 in this case) can form a linear combination with coefficients a to learn new tasks. Unlike the impressions, the resulting mask is value,, and not binary.
Specifically, ImpressLearn uses the first few tasks from a multi-task set to generate relatively small number of binary masks. For subsequent tasks, these impressions are used akin to basis: new tasks are learned through a simple linear combination the basis-impressions. This might be reminiscent of associative learning where impressions of previous scenarios are combined to cope with new ones. In order to increase the expressivity of this approach, we allocate one real-valued coefficient for each layer of weights of the underlying model. Learning these linear combinations is fast and simple; and storing the coefficients requires only trivial overhead per task. For instance, in the case of LeNNet-30-111 100 on RotatedMNIS" with a basis set of 10 masks, new tasks require only 3 x 10- = 30 parameters while a new SupSup mask at e.g. 10% sparsity requires 2 25 weight indices specify. We provide ample empirical evidence of the efficacy of our Impress approach on a variety of benchmarks, outlining the radical savings in parameters needed to be stored per task compared to SupSup.
Homogeneous masks: Somewhat surprisingly, we can even generate all basis-masks from the same initial task using different random seeds for the learning algorithm (but the same random backbone network). We show that with a sufficiently large number of such "homogeneous" " impressions, our algorithm learns linear combinations with close to benchmark accuracy on new tasks. We are reminded of an infant learning by taking different "snapshots" of the same object, to infer properties of another. This "homogeneous impression" method is particularly useful to address possible drifts in the data. Akin to ensembling, it leverages the power of linear combinations for transfer learning. An additional important advantage is that the homogeneous method has no limit on the number of basis-mant mask can generate ab initio (since this requires single task only).
To provide another benchmark for our approach, we also try our linear combination approach with set of completely random masks of desired sparsity. We demonstrate several benchmarks that we chose a sufficiently large set of such random basis masks, our optimization still yields good accuracy. While naturally, combinations of random masks lag behind the heterogeneous and also
2

---

Under review as a conference paper at ICLR 2022
the homogeneous setting, show that there is a trade-off between number of masks and their specificct monnraadommess). In settings where producing task-specific basis masks is costly and where we might not wish to perform the SupSup mask search, our linear combination optimization still can yield satisfactory results with a large number of random masks.
RotMNIST To illustrate our approach and its performance, Fig. shows the accuracy of our ImpressLearn approach, compared to SupSup on the Rotated MN ST task This allows to share some observations which generalize to a wide variety of benchmarks (see Sec. First, to show the power of the linear combination approach beyond pure transfer learning, we evaluate the accuracy of basis mask on tasks different from the one the mask was trained on: we get close to random accuracy ("X"' in Fig. [ leftmost box plot). Next we illustrate how a linear combination of small number of masks from different tasks (heterogeneous impressions) achieves close to benchmark accuracy on new tasks. In this case, combining few masks already gives performance close to, even exceeding, the SupSup benchmark. This followed by the performance of set of homogeneous masks, where all impressions are generated from the first task (middle of Fig. 2). We can see that when using homogeneous impressions, we need larger number of basis masks to achieve aceuracy comparable to the heterogeneous setting, but that ultimately here also we match SupSup accuracy, with a vastly smaller number of parameters than the SupSup approach. Lastly, in the right part of Fig. 2 we study how our linear combination approach performs tasks from the basis set when task labels are not provided at inference (basis-GN setting). Here, the our optimization finds the "correct" mask, or a linear combination of equal or better accuracy, providing an alternative to the entropy- entropy-based GN-inference of (Wortsman et al. 2020).
Example: LeNet-300-100
Supsup Base
Rardom Guess
66
25
Accrery
044
000
000
100
150
200
240
100
200
200
Figure 2: Left: Average validation performance on 10 new Rotated MNIST tasks by number of impressions and impression type. X indicates incorrect mask. Numbers 5, 10, 10,... indicate heteroge- neous and numbers with indicate a homogeneous impression set. Right: Average performance on tasks to min(IMRom 25) basis-Gi regime on basis tasks without task identity. All results are averaged over 3 different seeds and masks of density E (5, 8, 10, 15, 20, 25, 30} x 10-2
Unknown task labels at inference and new GN algorithm: Wortsman et al. 2020) showed that in the so called GN regime where task labels are given at training but not at inference, entropy minimization starting from a uniform combination of (binary) masks quickly singles out the "correct" mask for a task. Even though our masks for bon-b tasks are real-valued linear combinations of binary masks, we show that in the GN setting similar entropy minimization allows to infer the correct real-valued mask with the same approach. Moreover, in what we call the basis-GN regime where the unlabeled tasks come from the set of basis tasks, we show that our ImpressLearn optimization finds linear combination that either favors the correct mask yields an even better accuracy than the corresponding basis mask without using the entropy of the output. This means that the optimization routine of Impresslearr gives an alternative algorithm for inference in the case of unknown task labels at test time: it finds a linear combination of masks that often outperforms the corresponding basis mask.
In Section 2 we briefly review related work and general approaches to countering catastrophic forget- ting, highlighting research that motivated our approach. In Section 3 we lay out the ImpressLearn algorithm, including our modified objective function. In Section 4 we demonstrate the effectiveness variety of benchmark data sets and architectures to show close-to-bent -one-hbbbbnccaarkkkkk of ImpressLearn drastically reduced parameter count on new tasks, especially where the number performance with a of possible tasks is large. We discuss our results including the trade off's between homogeneous and
3

---

Under review as a conference paper at ICLR 2022
heterogeneous masks. Finally, in Section 5 we talk about limitations of our work, and areas for future research.
RELATED WORK
In practice, intelligence systems should be able to learn a variety of tasks incrementally and without ex- periencing c forgetting-degrrdinn performance on previously acquired skills (McCloskey & Cohen 1989), while possibly transferring current knowledge to facilitate future training. Often- times tasks are diverse and not available concurrently, making joint training impractical. Conversely, ordinary finetuning (continued training of pre-trained network) inevitably leads to catastrophic forgetting. Continual learning encompasses broad spectrum of algorithms and architectures that address these issues and propose systems capable of learning from an incremental stream of tasks while minimizing catastrophic forgetting. Most naturally, these techniques are categorized into three groups described below (Delange et al. 2021 Wortsman et al. 2020).
Regularization-basee methods: This class of algorithms trains on new tasks by finetuning weights but attempts to retain performance on previously learned tasks through regularization. A number of studies assess the importance of individual parameters for previous tasks and penalize their displacement accordingly during optimization. Pioneering this approach, [Serra et al. 2018) estimates parameter importance using saplace-upproximaaed posterior distribution after training on earlier tasks. IZenke et al. (2017) impose quadratic penalty proportional to the accumulated sensitivity of previous loss functions to perturbations in the corresponding parameter; AAljundi et al. (2018) use the same strategy but accumulate sensitivity of the network output to parameter perturbations instead. In contrast, Li & Hoiem (018)) regularize by means of distilling current knowledge on the incoming task's data and using it during finetuning. Regularization-ben additional memory overhead per task and hence are advantageous in capacity constrained settings. However, the plasticity of a network decreases with more tasks, imposing natural limit new tasks, which creates a trade-off between learning new tasks and catastrophic forgetting.
methods require no
Techniques in this category preserve performance prior tasks by replaying, rehearsing or otherwise utilizing representative samples from the corresponding data distributions. Most commonly, replay models store examples of seen data in a separate memory buffer (Rebuffi) et al. 2017 Rolnick et al. 2019:) Riemer et al. 2019); others maintain generators that approximate the original data distribution and provide
Replay methods:
exammlles (Atkinson al. 202T; [Shin et al., 2017). While the majority of algorithms in this group replay stored examples during optimization to mimic joint training, Lopez-Paz & Ranzato (2017) uses them to constrain optimization space and ensure positive knowledge transfer. Replay methods require additional memory to store data samples or allocate generators, however, these costs are usually kept fixed. For this reason, like regularization- based models, replay models exhibit poor stability trade-off with more tasks, but often come with increased memory requirements when compared to regularization based methods.
Parameter isolation methods: These methods allocate new parameters for incoming tasks and feature little to no interference between previously learned tasks. Rusu et al. (201) allocate a new copy of the network and enable forward transfer learning with lateral connections going into new modules. Ren et al.] (2017) combine individual learners in a decision tree and eliminate outdated models with tree pruning. A large body of recent algorithms piggyback on single backbone network shared by all tasks. As such, Wen et al. (2020) (BatchEnsemble) operate on a fixed pretrained network and, for each incoming task, optimize for a rank one parameter mask applied to the backbone at inference. Mallya & Lazebnik (2018) (PackNer) use pruning to assign subsets of free parameters of a backbone network to individual tasks by issuing one binary parameter mask per task. Assigned parameters are forever frozen at their trained values, limiting capacity of the network for future tasks. In a subsequent study, Maalya et al. (2018) (Piggyback) lift this limitation by directly optimizing per-task binary masks and applying them fixed pretrained network.
SupSup Our ImpressLearn algorithm is most closely related to yet another similar method called SupSup (Worsmman et al., 2020). This algorithm trains individual per-task binary masks but applies them to a cundooml-initialle network, leveraging the existence of supermasks (Zhou et al. (019). The mask optimization algorithm, edge-popup (Ramanujan et al. 2019) uses heaviside function to
4

---

Under review as a conference paper at ICLR 2022
binarize mask values the forward pass and employs straighttthrouge estimator when computing gradients. In addition. Wortsman et al. (2020) propose different training and inference modes depending on availability of task Identifiers; e.g., GG refers to the scenario when task identifies are known during both training and inference, while in GN they are available only during training. For the GN case Wortsman et al. (2020) introduce one-shot algorithm minimizing the output entropy, starting from a uniform linear combination of masks and optimizing the coefficients. While at first glance this algorithm resembles ours, there are essential differences as we use optimization of a refined linear combination to learn new tasks. While SupSup and other related methods suffer no catastrophic forgetting regardless of the number of tasks, they are required to store the corresponding parameter mask for each task, which is costly. SupSup try to address this by storing masks as attractors of Hopfield network, but it is unclear how feasible this approach is.
infer task identity by
3 APPROACH
Preliminaries and notation. We largely adopt the notation from Wortsman al. (2020). For the standard 1-way classification task from a set of tasks T, inputs x are mapped to a distribution over output neurons {1, .... 1). Let f be network architecture defined over the backbone weight matrix W, which is taken to be random but fixed. Similar to (Wortsman et al. 2020) we use the Edge-Popup training algorithm of Ramanujan et al. (2019) (based on earlier work by /Zouu et al. (2019)
train a binary mask M2 for task / E T. We further stratify each mask by layer: let d be the number of weight-layers of the network computing f. For i [L...., d] denote by .M. the part of the binary matrix corresponding to layer of the network such that M² = @,M Similarly, let W2 denote the submatrix of W corresponding to the ith layer. The sparsity E (0, 1] of mask M given by the fraction of is in the mask and is usually fixed in advance to D For each task t E T the SupSup algorithm aims to find a mask M of sparsity s to minimize E( C(,.)) f(x, M @ W)), where is the model's ' loss function (cross-entropy in most cases), y E (0, 1) 1)' is the one-hot encoding of the label and defines the element-wise product.
Supermask training with edge-popup: For each task t, the edge-popun pageeppopp algorithm in SupSup learns a score matrix S with the same dimensions as W via gradient descent as function of the (unchanged) weights and the loss function. It then generates the mask M by setting the top fraction of scores in S and the rest to O.
Inference without task labels (GN): The task inference idea presented in Wortsman et al. (2020) is based on the intuition that the correct mask for a task should give a highly certain (i.e. low entropy) model output. The algorithm hence starts with an equally weighted combination of masks and uses gradient descent on these coefficients using output entropy as the loss function. This quickly singles out the correct mask by up-ueige weigting the corresponding coefficient. We adopt this algorithm without change to our real-valued masks in the GN scenario.
The ImpressLearn algorithm. For ImpressLearn we define a set basis tasks To C T and call its compliment T. = T T₁ the set of "new" tasks. We randomly initialize and freeze once and for all a backbone network with weights w
Step 1: For each & E To basis-masks M leading to set M = {M M .M...., of basis-masks. We use up to 250 basis masks for various benchmark architectures and data sets.
the edge-popup algorithm in SupSup to create one several
Step 2: For each new task s E T, we define a coefficient-matris a E M|xd To find the layerwise linear combination of basis masks, we optimize
Σ af @ ow))) M'EM
& arg min L
v.f x,
'Note that often sparsity defined in complementary way the fraction of zeros in M S but we will
keep consistency with Wortsman 2020 weights are deactivated.
We consider sparsities (0.05, 0.5), where 0.05 means 95% of
Various standard initializations possible; use Kaiming normal distribution He et al. 2019 and set the biases zero.
5

---

Under review as a conference paper at ICLR 2022
Initial Condition on the on. A priori all basis masks have equal chance to contribute to new task. Moreover, testing a basis mask for task t on any other task t' * t gives close to random performance, meaning there is no direct knowledge transfer (see Section 4 Hence, uniform prior on the distibution of the a-coefficients per layer is a reasonable assumption. We treat each layer independently, setring 02,t M/[M| the start of the wradient-based optimization.
Regularization: While overfitting is less of a concern, given the small number of parameters in our optimization Eq. ①I want to enforce that our algorithm aims for sparse combination of masks when possible, especially when faced with one of the basis tasks, where we expect our algorithm to identify the "correct" mask among the basis masks. For some benchmarks we hence apply an Ll Ll-penalty for each layer for deviations from unit L1 cloeoffficient norm, to obtain the loss function:
2
M Σ 2=1 1=1
J
-
Heterogeneous vs homogeneous masks, random masks and overlap: In our heterogenous ap- proach, we create one mask for each task in Ty. In this setting we leverage knowledge transfer from all basis tasks for new task. However, in some settings with few tasks, limiting mask generation to one per task affects the viability of our method and doesn't 't allow for scaling benefits to become apparent. For example, for Sp1R--CIFAR-100 with 20 taks, can only generate at most 20 heterogeneous masks. We have hence also evaluated our approach with a set of basis masks all coming from the same task, so called homogeneous masks. priori is unclear whether masks produced on the same backbone network for the same task are sufficiently different to generate a diverse enough basis set. While previous work suggests that wider network architectures can support multiple suitable subnetworks for given task Ramanujan et al. (2019); (Frnklee & Carbin (219)), we find this effect is prominent even using relatively conservative architectures such as LeNet 300-100 with Permuted MNIST (Lecun et al. (1998). Our experiments show that masks are very sensitive to initial conditions of the popup scores and to data ordering: the overlap of homogeneous masks produced with different random seeds is close to the overlap of randomly picked masks. Hence, at sufficiently low sparsities, homogencous masks, even on the same backbone network, are close to independent.
Our approach can be extended to mix of homogeneous and heterogeneous impressions, though in this paper we only study the trade-off for "pure" homogeneous and heterogeneous settings.
To quantify the importance of task specific data in mask generation we also compare ImpressLearn with the scenario where in Step the basis masks are picked randomly instead of through the SupSup popup algorithm (see Sec. 4 This allows us study the trade-offs between random and task-specific tasks and gives an alternative in the case where the SupSup mask search is hard to implement. In this setting, for a fixed sparsity, masks are chosen uniformly at random (by picking edges one at a time without repetition until the desired sparsity is reached). This mask set is then used for Step 2 of Impress
optimize the a-coefficientss. While in general this approach requires larger base set than SupSup-based basee approaches, it avoids the SupSup optimization step, which in some scenarios could be costly (since it requires optimization of larger number of parameters than our Step 2).
Optimization and parallelization: We use the SupSup algorithm with random initializations for the edge popup scores and data ordering to generate the basis mask set M. One fetching attribute of using a set of basis masks is the ability to parallelize the training process. Our approach allows for all masks to share the same backbone network, both in the heterogeneous and the homogeneous setting. Thus one can parallelize the same model and train multiple tasks at the same time different GPUs, only constrained by the number of cores or GPU accelerators available.
To optimize the a for new tasks, we use stochastic gradient descent.
4 EXPERIMENTAL RESULTS
4.1 DATA, MODELS AND PARAMETERS
We evaluated ImpressLearn over the following classification datasets: MNIST Leeun et al. Permuted and Rotated; Split CIFAR- 100 (Krzhevsky)] (2112) and Split ImageNET (Deng et al. A detailed description of our experimental choices and infrastructure can be found in App. A.
1998):
2009

---

Under review as a conference paper at ICLR 2022
4.2 RESULTS
Our results are presented in Figs. (Scc. 4 and S We have also performed a couple of sanity checks evaluate our algorithm and understand which ingredients contribute its performance.
Evaluating with "incorrect" masks To make sure that any accuracy improvements from the linear combination are an effect of the optimization routine of the coefficients itself, and do not come from chance application of a suitable mask that does well on other tasks, we test impressions derived from one task on other tasks. We find that the incorrect mask fails have any predictive power on other tasks when used in isolation, giving roughly random accuracy of 10 + 3% for Permuted/Rotated wowoottteddIIS//Spppppia SaSpim (Figs. and and 20 2% on SplitCIFAR (Fig. benchmarks respectively.
Performance on unlabeled basis tasks basis-Ghi GN setting: evaluate the strength of our opti- mization routine, we have studied the "basis" "NN setting, by which we mean that our model is presented with one of the basis tasks without explicitly providing a task label (as opposed to the full GN setting where the unlabeled tasks come from the basis and the new set). We show that for most benchmark our algorithm is able to either find the correct mask or a linear combination of basis masks of equal often better accuracy. As such, our a-optimization algorithm alternative to the one-shot entropy based algorithm of SupSup for task inference in the GN setting. given in App. A.3
serve
More details and comparison for PermutedMNIST
Results: Overall, our results demonstrate the strength of Impress in various settings. We see that the Impressleern coefficient optimization is able learn new tasks well with fraction of the parameters required by other approaches. In line with expectations, we see a strong positive relationship between accuracy and the size of the impression set [.M,, with saturation reached different dlta-specifie sizes. While the number of heterogeneous masks required for good accuracy varies by dataset, ImpressLearn is particularly resource efficient when the number of possible tasks is large and it becomes costly to store a separate mask for each task (e.g. RotatedMNISTT and PermutedMNNSTT), We also see that compared to the heterogeneous scenario we need more homogeneous basis masks to achieve similar performance (and even more random masks, see App. A.4)
Suparp
2.
240
Figure 3: Left: Average performance on 10 new Pe PermutedMNIST tasks by number of impressions and impression type. X indicates incorrect mask. Numbers N indicate heterogencous and N homogeneous impression set. Right: Performance of our a-optimization in the basis-G) regime averaged min) 25) basis tasks. All results are averaged over 3 different seeds and masks of density E (5,8 8, 10, 15, 20, 25, 30) x 10-2
For the SplitImage-Net task we only show results for heterogeneous basis mask set of < 35 masks due to limitations on compute. We anticipate that a larger basis mask set will achieve performance closer to the SupSup benchmark and highlight again that approach scales to unlimited number of tasks with few extra parameters. Note also that for SplitImageNet, a-optimization yields better accuracy than the SupSup baseline the entropy based approach of (Wortmman et al. 2020), highlighting the power of our linear combination routine.
the basis-GN regime our
GN setting: We have verified several benchmarks that the one- shot algorithm of Wortsman et al. (2020) continues to work in our setting where the masks for new tasks can be viewed as real-valued masks obtained through linear combinations of binary (basis) masks. Our experiments show that the

---

Under review as a conference paper at ICLR 2022
one-shot algorithm continues to work without change. For PermutedMNIST for example, in the GN scenario the algorithm was able to find the correct mask 94.6 % the time in epochs, and 98.1 within 5 when inferring identity among 250 learned tasks.
5.0
44
Supsup Baseiie Guess
10
75"
19
Figure 4: Left: Average performance on 5 new Split CIFAR-100 tasks by number of impressions and impression type. X indicates incorrect mask. Numbers indicate heterogeneous and N homogeneous impression set. Right: Performance of a-optimization in the basis-GN regime averaged over min (I, 25) basis tasks. All results are averaged over 3 different seeds and masks of density (5, 8, 10, 15, 20, 25, 30} 10
Table 1: Parameter count for different architectures and data sets.
Dataset (Max Tasks)
Model
O(params)
Mask (kB)
Layers
la"|
T (kB)
PermutedMNIST (784 !)
LeNet 300-100
266K
64.9
100
3
300
2 1.17
55.5
RotatedMNIST (359)
LeNet 300-100
266K
64.9
5
3
15
0.96
67.4
Split CIFAR-100 (20)
ResNet-18
6.2M
1513
10
21
210
757.6
1.99
Wide ResNet-181
11.7M
2856
10
21
210
1429
1.99
Wide ResNet-34
21.8M
5322
10
37
370
2662
1.99
ImageNET (2100)
5e
25.6M
6250
75
53
3975
238.7
26.2
ResNet-1011
44.5M
10864
75
104
7800
418.47
25.9
ResNet- 152
60.2M
14697
75
155
11625
570.31
25.7
pre-Res
64.7M
15795
100
203
15225
623.6
25.3
VGG 16
137.9M
33667
75
16
1200
1207
27.89
WRN-50-2-bootleneee
68.9M
16821
75
53
3975
616.28
27.2
Table 2: Parameter tradeoff for different models and datasets. The maximum number of possible tasks for each data set is given in brackets in the first column. For ImageNet we have assumed that entire set of classes is split into 10-way tasks. For RotatedMNIST we have assumed a degree granularity of rotations. O(params) is the number of parameters of the network (weights only, assuming biases are set to zero). Mask Size is the space on disk required to store each additional SupSup mask as approximately chosen give either benchmark very good performance on new tasks. |a*) is the number of parameters (floating point) per additional new task, given basis of size [M]. & is the storage per task amortizing the cost of storing [M] basis masks over all possible tasks. Ca is the compression or savings ratio compared to SupSup. i marks those architectures we have not run due resource constraints. For those have interpolated the size the basis set [M] from those architectures/data sets that we ran.
16-bit integers. [M|
8

---

Under review as a conference paper at ICLR 2022
4.3 MODEL EPPICIENCY/PARAMETTI SAVINGS
Here, we present the parameter savings for our ImpressLearn approach, compared to the SupSup proposal for various architectures and data sets. Our approach has the fixed cost of storing the basis masks as well as the task-specific a coefficients for each new task. We amortize the basis masks across all tasks for comparison.
Table highlights the parameter savings of our approach for various data sets and architectures. Amortizing mask storage across tasks and accounting for storage of the a coefficients, we get particularly impressive savings of an order of magnitude or more for data sets with a large number of tasks, like PermutedMNIST and (general) ImageNet. We have also evaluated potential savings our approach would yield for several common architectures by giving an educated guess the number of required basis masks. In the case of ImageNet the number of potential tasks is so large that storage of basis masks per new task is very small even if we double the number of masks considered, and memory required to store the a remains small. Overall, we believe our approach affords considerable savings in memory at the expense of either no small loss in accuracy.
5 DISCUSSION
In this work, we propose a novel way to adapt an existing continual learning algorithm, leveraging principles from transfer learning generalize new tasks, allowing for scalable and parameter- efficient continual learning. Using simple linear combination of masks, or impressions, we see that even this basic setup is able learn new tasks effectively. We show that this effect is consistent across task types and network architectures, and that it achieves competitive performance while using significantly fewer parameters. This work highlights the advantages of re-using existing meta-features learned on previous tasks for future learning problems and opens up space of possibilities of applying transfer learning to protect against catastrophic forgetting.
One application of our approach could be protection against drifts in the data. In the homogencous setting, in anticipation of drift, one could learn a few basis masks and a linear combination, and then update the latter accordingly when anticipating that the underlying data changes slightly but steadily. To our knowledge, no other approach allows for such easy continued adjustments, as most of them fix masks weights for fixed tasks.
We have tried several variations of our approach using different types or masks:
Signed Binary Masks: M' E (-1,0, Allows for a little more flexibility but did not perform as well as 0,1 masks.
Real valued masks: M RIM Real valued masks performed much better at high sparsity (s > 0.75), but at lower sparsity the difference can be made up by adding more binary masks (which are cheaper to store). Some regularization benefits are lost.
We have also briefly explored a hybrid approach to understand the value of our linear combinations: inspired by early fine-tuning work where only the last layer of the network is retrained for each new task, we have created a hybrid setting where we employ our ImpressLearn algorithm in all layers except the last, and train a new SupSup mask in the last layer. This was done demonstrate that the power of our linear combination routine does not come from the number of extra a a-parameters we allow for each new task. We show that allowing even more parameters in the last layer (for an additional SupSup mask) does not improve performance much, and that hence the power of our approach does not reside simply in parameter finetuning for new tasks (see Sec. A.5).
Limitations: Our experimental results show that Impress works well on several benchmarks, and particularly shines when the number of new tasks large. scenarios where the number of different tasks is small (like SplitCIFAR with 20 tasks only) our approach will only give limited parameter savings, if any. In the case of SplitImageNet we were only able to evaluate our approach for relatively small number of masks and could not match benchmark performance. However, in particular in this case, our parameter savings are particularly impressive and highlight the power of transfer learning in this setting.
In future work, we plan to explore other settings where the linear combination optimization could be applied to leverage transfer learning for catastrophic forgetting.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
In keeping with conference guidelines, we have tried to ensure that our experimental results were free of any bias that would misconstrue the results. As specified in the paper, all experiments were run over multiple random seeds, on a variety of benchmarks using publicly available software and hardware. The code for the experiments is self contained and references appropriate instructions to install the relevant software dependencies. A copy of the experimental code will be made publicly available.
REFERENCES
Rahaf Aljundi, F. Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and T. Tuytelaars. Memory aware synapses: Learning what (not) to forget. ArXiv, abs/1711.09600 2018.
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony V. Robins. Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. Neurocomputingg 428: 291-307, 2021. doi: 10.101 //..///.ecccmm2000000 11.011...5 1.050. URL https: /ooi newcom. 2020 11.1111000 150
http///iioog/g/.//// wwg/oo.g/110. 101g/10.1016/3
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, pp- 1-1, 2021. doi: LLLL:0/TTAMM22222335544444
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 109/CVPR.2009, 5506848..
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019. URL https
/opeeneview net wttp://wwwwweeeeeeeeeeeeeeeeeee
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911.05722, 2019. URL http:
7/arxiv. www. httpg///bb/991.0.5535
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2288-2324 1998. doi: 10.1109/5.26791.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, (0(12))2935 2018. doi: 10.1 10.119//PAM1..2122273300
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learn- ing. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL httptts:::tt http://poooeeeeeeeeeeeeeeeeo neurins cccpaaper/2017/F1.. E87122778822b221111666666
7 - pdf
Pattie Maes, Maja J. Mataric, Jean-Arcady Meyer, Jordan Pollack, and Stewart W. Wilson. Incremen- fal Self-Improvement for Life-Time Multi-Agent Reinforcement Learning, pp. 516-525. The MIT Press. 1996.
Arun Mallya and S. Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. 2018 IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.). Computer Vision - ECCV 2018, pp. 72-88, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01122--0
10

---

Under review as a conference paper at ICLR 2022
Massimiliano Mancini, Elisa Ricci, Barbara Caputo, and Samuel Rota Bulo. Adding new tasks to single network with weight transformations using binary masks. In Proceedings of the European Conference Computer Vision (ECCV) Workshops, September 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower (ed.). Psychology of Learning and Motivation, volume 24, pp. 109-165. Academic Press, 1989. doi: https://doi.or// 10. URL
1016//80779-742210066633333333 atticcle/p11/8001974410888888
http://wwww.ccendddeeeeeeeeeeeee rect. wwt.aecetcccc/ccnnncee... scinnce/
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden randomly weighted neural network? Pruceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 11890-99, November 2019.
Roger Ratcliff.
Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97(2):285-308, 1990. URL https: org/
10.1037/0033-293 12SX.97.. 37.2.283
Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542, 2017. doi: 10.1 N0.119//CVP.2201..888
Boya Ren, Hongzhi Wang, Jianzhong Li, and Hong Gao. Life-long learning based on dynamic combination model. Applied Soft Computing, 56:398 404, 2017. ISSN 1568-4946. doi: https://oii. ogg/10.1016fj.asocc.0000 2017 03 000, 005. URL https: wtw.::::/www.iie..d.........dd.. rect. com/ science/
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In In International Conference on Learning Representations (ICLR), 2019.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL /ttps:://rrccccings. http://ccceeiinn........eeeeeeee neurips wttpuuuipssccppppp..... cc/papez/2019/
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, 111116666.4677.. 2016. URL http: org/ http///bbl/lle..eeeee// 04671
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings Machine Learning Research, pp. 4548-4557 PMLR, 10-15 Jul 2018. URL mlr. www.ppeesss..
wwwss/essvvve.... v wwt.//eeraaaaaaae html
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jíwon Kim. Continual learning with deep generative replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. http::t::: httpttpppppppppcpcnnggeooooeeeee Defbe980677666 www hrt.000006.....11.10000000.aaaa. Pape. pdf
Sebastian Thrun and Lorien Pratt. Lifelong Learning Algorithmss, pp. 181-209. Springer US, Boston, MA, 1998. ISBN 978- 978-1-4615-5529-2.
doi: 10. 101107/9988-1446155522222 URL
https://doi. www..orggo... 100
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020. URL htt.:/tttpss//pppeeevieeee hwtpeeiwwwweeeeeeeoooooeeeae flyrYDr

---

Under review as a conference paper at ICLR 2022
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Proceedings of the 34th Conference Neural Information Processing Systems, June, 2020. ArXiv. URL http:// arxiv.org/abo. 22066 14769
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995. PMLR, 06-11 Aug 2017. URL http::/ppze http://pxoceedingg mlr wtw...ppees/////eeeee html
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
A APPENDIX
A.1 HYPERPARAMETERS AND DESCRIPTION OF EXPERIMENTS
Here we provide the details of our experimental work.
Table 3: Dataset, Model and Hyperparameter Overiew.
Model
Dataset
[hhom
LR
(A)
Opt.
Batch Size
LeNet 300/100
MNIST
Rotated
250
0.002
0
RMSprop
128
Permuted
250
0.002
RMSprop
128
ResNet-18
Split CIFAR 100
75
0.02
0.005
ADAM
64
ResNet-50
SplitImageNet
0.0025
0.005
ADAM
96
For homogeneous basis masks, [.Rom is the largest mask set we tried. For ADAM, momentum was set to 0.9 and weight decay to 0.1.
We performed experiments for range of densities {0.05, 0.08, 0.1, 0.15, 0.2, 0.25, 0.3). For consistency, we maintained the same numerical ordering of tasks from each dataset across all experiments. Fach seed varied the random initialization of the popup scores and the training data ordering. For a heterogeneous basis set we used one seed to generate the entire basis set. In the homogeneous case seeded every mask to ensure mask diversity. The backbone network was fixed for one basis set and a-optimization for new tasks. All runs were performed on three different backbone networks and train/test splits to ensure that results were sufficiently general. The boxplots contain averages over all these settings and sparsities.
A.1.1 INFRASTRUCTURE
For all models except LeNet 300-100, we used GPU enabled hardware to expedite training time. Our experiments were performed SLURM cluster enabled with NVIDIA V100 Tesla and RTX 8000 GPUs. For the Imagenet experiments specifically, we were memory constrained and thus hard to tune batch size appropriately to fit within the 48 GB available the device.
12

---

Under review as a conference paper at ICLR 2022
A.2
IMAGENET RESULTS
Here show Impressl SplitImageNet.
Supsup Baseline
Guess
0.6
Accuracy
0.4
0.2
Figure 5: Left: Average performance on 5 new SplitImageNet tasks by number of heterogeneous impressions. X indicates incorrect mask. Right: Performance of our a-optimization in the basis GN regime averaged over min(IM). 25) basis tasks.
A.3 BASIS GN SETTING AND A NEW TASK-INFERENCE ALGORITHM
As shown in the experimental results in Figs, well when presented with basis-task without task label. In that case, our algorithm finds linear combination of masks that nearly always matches or outperforms the corresponding basis mask for this task. Our algorithm hence constitutes an alternative to entropy-based methods presented in (Worssman al. 2020) for the GN scenario (task labels given at training but not at inference).
and our a-optimization is able to perform
In Fig. 6 we show how our algorithm outperforms the baseline SupSup performance for Permut edMMIT..
0.86
79
0.82
0.80
0.76
ImpressLeam (Basis GN)
SupSup
SupSup (GG)
100
150
200
Task
Figure Comparison of inference accuracy between basis-GN, SupSup GG and SupSup GN on PermutedMNISS per seed and sparsity. All results are averaged over different seeds and masks of density {5. 8, 10, 15, 20, 25, 30) 10-2.
GN performance averaged over 10 different data splits and orderings
13

---

Under review as a conference paper at ICLR 2022
A.4 RANDOM BASIS MASKS
Here we compare performance of set of random basis masks homogeneous masks to illustrate the trade-offs in case the SupSup step is not available or difficult to implement (Figs. 7 g and (
Generally, in the limit of large number of masks, the random methods, while yielding inferior accuracy, is faring surprisingly well and could become an alternative for resource-consttranee settings.
0.3
Buscline
10
Random Guess
200
240
300
150
Figure 7: Random vs Homogeneous masks: Average validation performance on 10 unseen Ro- tated MNIST tasks by number of impressions and impression type.X indicates incorrect mask. Standard numbers indicate an impression set of random, untrained masks and N indicates homogeneous impression set. All results averaged over different seeds and masks density $15 8, 10, 15, 20, 25, 30) 10 10-?
25
02-
Supeup Tweeline Bandom Duess
Figure 8: Random vs Homogeneous masks: Average validation performance on 10 unseen Per- muted MNIST tasks by number of impressions and impression type. X indicates incorrect mask. Standard numbers indicate an impression set of random, untrained masks and N indicates a homogeneous impression set.All results averaged over different seeds and masks of density E {5, 8, 10, 15, 20, 25, 30) x 10
14

---

Under review as a conference paper at ICLR 2022
60
47
Supoup Duscline Random Duess
Figure 9: Random vs Homogeneous masks: Average performance on 5 unseen Split CIFAR- 100 tasks by number impressions and impression type. X indicates incorrect mask. Standard numbers indicate an impression set of random, untrained masks and N indicates homogeneous impression set.All results averaged over different seeds and masks of density {5, 8, 10, 15, 20, 25, 30} 10-
A.5 HYBRID APPROACH
Here we study hybrid setting where for each new task we employ our ImpressLearn algorithm in all layers except the last, and train new SupSup mask only on the last layer. This demonstrates that the power our linear combination routine does not come from the number cxtra a-parameters allow for each new task. We show that allowing additional SupSup mask) does not improve performance much, and that hence the power of our approach does not reside simply in parameter finetuning for tasks (Fig. 10
more parameters in the last layer (for an
0.
0.6
0.4
03
Supsup Random Guess
3h
36
10h
15h
301
30
35t
Figure Left: Hybrid vs Regular (Heterogeneous Impressions). Right: Hybrid vs Regular
(Homogencous Impressions). Average performance on 5 new Split CIFAR- 100 tasks by number of impressions and impression type. denotes hybrid model was used whereas A indicates homogeneous impression set. h indicates both. All results averaged over different seeds and masks of density E (5, 8, 10, 15, 20, 25, 30) x 10-2
15


Reviews:
[{"review_id": "uCgjkGccvOZ", "reviewer": "Reviewer_Tj8S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comment:\n\n1: The proposed approach is parameter efficient and requires very few parameters for each novel task (assuming that basis masks are learned) \n\n2: The paper follows the idea of SupSup, and the difference is how to combine the previously learned mask for the novel task. Also, the task prediction idea is the same as SupSup. Another contribution is learning homogeneous masks, which may be helpful in many scenarios, but the basic idea is the same. Overall, it seems that the paper has limited contribution compared to SupSup.\n\n\n3: the models are evaluated over the smaller architecture ( i.e. LeNet), and it's challenging to evaluate the proposed model using only the given experiment. For the larger architecture, only parameter efficiency are reported (Table-2). I would suggest the author, please report the result for the ResNet-18 architecture over the CIFAR-100 and Mini-ImageNet datasets. \n\n4: The results in Fig-3 are confusing. In the SupSup paper (Fig-3 SupSup paper), the result for the PermutedMNIST is ~94% (GN Scenario), while the paper reported the SupSup result of ~80%. How author got this result? As compared to the SupSup, the performance of the proposed model seems much lower. A similar pattern we observe for the CIFAR100 dataset, in Fig-4 performance is much poorer and for the GN scenario model only performs well for the very small task sequence (5, from Fig-4 right).\n\n5: What architecture is used for the ImageNet dataset experiment?\n\n6: The performance of the proposed model highly depends on the number of learned base tasks. Sometimes we have only a 5/10 task sequence, and if for the small task sequence, we learn 50 or 100 task sequence, it is much costly.  Also, we can observe that the homogeneous setting model does not perform well. Therefore, for the limited task sequence model may not be helpful.\n\n7: The results are only compared with the SupSup paper. I would suggest to the author; please consider the recent expansion based model [a,b,c] for the comparison. \n\n[a] Efficient Feature Transformations for Discriminative and Generative Continual Learning, CVPR-21\n\n[b] Ternary Feature Masks: zero-forgetting for task-incremental learning, CVPRW-21\n\n[c] Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors, AISTATS-21\n", "summary_of_the_paper": "The paper learns the binary basis mask for a few task sequence tasks, later linear combinations of the basis mask can be used as a mask for the new task. The parameter learned for combining the basis mask is significantly fewer than learning a novel mask for each new task sequence. The paper also proposes a homogeneous mask learning. For most of the datasets, we don't have the flexibility to learn an enormous basis mask and model performance highly depends on the number of learned basis tasks. The proposed model is evaluated over the MNIST, CIFAR100 and Split-ImageNet dataset over the smaller architecture. ", "summary_of_the_review": "The paper is lagging in experimental results with the recent baseline. Also, the contribution is limited compared to SupSup.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q4owfZYsIfH", "reviewer": "Reviewer_MxHs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is in general well-written and structured. \n\nThe proposed method follows the popular approach of alleviating forgetting through masking neurons for different tasks. As mentioned by the authors this has been explored in prior work such as PackNet and Piggyback, SupSup. Related work like [1,2] are not discussed. The main contribution over prior work is that this paper introduces a subspace of masks, that is spanned by a mask basis, which is referred to as “impression”. \n\nMy concern is that the contribution w.r.t. prior works is rather slim. In the experimental evaluation, the paper only compares to a single baseline, SupSup, and ignores the results of all other standard baseline methods mentioned in the related work, including replay and regularization-based methods, which achieve significantly higher performance on the evaluated benchmarks (e.g. FROMP [3] achieves >90% on P-MNIST). The contribution w.r.t. Supsup remains unclear beyond a reduction of the number of parameters, which comes at the cost of a moderate loss in performance. I would recommend the authors to motivate more clearly why the reduction in parameters is a problem of critical importance.\n\n[1] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization, 2020.\n[2] Chungkuk Yoo, Bumsoo Kang, and Minsik Cho. SNOW: Subscribing to Knowledge via Channel Pooling for Transfer & Lifelong Learning of Convolutional Neural Networks. 2019\n[3] Pan, P., Swaroop, S., Immer, A., Eschenhagen, R., Turner, R. E., and Khan, M. E. Continual deep learning by functional regularisation of memorable past, 2021.\n", "summary_of_the_paper": "The paper studies continual learning and that context the important and prominent problem of catastrophic forgetting. Similar to related work the paper achieves this through learning task-dependent masks on the neural network representation (effectively generating different sub-networks for different tasks).  The main contribution w.r.t. related work is that the masks are constrained to a sub-space which is spanned by a \"basis of masks\". The experiments show some reduction in the number of required parameters over a single baseline method.", "summary_of_the_review": "Overall, the contribution over related work seems limited to me, but I am open to feedback from the authors if my initial perception is not correct.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BZFpdS8V2Tw", "reviewer": "Reviewer_Xchu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\nThe paper clearly describes the main technical proposal of applying linear combination of masks to randomly initialized networks. The approach seems to efficiently scale to a larger number of tasks in the given problem setup. Together with the new homogeneous impression approach, the paper seems to describe unique and novel technical ideas to improve on SupSup (Wortzman 2020).\n\n## Weakness\n\nAlthough the approach seems to novel, there are several weaknesses in the current paper.\n\nThe first is the performance degradation compared to SupSup baseline in CIFAR / ImageNet benchmarks. Although Sec 5 discusses the limitation when there are small number of tasks, Fig 4 seems to indicate a serious performance disadvantage compared to SupSup, which does not look reasonable. Considering that MNIST variants are artificial benchmarks and not practical in any application, the performance degradation in CIFAR / ImageNet limits the significance of the work.\n\nAnother missing comparison is the Hopfield network in SupSup, as discussed in Sec 2.\n\nI wonder how strong is the catastrophic forgetting if we apply one of the discussed approaches in Sec 2; i.e., regularization-based, replay, or parameter isolation methods. As this paper only concerns the comparison to SupSup (fairly incremental), I feel it is unclear how impactful this work is in the broader continual learning context.\n\nRegarding the task setup, which also applies to Wortzman 2020, I feel impractical to only consider the fixed l-way classification from the same or similar domain. At least the setup should consider a variable number of categories per task.\n\n## Other\n\n- What is the solid / dashed lines within the boxes in Fig 2, 3, 4, and 5? Median and mean?", "summary_of_the_paper": "The paper describes an approach to continual learning by randomly initialized deep networks. The main idea is to extend the SupSup (Wortzman 2020) and apply linear combination of binary masks for efficiency. The paper also considers learning from multiple instances of the same task. The evaluation results indicate reasonable performance for the reduced parameters in common benchmarks.", "summary_of_the_review": "While the proposed idea seems novel and the results seem to indicate good efficiency in terms of model footprint, the results show performance degradation to the baseline, and I feel the submission has rooms for improvement for acceptance due to the narrow focus on comparison to SupSup, as pointed in the main review. In overall, I do not think the paper reaches the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9FU24TFMVf6", "reviewer": "Reviewer_7LBU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive aspects:\n- The proposed approach is scalable to a large number of tasks with a small parameter overhead than existing methods.\n- The paper is in general clearly written\n- The related work section covers most of the relevant papers in the field\n\nNegative aspects:\n- The scientific novelty of the paper is limited. It is an incremental work, relying heavily on [Wortsman et al., 2020]\n- The experimental validation is limited and not convincing. The paper is not compared with any other state-of-the art methods.", "summary_of_the_paper": "In the current paper, the authors propose a novel way to adapt an existing continual learning algorithm, leveraging principles from transfer learning.  More concretely, they learn an initial set of masks (or impressions) from a small number of basis tasks, and they use afterwards linear combinations of these masks in the learning process of new tasks. Therefore, their approach is able  to generalize to new tasks, allowing for scalable and parameter efficient continual learning (with much lower parameter overhead than existing methods).\n", "summary_of_the_review": "Please find below some of my concerns:\n1. First of all, please state clearly in the introduction in which aspects your paper is different from [Wortsman et al., 2020]\n2. In page 3, you mention for the first time the term: GN-setting. Where is the meaning of 'GN'?\n3. Please clarify the following statement (section 3): \"It then generates the mask M by setting the top s fraction of scores in S to 1 and the rest to 0\". How is this fraction chosen: heuristically, or do you use any criteria?\n4. Regarding the experimental settings:\n- State clearly, for each dataset, how do you define the tasks. How many basis tasks do you have for each dataset? How comes that for PermutedMNIST you have 250 learned tasks and the maximum could be 784 (Table 1)?\n- What is the relationship between number of masks and number of basis tasks: you have one mask per one basis task?\n5. Plots 2,3 and 4 are not clear\n6. Figures 3 and 4: min(jMj; 25) basis tasks. How did you choose 25? Why do you use this criterion for basis tasks?\n7. Please use a standard evaluation methodology in order to be able to interpret your results\n8. Compare your approach against (at least) the following related methods: Packnet, Piggyback, [Wortsman et al., 2020], Ternary Feature Masks (TFM) [Masana et al., 2021]\n\nM. Masana, T. Tuytelaars, and J. van de Weijer. Ternary Feature Masks: zero-forgetting for task-incremental learning. Proc. of CVPRW 2021, Workshop on Continual Learning", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach "ImpressLearn" to continual learning using the idea of task-specific masks. The idea builds upon another idea - SupSup (Wortzman 2020) - which uses a backbone network shared by all the tasks and binary task-specific masks. However, the number of parameters for an approach like SupSup can become excessively large when the number of tasks is very large. This paper presents a solution by having a small number of basis-masks and learning a weighted combination of these basis-masks to use as the task-specific mask for each task. The experimental results show that ImpressLearn yields significant parameter savings as compared to SupSup.

There were several concerns shared by all the reviewers, such as (1) Limited novelty as compared to SupSup, and (2) Limited experimental evaluation and not having enough baselines. From my own reading of the paper, I largely agree with the assessment of the other reviewers.

The authors responded to the original reviews and acknowledged some of the concerns raised by the reviewers. The reviewers read the authors' response but their assessment has remained unchanged.

The basic motivation and the idea is nice but offers limited novelty (especially as compared to SupSup). If the authors could improve the experimental evaluation (more baselines, larger datasets/networks, etc), it will be a much stronger paper. However, in its current shape, I as well as the other reviewers do not think that the paper is ready for publication.
Decision: Reject