Optimizing Reusable Knowledge for Continual Learning via Metalearning
Authors: Julio Hurtado
Alain Raymond
Alvaro Soto
Venue: NeurIPS 2021 Poster (2021)
Keywords: Continual Learning
Meta-learning
Knowledge transfer
Catastrophic forgetting

Abstract: When learning tasks over time, artificial neural networks suffer from a problem known as Catastrophic Forgetting (CF). This happens when the weights of a network are overwritten during the training of a new task causing forgetting of old information. To address this issue, we propose MetA Reusable Knowledge or MARK, a new method that fosters weight reusability instead of overwriting when learning a new task. Specifically, MARK keeps a set of shared weights among tasks. We envision these shared weights as a common Knowledge Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. Key components behind MARK are two-fold. On the one hand, a metalearning approach provides the key mechanism to incrementally enrich the KB with new knowledge and to foster weight reusability among tasks. On the other hand, a set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task. By using MARK, we achieve state of the art results in several popular benchmarks, surpassing the best performing methods in terms of average accuracy by over 10% on the 20-Split-MiniImageNet dataset, while achieving almost zero forgetfulness using 55% of the number of parameters. Furthermore, an ablation study provides evidence that, indeed, MARK is learning reusable knowledge that is selectively used by each task.

Optimizing Reusable Knowledge for Continual Learning via Metalearning
Julio Hurtado
Alain Raymond-Saer, Department of Computer Science
Department of Computer Science Pontificia Universidad Católica de Chile j jahurtadofuc.cl
Pontificia Universidad Católica de Chile
afraymon@uc.co cl
Alvaro Soto
Department of Computer Science Pontificia Universidad Católica de Chile asoto@ing, puc. .cl
Abstract
When learning tasks over time, artificial neural networks suffer from a problem known as Catastrophic Forgetting (CF). This happens when the weights of a network are overwritten during the training of a new task causing forgetting old information. To address this issue, we propose MetA Reusable Knowledge MARK, a new method that fosters weight reusability instead of overwriting when learning new task. Specifically, MARK keeps set of shared weights among tasks. We envision these shared weights as a common Knowledge Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. Key components behind MARK are two-fold. On the one hand, a metalearning approach provides the key mechanism to incrementally enrich the KB with new knowledge and to foster weight reusability among tasks. On the other hand, set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task. By using MARK, we achieve state of the art results in several popular benchmarks, surpassing the best performing methods in terms of average accuracy by over 10% on the Seo-ppiitMinilmmaeeee dataset, while achieving almost zero forgetfulness using 55% of the number of parameters. Furthermore, an ablation study provides evidence that, indeed, MARK is learning reusable knowledge that selectively used by each task.
Introduction
Among the cognitive abilities of humans, memory is of the most relevant. However, memories are fragile. In particular, studies from cognitive psychology show that memories can be lost when new knowledge disrupts old information, in process known task interference 2 In the of artificial neural networks (ANNs), task interference is behind the problem known as Catastrophic Forgetting (CF)
6. 7) In effect, model learns causing catastrophic forgetting of old information.
3 4
task, its weights overwritten
Unfortunately, in the case of ANNs, task interference is more severe than in the case of humans. This evident for the case of continual learning, i.e., the case when data from new tasks is presented sequentially the learner, who does not have access previous data. Under this training scheme, when a previously trained model is retrained on a new task, it usually suffers a significant drop in performance in the original task 8 10.. Previous works prevent CF have followed two main consists of avoiding the modification of parameters
strategies. The first strategy 10
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that are key to solve previous tasks. Specifically, when facing new task, regularization term ensures that critical parameters are modified as little as possible. In general, this approach shows satisfactory performance in problems that involve few tasks, however, when the number of tasks increases, problems such as accumulated drifting in weight values and interference among them, make this approach difficult to scale. The second strategy [14] 15
17 7 [8] consists of introducing [6]
structural changes to the architecture of a model. This includes methods that reserve part of the network capacity to learn each task [9]. and methods that use special memory units to recall critical training examples from previous tasks [77 The main problems with these methods are the extra model complexity and the need of efficient method to recall key information from previous tasks.
In contrast to these previous strategies, when learning new tasks, humans continuously associate previous experience to new situations, strengthening previous memories which helps to mitigate the CF problem [0 Taking inspiration from this mechanism, we propose MetA Reusable Knowledge or MARK, a new model based on learning strategy that, instead of mitigating weight overwriting or learning independent weights for different tasks, uses metalearning approach to foster weight reusability among tasks. In particular, we envision these shared weights Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. In this sense, the KB behind MARK is not given by external memory that encodes information in its vectors, but by a trainable model that encodes shared information in its weights. As a complementary mechanism to query this KB, MARK also includes set of trainable masks that are in charge of implementing a selective addressing scheme to query the KB.
common Knowledge
Consequently, to build and query its shared KB, MARK exploits two complementary learning strategies. On the one hand, a metalearning technique provides the key mechanism to meet two goals: i) encourage weight updates that are useful for multiple tasks, and ii) enrich the KB with new knowledge as the model learns new tasks. On the other hand, a set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task.
In terms of its inner operation, MARK works by first forcing the model reuse current knowledge via functions that detect the importance of each pattern learned in the KB, and later expanding its knowledge if past knowledge is not sufficient to successfully perform a task. As an intuition behind how MARK works (not to be taken literally), when facing different tasks related to the classification of animals, MARK may take advantage of its metalearning approach to learn relevant features to discriminative concepts such as fur, paws, fin, and ears. Afterwards, when solving task related to the discrimination of horses vs zebras, it might use a mask to focus on features related to fur rather than fins or ears.
We perform extensive experiments to analyze the inner workings behind MARK. In particular, we demonstrate that, indeed, MARK is able to accumulate knowledge incrementally as it learns new tasks. Furthermore. in terms of the pieces of knowledge used to solve each task, our experiments confirm that the use of task dependent feature masks results critical to the success of the method. We summarize our contributions follows:
We propose MARK, a novel method for continual learning scenarios that is based on two com- plementary mechanisms: i) A new method to build a KB that incrementally accumulates relevant knowledge from several tasks while avoiding CF problems, ii) A mechanism to query this KB to extract relevant knowledge to solve new tasks.
An extensive experimental evaluation and public implementation of MARK that demonstrate its advantages with respect to previous works.
2 Continual Learning Scenario
Following previous work on continual learning [21]. we consider a task incremental scenario, where task identity is provided at training time. Each task & consists of new data distribution D' = (X', Y", T where X denotes the input instances, Y" denotes the instance labels, and T" is a task ID. This task ID required during training and inference. The goal is train a classification
model f X - Y using data from sequence of 'T' tasks: D = (D' ...., ) Following the usual setup for continual learning [21], each task is presented sequentially to the model. Our scenario allows for unlimited use of data from the current task, but after switching to a new task, this data is no longer available. Furthermore, there is no intersection between the classes of different tasks, but they share similar domain. This domain intersection is key to take ad advantage of common patterns among

---

the tasks. Next section presents the details behind MARK, our proposed model for incremental learning in the previous scenario.
Method
When learning tasks sequentially, humans build upon previous knowledge, leading to incremental learning. In contrast, in a similar scenario, ANNs devote all of their resources the current task, leading to the problem of CF. Taking inspiration from the behavior of humans, an appealing idea is to provide the model with KB that, as the model faces new tasks, incrementally captures relevant knowledge. Using this shared KB, the model can associate previous experience to new situations, mitigating the CF problem. To implement this idea, have address two main challenges: How do we build this KB incrementally?, and ii) How do we query this KB to access relevant pieces of knowledge?
To address the first challenge, we leverage metalearning in order to train a KB from data. Specifically, metalearning strategy known episodic training [22]. This strategy consists of sampling from a task pairs of support and query sets to simulate training data from a large number of mini- tasistaks. These mini-tasks are then used to bias the learner to improve its ability to generalize to future tasks. In turn, this generalization leads to our goal: to capture relevant knowledge that can be reused to face new tasks.
To address the second challenge, on top of the KB, we train gank-ggennatiing functions for each task. These masks provide suitable mechanism to selectively access the information encoded in the weights of the KB. We can envision these masks as a query that is used to access the intermediate activations of the KB. Following 23) these masks depend solely on each specific task and input. In this way, given an input, MARK uses the corresponding mask to query the KB generating a feature vector. This vector is then used by a task dependent classification head to output prediction. We describe next the details behind this operation.
3.1 Model Architecture
Figure shows schematic view of the operation behind MARK, where the main modules are:
Feature Extractor (F"): This module is charge of providing an initial embedding for each input X,, i.c., F takes input X; and outputs vector representation F In our case, we test our method using visual recognition applications, therefore, F is given by a convolutional model. is important to note that model Fs can be shared among tasks it can be specific to each task.
Knowledge Base (K B): This is the main module behind MARK. It is in charge of accumulating relevant knowledge as the model faces new tasks. In our implementation, we use a convolutional architecture with B blocks. This part of the model is shared across tasks.
functions )) These modules take an input feature vector F and produce an instance and task-dependent mask M for each block of the KB. Each mask consists of a set of scalars, one for each channel in the convolutional blocks of the KB, that multiply the activation of each channel. These masks are critical to select what knowledge is relevant to each instance and task. In our implementation. we use fully connected layers.
Classifier (C"): These modules correspond tusk-dependent classification heads. Its input FiKE is given by flattening operation over the output of the last block of the KB. Given the task ID of an input X,, the corresponding head outputs the model prediction. our implementation, we use fully connected layers.
Specific details can be found in Section A.I As shown in Figure [ the flow of information in MARK is as follows. Input X, goes into F' to extract the representation TV. This representation is then used by M to produce the set of masks that condition each of the blocks in the KB. The same input X, enters the mask-conditioned KB leading to vector Ft K used by the classification head. Finally, classifier C" generates the model prediction, where t is the task ID associated to input X,-
3.2 MARK Training
Algorithm D describes the training process behind MARK. The first step consists of initializing the KB by training it end-to-end on the first task, without using metalcarning and mask functions. In
3

---

KB
Block
Block 2
Block
Xi
F
Figure 1: Given an input X; from task t, (1) We use feature extractor Ft to obtain F (2) F is then passed to mask function M' to generate mask M Afterwards, (3) the same input X; enters the KB, which has intermediate activations modulated by Mý. Finally, (4) the modulated features go through a task-dependent classifier C that performs the class prediction for X,.
other words, we perform the KB initialization using the regular training procedure of a convolutional neural network for classification task. After this, train MARK sequentially each task by alternating three main steps:
1. KB Querying. We train task dependenndnt generrennn functions that used to query the KB using vector F. Also, we concurrently train the task classifier for the current task. Notice that, leaving aside the KB initialization, in this step each new task is trained using only accumulated knowledge from previous tasks.
KB Update. We use metalearning strategy to update the weights in the KB. This scheme allows fostering KB updates that favor the acquisition of knowledge that can be reused to face new tasks.
KB Querying. After updating the KB using knowledge from the current task, we repeat the querying process to finetune the mask-generating functions and task classifier using these new knowledge. Notice that during this step the KB is kept fixed.
The intuition behind the application of the three previous steps is follows. We initially query the KB using the accumulated knowledge from previous tasks. This forces mask functions and classifiers to reuse the available knowledge. When that knowledge is exhausted, we proceed to add knowledge from the current task into the KB. Finally, we take advantage of this newly updated KB to obtain final mask-functions and classifiers for given task. We describe next the main steps behind the process to query and update the KB.
Algorithm 1: MARK -Training Process
Algorithm 2: KB-Update
Components:
Components:
D Dataset for task t.
KB: Knowledge Base.
pt Feature extractor for task t.
C* Temporal Classifier for batch
K B: Knowledge Base.
Easter: number of training updates for KB (outer loop)
M Mask Mene-atng function for task t.
C³: Classifier for task L.
Finner: number of inner loop epochs.
KB Train (KB))))
/*Init KB o/
for e Flouter do Generate K batches of data from D)
for task t + to do
[=]. Obtain feature vectors F/ay
for k in K do K +- KKB Copp of KBB*/ Initialize(C*) Train R and C* with batch & Emaer epochs.
Ff + (-(FF (F'(D))
(1,...,|1"I
f+2. Train M and C using current KB*/
Trainç M & C*)
1²3. Update KB using metalearning")
KB- + KB-Update A'llorrthmm 2
end
444. Finetune M and O' with updated KB*/
VKB + Ernner Σ (KKB -KB) KB - KB - aVKB
Train & C*)
end
end
Output: Trained modules KB, M C'.
Output: KB
/*output updated K B%

---

3.2.1 KB Querying
Once we obtain feature vectors through the use of feature extractor Ft, the model can learn which are the modules in the KB that can best solve the current task. In this training stage, the model trains functions to learn how to use the knowledge available in the KB, focusing solely on reusing knowledge from previous tasks, without modifying the KB, particular, in this step we only train M' and C Both are trained end-to-end, while keeping the KB weights frozen.
As we generate masks for each intermediate activation of our model, strictly speaking we have a total of B mask-generating functions. However, to facilitate the notation, we subsume all such functions under the term M and think of its output as the concatenation of the results of those B functions. Eq. shows the function M where a mask Mf is obtained given an input X; from task t. Our implementation of these functions consists of a linear function with parameters Wi,, and an activation function p. which we implement ReLU.
Mf = M'(E = ((WAMM)FF))
Masks generated in this process encouraged to have two effects: first, give signal of how important a specific module from the KB is for the current input; second, to make sure that gradient updates are done where they really matter. If an activation map is irrelevant for certain task, then the value of the corresponding mask will be zero, making the gradient update associated to that activation being zero as well.
3.2.2 KB Update
The purpose of this training step is to add new knowledge from the current task to the KB, As a further goal, aim enrich the KB with features that are highly general, i.c., they can be used to solve several tasks. To achieve this, we use metalearning as a way to force the model to capture knowledge that can be reused face new tasks.
Figure an schematic view of the metaleaming procedure that we use train MARK. This procedure consists of an adaptation of the training process presented in (24). Specifically, we create set of K mini-tasks, where each mini-task consists randomly sampling from the current task set of H classes and h training instances per class. This allows to create mini-task that differs from the main task, finding weights not specific to it. We train one copy of the model using each mini-task for Einne epochs. We refer to the copy & trained for epochs as K KB For each mini-task use temporary classifier O* initialized with the parameters of C*. We discard this classifier after the final iteration of the inner-loop.
Following 255,, our method for episodic training consists of two nested loops, an inner and an outer loop. The inner loop is in charge of training the copies of KB for the current mini-task while the outer loop is in charge of updating the KB weights following a gradient direction that leads to fast adaptation new mini-tasks. During each inner loop, K B and C* are trained end-to-end for Einner epochs.
To simulate the outer-loop and update the K B, we follow Eq. 2 Specifically, for each k, we average the difference between the parameters of the K B before K KB and after K KBB. Pinner in the inner loop (see Eq. 2 This is simply an average of the sum of the cumulative gradients for each model K KB This is repeated Fouter times.
K
KB = KB dVKB
VKB
Th 7K(K BB. Binner
K KB))
(2)
Einner
The subtraction in Eq. is weighted by %k- We compute ?/ shown in Eq. taaking reference the accuracy of each model on a validation batch from the same task t.
aCCk
(3)
ΣS ace;
By updating weights with a weighted average solutions of all meta-tasks, we expect that the new values of the full model should be (on average) appropriate solution for different tasks.
5

---

B
K KBgaa
K KBB
VKB
VKB
VKBK
KB
KB
(a) Generate mini-tasks.
(b) Train copy of KB in each mini-task. (c) Update KB using gradients.
Figure 2: KB update using metalcarning. a) Given a task t, we randomly generate a set of K mini-tasks, where each mini-task consists of subset of classes from the original task. b) For each mini-task, we train an independent copy of the current KB for a fixed amount of epochs, leading to K models. e) Afterwards, models using hold-out set of training examples. Finally, we update the KB using a weighted average of these gradients.
calculate gradients with respect to the loss function of each of these
4 Experiments
For our experiments, we use two benchmarks used in previous works [18, 26]. The first one is 20-Split CIFAR-100 27 that consists of splitting CIFAR- 100 into 20 tasks, each with 5 classes. The second one 20-Split Minilmagenet [22] that consists of dividing its 100 classes into 20 tasks.
In our experiments, we compare MARK with recent state of the art methods: Hard Attention to the Tasks (HAT) 77 A-GEM (28), Adversarial Continual Learning (ACL) [8], GPM [2),, Experience Replay and SupSup 30 We also include a Multitask baseline to serve performance, where all tasks are trained jointly, the Appendix, we include a comparison of size of these alternative models with respect to MARK. All hyperparameters used for each method are also included in the Section
upper bound for
In terms of the feature embedding Fr used to represent input instances, we test the following approaches:
we train F for each task adding a classifier on top that is trained using D After training F*, this classifier discarded.
MARK-Tashe
MARK-Random: fd consists of a set of random weights. All tasks share the same Ft
MARK-Resnet: all tasks share Resnet-18 pre-trained on Imagenet as feature extractor.
We run all of our experiments using different seeds. In terms of hyperparameters, we use SGD with learning rate of 0.01 and batch size of 128. Each task is trained for 50 epochs. To update the KB, we use 10 meta-tasks (K), trained for Einner =40 epochs, each with a learning rate of 0.001. We repeat this training stage 20 times (Eouter). Code is released at: httpa://github.. com/ JuliousHurtade meta- seraiiinng-ssuyy
4.1 Metrics
To quantify the performance of our method, we use two metrics: average accuracy (Acc) and backward transfer (BWT). These are given by:
T
Acc = T
Σ 1=1
ACCT,
BWT
AccT, Acc
(4)
Acc measures average performance over the T tasks after the sequential learning. BWT measures how much performance is lost on previous tasks after sequential learning. As measure of efficiency, we also consider the amount of memory used by each method, considering number of parameters and temporal information needed by each method.
4.2 Results
We first analyze the general performance of our method with respect to the alternative approaches. We highlight the version of MARK where we pre-train F using data from each task (MARK-Tash)

---

Table Results using 20-split CIFAR-100 and 20-split Minilmagenet datasets. Standard deviation for 3 runs is listed in parentheses.
CIFARI00
Mini-Imagenet
Method
Acc (std.)%
BWT%
Mem %
Ace (std.)%
BWT%
Mem
HAT
76.96 (# 1.2)
0.01
146%
59.45 (#0.1)
-0.04
198%
A-GEM
61.88 (# 0.2)
-1.97
205%
52.43 (#3.1)
-15.23
165%
ACL
78.08 (# 1.3)
134%
62.07 (=0.5)
195%
GPM
76.67 3.1)
-0.42
66%
60.41 (=0.6)
0
70%
Exp. Replay
65.90 1.2)
-16.9
130%
SupSup
77.58 (# 1.3)
26/1 1
Multitask
84.08 (=: 1.2)
0
159%
74.44 (+116)
0
99%
60.91 (# 0.4)
-1.73
41%
33.02 (=0.7)
1.29
43%
MARK-Taa -askk
78.31 (# 0.3)
-0.27
100%
69.43 (#1.6)
-0.39
100%
MARK-Resnet
86.29 (+ 0.1)
-3.05
345%
93.55 (+0.2)
-2.52
126%
While Table 2 shows that MARK-Resnet leads to better results, it uses additional information due to its pre-training using ImageNet. is important note that embeddings Fi are directly used to classify the corresponding instances. Instead, they are applied to generate the masks used to query the KB. Aside from its impact on accuracy, we observe no meaningful impact of the input representation on BWT.
As shown in Table for both datasets, MARK-Task outperforms all competing methods in terms of average accuracy, while showing no sign of CF (BWT is close to zero). This is especially remarkable for Mini-Imagennet, as competing methods versus our simple convolutional model. It also noteworthy that MARK-Taa accomplishes this while using almost half less memory than its closest competitor, ACI., This is because MARK-Task reuses the same stored KB for all tasks, while for each task it needs to store a small mask generating function and classifier. Other methods either need store extra parameters for adversarial training 18] or they require access to past gradients (28 (29
complex AlexNet-based architectures during training
4.2.1 Evolution of Weight Updates
We hypothesize that the positive results achieved by MARK are due to the knowledge stored in its KB being highly reusable. To test this, we analyze the updated weight during training of all tasks. As a baseline, we consider version of MARK that does not include metalearning and mask-filtering steps. In this experiment, weights are considered updated if their average deviation is over a certain threshold after training task. Please see the appendix for specific details.
Figure3Jshows shows the percentage of weights that change after training each task. We observe two distinct phenomena: 1) MARK changes rather small number of weights compared the bascline, and 2) As more tasks are trained, the amount of weight updates dwindles.
We attribute both phenomena to MARK's ability reuse previous knowledge to tackle new tasks. Thus updates should be needed mostly when learning new knowledge. The dwindling amount of updates can be linked MARK's ability to develop its KB incrementally, thus tasks are far more likely to be tackled with knowledge already in the KB. We believe that task similarity plays a crucial part in this behaviour, with each task learned facilitating MARK's ability to find similarities for future tasks. However, to elucidate how important task similarity to achieve these results is beyond the scope of this study. We leave this interesting avenue for future work.
4.2.2 Task Learning Speed
We hypothesize that if the KB is storing reusable knowledge, then learning speed for new tasks should increase as the KB incrementally contains more knowledge. We analyze the accuracy curves for each task when using MARK versus a model that is sequentially trained without including any mechanism
Memory usage of SupSup depends whether the model is generated from seed stored fully.
7

---

3.0
0.8
5.6
Bass-re
0.6
5.2
0.
5.0
10
(a) 20-Split CIFAR-100
(b) 20-Split MinilmageNet
Figure 3: Percentage of updated weights in different blocks of the KB during sequential training. As baseline, we consider a version MARK that do not include metalearning and mask-filtering Using MARK fewer updates are required, suggesting that new tasks add less knowledge to the KB due to incremental learning.
to avoid CF. Given that MARK trains its masks and classifiers two times task, to be fair, we train the baseline for twice as many epochs as MARK. Consequently, we report results for the baseline every two epochs. Figure shows shows the comparison in speed between the two models. We observe that indeed, on average, MARK achieves higher accuracy and stabilizes quicker than baseline. This provides further evidence about the ability of MARK to encode reusable knowledge.
Suse Rtc MANK
0.32
0.31
0.00
-0.93
12
Epechs
(a) Training Speed
(b) Accuracy Gain
Figure 4: (a) Average test accuracy in the CIFAR-100 dataset achieved by MARK and a baseline model that is sequentially trained without including any mechanism avoid CF. MARK achieves both greater accuracy and faster stabilization. (b) Accuracy in a task when we re-training the task after the model complete its sequential learning using all available tasks. We observe an increase in performance for most tasks, suggesting successful knowledge accumulation in the KB.
4.2.3 Importance Metalcarning and Mask Functions
We study the impact the metalearning strategy used to update the KB and the mask mask-genneatinng functions used to query the KB. To do this, compare MARK against three ablations:
Simple sequential learning with no metalearning or mask-generating functions. We use the same architecture as the KB.
Baseline:
Baseline + MI: We improve the baseline by adding metalearning, i.e., KB update.
Baseline Mask: We improve the baseline by adding task-specific mask functions.
Figure shows that the baseline suffers from both significant forgetting and reduced performance.
In contrast, when we include metalearning (Baseline + ML) forgetting is reduced. Similarly, when we only add mask-functions (Baseline + Mask), there is boost in performance but forgetting also increases. By using metalearning and mask-functions, our full MARK-Tase model achieves high accuracy with almost no forgetting.

---

Two important observations can be extracted from these experiments: 1) The forgetfulness of Baseline + ML is more significant than that obtained in MARK (-3% v/s -0.25%) showing that learning to use prior knowledge through masks can also help reduce forgetfulness. 2) The maximum accuracy averaged over tasks (without forgetting) is higher in MARK than Baseline + Mask (78% v/s 74%), showing that training with metalearning helps knowledge transfer between tasks. This behaviour might seem counterintuitive as we metalearn only using the current task. However, an important part of MARK is forcing the reuse of previously useful features which biases solutions to start from point which is useful for previous tasks. It is also worthwhile to note that it is known [24] that a REPTILE-styL update (as is used in MARK) optimizes for synergistic gradient updates between minitasks. This might also help learn features that are more reusable, which might explain part of the success of MARK.
4.2.4 Incremental Construction of KB
We expect that, as MARK faces new tasks, its KB should be enriched with new knowledge. To test this hypothesis, we analyze the effect of training again each task after the model completes its sequential learning using all available tasks. As expected, Figure44jshowe that indeed, for most tasks, there is absolute increase of 1.24% in performance when they revisited after completing the sequential learning cycle. However, this increase might have to do with relearning what was forgotten rather than learning new knowledge. Thus, also compared the difference between the maximum accuracy achieved during training for a task versus its final accuracy after retraining. We observed an absolute 0.97% average increase in accuracy well, which shows that indeed new knowledge was used by earlier tasks.
Forgerting
Accurary
Forgersig
0.50
(a) Baseline
(b) Baseline Mermnngrning
Forgeting
Forgeting
1.50
1,50
275
575
3.50
0.50
335
025
(c) Baseline Masks
(d) Full model: MARK-Task
Figure 5: CF and accuracy for different versions of MARK-Task tested 20-Split CIFAR100. (a) Without using metalearning and mask-functions, performance is low and CF is high. (b) Adding only metalearning, performance still low, but there almost no forgetting. mask-functions, performance increases but forgetting is still high. (d) MARK-Task, our full model, achieves high performance with almost no forgetting.
(c) Adding only
5 Limitations
Our method relies on base assumption that there is common structure to be learned between tasks. Thus, if this does not happen it will perform poorly (as any method which expects to reuse parameters for different tasks). Therefore, MARK is sensitive to the quality and structure of the first task.
9

---

As our method relies on task-specific modules, the amount of parameters scales linearly with the amount of tasks, possibly hindering practical use in settings with a large amount of tasks.
is not a straightforward task to extend our model to use memories for replay, since our focus has been the scenario where can't 't have access past data after switching tasks. It is unclear where replay should happen, nor what should be stored as both masks and samples could work well.
6 Related Work
Previous works have tackled the problem of CF using different strategies. Memory-based methods mitigate CF inserting data from past tasks into the training process of new tasks, continuously re- training previous tasks [31, either with raw samples [32 26], or minimizing gradient interference
33 (8))
Other works such as
+4 35
train generator functions (GANs) or autoencoders [36] to
generate elements from past distributions. They seek meemor--fficiency by generating examples. Similarly, [37 38) seek to be memory-efficien by saving feature vectors of instances from previous tasks, while learning a transformation from the feature space of past tasks to current ones.
Other methods focus on limiting the flexibility of new tasks to modify the current model. The typical approach penalizes weight modifications or freezes subset of the model. This can be achieved by adding weight regularizations [10, 8 or using masks to freeze parts of the model [16] 15]. Another
alternative is to increase network capacity by adding extra parameters 9 18). Works similar to ours have proposed to find path of relevant weights to solve the task
[39] freezing used weights, and
limiting learning new tasks. Others use different functions as components in the network, either Hypemettwrks [40), Deep Artificial Neurons (DANs) 41, or Compositional Structures the network components are more flexible during learning.
42 that
Metalcarning 43 is the idea of learning to learn. It has been used as a training strategy in sequence learning 44
50 In the case of [44 45), the objective of meta-learning is to learn starting point from where future sequences can learn sub-components of the model the so-called inner-loop. In the case of MARK, we use the meta- learning strategy while learning new tasks, encouraging the reuse of the weights before and after the modification of the KB.
and continual learning scenarios 46
The idea of conditioning the weights of a model has being used before. [T]] normalizes the output of a convolution layer by squeezing the activation maps via linear functions. 23 conditions the output of convolutional layers using the question in VQA problem. However, as far as we know, this idea has not been combined with metalcariin to update KB.
Conclusions
In this work, we present MARK, novel method for continual learning scenarios that is based on the construction and query of KB that uses metalcarning to incrementally accumulate relevant knowledge from different tasks. Our experiments indicate that the use of metalearning build the KB is the crucial factor to mitigate CF, while the use of mask-functions to query the KB the key factor to achieve high performance. Specifically, MARK achieves state of the results in both 20-Split CIFAR-100 and 20-Split MinilmageNet, while suffering almost no BWT. Compared ACL, the best performing alternative method, MARK 55% of the parameters, 51% of the memory requirements, while achieving almost a 12% increase in average accuracy on the 20-Split MinilmageNet benchmark.
Acknowledgments and Disclosure of Funding
This work was supported by National Agency of Research and Development (ANID) via National Doctorate Scholarships. This project has also received funding from the Millennium Institute of Foundational Research Data (IMFD).
References
[1] Alan D Baddeley. Human memory: Theory and practice. Psychology Press, 1997.
10

---

[2] Saul A McLeod. Forgetting, http:wwwwwsmpppyyyyydooggggggggg 2008. [Online; accessed 30- 30-Sept-2019).
[3] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, 2017.
[4] Nicolas Y Masse, Gregory D Grant, and David Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 2018.
[5] Sang-Wop Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Wor Ha, and Byoung-Take Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, 2017.
[6] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In IEEE International Conference on Computer Vision, 2017.
[7] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, 2018.
[8] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
[9] Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.66616 2016.
[10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
et al.
[11] Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. In International Conference on Learning Representations, 2019.
[12] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, 2018.
[13] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[14] Zhizhong Li and Derek Hoiem. Leaminng without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
[15] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[16] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision. 2018.
[17] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting via model adaptation. In International Conference on Learning Representations, 2019.
[18] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. Proceedings of the European Conference on Computer Vision 2020.

---

[19] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019.
[20] Jeffrey D. Karpicke. A powerful way to improve learning and memory, 2016 (accessed June, 2020).
[21] Gido M van de Ven and Andreas S Toolias. Three scenarios for continual learning. arXiv preprint arXiv: 1004:0/999007755 2019.
[22] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 2016.
[23] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Association for the Advance of Artifical Intelligence, 2018.
[24] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint /803, 029999 2018.
[25] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning. 2017.
[26] Arslan Chaudhry. Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. International Conference on Machine Learning, 2019.
[27] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. http://wwww.csstootoooooo krim/cifar.tml, 2009.
[28] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representations, 2018.
[29] Gobinda Saha and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
[30] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Raste- gari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Larochelle, M. Ran- zato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15173-15184 84. Curran Associates, Inc., 2020.
[31] Sayna Ebrahimi, Suzanne Petryk, Akash Gokul, William Gan, Joseph E. Gonzalez, Mareus Rohrbach, and trevor darrell. Remembering for the right reasons: Explanations reduce catas- trophic forgetting. In International Conference on Learning Representations, 2021.
[32] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
[33] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, 2017.
[34] Timothée Lesort, Hugo Gaselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. International Joint Conference on Neural Networks. IEFF, 2019.
[35] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in neural information processing systems, 2017.
[36] Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations, 2018.
[37] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid, Memory-efficient incremental learning through feature adaptation. In Proceedings of the European Conference on Computer Vision. Springer, 2020.
12

---

[38] Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compression with adaptive quantization modules. In International Conference on Machine Learning, pages 1240-1250. PMLR, 2020.
[39] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 0rXhv:1701.08734 2017.
[40] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
[41] Blake Camp. Jaya Krishna Mandivarapu, and Rolando Estrada. Continual learning with deep artificial neurons. arXiv preprint vrXiv22011.00035 2020.
[42] Jorge A Mendez and Eric Faton. Lifelong learning of compositional structures. International Conference Learning Representations, 2021.
[43] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn. Springer, 1998.
[44] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, 2019.
[45] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth Stanley, Jeff Clune, and Nick Cheney, Learning to continually learn. European Conference on Artificial Intelligence, 2020.
[46] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 2020.
[47] Joseph K J and Vinceth N Balasubramanian. Meta-consolidation for continual learning. In Advances in Neural Information Processing Systems, 2020.
[48] Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2020.
[49] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906. 05201, 2019.
[50] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. arXiv preprint arXiv:2009.009) 00919, 2020.
[51] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiatien networks. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
[52] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Grafficti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven. Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning, 2021.
[53] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions machine learning, arxiv preprint arXiv: 1910.090000000 2019.
13


Reviews:
[{"review_id": "Pl9GEXERlTM", "reviewer": "Reviewer_fktr", "summary": "The paper proposes a new architecture for task-based continual learning. It comprises two main parts : \n1) a Knowledge Base (equivalent to a regular CNN) and \n2) a Mask function with task-specific weights, which generates a per-example set of masks to apply on the weights of the knowledge base.  \nWith the masked weights and the task-specific head a prediction is then obtained. \n\nTo train the model (for tasks > 1), the authors propose to alternate between training mask function keeping the KB fixed, then updating the KB with a fixed mask function. For the KB update, the authors develop a Reptile-like algorithm where artificial \"tasks\" are created by performing multiple updates on data coming from a subset of the classes in the current tasks. \n\nThe authors evaluate their method in the multihead setting on split CIFAR100 and MiniImagenet datasets. They further provide two ablations to evaluate how their contribution affects training speed and how many weights are changed. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Originality__:  This works presents a novel combination of well known techniques (masking for CL, reptile-like updates for CL). It is my opinion that some similar work should be cited for this submission \n\n[1] Wortsman, Mitchell, et al. \"Supermasks in superposition.\" NeurIPS 2020. The authors learn a set of task-specific binary masks over a fixed set of weights.  The authors also look at similar datasets, I strongly recommend that the authors add this baseline to their paper. \n\n__Quality__:  The method presented is sound. __My main criticism of the paper lies in the experimental section__. For all (3) baselines considered, the authors took numbers from the respective papers. This is not problematic in itself, _provided all papers use the exact same training and evaluation protocol_. This is not case the case here, as for example AGEM works in the single epoch setting (resulting in ~2% of the compute used for the author's 50 epoch setting). Moreover, no information is given as to how hyperparameter selection was performed (was a similar compute budget given to each method ?) Overall the empirical section does not meet the bar for this conference. \n\nFor me to accept this paper, the authors should \n-> Include more baselines (at the very least [1]). Expansion based methods (e.g. Progressive Neural networks) also perform very well in the setting explored by the authors, and they have no forgetting by design. A simple replay based method should also be included. \n->  Be very explicit as how the hyperparameter selection for different methods was performed. \n\n\n__Clarity__: Overall the paper is well written and easy to follow. The figures are well made as well.  \n\n__Significance__:  The setting explored by the authors is very relevant (even if only two datasets are explored)\n\n\n**edit** : after having clarified some of my concerns, I have increased my score by one point\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes / ok", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5h"}, "extra_text": {}}, {"review_id": "LNJUQ9bL7N", "reviewer": "Reviewer_ZFES", "summary": "This paper develops a method (MARK) for tackling catastrophic forgetting in neural networks that maintains a knowledge base (KB) in the form of a convnet, which is shared across training of all tasks, and which, when combined with dynamic feature masking and a form of meta-learning to update the KB, enables MARK to outperform competing methods when trained on sequential image classification tasks. In more detail, MARK classifies an image as follows: the image is passed through a task-specific feature extractor, producing an embedding which is used to generate masks (via a task-specific MLP) over the feature maps of the KB convnet (through which the image is passed separately), the output of which is then passed to a task-specific classifier. When a new task is encountered, the first step (“KB querying”) updates the task-specific parameters over several epochs, while keeping the KB parameters fixed; the second step updates the KB parameters using the meta-learning algorithm Reptile, where the inner loop updates the KB parameters and the task-specific classifiers over a batch of “mini-tasks”, each of which are created by sampling a subset of images from a subset of classes in the current task. Finally the KB querying task is repeated before moving to the next task. Experiments are conducted on 20-way split cifar100 and split-mini-imagenet, showing that MARK outperforms a number of competing methods in terms of average accuracy and forgetting, while using less memory in total. Experiments are run using different types of feature extractor (random fixed and one pretrained on imagenet), and with ablations of the meta-learning update and the mask-generating features, demonstrating that the former is most important for mitigating forgetting, and the latter for achieving a good maximum task performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a novel method that combines meta-learning and dynamic feature masking to yield strong empirical results in a couple of standard continual learning benchmarks, and thus it would be a useful contribution to the CL community. For this reason, I recommend it for acceptance, but only marginally, predominantly because several design choices and details of the implementation are not clearly motivated, and I was left unsure about why the model works, despite the effort made in the paper to demonstrate the importance of each component with ablations. Below are more detailed comments:\n- If one were following a typical meta-learning setup, one would expect the KB querying step to match the inner loop of the meta-learning KB update step, but there are at least two differences: (i) the KB parameters are kept fixed in the KB querying step, but are updated in the inner loop of the KB update, and (ii) the mask-generating features are updated in the KB-querying step but not in the inner loop of the KB update step. Could the authors explain the rationale behind these design choices that seem to differ, as far as I am aware, from the usual meta-learning setup (particularly (i))?\n- In the ablation experiments, it is curious to see that, while the mask generating functions have a large impact on maximum performance per task, meta-learning the KB is most important for reducing forgetting. What is the intuition here? One would intuitively expect meta-learning to impact the speed of learning of new tasks (as is shown in the paper), but why should it reduce forgetting?\n- The KB update\n    - Why are the updates weighted by gamma_k? Does this not encourage more adaptation to tasks that the model is already good at?\n    - What if after the first KB Querying update, the model is already good at performing task t - is there a danger of it not getting a signal at all in the KB update?\n- In the baselines that ablate the meta-learning update, is the KB update still updated using SGD or is it kept fixed after training on the first task? If it is kept fixed, then an important ablation is missing, which is to train the KB with SGD along with the rest of the parameters; otherwise, we don’t know how important the meta-learning is for the performance of the model.\n- The KB is trained fully on first task. What if first task is not that relevant for the subsequent ones? Maybe an ablation with a randomly initialised KB could be useful to understand the importance of this?\n- In Figure 4b, is the accuracy gain over the maximum performance achieved in that task, or over the final performance having trained on all tasks? If it’s the latter, to what extent can you attribute this gain to just recovering forgotten knowledge, as opposed to new knowledge accumulated over training of other tasks?\n- Algorithm 2 is a bit confusing:\n    - S_F is not initialised.\n    - C_k is not mentioned here, but in the main text it says that C_t is copied and trained as well in the inner loop.\n- Even though it makes use of extra data, it is encouraging to see the MARK-Resnet (where the feature encoder is pretrained on Imagenet) perform so well in a sequential setting.\n- Comments on related work:\n    - It is mentioned in line 293 that previously “metalearning has not been used to increment a KB as in our model”. However, in [1] and [2], meta-learning is used to update a representation learning network or a neuromodulatory network, both of which can be thought of as a knowledge bases. In [1] the representation learning network is used directly in inference, and in [2] the neuromodulatory network gates the representation learning network, which bears similarities to the dynamic masking in MARK.\n    - As well as HAT, which is cited in the paper, two other methods that use masks for continual learning are [3] and [4], which might be worth mentioning.\n\nMinor comments:\n- Line 199. The name of the referred method is wrong here: “Hard attention tasks” -> “Hard attention to the task”\n- Line 217 and 230. “BTW” -> “BWT”\n- Line 230 “specially” -> ”especially”\n- Line 264. “both,” -> “both”\n- Figure 3 y axes seem to be mislabelled - the range is from 0 to 1 but the axis labelled as “percentage” - should it not be “proportion”, or otherwise the scale should be from 0 to 100?\n\n[1] Javed, Khurram, and Martha White. \"Meta-learning representations for continual learning.\" Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.\n\n[2] Beaulieu, Shawn, et al. \"Learning to Continually Learn.\" ECAI 2020. IOS Press, 2020. 992-1001.\n\n[3] Wortsman, Mitchell, et al. \"Supermasks in Superposition.\" Advances in Neural Information Processing Systems 33 (2020).\n\n[4] Wen, Yeming, Dustin Tran, and Jimmy Ba. \"BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning.\" International Conference on Learning Representations. 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is not much discussion of the limitations of the paper. It is worth mentioning that, while the method uses less memory than the methods compared to, it does suffer from the fact that the number of parameters increases with each task, which may not be scalable for a large number of tasks, especially since the task-specific feature extractors are not insignificantly small.\nThere is a section in the appendix detailing the CO2 impact of the experiments.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "KQV7HdpJyeg", "reviewer": "Reviewer_KoD1", "summary": "This paper is introducing a meta-learning framework into continual learning to promote learning resuable kwowledge from a sequence of tasks. The resuable kwowledge is simply defined by a set of shared model weights. The authors claim these weights as a common knowledege base (KB), which is not only reused but also incrrementally / continually enriched with new knowledge from new tasks, and propose to selectively / adaptively use them by utilzing some masking mechanism.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The sharable weights / modules are defined as a knowledge base (KB), but it doesn't really have some properties that the exact KB has, and this could probably be misleading naming. \n- The role of KB is proposed to incrementally store meaningful knowledge into a set of weights and not only aovid catastrophic forgetting but also improve the learning of upcoming new tasks. To make it happend, the KB is updated through meta-learning based on a gradient-based method. (There were several works that similarly utilize a meta-learning framework to improve continual learning.) \n- The ablation study shows that meta-learning reduces the forgetting problem. However, it is unclear why this meta-learning promotes to incrementally update the KB without catastrophic forgetting. The meta-learing objective is defined by mini-tasks that are sampled only from the current task, not caring about previous tasks. \n- Moreover, the KB seems not to improve the learning of new tasks (i.e. accuracy), but act as a regularizer to avoid catastrophic forgetting. Because, the ablation study shows that the masking mechanism, which is highly related to feature modulation methods (i.e FiLM, DLN, CBN, SENet), is higly improving the accruacy.\n- In terms of network architecutre, $F^t$, $M^t$ and $C^t$ are already a single classifier for task $t$. For example, if we asssume KB is very small or almost constant regardless of input $x$, then the mask variables become a representation itself. That way, it seems like the method is actually incrementally increasing the network size with having task labels.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- For the results, it would be good to compare the results with some possible upper bounds such as results from multi-task learning setting. \n- for line 142, the naming is same as line 136, so it would be good to have another name for it.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "2Kq6GusL7_S", "reviewer": "Reviewer_FTM8", "summary": "The paper presents a continual learning method that uses a shared network (knowledge base) to learn combined knowledge across multiple tasks. This shared network is trained using meta-learning, a version similar to Reptile, on the dataset of the current task. In addition to the knowledge base, the paper proposed parameterized masking operators for each task, that mask out the irrelevant features in the knowledge base for a given task. The overall training proceeds in three steps. In step 1, the knowledge base is kept fixed, and the masking network and the classifier for the given task is trained. In step 2, for the trained masking network and the classifier, the knowledge base is updated using a meta-learning approach. Finally, in step 3, the masks and classifiers are finetuned using the updated knowledge base. Experiments are reported on Split-CIFAR 100 and miniImageNet dataset, where the proposed method is shown to perform better than the reported baselines. Some ablation of the method is also performed where it is shown that both the mate-learning and masking contribute to the overall better performance of the algorithm. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n\n- The paper is written well and easy to understand. \n- The proposed approach strongly outperforms the baseline. \n- Most of the ablations are done nicely and gives a better picture of why the proposed method is working. \t\n\nNegatives/ Questions:\n\n- Motivation for meta-learning: The motivation of using meta-learning is not clear to me. The way it has been used in this work, meta-learning is only performed at a given task-level (by splitting the task into tiny subtasks). In my opinion, the assumption that meta-learning would help different tasks exploit the common structure would only work if the training involves all those tasks jointly. As per my understanding of the paper, the authors perform meta-learning incrementally. The reason they still see the strong results could be that, in the benchmarks they considered, different tasks are not really different (different cifar-100 classes etc.). In a setup, where tasks differ the proposed method might see the forgetting problem in their knowledge base. \n\n- Too many Task-specific parts in the approach : Currently the authors assume a task-specific feature extractor, mask and the classifier head. I wonder whether the method would even need a knowledge base. Perhaps two kinds of ablations can be performed here; 1) combine the feature extractor, masking network and the classifier for each task, train it end-to-end and report numbers, 2) add the knowledge-base, as the authors currently have, but keep it random (i.e) don’t train it, meta- or otherwise, and see how the network performs. It is difficult to discern from section 4.2.3 whether the reported ablations do any of the above. I see that the appendix makes an effort to understand a similar phenomenon but both the ‘feature’ and ‘no-retraining’ baselines differ slightly to what I am describing here. \n\n- Architecture details: Please add the forward pointer to the appendix for the architecture details of all the components (knowledge-base, mask generator, feature extractor etc). \n\n- Section 2:  Section is sparse in the details of the setup. Please add information regarding whether the authors assume an online access to the data of each task (single-pass setup). At test time what are the constraints, do the authors assume an identifier for the corresponding task. \n\n- Baseline comparisons: Some of the methods, AGEM for example, might not be directly comparable as it assumes a single-pass through the data. \n\n- Why baseline + MetaLearning would decrease forgetting: Do the authors have any intuition on why just the vanilla meta-learning on top of baseline would reduce forgetting, especially the way it is done (in a task-specific manner, see my comments above)? I would expect masking to be a better inhibitor of forgetting. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a continual learning method that maintains a knowledge base (KB) shared across all tasks and utilizes a form of meta learning to update the KB and dynamic feature masking to selectively choose relevant weights for each task from the KB. The three training stages are explained, and empirical results are provided that show competitive performance to alternatives. Additionally, an ablation study is provided showing that meta learning is most important for reducing catastrophic forgetting while the masking mechanism contributes to achieving higher accuracy.

The proposed method is an interesting combination of techniques that end up working synergistically to improve both overall performance and mitigate catastrophic forgetting. While the ablation study helps give an idea about which component more significantly contributes to what performance aspect, the writeup may benefit from more explanation  of the design choices and the intuition we should gain from the ablation study, making it more clear why the method works. The experiments section could be improved by clarifying the evaluation setup and comparing all methods on the same setup. Some of these aspects were discussed in the rebuttal, convincing some reviewers to update their scores. 

Decision: Accept (Poster)