Optimizing Reusable Knowledge for Continual Learning via Metalearning
Authors: Julio Hurtado
Alain Raymond
Alvaro Soto
Venue: NeurIPS 2021 Poster (2021)
Keywords: Continual Learning
Meta-learning
Knowledge transfer
Catastrophic forgetting

Abstract: When learning tasks over time, artificial neural networks suffer from a problem known as Catastrophic Forgetting (CF). This happens when the weights of a network are overwritten during the training of a new task causing forgetting of old information. To address this issue, we propose MetA Reusable Knowledge or MARK, a new method that fosters weight reusability instead of overwriting when learning a new task. Specifically, MARK keeps a set of shared weights among tasks. We envision these shared weights as a common Knowledge Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. Key components behind MARK are two-fold. On the one hand, a metalearning approach provides the key mechanism to incrementally enrich the KB with new knowledge and to foster weight reusability among tasks. On the other hand, a set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task. By using MARK, we achieve state of the art results in several popular benchmarks, surpassing the best performing methods in terms of average accuracy by over 10% on the 20-Split-MiniImageNet dataset, while achieving almost zero forgetfulness using 55% of the number of parameters. Furthermore, an ablation study provides evidence that, indeed, MARK is learning reusable knowledge that is selectively used by each task.

Optimizing Reusable Knowledge for Continual Learning via Metalearning
Julio Hurtado
Alain Raymond-Saer, Department of Computer Science
Department of Computer Science Pontificia Universidad Católica de Chile j jahurtadofuc.cl
Pontificia Universidad Católica de Chile
afraymon@uc.co cl
Alvaro Soto
Department of Computer Science Pontificia Universidad Católica de Chile asoto@ing, puc. .cl
Abstract
When learning tasks over time, artificial neural networks suffer from a problem known as Catastrophic Forgetting (CF). This happens when the weights of a network are overwritten during the training of a new task causing forgetting old information. To address this issue, we propose MetA Reusable Knowledge MARK, a new method that fosters weight reusability instead of overwriting when learning new task. Specifically, MARK keeps set of shared weights among tasks. We envision these shared weights as a common Knowledge Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. Key components behind MARK are two-fold. On the one hand, a metalearning approach provides the key mechanism to incrementally enrich the KB with new knowledge and to foster weight reusability among tasks. On the other hand, set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task. By using MARK, we achieve state of the art results in several popular benchmarks, surpassing the best performing methods in terms of average accuracy by over 10% on the Seo-ppiitMinilmmaeeee dataset, while achieving almost zero forgetfulness using 55% of the number of parameters. Furthermore, an ablation study provides evidence that, indeed, MARK is learning reusable knowledge that selectively used by each task.
Introduction
Among the cognitive abilities of humans, memory is of the most relevant. However, memories are fragile. In particular, studies from cognitive psychology show that memories can be lost when new knowledge disrupts old information, in process known task interference 2 In the of artificial neural networks (ANNs), task interference is behind the problem known as Catastrophic Forgetting (CF)
6. 7) In effect, model learns causing catastrophic forgetting of old information.
3 4
task, its weights overwritten
Unfortunately, in the case of ANNs, task interference is more severe than in the case of humans. This evident for the case of continual learning, i.e., the case when data from new tasks is presented sequentially the learner, who does not have access previous data. Under this training scheme, when a previously trained model is retrained on a new task, it usually suffers a significant drop in performance in the original task 8 10.. Previous works prevent CF have followed two main consists of avoiding the modification of parameters
strategies. The first strategy 10
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that are key to solve previous tasks. Specifically, when facing new task, regularization term ensures that critical parameters are modified as little as possible. In general, this approach shows satisfactory performance in problems that involve few tasks, however, when the number of tasks increases, problems such as accumulated drifting in weight values and interference among them, make this approach difficult to scale. The second strategy [14] 15
17 7 [8] consists of introducing [6]
structural changes to the architecture of a model. This includes methods that reserve part of the network capacity to learn each task [9]. and methods that use special memory units to recall critical training examples from previous tasks [77 The main problems with these methods are the extra model complexity and the need of efficient method to recall key information from previous tasks.
In contrast to these previous strategies, when learning new tasks, humans continuously associate previous experience to new situations, strengthening previous memories which helps to mitigate the CF problem [0 Taking inspiration from this mechanism, we propose MetA Reusable Knowledge or MARK, a new model based on learning strategy that, instead of mitigating weight overwriting or learning independent weights for different tasks, uses metalearning approach to foster weight reusability among tasks. In particular, we envision these shared weights Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. In this sense, the KB behind MARK is not given by external memory that encodes information in its vectors, but by a trainable model that encodes shared information in its weights. As a complementary mechanism to query this KB, MARK also includes set of trainable masks that are in charge of implementing a selective addressing scheme to query the KB.
common Knowledge
Consequently, to build and query its shared KB, MARK exploits two complementary learning strategies. On the one hand, a metalearning technique provides the key mechanism to meet two goals: i) encourage weight updates that are useful for multiple tasks, and ii) enrich the KB with new knowledge as the model learns new tasks. On the other hand, a set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task.
In terms of its inner operation, MARK works by first forcing the model reuse current knowledge via functions that detect the importance of each pattern learned in the KB, and later expanding its knowledge if past knowledge is not sufficient to successfully perform a task. As an intuition behind how MARK works (not to be taken literally), when facing different tasks related to the classification of animals, MARK may take advantage of its metalearning approach to learn relevant features to discriminative concepts such as fur, paws, fin, and ears. Afterwards, when solving task related to the discrimination of horses vs zebras, it might use a mask to focus on features related to fur rather than fins or ears.
We perform extensive experiments to analyze the inner workings behind MARK. In particular, we demonstrate that, indeed, MARK is able to accumulate knowledge incrementally as it learns new tasks. Furthermore. in terms of the pieces of knowledge used to solve each task, our experiments confirm that the use of task dependent feature masks results critical to the success of the method. We summarize our contributions follows:
We propose MARK, a novel method for continual learning scenarios that is based on two com- plementary mechanisms: i) A new method to build a KB that incrementally accumulates relevant knowledge from several tasks while avoiding CF problems, ii) A mechanism to query this KB to extract relevant knowledge to solve new tasks.
An extensive experimental evaluation and public implementation of MARK that demonstrate its advantages with respect to previous works.
2 Continual Learning Scenario
Following previous work on continual learning [21]. we consider a task incremental scenario, where task identity is provided at training time. Each task & consists of new data distribution D' = (X', Y", T where X denotes the input instances, Y" denotes the instance labels, and T" is a task ID. This task ID required during training and inference. The goal is train a classification
model f X - Y using data from sequence of 'T' tasks: D = (D' ...., ) Following the usual setup for continual learning [21], each task is presented sequentially to the model. Our scenario allows for unlimited use of data from the current task, but after switching to a new task, this data is no longer available. Furthermore, there is no intersection between the classes of different tasks, but they share similar domain. This domain intersection is key to take ad advantage of common patterns among

---

the tasks. Next section presents the details behind MARK, our proposed model for incremental learning in the previous scenario.
Method
When learning tasks sequentially, humans build upon previous knowledge, leading to incremental learning. In contrast, in a similar scenario, ANNs devote all of their resources the current task, leading to the problem of CF. Taking inspiration from the behavior of humans, an appealing idea is to provide the model with KB that, as the model faces new tasks, incrementally captures relevant knowledge. Using this shared KB, the model can associate previous experience to new situations, mitigating the CF problem. To implement this idea, have address two main challenges: How do we build this KB incrementally?, and ii) How do we query this KB to access relevant pieces of knowledge?
To address the first challenge, we leverage metalearning in order to train a KB from data. Specifically, metalearning strategy known episodic training [22]. This strategy consists of sampling from a task pairs of support and query sets to simulate training data from a large number of mini- tasis These mini-tasks are then used to bias the learner to improve its ability to generalize to future tasks. In turn, this generalization leads to our goal: to capture relevant knowledge that can be reused to face new tasks.
To address the second challenge, on top of the KB, we train gank-ggennatiing functions for each task. These masks provide suitable mechanism to selectively access the information encoded in the weights of the KB. We can envision these masks as a query that is used to access the intermediate activations of the KB. Following 23) these masks depend solely on each specific task and input. In this way, given an input, MARK uses the corresponding mask to query the KB generating a feature vector. This vector is then used by a task dependent classification head to output prediction. We describe next the details behind this operation.
3.1 Model Architecture
Figure shows schematic view of the operation behind MARK, where the main modules are:
Feature Extractor (F"): This module is charge of providing an initial embedding for each input X,, i.c., F takes input X; and outputs vector representation F In our case, we test our method using visual recognition applications, therefore, F is given by a convolutional model. is important to note that model Fs can be shared among tasks it can be specific to each task.
Knowledge Base (K B): This is the main module behind MARK. It is in charge of accumulating relevant knowledge as the model faces new tasks. In our implementation, we use a convolutional architecture with B blocks. This part of the model is shared across tasks.
functions )) These modules take an input feature vector F and produce an instance and task-dependent mask M for each block of the KB. Each mask consists of a set of scalars, one for each channel in the convolutional blocks of the KB, that multiply the activation of each channel. These masks are critical to select what knowledge is relevant to each instance and task. In our implementation. we use fully connected layers.
Classifier (C"): These modules correspond tusk-dependent classification heads. Its input FiKE is given by flattening operation over the output of the last block of the KB. Given the task ID of an input X,, the corresponding head outputs the model prediction. our implementation, we use fully connected layers.
Specific details can be found in Section A.I As shown in Figure [ the flow of information in MARK is as follows. Input X, goes into F' to extract the representation TV. This representation is then used by M to produce the set of masks that condition each of the blocks in the KB. The same input X, enters the mask-conditioned KB leading to vector Ft K used by the classification head. Finally, classifier C" generates the model prediction, where t is the task ID associated to input X,-
3.2 MARK Training
Algorithm D describes the training process behind MARK. The first step consists of initializing the KB by training it end-to-end on the first task, without using metalcarning and mask functions. In
3

---

KB
Block
Block 2
Block
Xi
F
Figure 1: Given an input X; from task t, (1) We use feature extractor Ft to obtain F (2) F is then passed to mask function M' to generate mask M Afterwards, (3) the same input X; enters the KB, which has intermediate activations modulated by Mý. Finally, (4) the modulated features go through a task-dependent classifier C that performs the class prediction for X,.
other words, we perform the KB initialization using the regular training procedure of a convolutional neural network for classification task. After this, train MARK sequentially each task by alternating three main steps:
1. KB Querying. We train task dependenndnt generrennn functions that used to query the KB using vector F. Also, we concurrently train the task classifier for the current task. Notice that, leaving aside the KB initialization, in this step each new task is trained using only accumulated knowledge from previous tasks.
KB Update. We use metalearning strategy to update the weights in the KB. This scheme allows fostering KB updates that favor the acquisition of knowledge that can be reused to face new tasks.
KB Querying. After updating the KB using knowledge from the current task, we repeat the querying process to finetune the mask-generating functions and task classifier using these new knowledge. Notice that during this step the KB is kept fixed.
The intuition behind the application of the three previous steps is follows. We initially query the KB using the accumulated knowledge from previous tasks. This forces mask functions and classifiers to reuse the available knowledge. When that knowledge is exhausted, we proceed to add knowledge from the current task into the KB. Finally, we take advantage of this newly updated KB to obtain final mask-functions and classifiers for given task. We describe next the main steps behind the process to query and update the KB.
Algorithm 1: MARK -Training Process
Algorithm 2: KB-Update
Components:
Components:
D Dataset for task t.
KB: Knowledge Base.
pt Feature extractor for task t.
C* Temporal Classifier for batch
K B: Knowledge Base.
Easter: number of training updates for KB (outer loop)
M' Mask Mene-atng function for task t.
C³: Classifier for task L.
Finner: number of inner loop epochs.
KB Train (KB))))
/*Init KB o/
for e Flouter do Generate K batches of data from D)
for task t + to do
[=]. Obtain feature vectors F/ay
for k in K do K +- KKB Copp of KBB*/ Initialize(C*) Train R and C* with batch & Emaer epochs.
Ff + (-(FF (F'(D))
(1,...,|D"I
f+2. Train M and C using current KB*/
Trainç M & C*)
1²3. Update KB using metalearning")
KB- + KB-Update A'llorrthmm 2
end
444. Finetune M and O' with updated KB*/
VKB + Ernner Σ (KKB -KB) KB - KB - aVKB
Train & C*)
end
end
Output: Trained modules KB, M C'.
Output: KB
/*output updated K B%

---

3.2.1 KB Querying
Once we obtain feature vectors through the use of feature extractor Ft, the model can learn which are the modules in the KB that can best solve the current task. In this training stage, the model trains functions to learn how to use the knowledge available in the KB, focusing solely on reusing knowledge from previous tasks, without modifying the KB, particular, in this step we only train M' and C Both are trained end-to-end, while keeping the KB weights frozen.
As we generate masks for each intermediate activation of our model, strictly speaking we have a total of B mask-generating functions. However, to facilitate the notation, we subsume all such functions under the term M and think of its output as the concatenation of the results of those B functions. Eq. shows the function M where a mask Mf is obtained given an input X; from task t. Our implementation of these functions consists of a linear function with parameters Wi,, and an activation function p. which we implement ReLU.
Mf = M'(E = ((WAMM)FF))
Masks generated in this process encouraged to have two effects: first, give signal of how important a specific module from the KB is for the current input; second, to make sure that gradient updates are done where they really matter. If an activation map is irrelevant for certain task, then the value of the corresponding mask will be zero, making the gradient update associated to that activation being zero as well.
3.2.2 KB Update
The purpose of this training step is to add new knowledge from the current task to the KB, As a further goal, aim enrich the KB with features that are highly general, i.c., they can be used to solve several tasks. To achieve this, we use metalearning as a way to force the model to capture knowledge that can be reused face new tasks.
Figure an schematic view of the metaleaming procedure that we use train MARK. This procedure consists of an adaptation of the training process presented in (24). Specifically, we create set of K mini-tasks, where each mini-task consists randomly sampling from the current task set of H classes and h training instances per class. This allows to create mini-task that differs from the main task, finding weights not specific to it. We train one copy of the model using each mini-task for Einne epochs. We refer to the copy & trained for epochs as K KB For each mini-task use temporary classifier O* initialized with the parameters of C*. We discard this classifier after the final iteration of the inner-loop.
Following 255,, our method for episodic training consists of two nested loops, an inner and an outer loop. The inner loop is in charge of training the copies of KB for the current mini-task while the outer loop is in charge of updating the KB weights following a gradient direction that leads to fast adaptation new mini-tasks. During each inner loop, K B and C* are trained end-to-end for Einner epochs.
To simulate the outer-loop and update the K B, we follow Eq. 2 Specifically, for each k, we average the difference between the parameters of the K B before K KB and after K KBB. Pinner in the inner loop (see Eq. 2 This is simply an average of the sum of the cumulative gradients for each model K KB This is repeated Fouter times.
K
KB = KB dVKB
VKB
Th 7K(K BB. Binner
K KB))
(2)
Einner
The subtraction in Eq. is weighted by %k- We compute ?/ shown in Eq. taaking reference the accuracy of each model on a validation batch from the same task t.
aCCk
(3)
ΣS ace;
By updating weights with a weighted average solutions of all meta-tasks, we expect that the new values of the full model should be (on average) appropriate solution for different tasks.
5

---

B
K KBgaa
K KBB
VKB
VKB
VKBK
KB
KB
(a) Generate mini-tasks.
(b) Train copy of KB in each mini-task. (c) Update KB using gradients.
Figure 2: KB update using metalcarning. a) Given a task t, we randomly generate a set of K mini-tasks, where each mini-task consists of subset of classes from the original task. b) For each mini-task, we train an independent copy of the current KB for a fixed amount of epochs, leading to K models. e) Afterwards, models using hold-out set of training examples. Finally, we update the KB using a weighted average of these gradients.
calculate gradients with respect to the loss function of each of these
4 Experiments
For our experiments, we use two benchmarks used in previous works [18, 26]. The first one is 20-Split CIFAR-100 27 that consists of splitting CIFAR- 100 into 20 tasks, each with 5 classes. The second one 20-Split Minilmagenet [22] that consists of dividing its 100 classes into 20 tasks.
In our experiments, we compare MARK with recent state of the art methods: Hard Attention to the Tasks (HAT) 77 A-GEM (28), Adversarial Continual Learning (ACL) [8], GPM [2),, Experience Replay and SupSup 30 We also include a Multitask baseline to serve performance, where all tasks are trained jointly, the Appendix, we include a comparison of size of these alternative models with respect to MARK. All hyperparameters used for each method are also included in the Section
upper bound for
In terms of the feature embedding Fr used to represent input instances, we test the following approaches:
we train F for each task adding a classifier on top that is trained using D After training F*, this classifier discarded.
MARK-Tashe
MARK-Random: fd consists of a set of random weights. All tasks share the same Ft
MARK-Resnet: all tasks share Resnet-18 pre-trained on Imagenet as feature extractor.
We run all of our experiments using different seeds. In terms of hyperparameters, we use SGD with learning rate of 0.01 and batch size of 128. Each task is trained for 50 epochs. To update the KB, we use 10 meta-tasks (K), trained for Einner =40 epochs, each with a learning rate of 0.001. We repeat this training stage 20 times (Eouter). Code is released at: tttpa://github.. com/ JuliousHurtade meta- seraiiinng-ssuyy
4.1 Metrics
To quantify the performance of our method, we use two metrics: average accuracy (Acc) and backward transfer (BWT). These are given by:
T
Acc = T
Σ 1=1
ACCT,
BWT
AccT, Acc
(4)
Acc measures average performance over the T tasks after the sequential learning. BWT measures how much performance is lost on previous tasks after sequential learning. As measure of efficiency, we also consider the amount of memory used by each method, considering number of parameters and temporal information needed by each method.
4.2 Results
We first analyze the general performance of our method with respect to the alternative approaches. We highlight the version of MARK where we pre-train F using data from each task (MARK-Tash)

---

Table Results using 20-split CIFAR-100 and 20-split Minilmagenet datasets. Standard deviation for 3 runs is listed in parentheses.
CIFARI00
Mini-Imagenet
Method
Acc (std.)%
BWT%
Mem %
Ace (std.)%
BWT%
Mem
HAT
76.96 (# 1.2)
0.01
146%
59.45 (#0.1)
-0.04
198%
A-GEM
61.88 (# 0.2)
-1.97
205%
52.43 (#3.1)
-15.23
165%
ACL
78.08 (# 1.3)
134%
62.07 (=0.5)
195%
GPM
76.67 3.1)
-0.42
66%
60.41 (=0.6)
0
70%
Exp. Replay
65.90 1.2)
-16.9
130%
SupSup
77.58 (# 1.3)
26/1 1
Multitask
84.08 (=: 1.2)
0
159%
74.44 (+116)
0
99%
60.91 (# 0.4)
-1.73
41%
33.02 (=0.7)
1.29
43%
MARK-Taa -ask
78.31 (# 0.3)
-0.27
100%
69.43 (#1.6)
-0.39
100%
MARK-Resnet
86.29 (+ 0.1)
-3.05
345%
93.55 (+0.2)
-2.52
126%
While Table 2 shows that MARK-Resnet leads to better results, it uses additional information due to its pre-training using ImageNet. is important note that embeddings Fi are directly used to classify the corresponding instances. Instead, they are applied to generate the masks used to query the KB. Aside from its impact on accuracy, we observe no meaningful impact of the input representation on BWT.
As shown in Table for both datasets, MARK-Task outperforms all competing methods in terms of average accuracy, while showing no sign of CF (BWT is close to zero). This is especially remarkable for Mini-Imagennet, as competing methods versus our simple convolutional model. It also noteworthy that MARK-Taa accomplishes this while using almost half less memory than its closest competitor, ACI., This is because MARK-Task reuses the same stored KB for all tasks, while for each task it needs to store a small mask generating function and classifier. Other methods either need store extra parameters for adversarial training 18] or they require access to past gradients (28 (29
complex AlexNet-based architectures during training
4.2.1 Evolution of Weight Updates
We hypothesize that the positive results achieved by MARK are due to the knowledge stored in its KB being highly reusable. To test this, we analyze the updated weight during training of all tasks. As a baseline, we consider version of MARK that does not include metalearning and mask-filtering steps. In this experiment, weights are considered updated if their average deviation is over a certain threshold after training task. Please see the appendix for specific details.
Figure3Jshows shows the percentage of weights that change after training each task. We observe two distinct phenomena: 1) MARK changes rather small number of weights compared the bascline, and 2) As more tasks are trained, the amount of weight updates dwindles.
We attribute both phenomena to MARK's ability reuse previous knowledge to tackle new tasks. Thus updates should be needed mostly when learning new knowledge. The dwindling amount of updates can be linked MARK's ability to develop its KB incrementally, thus tasks are far more likely to be tackled with knowledge already in the KB. We believe that task similarity plays a crucial part in this behaviour, with each task learned facilitating MARK's ability to find similarities for future tasks. However, to elucidate how important task similarity to achieve these results is beyond the scope of this study. We leave this interesting avenue for future work.
4.2.2 Task Learning Speed
We hypothesize that if the KB is storing reusable knowledge, then learning speed for new tasks should increase as the KB incrementally contains more knowledge. We analyze the accuracy curves for each task when using MARK versus a model that is sequentially trained without including any mechanism
Memory usage of SupSup depends whether the model is generated from seed stored fully.
7

---

3.0
0.8
5.6
Bass-re
0.6
5.2
0.
5.0
10
(a) 20-Split CIFAR-100
(b) 20-Split MinilmageNet
Figure 3: Percentage of updated weights in different blocks of the KB during sequential training. As baseline, we consider a version MARK that do not include metalearning and mask-filtering Using MARK fewer updates are required, suggesting that new tasks add less knowledge to the KB due to incremental learning.
to avoid CF. Given that MARK trains its masks and classifiers two times task, to be fair, we train the baseline for twice as many epochs as MARK. Consequently, we report results for the baseline every two epochs. Figure shows shows the comparison in speed between the two models. We observe that indeed, on average, MARK achieves higher accuracy and stabilizes quicker than baseline. This provides further evidence about the ability of MARK to encode reusable knowledge.
Suse Rtc MANK
0.32
0.31
0.00
-0.93
12
Epechs
(a) Training Speed
(b) Accuracy Gain
Figure 4: (a) Average test accuracy in the CIFAR-100 dataset achieved by MARK and a baseline model that is sequentially trained without including any mechanism avoid CF. MARK achieves both greater accuracy and faster stabilization. (b) Accuracy in a task when we re-training the task after the model complete its sequential learning using all available tasks. We observe an increase in performance for most tasks, suggesting successful knowledge accumulation in the KB.
4.2.3 Importance Metalcarning and Mask Functions
We study the impact the metalearning strategy used to update the KB and the mask-genneatinng functions used to query the KB. To do this, compare MARK against three ablations:
Simple sequential learning with no metalearning or mask-generating functions. We use the same architecture as the KB.
Baseline:
Baseline + MI: We improve the baseline by adding metalearning, i.e., KB update.
Baseline Mask: We improve the baseline by adding task-specific mask functions.
Figure shows that the baseline suffers from both significant forgetting and reduced performance. In contrast, when we include metalearning (Baseline + ML) forgetting is reduced. Similarly, when we only add mask-functions (Baseline + Mask), there is boost in performance but forgetting also increases. By using metalearning and mask-functions, our full MARK-Tase model achieves high accuracy with almost no forgetting.

---

Two important observations can be extracted from these experiments: 1) The forgetfulness of Baseline + ML is more significant than that obtained in MARK (-3% v/s -0.25%) showing that learning to use prior knowledge through masks can also help reduce forgetfulness. 2) The maximum accuracy averaged over tasks (without forgetting) is higher in MARK than Baseline + Mask (78% v/s 74%), showing that training with metalearning helps knowledge transfer between tasks. This behaviour might seem counterintuitive as we metalearn only using the current task. However, an important part of MARK is forcing the reuse of previously useful features which biases solutions to start from point which is useful for previous tasks. It is also worthwhile to note that it is known [24] that a REPTILE-styL update (as is used in MARK) optimizes for synergistic gradient updates between minitasks. This might also help learn features that are more reusable, which might explain part of the success of MARK.
4.2.4 Incremental Construction of KB
We expect that, as MARK faces new tasks, its KB should be enriched with new knowledge. To test this hypothesis, we analyze the effect of training again each task after the model completes its sequential learning using all available tasks. As expected, Figure44jshowe that indeed, for most tasks, there is absolute increase of 1.24% in performance when they revisited after completing the sequential learning cycle. However, this increase might have to do with relearning what was forgotten rather than learning new knowledge. Thus, also compared the difference between the maximum accuracy achieved during training for a task versus its final accuracy after retraining. We observed an absolute 0.97% average increase in accuracy well, which shows that indeed new knowledge was used by earlier tasks.
Forgerting
Accurary
Forgersig
0.50
(a) Baseline
(b) Baseline Mermnngrning
Forgeting
Forgeting
1.50
1,50
275
575
3.50
0.50
335
025
(c) Baseline Masks
(d) Full model: MARK-Task
Figure 5: CF and accuracy for different versions of MARK-Task tested 20-Split CIFAR100. (a) Without using metalearning and mask-functions, performance is low and CF is high. (b) Adding only metalearning, performance still low, but there almost no forgetting. mask-functions, performance increases but forgetting is still high. (d) MARK-Task, our full model, achieves high performance with almost no forgetting.
(c) Adding only
5 Limitations
Our method relies on base assumption that there is common structure to be learned between tasks. Thus, if this does not happen it will perform poorly (as any method which expects to reuse parameters for different tasks). Therefore, MARK is sensitive to the quality and structure of the first task.
9

---

As our method relies on task-specific modules, the amount of parameters scales linearly with the amount of tasks, possibly hindering practical use in settings with a large amount of tasks.
is not a straightforward task to extend our model to use memories for replay, since our focus has been the scenario where can't 't have access past data after switching tasks. It is unclear where replay should happen, nor what should be stored as both masks and samples could work well.
6 Related Work
Previous works have tackled the problem of CF using different strategies. Memory-based methods mitigate CF inserting data from past tasks into the training process of new tasks, continuously re- training previous tasks [31, either with raw samples [32 26], or minimizing gradient interference
33 (8))
Other works such as
+4 35
train generator functions (GANs) or autoencoders [36] to
generate elements from past distributions. They seek meemor--fficiency by generating examples. Similarly, [37 38) seek to be memory-efficien by saving feature vectors of instances from previous tasks, while learning a transformation from the feature space of past tasks to current ones.
Other methods focus on limiting the flexibility of new tasks to modify the current model. The typical approach penalizes weight modifications or freezes subset of the model. This can be achieved by adding weight regularizations [10, 8 or using masks to freeze parts of the model [16] 15]. Another
alternative is to increase network capacity by adding extra parameters 9 18). Works similar to ours have proposed to find path of relevant weights to solve the task
[39] freezing used weights, and
limiting learning new tasks. Others use different functions as components in the network, either Hypemettwrks [40), Deep Artificial Neurons (DANs) 41, or Compositional Structures the network components are more flexible during learning.
42 that
Metalcarning 43 is the idea of learning to learn. It has been used as a training strategy in sequence learning 44
50 In the case of [44 45), the objective of meta-learning is to learn starting point from where future sequences can learn sub-components of the model the so-called inner-loop. In the case of MARK, we use the meta- learning strategy while learning new tasks, encouraging the reuse of the weights before and after the modification of the KB.
and continual learning scenarios 46
The idea of conditioning the weights of a model has being used before. [T]] normalizes the output of a convolution layer by squeezing the activation maps via linear functions. 23 conditions the output of convolutional layers using the question in VQA problem. However, as far as we know, this idea has not been combined with metalcariin to update KB.
Conclusions
In this work, we present MARK, novel method for continual learning scenarios that is based on the construction and query of KB that uses metalcarning to incrementally accumulate relevant knowledge from different tasks. Our experiments indicate that the use of metalearning build the KB is the crucial factor to mitigate CF, while the use of mask-functions to query the KB the key factor to achieve high performance. Specifically, MARK achieves state of the art results in both 20-Split CIFAR-100 and 20-Split MinilmageNet, while suffering almost no BWT. Compared ACL, the best performing alternative method, MARK 55% of the parameters, 51% of the memory requirements, while achieving almost a 12% increase in average accuracy on the 20-Split MinilmageNet benchmark.
Acknowledgments and Disclosure of Funding
This work was supported by National Agency of Research and Development (ANID) via National Doctorate Scholarships. This project has also received funding from the Millennium Institute of Foundational Research Data (IMFD).
References
[1] Alan D Baddeley. Human memory: Theory and practice. Psychology Press, 1997.
10

---

[2] Saul A McLeod. Forgetting, http:wwwwwsmpppyyyyydooggggggggg 2008. [Online; accessed 30- 30-Sept-2019).
[3] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, 2017.
[4] Nicolas Y Masse, Gregory D Grant, and David Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 2018.
[5] Sang-Wop Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Wor Ha, and Byoung-Take Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, 2017.
[6] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In IEEE International Conference on Computer Vision, 2017.
[7] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, 2018.
[8] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
[9] Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.66616 2016.
[10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
et al.
[11] Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. In International Conference on Learning Representations, 2019.
[12] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, 2018.
[13] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[14] Zhizhong Li and Derek Hoiem. Leamiing without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
[15] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[16] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision. 2018.
[17] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting via model adaptation. In International Conference on Learning Representations, 2019.
[18] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. Proceedings of the European Conference on Computer Vision 2020.

---

[19] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019.
[20] Jeffrey D. Karpicke. A powerful way to improve learning and memory, 2016 (accessed June, 2020).
[21] Gido M van de Ven and Andreas S Toolias. Three scenarios for continual learning. arXiv preprint arXiv: 1004:v/999007755 2019.
[22] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 2016.
[23] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Association for the Advance of Artifical Intelligence, 2018
[24] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint /803, 029999 2018.
[25] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning. 2017.
[26] Arslan Chaudhry. Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. International Conference on Machine Learning, 2019.
[27] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. http://wwww.csstootoooooo krim/cifar.tml, 2009.
[28] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representations, 2018.
[29] Gobinda Saha and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
[30] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Raste- gari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Larochelle, M. Ran- zato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15173-15184 84. Curran Associates, Inc., 2020.
[31] Sayna Ebrahimi, Suzanne Petryk, Akash Gokul, William Gan, Joseph E. Gonzalez, Mareus Rohrbach, and trevor darrell. Remembering for the right reasons: Explanations reduce catas- trophic forgetting. In International Conference on Learning Representations, 2021.
[32] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
[33] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, 2017.
[34] Timothée Lesort, Hugo Gaselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. International Joint Conference on Neural Networks. IEFF, 2019.
[35] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in neural information processing systems, 2017.
[36] Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations, 2018.
[37] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid, Memory-efficient incremental learning through feature adaptation. In Proceedings of the European Conference on Computer Vision. Springer, 2020.
12

---

[38] Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compression with adaptive quantization modules. In International Conference on Machine Learning, pages 1240-1250. PMLR, 2020.
[39] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 0rXhv:1701.08734 2017.
[40] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
[41] Blake Camp. Jaya Krishna Mandivarapu, and Rolando Estrada. Continual learning with deep artificial neurons. arXiv preprint vrXiv2201..00035 2020.
[42] Jorge A Mendez and Eric Faton. Lifelong learning of compositional structures. International Conference Learning Representations, 2021.
[43] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn. Springer, 1998.
[44] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, 2019.
[45] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth Stanley, Jeff Clune, and Nick Cheney, Learning to continually learn. European Conference on Artificial Intelligence, 2020.
[46] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 2020.
[47] Joseph K J and Vinceth N Balasubramanian. Meta-consolidation for continual learning. In Advances in Neural Information Processing Systems, 2020.
[48] Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2020.
[49] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906. 05201, 2019.
[50] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. arXiv preprint arXiv:2009.009) 00919, 2020.
[51] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiatien networks. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
[52] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Grafficti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven. Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning, 2021.
[53] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions machine learning, arxiv preprint arXiv: 1910.090000000 2019.
13


Reviews:
[{"review_id": "lWoQDywX8bL", "reviewer": "Reviewer_GtkW", "summary": "This paper mainly solves the catastrophic forgetting problem in task-based continual learning. In particular, this paper proposes a new meta-learning based method, MARK, which uses knowledge base to store some useful knowledge and uses meta-learning to update Knowledge Base. In addition, the knowledge of KB is selected through a set of masks. Finally, the experimental results of this paper show that MARK better than existing methods on two datasets. The main innovation of this work is to use meta-learning to update the knowledge base to simulate human learning methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using meta-learning to update the knowledge base and combining with mask for feature selection is not novel enough. The main innovation of this paper is to propose \"meta-learning to update the knowledge base\", but the method of updating KB only with the data of the new task still has CF for KB. Therefore, I don’t know that the use of meta-learning in this paper can get highly general features by KB. \n\nIn content: Refer to the question in Originality, hoping to get a clearer explanation. An additional suggestion is that if you use part of the data of the previous task (caching or generation) when updating the knowledge warehouse, will you achieve better results (e.g., use the replay based method)? \nExperimentally: \n1. Please show the experimental results of GPM on CIFA-100? As far as I know, the GPM code is very easy to run CIFAR-100. \n2. Because of meta-learning, how different is the training time between MARK and baseline? \n3. There are relatively few baseline models for experimental comparison. CONTINUAL LEARNING WITH HYPERNETWORKS. ICLR 2020. Continual Learning by Using Information of Each Class Holistically. AAAI 2021 \nClarity:\n\n This paper is clearly written. But I suggest that section 2 should be a subsection of section 3.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential negative societal impact should be addressed clearly.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical issue with this paper.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "cu0cQZgVPW_", "reviewer": "Reviewer_AG6A", "summary": "This paper introduces a novel method MARK which utilizes the knowledge base (KB) that can facilitate learning a novel task and reducing the forgetting issue in continual learning and a mask function that selects proper reusable features in KB. The learning procedure consists of four steps: 1) Obtain feature vectors to train the mask function, 2) train a mask function and classifier to learn a novel task, 3) update KB using meta-learning, 4) fine-tune the mask function and classifier to utilize the information of updated KB. When updating the KB, the dataset for a novel task separated to many subtasks, and using the adaptation technique proposed in [24], KB meta-learns the features that can generalize to different tasks. In experiments, the proposed methods achieves the state-of-the-art accuracy in CIFAR and mini-ImageNet dataset. By ablating the main components (KB and mask functions), authors also show the importance of each component.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n- This paper proposes a novel method that utilizes the knowledge base which contains well adaptable features to a novel tasks and the knowledge for old tasks. In experiment, by showing the robustness of weight changes with respect to the sequential learning, using knowledge base can effectively resolve the stability and plasticity dilemma in continual learning\n\n- The proposed method outperforms other baselines with much fewer parameters than others.\n\nCons\n- In my opinion, I wonder MARK achieves the state-of-the-art performance. It would be great to compare MARK with a recently proposed method CTN[Ref 1] which also achieves great performance in CIFAR and mini-ImageNet dataset in task-incremental learning setting. \n\n- Though authors show that the deviation of weights in KB is much smaller than baseline, it maybe effective only in similar task distributions because there is no clear components  for reducing the forgetting issue in KB. Therefore, it would be great to show the effectiveness of KB in other continual learning scenario which consists of different distribution of tasks (e.g. sequence of 8 tasks proposed in HAT[7]) by comparing the results of MARK with other baselines (e.g. HAT)\n\nQuestions\n- Does MARK can applicable to RL? If it does, it could give a great impact on continual learning in RL.\n\n- If KB continuously updates its representation, is there any possibilities for mismatch between the current KB and the mask function for previous tasks?\n\n[Ref 1] \"Contextual Transformation Networks For Online Continual Learning\", Pham et al., ICLR, 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48 hours"}, "extra_text": {}}, {"review_id": "GaOUj6rVaW1", "reviewer": "Reviewer_phLM", "summary": "This paper investigates how to dynamically select reusable knowledge from past experience via meta learning and trainable masks.\nThe main contribution is the proposition of a meta learning mechanism for enriching the sharing knowledge and trainable masks for selecting reusable parameters.\nThe experiments on 20-Split CIFAR-100 and 20-Split MiniImageNet have been conducted to validate the performances of the proposed method.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n  This paper proposes a new method to increase sharable parameters between related tasks via a meta learning mechanism and to dynamically select reusable parameters using trainable masks.\n  Relationship to previous works has been explained and relevant literature has been cited appropriately.\n\nQuality:\n  The work is relevant to audience at Neurips and provides reasonable details for reproducibility.\n  The authors have conducted ablation study to evaluate the effectiveness of the proposed method, and have done experiments on CL benchmarks to compare the performances between the proposed method and the best performing methods.\n\nClarity:\n The paper is overall clearly written and the method is adequately described.\nIt provides reasonable details for reproducibility.\n\nSignificance:\n The proposed method outperforms the SOTA CL methods in terms of average accuracy by over 10%.\n\n[After Rebuttal]\nAfter reading other reviewers' comments and the authors responses, I'd like to keep my rating 6 because the authors have addressed an important question in CL and most reviewers' concerns in their responses. They also provide enough details/codes for readers to reproduce the results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hr"}, "extra_text": {}}, {"review_id": "9hj1A-WewZf", "reviewer": "Reviewer_1x2r", "summary": "This paper proposes a mechanism for continual learning under discrete tasks with known task boundaries. The method is based on learning a shared feature extractor, called Knowledge Base (KB) in the paper. Given the KB, task adaptation occurs by training a mask over the KB’s activations and a final linear classifier for that task. As far as I know, the proposed method is novel and empirical results on standard benchmarks are encouraging.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper could do with a little polish, overall it is clearly written and the high-level idea of the method is easy grasp. While the idea of using parameters as a so-called knowledge base has been explored in the past, for instance in the HAT paper as the authors note, I find the method proposed in this paper interesting and the results are convincing.\n\nIn particular, given current parameters of a feature extractor (the KB), the authors first tune a task-specific mask and final classifier on the task, with the rationale of forcing the classifier to use existing knowledge. They then tune the KB, holding the mask and final classifier fixed, by a meta-learning strategy; the idea being that this lets the KB absorb new information that generalises across tasks. Finally, they perform some additional tuning of the mask and classifier to take advantage of the new information. \n\nInterestingly, rather than simply adapting the KB to the task, the authors construct a meta-learning problem out of the current task, by randomly sampling classes and observations into query and support sets. The KB is then trained with a variant of Reptile that, as opposed to taking a simple average, weighs the gradient contribution of each mini-task by their final accuracy. On paper, I wouldn’t expect this to work, because the mini-tasks are all constructed from the same task. However, the authors show in an ablation that this meta-learning procedure really does help prevent forgetting. \n\nDetailed comments and questions:\n\nL 49: all continual learning methods that share parameters store shared knowledge in their parameters, claiming this as a novel contribution is a bit strong.\n\n\nL 61: you mention that meta-learning approach learns discriminative features, such as claws and fur. This is a strong claim that needs to be verified. Does it really learn distinct sub-concepts?\n\n\nYou add a task index to the problem description, but I do not believe that the task ID is actually used by the method?  \n\n\nL 97: I’m not sure training on mini-tasks from one task necessarily ‘bias the learner to improve its ability to generalize’. If the task is very specific (say, MNIST images), so will all mini-tasks be, and there is no guarantee that they will actually generalize to new tasks.\n\n\nL 109: it would be helpful to say up front what the encoder F is, rather than leaving the reading wondering for 3 pages.\n\n\nL 163: a mask typically means something that takes on values 0,1. A more common choice would have been a sigmoid activation, or parameterising a Bernoulli distribution. I think this choice need stronger motivation, and you should also qualify that what you mean by mask isn’t quite the typical definition.\n\n\nEq. 1, if the mask functions is a linear map from the feature space of a convnet to the all hidden activations in the KB, the number of parameters in the mask should be very large. It would be nice to see a comment on the size of the masking function.\n\n\nL 182. The update you use is essentially the same as [24], which should be cited.\n\n\nEq. 3. This seems rather unmotivated. Could you provide an ablation to show that weighting the batches makes a difference? One thing to note is that this biases the meta-update towards *easier* tasks, which may not be what you want - easier tasks are things the KB already knows. Hard tasks are things it needs to learn.\n\n\nL 239. What does the ‘amount of weight updates’ mean? The norm of the parameter delta? Or the number of non-zero gradient elements? Or something else?\n\n\nFigure 3. It seems odd to me that the baseline is at almost 100% on task 1, when the KB isn’t. Since the KB is trained in the standard fashion on task 1, shouldn’t it be equal to the baseline on task 1?\n\n\nAblations: I would move the baseline+mask ablation to the main text to have a complete ablation analysis.\n\nYou're missing a discussion of the limitations of this method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No discussion, this needs to be added. For instance, here are a few question I have: What would break the KB? For instance, what if tasks are more distinct that what is uses in the experiments? My guess is that the meta-learning strategy would break down, since the mini-tasks would become too specific. Are there ways to prevent that? Did you consider adding task replay to the meta-update step?", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After viewing the review and rebuttal and skimming through the paper I agree that the paper proposes a few components that are at least quite interesting, and that I feel the community can build on easily moving forward. 

One of the main arguments against the work is that it assumes similarity between tasks. I think partially the answer to this is through the benchmarks considered (Cifar-100 and mini-Imagenet). I find these benchmarks as valid benchmarks (proposed in other works) so the implicit similarity between tasks is the one exploited by previously existing works as well. 

What I do agree with though is that while this assumption is crucial for the method, we do not have a good understanding of it (and the paper does not provide one or build ablations towards one). E.g. how similar things needs to be for this to work (or how dissimilar). Explicitly trying to find that task that would break this assumption would greatly improve our understanding on this assumption (at least in an empirical way). One could of course compound current method with a more traditional CL method to add that additional guaranty against forgetting in KB or deal with scenarios where the distributions become less and less related. 

However I find this applications of ideas from meta-learning directly in a continual learning setting interesting, and in particular I find the results quite surprising so I'm thinking the community will find them too.  

Unfortunately after thinking carefully about this work, I do not think it can be accepted in its current format. But I think is a very relevant and interesting work to the community. And I urge the authors to resubmit a new version where you add some more ablation or explicitly try to understand when the method breaks, what part of the similarity between tasks you explore, how do the learned representation change. It definitely is a borderline paper in my view, with the main worry being that the current version mostly harms the impact the paper can have (if for e.g. we could understand better what is going on). I think an improved version of this would make for a really interesting paper. 
Decision: Accept (Poster)