Optimizing Reusable Knowledge for Continual Learning via Metalearning

Abstract: When learning tasks over time, artificial neural networks suffer from a problem known as Catastrophic Forgetting (CF). This happens when the weights of a network are overwritten during the training of a new task causing forgetting of old information. To address this issue, we propose MetA Reusable Knowledge or MARK, a new method that fosters weight reusability instead of overwriting when learning a new task. Specifically, MARK keeps a set of shared weights among tasks. We envision these shared weights as a common Knowledge Base (KB) that is not only used to learn new tasks, but also enriched with new knowledge as the model learns new tasks. Key components behind MARK are two-fold. On the one hand, a metalearning approach provides the key mechanism to incrementally enrich the KB with new knowledge and to foster weight reusability among tasks. On the other hand, a set of trainable masks provides the key mechanism to selectively choose from the KB relevant weights to solve each task. By using MARK, we achieve state of the art results in several popular benchmarks, surpassing the best performing methods in terms of average accuracy by over 10% on the 20-Split-MiniImageNet dataset, while achieving almost zero forgetfulness using 55% of the number of parameters. Furthermore, an ablation study provides evidence that, indeed, MARK is learning reusable knowledge that is selectively used by each task.


Reviews and author responses:

 We appreciate the reviewer's participation in the discussion process.

We want to clarify that we did resolve most of the comments from the reviewer. We added extra experiments due to the concern of comparing ourselves with more recent methods (GPM: 76.67% (std 3.1%) BWT: -0.42%, Experience Replay: 65.9% (std 1.24%) BWT: -16.9%, SupSup: 77.58% (std 1.3%) BWT: 0), and answer both questions. 

In terms of the specific concern about the assumption that tasks share a similar domain, as we state in the rebuttal, in our work we follow the common practice in continual learning, testing our method in common benchmarks (datasets coming from a similar domain). However, we agree with the reviewer that it will be important to run experiments beyond this setup, we did not have the time to do so during the rebuttal period, but we will include this in the final version of our paper.

This paper investigates how to dynamically select reusable knowledge from past experience via meta learning and trainable masks.
The main contribution is the proposition of a meta learning mechanism for enriching the sharing knowledge and trainable masks for selecting reusable parameters.
The experiments on 20-Split CIFAR-100 and 20-Split MiniImageNet have been conducted to validate the performances of the proposed method.

  Originality:
  This paper proposes a new method to increase sharable parameters between related tasks via a meta learning mechanism and to dynamically select reusable parameters using trainable masks.
  Relationship to previous works has been explained and relevant literature has been cited appropriately.

Quality:
  The work is relevant to audience at Neurips and provides reasonable details for reproducibility.
  The authors have conducted ablation study to evaluate the effectiveness of the proposed method, and have done experiments on CL benchmarks to compare the performances between the proposed method and the best performing methods.

Clarity:
 The paper is overall clearly written and the method is adequately described.
It provides reasonable details for reproducibility.

Significance:
 The proposed method outperforms the SOTA CL methods in terms of average accuracy by over 10%.

[After Rebuttal]
After reading other reviewers' comments and the authors responses, I'd like to keep my rating 6 because the authors have addressed an important question in CL and most reviewers' concerns in their responses. They also provide enough details/codes for readers to reproduce the results. Yes

 After reading other reviewers' comments and the authors responses, I'd like to keep my rating 5 because the authors did not address my question before, and the algorithm has strong assumption that tasks share a similar domain.

This paper mainly solves the catastrophic forgetting problem in task-based continual learning. In particular, this paper proposes a new meta-learning based method, MARK, which uses knowledge base to store some useful knowledge and uses meta-learning to update Knowledge Base. In addition, the knowledge of KB is selected through a set of masks. Finally, the experimental results of this paper show that MARK better than existing methods on two datasets. The main innovation of this work is to use meta-learning to update the knowledge base to simulate human learning methods.  The idea of using meta-learning to update the knowledge base and combining with mask for feature selection is not novel enough. The main innovation of this paper is to propose "meta-learning to update the knowledge base", but the method of updating KB only with the data of the new task still has CF for KB. Therefore, I don’t know that the use of meta-learning in this paper can get highly general features by KB. 

In content: Refer to the question in Originality, hoping to get a clearer explanation. An additional suggestion is that if you use part of the data of the previous task (caching or generation) when updating the knowledge warehouse, will you achieve better results (e.g., use the replay based method)? 
Experimentally: 
1. Please show the experimental results of GPM on CIFA-100? As far as I know, the GPM code is very easy to run CIFAR-100. 
2. Because of meta-learning, how different is the training time between MARK and baseline? 
3. There are relatively few baseline models for experimental comparison. CONTINUAL LEARNING WITH HYPERNETWORKS. ICLR 2020. Continual Learning by Using Information of Each Class Holistically. AAAI 2021 
Clarity:

 This paper is clearly written. But I suggest that section 2 should be a subsection of section 3.
 The potential negative societal impact should be addressed clearly.

 After reading other reviewers' comments and the authors responses, I'd like to raise my rating to 5 because the authors have addressed my questions before, while the novelty of the proposed method is not great enough for a positive score.

 After reading other reviewers' comments and the authors responses, I'd like to keep my rating 6 because the authors have addressed an important question in CL and most reviewers' concerns in their responses.

 We appreciate the comments, below our answers:

**Main Review:**
1. We are adding extra baselines such as SupSup[1] and Experience Replay, as alternatives. Since these are experiments that we have been doing after the deadline of the conference and we have them at hand. We are sorry for not being able to run experiments in the works you suggested, but we hope that the former can supply the need for more baselines.

2. The idea that tasks share common structure is a fundamental notion for not only our method, but for any method that expects to reuse or share information among tasks. We explicitly mention this in Section 2: *“Furthermore, there is no intersection between the classes of different tasks, but they share a similar domain. This domain intersection is key to take advantage of common patterns among the tasks.”*. We agree that if this is not the case, such methods -including our own- will not perform adequately. 

**Answers to Questions:**

1. We see no impediment on using MARK for RL, however, none of us are experts in this area, and cannot anticipate the practical hurdles that need to be overcome for its use there.

2. While this could potentially happen, we think the forcing of feature reuse during training alleviates this problem as relevant past features remain relevant during the whole of training.

[1] Wortsman, Mitchell, et al. "Supermasks in Superposition." Advances in Neural Information Processing Systems 33 (2020)

 We appreciates the comments and questions of the reviewer.

**Main Review:**

We agree with the reviewer that our method has a close relation with previous works to face continual learning scenarios. As an example, it has a similarity with OML (ref [43]) in the sense that it includes a dedicated mechanism to enrich a common KB among the tasks. It also has a similarity with ANML (ref [44]) in the sense that it includes a dedicated mechanism to modulate the use of features selected from a second network. In contrast to our work, previous methods rely on different mechanisms to implement and integrate these previous ideas.  In this context, our main contribution is to propose a novel method to integrate these mechanisms in a synergistic way: an initial “KB-querying step” attempts to solve the current task using previous knowledge from the KB; then a “KB-updating step” exploits the new task to enrich the KB; finally, a second “KB-querying step” uses the updated KB to solve the current task. As far as we know, these are novel ideas, and we support their relevance with strong results on common benchmarks. In particular, the ablation analysis in Fig. 5 shows that the synergistic combination of the previous steps is key to improve performance.

In terms of how our method fosters the learning of general features by the KB and how it can mitigate CF, we believe that this is due to the synergistic combination of the metalearning and mask-modulation mechanisms. Specifically, there are two main reasons that can explain this effect: i) As shown in previous works, the use of an episodic training strategy fosters weight reusability. In other words, training based on an inner and outer loops that operate over mini-tasks, biases the learner to improve its ability to generalize to further tasks, fostering the learning of more generic rather than task-specific features; ii) During metalearning, the feature selector masks are kept fixed, therefore, only the KB features that are relevant to solve the current task are updated, enriching the current KB while avoiding CF. We believe that this is a key mechanism to mitigate CF because gradient updates are performed only where they really matter, helping to preserve previous knowledge. If an activation map is irrelevant for a certain task, then the value of the corresponding mask will be zero, making the gradient update associated to that activation being zero as well.

We agree that using data from previous tasks is an interesting idea to improve performance, especially for the application of the metalearning step. However, our current research focuses on a scenario where we don’t have access to previous data (originally we were focused on data privacy concerns). 

**Experimentally**:

1. We run GPM on CIFAR-100 with 20 tasks and have added the results to our paper. Results are: CIFAR100: 76.67% accuracy. BWT: -0.42%.

2. For CIFAR100: Baseline: 10 minutes, MARK: 28.5 minutes.

3. We are adding extra baselines such as SupSup[1] and Experience Replay, as alternatives. Since these are experiments that we have been doing after the deadline of the conference and we have them at hand. We are sorry for not being able to run experiments in the works you suggested, but we hope that the former can supply the need for more baselines. The results obtained in CIFAR100 are: Experience Replay: 65.9% (std 1.24%) BWT: -16.9%, SupSup:  77.58% (std 1.3%) BWT: 0

*This paper is clearly written. But I suggest that section 2 should be a subsection of section 3.*

Thanks, we will consider this suggestion for the final version of our work. 

**Limitations**:

We will add the limitations sections, explaining in detail the limitations raised by the reviewers

[1] Wortsman, Mitchell, et al. "Supermasks in Superposition." Advances in Neural Information Processing Systems 33 (2020).


 We appreciate the hard work made by the reviewer. Very detailed and clear.

**Main Review:**

1. (L49) We agree that all methods that share parameters share knowledge, however, not all such methods encourage and allow this knowledge to be reused for new tasks. Our focus is on  fostering reusability unlike other methods.

2. (L61) 
    - Our intention was not for this phrase to be taken literally but we include this example as a way to explain the main intuitions behind the construction of our model. Therefore, we will revise this text indicating this comment.
    - The task ID is not used explicitly but it is needed to select the appropriate classifiers, feature extractors, and mask functions during training and inference.

3. (L97) We agree with the reviewer that if a task is very specific, our proposal will not perform adequately. The idea that tasks share common structure is a fundamental notion not only for our method, but for any method that expects to reuse or share information among tasks. As we mention in Section 2: “*there is no intersection between the classes of different tasks, but they share a similar domain. This domain intersection is key to take advantage of common patterns among the tasks*”. We agree that if this is not the case, continual learning  methods -including our own- will not perform adequately.
However, as shown in previous works, the use of an episodic training strategy fosters weight reusability. In other words, training based on an inner and outer loops that operate over mini-tasks, biases the learner to improve its ability to generalize to further tasks, fostering the learning of more generic rather than task-specific features. This impacts not only future but also previous tasks 

4. (L109) We are sorry if the first pages of our manuscript are confusing about the definition of F_t, we will fix this.

5. (L163) We understand and agree that it is an issue that can causes confusion due to how the concept of mask has been used in the past. In early versions of our work, we started using sigmoid activations to create the masks, but results were worse (around 8% reduction in accuracy). Trying to understand this effect, following works such as Squeeze-and-Excitation Networks (ref [50]), we test alternative activation functions. This leads us to the masks generated using ReLU. As a relevant insight, we notice that the output of the ReLU function to mask each visual channel  is usually in the [0,1] range, which is an interesting issue that we plan to study as part of our future work

6. (Eq. 1) The generated mask does not mask each activation individually but it rather masks complete channels in the KB. For example, if there is a convolutional layer of output size CxHxW, our masking function would create a C-dimensional vector, not a CxHxW vector. Therefore the number of parameters of each mask is not large at all. This is similar to the operation of the method proposed in [50].

7. (L182) The paragraph before this line states *“This procedure consists of an adaptation of the training process presented in [24].”*

8. (Eq. 3) We agree with you that there might be a bias for easier tasks. However,  as we only run one model per mini task we are unable to distinguish between a bad model or a hard task. Thus, we chose to bias our solution to models that work well on their given tasks. There is an underlying assumption on our part that we expect mini tasks to be of roughly the same difficulty. We did try to use different weighting mechanisms: in particular, we tried using the same weight for each task and using the opposite of gamma, i.e., (1 - gamma). For equal weighting, we find that the solution achieves 77.78% BWT: -0.37% . By using (1-gamma) we find the solution achieves 77.8% BWT:-0.09%. So we find that weighting is slightly better using our current method, but it does not have a significant impact over the overall result. 

9. (L239) This explanation was meant to be in the appendix. We will gladly explain: For every task we compare the model before training on task t and after training. We compare the mean of the absolute difference between the “before” and “after” model against a threshold that is dependent on the standard deviation of the weights of the “before” model. If it’s above this threshold we assume the weight was updated, else we don’t.

10. (Figure 3) Yes, we can explain this. As an example, for task 1, the KB is fully trained in the standard fashion. Let’s call this model A. For MARK, we run our KB querying and KB update steps for task 1 as well, starting from model A. This leads to model B. For our baseline, starting from model A we train again on task 1 and obtain model C. The percentages reflected in the figure are: Model B vs A (MARK) and model C vs A (baseline).

**Limitations And Societal Impact:**

We agree completely. We will add a Limitations section to our paper, which includes this. Regarding the specific limitations listed here. The idea that tasks share common structure is a fundamental notion for not only our method, but for any method that expects to reuse or share information among tasks. We explicitly mention this in Section 2: *“Furthermore, there is no intersection between the classes of different tasks, but they share a similar domain. This domain intersection is key to take advantage of common patterns among the tasks.”*. We agree that if this is not the case, such methods -including our own- will not perform adequately. We will add this to the Limitations. Regarding using some sort of replay, our investigation was focused on a scenario where we don’t have access to previous data (originally we were focused on data privacy concerns), so it would seem counterproductive to go in this direction. However, we understand other researchers might want to pursue this in a different direction and we see this as a definite reasonable extension to our work. 


 We welcome positive feedback regarding our work. We invite the reviewers to review the questions and comments of the other reviewers if they clarify any unwritten doubts about our method.

This paper proposes a mechanism for continual learning under discrete tasks with known task boundaries. The method is based on learning a shared feature extractor, called Knowledge Base (KB) in the paper. Given the KB, task adaptation occurs by training a mask over the KB’s activations and a final linear classifier for that task. As far as I know, the proposed method is novel and empirical results on standard benchmarks are encouraging.  While the paper could do with a little polish, overall it is clearly written and the high-level idea of the method is easy grasp. While the idea of using parameters as a so-called knowledge base has been explored in the past, for instance in the HAT paper as the authors note, I find the method proposed in this paper interesting and the results are convincing.

In particular, given current parameters of a feature extractor (the KB), the authors first tune a task-specific mask and final classifier on the task, with the rationale of forcing the classifier to use existing knowledge. They then tune the KB, holding the mask and final classifier fixed, by a meta-learning strategy; the idea being that this lets the KB absorb new information that generalises across tasks. Finally, they perform some additional tuning of the mask and classifier to take advantage of the new information. 

Interestingly, rather than simply adapting the KB to the task, the authors construct a meta-learning problem out of the current task, by randomly sampling classes and observations into query and support sets. The KB is then trained with a variant of Reptile that, as opposed to taking a simple average, weighs the gradient contribution of each mini-task by their final accuracy. On paper, I wouldn’t expect this to work, because the mini-tasks are all constructed from the same task. However, the authors show in an ablation that this meta-learning procedure really does help prevent forgetting. 

Detailed comments and questions:

L 49: all continual learning methods that share parameters store shared knowledge in their parameters, claiming this as a novel contribution is a bit strong.


L 61: you mention that meta-learning approach learns discriminative features, such as claws and fur. This is a strong claim that needs to be verified. Does it really learn distinct sub-concepts?


You add a task index to the problem description, but I do not believe that the task ID is actually used by the method?  


L 97: I’m not sure training on mini-tasks from one task necessarily ‘bias the learner to improve its ability to generalize’. If the task is very specific (say, MNIST images), so will all mini-tasks be, and there is no guarantee that they will actually generalize to new tasks.


L 109: it would be helpful to say up front what the encoder F is, rather than leaving the reading wondering for 3 pages.


L 163: a mask typically means something that takes on values 0,1. A more common choice would have been a sigmoid activation, or parameterising a Bernoulli distribution. I think this choice need stronger motivation, and you should also qualify that what you mean by mask isn’t quite the typical definition.


Eq. 1, if the mask functions is a linear map from the feature space of a convnet to the all hidden activations in the KB, the number of parameters in the mask should be very large. It would be nice to see a comment on the size of the masking function.


L 182. The update you use is essentially the same as [24], which should be cited.


Eq. 3. This seems rather unmotivated. Could you provide an ablation to show that weighting the batches makes a difference? One thing to note is that this biases the meta-update towards *easier* tasks, which may not be what you want - easier tasks are things the KB already knows. Hard tasks are things it needs to learn.


L 239. What does the ‘amount of weight updates’ mean? The norm of the parameter delta? Or the number of non-zero gradient elements? Or something else?


Figure 3. It seems odd to me that the baseline is at almost 100% on task 1, when the KB isn’t. Since the KB is trained in the standard fashion on task 1, shouldn’t it be equal to the baseline on task 1?


Ablations: I would move the baseline+mask ablation to the main text to have a complete ablation analysis.

You're missing a discussion of the limitations of this method. No discussion, this needs to be added. For instance, here are a few question I have: What would break the KB? For instance, what if tasks are more distinct that what is uses in the experiments? My guess is that the meta-learning strategy would break down, since the mini-tasks would become too specific. Are there ways to prevent that? Did you consider adding task replay to the meta-update step?

This paper introduces a novel method MARK which utilizes the knowledge base (KB) that can facilitate learning a novel task and reducing the forgetting issue in continual learning and a mask function that selects proper reusable features in KB. The learning procedure consists of four steps: 1) Obtain feature vectors to train the mask function, 2) train a mask function and classifier to learn a novel task, 3) update KB using meta-learning, 4) fine-tune the mask function and classifier to utilize the information of updated KB. When updating the KB, the dataset for a novel task separated to many subtasks, and using the adaptation technique proposed in [24], KB meta-learns the features that can generalize to different tasks. In experiments, the proposed methods achieves the state-of-the-art accuracy in CIFAR and mini-ImageNet dataset. By ablating the main components (KB and mask functions), authors also show the importance of each component.  Pros

- This paper proposes a novel method that utilizes the knowledge base which contains well adaptable features to a novel tasks and the knowledge for old tasks. In experiment, by showing the robustness of weight changes with respect to the sequential learning, using knowledge base can effectively resolve the stability and plasticity dilemma in continual learning

- The proposed method outperforms other baselines with much fewer parameters than others.

Cons
- In my opinion, I wonder MARK achieves the state-of-the-art performance. It would be great to compare MARK with a recently proposed method CTN[Ref 1] which also achieves great performance in CIFAR and mini-ImageNet dataset in task-incremental learning setting. 

- Though authors show that the deviation of weights in KB is much smaller than baseline, it maybe effective only in similar task distributions because there is no clear components  for reducing the forgetting issue in KB. Therefore, it would be great to show the effectiveness of KB in other continual learning scenario which consists of different distribution of tasks (e.g. sequence of 8 tasks proposed in HAT[7]) by comparing the results of MARK with other baselines (e.g. HAT)

Questions
- Does MARK can applicable to RL? If it does, it could give a great impact on continual learning in RL.

- If KB continuously updates its representation, is there any possibilities for mismatch between the current KB and the mask function for previous tasks?

[Ref 1] "Contextual Transformation Networks For Online Continual Learning", Pham et al., ICLR, 2021 None

Meta-review: After viewing the review and rebuttal and skimming through the paper I agree that the paper proposes a few components that are at least quite interesting, and that I feel the community can build on easily moving forward. 

One of the main arguments against the work is that it assumes similarity between tasks. I think partially the answer to this is through the benchmarks considered (Cifar-100 and mini-Imagenet). I find these benchmarks as valid benchmarks (proposed in other works) so the implicit similarity between tasks is the one exploited by previously existing works as well. 

What I do agree with though is that while this assumption is crucial for the method, we do not have a good understanding of it (and the paper does not provide one or build ablations towards one). E.g. how similar things needs to be for this to work (or how dissimilar). Explicitly trying to find that task that would break this assumption would greatly improve our understanding on this assumption (at least in an empirical way). One could of course compound current method with a more traditional CL method to add that additional guaranty against forgetting in KB or deal with scenarios where the distributions become less and less related. 

However I find this applications of ideas from meta-learning directly in a continual learning setting interesting, and in particular I find the results quite surprising so I'm thinking the community will find them too.  

Unfortunately after thinking carefully about this work, I do not think it can be accepted in its current format. But I think is a very relevant and interesting work to the community. And I urge the authors to resubmit a new version where you add some more ablation or explicitly try to understand when the method breaks, what part of the similarity between tasks you explore, how do the learned representation change. It definitely is a borderline paper in my view, with the main worry being that the current version mostly harms the impact the paper can have (if for e.g. we could understand better what is going on). I think an improved version of this would make for a really interesting paper. 

Decision: accept