Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper is introducing a meta-learning framework into continual learning to promote learning resuable kwowledge from a sequence of tasks. The resuable kwowledge is simply defined by a set of shared model weights. The authors claim these weights as a common knowledege base (KB), which is not only reused but also incrrementally / continually enriched with new knowledge from new tasks, and propose to selectively / adaptively use them by utilzing some masking mechanism.
main_review: - The sharable weights / modules are defined as a knowledge base (KB), but it doesn't really have some properties that the exact KB has, and this could probably be misleading naming. 
- The role of KB is proposed to incrementally store meaningful knowledge into a set of weights and not only aovid catastrophic forgetting but also improve the learning of upcoming new tasks. To make it happend, the KB is updated through meta-learning based on a gradient-based method. (There were several works that similarly utilize a meta-learning framework to improve continual learning.) 
- The ablation study shows that meta-learning reduces the forgetting problem. However, it is unclear why this meta-learning promotes to incrementally update the KB without catastrophic forgetting. The meta-learing objective is defined by mini-tasks that are sampled only from the current task, not caring about previous tasks. 
- Moreover, the KB seems not to improve the learning of new tasks (i.e. accuracy), but act as a regularizer to avoid catastrophic forgetting. Because, the ablation study shows that the masking mechanism, which is highly related to feature modulation methods (i.e FiLM, DLN, CBN, SENet), is higly improving the accruacy.
- In terms of network architecutre, $F^t$, $M^t$ and $C^t$ are already a single classifier for task $t$. For example, if we asssume KB is very small or almost constant regardless of input $x$, then the mask variables become a representation itself. That way, it seems like the method is actually incrementally increasing the network size with having task labels.
limitations_and_societal_impact: - For the results, it would be good to compare the results with some possible upper bounds such as results from multi-task learning setting. 
- for line 142, the naming is same as line 136, so it would be good to have another name for it.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a continual learning method that uses a shared network (knowledge base) to learn combined knowledge across multiple tasks. This shared network is trained using meta-learning, a version similar to Reptile, on the dataset of the current task. In addition to the knowledge base, the paper proposed parameterized masking operators for each task, that mask out the irrelevant features in the knowledge base for a given task. The overall training proceeds in three steps. In step 1, the knowledge base is kept fixed, and the masking network and the classifier for the given task is trained. In step 2, for the trained masking network and the classifier, the knowledge base is updated using a meta-learning approach. Finally, in step 3, the masks and classifiers are finetuned using the updated knowledge base. Experiments are reported on Split-CIFAR 100 and miniImageNet dataset, where the proposed method is shown to perform better than the reported baselines. Some ablation of the method is also performed where it is shown that both the mate-learning and masking contribute to the overall better performance of the algorithm. 

main_review: Positives:

- The paper is written well and easy to understand. 
- The proposed approach strongly outperforms the baseline. 
- Most of the ablations are done nicely and gives a better picture of why the proposed method is working. 	

Negatives/ Questions:

- Motivation for meta-learning: The motivation of using meta-learning is not clear to me. The way it has been used in this work, meta-learning is only performed at a given task-level (by splitting the task into tiny subtasks). In my opinion, the assumption that meta-learning would help different tasks exploit the common structure would only work if the training involves all those tasks jointly. As per my understanding of the paper, the authors perform meta-learning incrementally. The reason they still see the strong results could be that, in the benchmarks they considered, different tasks are not really different (different cifar-100 classes etc.). In a setup, where tasks differ the proposed method might see the forgetting problem in their knowledge base. 

- Too many Task-specific parts in the approach : Currently the authors assume a task-specific feature extractor, mask and the classifier head. I wonder whether the method would even need a knowledge base. Perhaps two kinds of ablations can be performed here; 1) combine the feature extractor, masking network and the classifier for each task, train it end-to-end and report numbers, 2) add the knowledge-base, as the authors currently have, but keep it random (i.e) don’t train it, meta- or otherwise, and see how the network performs. It is difficult to discern from section 4.2.3 whether the reported ablations do any of the above. I see that the appendix makes an effort to understand a similar phenomenon but both the ‘feature’ and ‘no-retraining’ baselines differ slightly to what I am describing here. 

- Architecture details: Please add the forward pointer to the appendix for the architecture details of all the components (knowledge-base, mask generator, feature extractor etc). 

- Section 2:  Section is sparse in the details of the setup. Please add information regarding whether the authors assume an online access to the data of each task (single-pass setup). At test time what are the constraints, do the authors assume an identifier for the corresponding task. 

- Baseline comparisons: Some of the methods, AGEM for example, might not be directly comparable as it assumes a single-pass through the data. 

- Why baseline + MetaLearning would decrease forgetting: Do the authors have any intuition on why just the vanilla meta-learning on top of baseline would reduce forgetting, especially the way it is done (in a task-specific manner, see my comments above)? I would expect masking to be a better inhibitor of forgetting. 

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new architecture for task-based continual learning. It comprises two main parts : 
1) a Knowledge Base (equivalent to a regular CNN) and 
2) a Mask function with task-specific weights, which generates a per-example set of masks to apply on the weights of the knowledge base.  
With the masked weights and the task-specific head a prediction is then obtained. 

To train the model (for tasks > 1), the authors propose to alternate between training mask function keeping the KB fixed, then updating the KB with a fixed mask function. For the KB update, the authors develop a Reptile-like algorithm where artificial "tasks" are created by performing multiple updates on data coming from a subset of the classes in the current tasks. 

The authors evaluate their method in the multihead setting on split CIFAR100 and MiniImagenet datasets. They further provide two ablations to evaluate how their contribution affects training speed and how many weights are changed. 
main_review: __Originality__:  This works presents a novel combination of well known techniques (masking for CL, reptile-like updates for CL). It is my opinion that some similar work should be cited for this submission 

[1] Wortsman, Mitchell, et al. "Supermasks in superposition." NeurIPS 2020. The authors learn a set of task-specific binary masks over a fixed set of weights.  The authors also look at similar datasets, I strongly recommend that the authors add this baseline to their paper. 

__Quality__:  The method presented is sound. __My main criticism of the paper lies in the experimental section__. For all (3) baselines considered, the authors took numbers from the respective papers. This is not problematic in itself, _provided all papers use the exact same training and evaluation protocol_. This is not case the case here, as for example AGEM works in the single epoch setting (resulting in ~2% of the compute used for the author's 50 epoch setting). Moreover, no information is given as to how hyperparameter selection was performed (was a similar compute budget given to each method ?) Overall the empirical section does not meet the bar for this conference. 

For me to accept this paper, the authors should 
-> Include more baselines (at the very least [1]). Expansion based methods (e.g. Progressive Neural networks) also perform very well in the setting explored by the authors, and they have no forgetting by design. A simple replay based method should also be included. 
->  Be very explicit as how the hyperparameter selection for different methods was performed. 


__Clarity__: Overall the paper is well written and easy to follow. The figures are well made as well.  

__Significance__:  The setting explored by the authors is very relevant (even if only two datasets are explored)


**edit** : after having clarified some of my concerns, I have increased my score by one point

limitations_and_societal_impact: Yes / ok
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper develops a method (MARK) for tackling catastrophic forgetting in neural networks that maintains a knowledge base (KB) in the form of a convnet, which is shared across training of all tasks, and which, when combined with dynamic feature masking and a form of meta-learning to update the KB, enables MARK to outperform competing methods when trained on sequential image classification tasks. In more detail, MARK classifies an image as follows: the image is passed through a task-specific feature extractor, producing an embedding which is used to generate masks (via a task-specific MLP) over the feature maps of the KB convnet (through which the image is passed separately), the output of which is then passed to a task-specific classifier. When a new task is encountered, the first step (“KB querying”) updates the task-specific parameters over several epochs, while keeping the KB parameters fixed; the second step updates the KB parameters using the meta-learning algorithm Reptile, where the inner loop updates the KB parameters and the task-specific classifiers over a batch of “mini-tasks”, each of which are created by sampling a subset of images from a subset of classes in the current task. Finally the KB querying task is repeated before moving to the next task. Experiments are conducted on 20-way split cifar100 and split-mini-imagenet, showing that MARK outperforms a number of competing methods in terms of average accuracy and forgetting, while using less memory in total. Experiments are run using different types of feature extractor (random fixed and one pretrained on imagenet), and with ablations of the meta-learning update and the mask-generating features, demonstrating that the former is most important for mitigating forgetting, and the latter for achieving a good maximum task performance.

main_review: The paper introduces a novel method that combines meta-learning and dynamic feature masking to yield strong empirical results in a couple of standard continual learning benchmarks, and thus it would be a useful contribution to the CL community. For this reason, I recommend it for acceptance, but only marginally, predominantly because several design choices and details of the implementation are not clearly motivated, and I was left unsure about why the model works, despite the effort made in the paper to demonstrate the importance of each component with ablations. Below are more detailed comments:
- If one were following a typical meta-learning setup, one would expect the KB querying step to match the inner loop of the meta-learning KB update step, but there are at least two differences: (i) the KB parameters are kept fixed in the KB querying step, but are updated in the inner loop of the KB update, and (ii) the mask-generating features are updated in the KB-querying step but not in the inner loop of the KB update step. Could the authors explain the rationale behind these design choices that seem to differ, as far as I am aware, from the usual meta-learning setup (particularly (i))?
- In the ablation experiments, it is curious to see that, while the mask generating functions have a large impact on maximum performance per task, meta-learning the KB is most important for reducing forgetting. What is the intuition here? One would intuitively expect meta-learning to impact the speed of learning of new tasks (as is shown in the paper), but why should it reduce forgetting?
- The KB update
    - Why are the updates weighted by gamma_k? Does this not encourage more adaptation to tasks that the model is already good at?
    - What if after the first KB Querying update, the model is already good at performing task t - is there a danger of it not getting a signal at all in the KB update?
- In the baselines that ablate the meta-learning update, is the KB update still updated using SGD or is it kept fixed after training on the first task? If it is kept fixed, then an important ablation is missing, which is to train the KB with SGD along with the rest of the parameters; otherwise, we don’t know how important the meta-learning is for the performance of the model.
- The KB is trained fully on first task. What if first task is not that relevant for the subsequent ones? Maybe an ablation with a randomly initialised KB could be useful to understand the importance of this?
- In Figure 4b, is the accuracy gain over the maximum performance achieved in that task, or over the final performance having trained on all tasks? If it’s the latter, to what extent can you attribute this gain to just recovering forgotten knowledge, as opposed to new knowledge accumulated over training of other tasks?
- Algorithm 2 is a bit confusing:
    - S_F is not initialised.
    - C_k is not mentioned here, but in the main text it says that C_t is copied and trained as well in the inner loop.
- Even though it makes use of extra data, it is encouraging to see the MARK-Resnet (where the feature encoder is pretrained on Imagenet) perform so well in a sequential setting.
- Comments on related work:
    - It is mentioned in line 293 that previously “metalearning has not been used to increment a KB as in our model”. However, in [1] and [2], meta-learning is used to update a representation learning network or a neuromodulatory network, both of which can be thought of as a knowledge bases. In [1] the representation learning network is used directly in inference, and in [2] the neuromodulatory network gates the representation learning network, which bears similarities to the dynamic masking in MARK.
    - As well as HAT, which is cited in the paper, two other methods that use masks for continual learning are [3] and [4], which might be worth mentioning.

Minor comments:
- Line 199. The name of the referred method is wrong here: “Hard attention tasks” -> “Hard attention to the task”
- Line 217 and 230. “BTW” -> “BWT”
- Line 230 “specially” -> ”especially”
- Line 264. “both,” -> “both”
- Figure 3 y axes seem to be mislabelled - the range is from 0 to 1 but the axis labelled as “percentage” - should it not be “proportion”, or otherwise the scale should be from 0 to 100?

[1] Javed, Khurram, and Martha White. "Meta-learning representations for continual learning." Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.

[2] Beaulieu, Shawn, et al. "Learning to Continually Learn." ECAI 2020. IOS Press, 2020. 992-1001.

[3] Wortsman, Mitchell, et al. "Supermasks in Superposition." Advances in Neural Information Processing Systems 33 (2020).

[4] Wen, Yeming, Dustin Tran, and Jimmy Ba. "BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning." International Conference on Learning Representations. 2019.

limitations_and_societal_impact: There is not much discussion of the limitations of the paper. It is worth mentioning that, while the method uses less memory than the methods compared to, it does suffer from the fact that the number of parameters increases with each task, which may not be scalable for a large number of tasks, especially since the task-specific feature extractors are not insignificantly small.
There is a section in the appendix detailing the CO2 impact of the experiments.
review_title: null

Meta-review: The paper proposes a continual learning method that maintains a knowledge base (KB) shared across all tasks and utilizes a form of meta learning to update the KB and dynamic feature masking to selectively choose relevant weights for each task from the KB. The three training stages are explained, and empirical results are provided that show competitive performance to alternatives. Additionally, an ablation study is provided showing that meta learning is most important for reducing catastrophic forgetting while the masking mechanism contributes to achieving higher accuracy.

The proposed method is an interesting combination of techniques that end up working synergistically to improve both overall performance and mitigate catastrophic forgetting. While the ablation study helps give an idea about which component more significantly contributes to what performance aspect, the writeup may benefit from more explanation  of the design choices and the intuition we should gain from the ablation study, making it more clear why the method works. The experiments section could be improved by clarifying the evaluation setup and comparing all methods on the same setup. Some of these aspects were discussed in the rebuttal, convincing some reviewers to update their scores. 

Decision: Accept (Poster)