Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper investigates the problem of sequential learning of NLP tasks. Concretely, it proposes a novel method, AFK, to alleviate catastrophic forgetting and enable knowledge transfer across related tasks. The paper argues that most of the existing works solely focus on catastrophic forgetting problems and there is no explicit mechanism to facilitate the transfer across tasks. To address this issue, the paper designs a novel architecture as a part of the AFK method, CBA (Continual learning BERT Adapter). CBA consists of a capsule network to model different tasks and a transfer routing algorithm to support knowledge transfer across tasks. For experimentation, the paper considers three text classification datasets - Document Sentiment Classification (10 tasks), Aspect Sentiment Classification (19 tasks), and 20News (10 tasks) and evaluates AFK on task-incremental learning scenarios (separate classification head per task). In comparison with the considered baselines, the paper reports improved overall performance and minimal forgetting for AFK.
main_review: Naive fine-tuning of the BERT model is known to undergo forgetting. To address this issue, several works have been proposed with Adapters being more compact and modular. Building upon this line of work, the paper proposes a continual learning variant of the BERT Adapter (CBA). Instead of a simple feedforward bottleneck layer in classical Adapters, the paper introduces well-motivated layers to CBA: (i) a capsule per task (TK-layer), (ii) a transfer capsule layer (TR-layer) mainly consisting of similarity estimator and task router to control the influence of the previous tasks while learning the current task, (iii) task-specific modulation layer to mask neurons corresponding to the previous layer, thus, minimizing the forgetting. However, based upon the similar motivation to AFK, i.e, using knowledge from the previous tasks while learning the current task, AdapterFusion [1] work proposes to learn a task-specific composition of adapters from previous tasks. The paper completely ignores this relevant line of work and ends up just comparing with a single adapter per task or applying existing CL methods to vanilla adapters.

In section 5.4, the paper compares standalone training (SDL) with NFH (continual learning with no forgetting handled). Based upon this comparison, the paper suggests that ASC and DSC tasks have similarities. However, there is no explicit metric to quantify this similarity (for example, some form of clustering on representations extracted from pre-trained BERT models?). The paper seems to make an implicit assumption that similar tasks undergo less forgetting. Based on the recent controlled study [4], the relationship between task similarity and forgetting is less straightforward. The paper should review this work and reformulate its claims.

The paper uses the term Forward Transfer without any formal definition. Based upon the textual description it seems that the paper wanted to refer to the Learning Accuracy metric which is considered in the literature. Moreover, in the continual learning literature, Forward Transfer refers to the zero-shot evaluation of the model from previous tasks. Please review [2, 3] to be consistent with the existing terminology and restrain from overloading existing terms to avoid confusion.

The paper motivates that there is a need to explicitly model knowledge transfer between tasks. However, it is unclear from Table 2 whether there is any positive backward transfer happening. At least AdapterFusion guarantees that there would be no forgetting (refer [2] for the definition of the backward transfer). Therefore, it is important to lay down a clear distinction with the existing work and explicitly focus on analysis to validate the proposition being in the paper.

Overall, the paper attempts to solve an important problem of continual learning in the context of the pre-trained Transformers. With the rise of larger models, it is important to consider the training and inference time of the methods. Given the paper builds upon the parameter-efficient transfer learning, the number of parameters would not grow dramatically with the number of tasks. However, with the use of a capsule network, the proposed approach is expensive and the paper rightly acknowledges it to be the limitation of their work.

[1] Pfeiffer, Jonas, et al. "AdapterFusion: Non-Destructive Task Composition for Transfer Learning." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021. 

[2] Riemer, Matthew, et al. "Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference." International Conference on Learning Representations. 2018. 

[3] Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." Advances in neural information processing systems 30 (2017): 6467-6476.

[4] Ramasesh, Vinay Venkatesh, Ethan Dyer, and Maithra Raghu. "Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics." International Conference on Learning Representations. 2020.

Typos: In Eq (6), should it be $u_{j|t}^{(t)}$ instead of $v_{j|t}^{(t)}$.


limitations_and_societal_impact: Yes, the paper discusses the limitations of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed a new continual learning method AFK, aiming to prevent catastrophic forgetting and exhibit knowledge transfer. Capsule Network is used to share task-specific knowledge in the continual learning process. Experimental results show that AFK outperforms previous SOTA approaches on many different continual learning sentiment classification datasets.
main_review: In this paper, the authors aim to address not only catastrophic forgetting which is intensively explored in previous work but also knowledge transfer.
Using the routing algorithm in capsule network to encourage knowledge transfer is novel and reasonable. This may be the first paper to incorporate capsule network into continual learning (CL).
The results support the effectiveness of the proposed method which outperforms the other CL methods.
The writing of this paper is clear.

Suggestions:
1. In Table 2, it’s better to use bold numbers to highlight the best score in each part. A bunch of tiny digits in this large table is not easy for the reader to focus on the important parts.
2. In continual learning, we usually want to see the visualization of accuracy for each old task, which may go up and down in the whole training process. It allows us to observe the details of catastrophic forgetting/knowledge transfer for each method. If the authors can show the details with some figures, it would be easier for readers to grasp the experiment results.
3. For text classification, there is a strong continual learning method called LAMOL [1]. I suggest the authors also compared AFK with their results.
4. In Table 2, the authors only show non-CL (single-task) results and CL results. However, the multi-task learning model should also be considered as a strong upper bound of CL. It can tell us how well the model can achieve without the continual learning setting (under the same network capacity). I suggest that the authors can include the multi-task results using a model similar to the AFK model.
5. Capsule network and the routing algorithm are the backbone of the proposed method, but they are not shown in the title. I think it is better to use “capsule” in your title (or in the abbreviation AFK) to make it more clear and consistent with your proposed content.

[1] Fan-Keng Sun, Cheng-Hao Ho, Hung-Yi Lee. LAMOL: LAnguage MOdeling for Lifelong Language Learning. ICLR 2020.

limitations_and_societal_impact: The authors have adequately addressed the limitations (e.g. large capsule slow down training). I did not see any potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of continual learning in a popular NLP task, text classification.  It proposes a framework called AFK, that utilizes capsule networks to dynamically route task-specific knowledge among task-shared knowledge. The author claims two major benefits of doing this: (1) this essentially establishes an attention effect that allows the model to more effectively conduct forward transfer, and (2) the assignment of new parameters for new tasks and the masking mechanism can mitigate catastrophic forgetting. Overall, the application of capsule network in continual learning is novel and the framework obtains good performance. However, I feel that more comprehensive analysis could further supported the presentation, and some could be critical for understanding these models (see below).
main_review: Pros:
1. The proposed idea of using capsule network for better knowledge transfer in continual learning is novel.
2. Experimental results do show the efficacy of the method.
3. The paper includes an amazing amount of baselines, which could allow us to compare among them as well.

Cons:
1. Not too much analysis is included for understanding model behaviors. For example, since the method is designed to capture task transferability for better knowledge transfer, can you design some analysis to show this? Do similar tasks enjoy similar metrics? Can we tell which task(s) benefit a specific task? 
2. Experimental settings can be more comprehensive. While a lot of baselines are included, interestingly a naive MTL model is not, which is usually considered as the upper bound for continual learning. In addition, the paper only conduct experiments on text classifications, and thus it is also interesting to explore other types or even mixture of different types of tasks, which should be more challenging and practical.
3. Motivation is not enough. I am not sure if I missed this, but why capsule network in particular for your AFK framework? Based on my understanding, you can also do, for example, a mixture-of-expert structure with adapters? It is of course nothing wrong to explore something new, but could you please give a bit more motivation for this? It would also be great if you can present some comparison with MOE too.

Questions:
1. Memory-replay-based methods are known to be robust to catastrophic forgetting. Do you have any idea why they didn't do well in your settings?
2. [1] also explores comparing forward and backward knowledge transfer. In particular, they found negative transfer to the latest task seen. Do you have similar observations?

Reference:
[1] Efficient Meta Lifelong-Learning with Limited Memory. Wang et al., EMNLP 2020

Some small suggestions:
1. Please highlight some results in Table 2. It is quite painful to search for numbers in a giant table like this.
2. I would recommend using a different name. AKF ((Achieving Forgetting avoidance and Knowledge transfer in continual learning) is too broad and doesn't sound like a method name to me. 




limitations_and_societal_impact: None.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies the continual learning of a sequence of NLP tasks. A novel model called AFK is proposed, which contains a series of hand-crafted modules to prevent forgetting and help with knowledge transfer. A number of baselines are compared with, and strong CL performance are shown in the experiments.
main_review: I'm mainly voting to accept this paper because the model architecture is novel, and experiment results are strong. However, I admit I'm not an expert in the CL field, so my judgement is not confident. 

My major problem with the paper is the writing: especially section 4. The architecture seems overly complicated to me, I don't even totally understand them after I read them twice. I hope the authors can add more intuition or high-level descriptions for the model (like what's the reason behind this design?). It would also be good to show some examples about how the modules work in practice. (Figure 2B is a good example)

Questions:

Re "Each capsule in the TK-Layer represents a task, and it prepares the low-level features derived from each task", does that mean the capsule for the first will be frozen when training for other tasks?

Is the introduction of capsules necessary in the model design ? It seems to me most of the modules are standard NN modules, and the system is trained end-to-end. Could you give more intuition about how capsules work, and how they are different from other modules ? (I'm not very familiar with capsules.) For example, can I just understand the capsule as a MLP? If that's the case why you call it capsule ?

About presentation:
I think it would be helpful to provide something like average accuracy in Table to help readers. Also, it would be helpful to show how other methods "forgets" about previous tasks, while AFK can remember them in the trajectory of learning.

Citations:
This paper talks about knowledge transfer https://arxiv.org/abs/1910.07117 , from a digalogue model perspective.

limitations_and_societal_impact: The authors discuss how the architecture could slow down training briefly in the conclusion. It would be helpful to give more details.
review_title: null

Meta-review: The paper studies continually learning of a sequence of natural language processing (NLP) tasks. The authors claim that state of the art approaches that facilitate knowledge transfer (e.g. fine-tuning a BERT-like language model) suffer from serious catastrophic forgetting in the continual learning setting and present a novel model called AFK to achieve knowledge transfer while avoiding catastrophic forgetting in NLP tasks. AFK  makes use of a combination of Capsule networks to model each task, a transfer routing algorithm to identify and transfer knowledge across tasks to improve accuracy and task masks to prevent catastrophic forgetting. The effectiveness of the proposed method is demonstrated in empirical evaluations. 
 
The reviewers raised some points that were satisfactorily addressed in the rebuttal, including additional experiments. The rebuttal helped convince the reviewers of the merits of the paper. 

Decision: Accept (Poster)