Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of task continual learning wherein new tasks are continuously learned in an incremental fashion. Specifically, the paper aims to achieve two key objectives for a successful continual learning system: preventing catastrophic forgetting of the previous tasks and enabling knowledge transfer to new tasks. To achieve this, the paper proposes the TST approach, Task-continual learning based on Sub-networks and Task similarity. The main idea is to start with a mask pool, then select the relevant mask for the given task with gradient-based importance measure, and fine-tune it with the current task data. With the proper selection of the relevant mask, one can alleviate the interference from unrelated tasks and simultaneously enable backward/ forward transfer to related/ current tasks. The paper conducts an extensive set of experiments on a wide range of NLP tasks - sentence-level classification, token-level classification, and generation and compares the proposed TST approach with relevant baselines. In summary, the paper shows that TST improves the overall accuracy of the system after sequentially training on different tasks, reduces forgetting, and in some cases (ASC and NER) even enables positive backward transfer.  
strength_and_weaknesses: Strengths:

1. The paper attempts to solve an important problem of task continual learning with a specific focus on reducing forgetting and enabling forward transfer objectives. These objectives constitute important desiderata for realistic continual learning systems.

2. The proposed TST approach employs parameter-efficient adapter modules to learn sub-networks for different tasks, enabling their approach to scale to a large number of tasks. Also, the proposed task similarity detection method based on gradients is easy to compute and seems effective.

3. The paper conducts a rigorous set of experiments spanning different NLP tasks and compares their performance with several relevant baselines.

Weakness:

1. Although the paper conducts several experiments, there are some open questions regarding the statistical significance of the results provided in Table 1. Specifically, comparing the results with SupSup, it is unclear whether TST is the clear winner (see SUM, CCD, DRG). As the paper reports, the average results over 5 random task orderings, please consider reporting significant test results. 

2. Apart from alleviating forgetting of the previous task, the paper claims that the TST approach also encourages knowledge transfer. Results presented in Table 1 do not provide any convincing evidence for the same. The paper should consider reporting the learning accuracy as mentioned in [1]. For example, to understand the knowledge transfer from Tasks 1 and 2  while learning Task 3, one should consider reporting average performance for tasks after learning on their datasets. Also, if the paper argues that there is positive backward transfer in that case to compute the forgetting, the drop in the performance should be from the max performance for previous tasks and not just right after Task i. Please see the definitions of forgetting and learning accuracy in Eq(1) in [1] for reference.

References: 

[1] Mehta, Sanket Vaibhav, Darshan Patil, Sarath Chandar, and Emma Strubell. "An empirical investigation of the role of pre-training in lifelong learning." arXiv preprint arXiv:2112.09153 (2021).

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Moreover, the proposed solution is grounded in recent works on parameter-efficient transfer learning-based approaches. Also, the proposed mask importance measure is built on existing pruning literature and is successfully applied to the continual learning problem. The paper also provides its code thereby convincing the reader of the reproducibility of their experiments. 

Typos in Eq(4) and Eq(5): should it be $\nabla_{g_{(k)}}^n$?

summary_of_the_review: Overall, I rate this paper marginally below the acceptance threshold. It is interesting to see the efficacy of the proposed approach for task continual learning. However, with the currently reported numbers, it is unclear how significant the improvements are over considered baselines. The addition of the statistical significance test results would be more convincing and I would be happy to increase my score during the discussion phase.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work introduces a new continual learning method TST.
TST considers both preventing catastrophic forgetting (CF) and encouraging knowledge transfer (KT).
To prevent CF, TST directly use one existing CL method called SupSup.
To encourage KT, TST borrows the idea of gradient-based importance score and applies this score to assign sub-network masks for sequential tasks.
Experimental results show that TST meets its motivation.
Concretely, it improves the CL performance on similar tasks and keeps low forgetting rates.

strength_and_weaknesses: **Strength**

* The motivation behind the design of TST (i.e., assigning sub-network masks based on task similarity) is promising.
* The design of importance score (Equ. 2-5) and the choosing strategy (Equ. 7) is interesting. Moreover, the analysis results in Table 3 and 4 show the effectiveness of these designs.

**Weakness**

* The impt mask defined in Equ. 2 and used in Equ. 3 is a little weird. It seems that the elements in impt mask are not 0/1 and could be extremely larger than 1. Therefore, multiplying this mask to the original weight matrix may strongly affect the functionality of the original modular (i.e., the adapter), and the gradients (which will be used to calculate the importance score) can also be influenced.
* Lack of further analysis of the mechanism behind TST.  I suppose the most important and interesting part in this work is the designs in Equ. 2-7. I agree that these designs sounds good, but I feel it is too amazing for a neural model to perform in a so interpretable way.
This is my major concerns to this work. I list several concrete questions as follows and will change my score according to your response.

**Questions**

* Q1: **Did you cherry pick the visualization in Figure 3?** Your analysis in 'Effectiveness of task similarity detection' tried to demonstrate that the actions (or called the choices of masks) in TST are in line with the intuition that the similar tasks share the mask while dissimilar tasks choose different masks.
Therefore, I'm wondering if these analysis are from the cherry picked model or general ones.
For instance, for the two tasks 'icsi' and 'ami' in SUM, when the random seed and task sequence order are changed, can TST still find them similar?
* Q2: **Does Equ. 7 work correctly at the beginning of the task sequences?** Since masks are randomly initialized, the importance scores calculated in the first several tasks could be hugely affected by this randomness, and the comparison in Equ. 7 might also be influenced. Moreover, I notice that for (a) (b) (e) in Figure 3, TST found most similar tasks at the beginning of the task sequences. Therefore, I'm wondering whether these 'similar tasks' are superficial correlations according to their order in task sequences.

**Minor Questions**

* How are the adapter used in TST trained? Since the paper claims that only the mask is trained, I wonder if the adapter is just randomly initialized without further training?
clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality**: The analysis are not clearly demonstrated.

**Novelty**: The novelty of the proposed method is enough.

**Reproducibility**: The author upload code for reproduction.
summary_of_the_review: **Post-revision update:**
The response make me further worried about the relation between the 'performance improvement' and the story-telling of 'task similarity detection'.
The insight about 'task similarity detection' is not well-supported.
Thus, I decrease the overall score.


I have some concerns for the effectiveness of the proposed method and therefore give a borderline score.
I list several questions and will change my score according to the response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This works address two majors challenges in continual learning: catastrophic forgetting  and knowledge transfer (KT) across tasks applied to NLP tasks.
- In doing so, the approach is based on sub-networks, and computes task similaity via gradients and mask importance where catastrophic forgetting is controlled using a different mask while knowledge transfer across tasks is encourage using the same mask across tasks.
- The proposed approach is applied to several NLP tasks using 5 datasets, and evalued against exhaustive baselines. The paper show improved performance in terms of controlling catastrophic forgetting  and performing knowledge transfer (KT) across tasks
- The paper is clearly written and well motivated 

strength_and_weaknesses: Strengths:
- paper address key questions in continual learning and applied to NLP tasks
- the approach is incremental yet effective
- novel ideas in computing task similarity using mask importance applied to presenting CF and encouraging knowledge transfer, however inspired from existing works
- paper is well motivated and clearly written 
- evaluates the proposed approach on 5 different NLP data sets 
- paper employs a good set of related baselines to empirical comparisons 
- sound quantitate analysis and ablation study 

Weaknesses:
- incremental novel steps (limited novelty) yet effective and simple 
- unclear / missing experimental setup: how is the data split over time across tasks in Cl setup? Does a different sequence of tasks leads to different performances?
- unclear how the approach address the question: how new entities are learned over time, not seen in the historical tasks? How does the SoftMax adjust and the parameters shared over a sequence of NRE tasks as new entities evolve?

Question:
- See Weaknesses section 
- How does the approach scale to unsupervised tasks - such as topic modeling in continual learning paradigm [1]? How does the task similarity be applied on document/text latent representations: topics?
- Include additional existing works such as [2] in continual topic modeling: Unsupervised Continual learning for NLP that overcomes CF via both regularization approach and data replay and computes task similarity using topics - that represents data documents [1] and perform KT.
It is interesting to see how the tasks similarity approaches scales to unsupervised continual learning (close to real world). 


References:
[1] Topic modeling using topics from many domains, lifelong learning and big data. ICML 2014.
[2] Neural Topic Modeling with Continual Lifelong Learning. ICML 2020.


clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: good
Novelty: incremental
Reproducibility: yes
summary_of_the_review: Please see above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Prior work in continual learning has focused on addressing the catastrophic forgetting (CF) problem. However, this paper notes that "knowledge transfer" (KT) between tasks is also important, and proposes a new method which "addresses both issues." 

To address CF, each task is associated with a learned mask identifying a task-specific sub-network. The mechanism by which this is accomplished is “borrowed from (Wortsman et al., 2020)” and consists of boolean gates indicating which parameters in the network should be employed for which tasks.    

To address KT, the idea is to measure the importance of the mask of the previous task relative to the current task. This uses an idea from network pruning (Michel et al., 2019) and uses gradients as measure of parameter importance. A contribution of this work is to extend this idea to entire task-specific masks.

The experiments consist of five benchmarks: (1) 19 tasks for aspect sentiment classification, (2) 5 tasks for continual classification dataset, (3) 6 tasks from ConvoSum, (4) 5 tasks for dialogue response generation and (5) 5 named-entity recognition tasks. On these benchmarks, 11 baselines are compared, including continual learning and non continual learning methods.

strength_and_weaknesses: The experiments are fairly systematic in terms of considering a large number of baselines and datasets. However, it should be noted that the improvements over certain CL baselines like SupSup (Wortsman et al., 2020) are very similar in performance (in one case, SupSup achieves 11.63, while the proposed method gets 11.36 and is incorrectly labelled in bold as the best performance).

I would have liked to see more motivation for the proposed masking approach. Specifically, looking at the gradients does not strike me as the most obvious or computationally expedient mechanism to compute task similarity. For example, task similarity could be measured instead on the basis of simple statistics of the feature representations (e.g., multivariate Gaussian fit to each task’s activations).  In general, additional discussion of the limitations of the proposed approach and discussion of alternatives would benefit the paper.

I would have liked to see a multi-task “oracle” that has access to all tasks at once during training. This would help characterise the potential performance benefit of knowledge transfer separate from the impact of catastrophic forgetting.

clarity,_quality,_novelty_and_reproducibility: The technical presentation was hard to follow at times. For example, exactly how masks are applied to parameter sets is not clear from this paper alone; I had to go read the cited papers to understand the ideas being applied here (Wortsman et al, 2020; Houlsby et al., 2019). In several places, the gradient is used to score the importance of different masks, but I had trouble figuring out what the gradient is being evaluated on, and then how it is aggregated across examples to yield a single task importance score. To this end, Figure 1 seeks to illustrate how the whole framework operates, but I didn’t find it particularly helpful (it could also benefit from proper typesetting).
summary_of_the_review: Overall, this paper provides a sensible framework for continual learning, providing mechanisms to address both CF and KT. My main concern is the lack of novelty: the main technical contribution is how task similarity is estimated but even this is largely based on prior work. Furthermore, while experiments are fairly comprehensive, in some cases the improvement over baselines is inconsistent or relatively small (e.g., SupSup).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new method called TST:  Task-CL based on Sub-networks and Task similarity, which is a continual learning method that is based on subnetworks and task similarity.  The authors claim that this method overcomes catastrophic forgetting while also helping in knowledge transfer.  

Strengths:  The problem space is very important, the methods are elegant and seem to be generally quite easy to use, and the reviewers like the experimental setup and it is well written.

Weaknesses: Reviewers have questions about the significance of the results, questions about how convincing the knowledge transfer experiments are.  Reviewer iK2Y have concerns about how justified the purported improvements are due to the mentioned insights of the TST formulation.
justification_for_why_not_higher_score: Given the reviews and the interaction with the authors, I felt that the paper could be improved if the feedback from the reviewers are used to improve the paper.
Decision: Reject