Learning with Preserving for Continual Multitask Learning

Authors: Siwoo Bae, Hanchen David Wang, Zirong Chen, Meiyi Ma
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: continual learning, continual multitask learning, representation learning, knowledge distillation

Abstract:
Artificial Intelligence (AI) drives advancements across fields, enabling capabilities previously unattainable. Modern intelligent systems integrate increasingly specialized tasks, such as improving tumor classification with tissue recognition or advancing driving assistance with lane detection. Typically, new tasks are addressed by training single-task models or re-training multitask models, which becomes impractical when prior data is unavailable or new data is limited. This paper introduces Continual Multitask Learning (CMTL), a novel problem category critical for future intelligent systems yet overlooked in current research. CMTL presents unique challenges beyond the scope of traditional Continual Learning (CL) and Multitask Learning (MTL). To address these challenges, we propose Learning with Preserving (LwP), a novel approach for CMTL that retains previously learned knowledge while supporting diverse tasks. LwP employs a Dynamically Weighted Distance Preservation loss function to maintain representation integrity, enabling learning across tasks without a replay buffer. We extensively evaluate LwP on three benchmark datasets across two modalities—inertial measurement units of multivariate time series data for quality of exercises assessment and image datasets. Results demonstrate that LwP outperforms existing continual learning baselines, effectively mitigates catastrophic forgetting, and highlights its robustness and generalizability in CMTL scenarios.

Decision: reject
Reviewer scores: 3, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730408084225

thread_type:
official_review

<thread_object>
creator: reviewer_kdbt

<content>
summary:
This paper introduces Learning with Preserving (LwP), a novel approach to continual multitask learning (CMTL) that addresses limitations in traditional continual and multitask learning methods by preserving previously learned knowledge across diverse tasks. LwP employs a Dynamically Weighted Distance Preservation (DWDP) loss function, which maintains representation integrity for both prior and future tasks without relying on a replay buffer.

strengths:
1. The idea is good - multi-task continual learning and is an essential problem in the space of continual learning.
2. The dynamic weighting is an interesting method, but a little ensure if pair-wise comparison is optimal when the dataset is big.
3. Adequate sets of experiments across various metrics.

weaknesses:
1. Why does the authors consider three separate datasets and not a combination of them? The latter would be more representative of real-world scenarios.
Eg: first 3 tasks CelebA, next 3 tasks PhysiQ and so on, which is more representative of a realistic scenario.
2. How is Fig 2 visualized? what exactly it is meant to represent? Is this is a conceptual diagram, a visualization of actual data?

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582371821

thread_type:
official_review

<thread_object>
creator: reviewer_9jhi

<content>
summary:
The paper aims to address the continual multi task problem. The paper proposes a LwP loss in addiction to current loss and loss to preserve old preditions. LwP tries to preserve the knowledge in the implicit knowlege space. The paper also propose to masks the loss on LwP if the labels are different and in that case it is not nessory to have this preserving loss. 

The paper then goes to evaulate this approch on various small scale benchmarks, and specilay on image datasets, it shows a clear gains over previous approches. The paper also shows the BWT metric for all the continual learning methods, and t-sne plots for the latent space.

strengths:
Performance on all the benchmarks are impressive. Figure 5 clearly shows the minimal loss in performance in the previous tasks as the learning progress. 

the benifits of Learning with Preserving (LwP) loss as a regulaization is very solid, and can be seen on the figure 5 and table 1, and compared to other appoches LwP performs considerably well. 

the evaulation is done with a good coverage, with 3 vision benchmarks, and show the distributions of these latents in t-sne plots. the paper also measures the backward transfer values of the continual learning methods.

weaknesses:
It is not clear, how this CMTL problem is novel, it is same as in early LwF papers, and the paper claims this is one of the contibutions. please adress this in the rebuttal. 

while the results are impressive, i am bit scaptical on the scale of the datasets, all have been trained on smaller scale and low resolution. would be nice to show some results on larger resolution images and models. Also would be nice to show that this approch can work for other archituctres like vit. I belive it should work without any problems. I still think resnet 18 is too small model in the current landscape to validate anything concretely.

Also there is not enough ablations to varify the contibutions of dynamic weighting, that would be helpful to validate this claim.

questions:
please look at my strengths and weakness sections, and if you can adress the weakness section, i am happy to change my ratings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611108338

thread_type:
official_review

<thread_object>
creator: reviewer_minz

<content>
summary:
This paper introduces Learning with Preserving (LwP), a novel framework designed for Continual Multitask Learning (CMTL), which involves learning different tasks sequentially while preserving shared representations. The paper evaluates LwP on three benchmark datasets across two modalities, demonstrating its competitive performance compared to existing continual learning methods.

strengths:
1.	The paper proposes a new scenario of continual learning, CMTL, highlighting its unique challenges and significance in practical applications.
2.	The LwP framework is innovative in preserving previously learned knowledge in a way that remains applicable and beneficial across diverse tasks.
3.	The experimental results suggest that LwP demonstrates competitive performance compared to existing continual learning methods.

weaknesses:
1.	How does the proposed method address the fundamental challenges in continual learning, such as catastrophic forgetting or the stability-plasticity dilemma?
2.	The Dynamically Weighted Distance Preservation (DWDP) loss is an innovative contribution. However, it would be valuable to delve deeper into the theoretical foundations of DWDP, exploring its relationship to other distance-preserving techniques and providing additional insights into why it is effective for preserving implicit knowledge.
3.	A point of concern is that, continuous learning methods in the comparison experiment are not state-of-the-art, and therefore may not effectively substantiate the validity of the method proposed in this paper.
4.	Further exploration is needed for more experimental settings, such as investigating the performance of a model when continuously learning five tasks in the presence of five base tasks.
5.	The section on the extension to learning problems (pages 19-20) provides a valuable insight into the theoretical underpinnings of LwP, but it could be integrated more seamlessly into the main body of the paper to enhance its readability and coherence.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730854754948

thread_type:
official_review

<thread_object>
creator: reviewer_qi2x

<content>
summary:
This paper introduces a new problem setting called Continual Multitask Learning (CMTL) and proposes a novel method called Learning with Preserving (LwP) to address it. CMTL is defined as a scenario where a model needs to learn multiple different tasks sequentially, with input data coming from the same distribution but each task having distinct label spaces. The proposed LwP method aims to preserve previously learned knowledge in the shared representation space without requiring a replay buffer of old data. It uses a novel Dynamically Weighted Distance Preservation (DWDP) loss to maintain the integrity of representations. Extensive experiments demonstrate LwP's strong performance and generalization abilities in CMTL scenarios.

strengths:
1. A new continual learning setting is introduced.
2. A method tailored to the new setting is designed.

weaknesses:
1. Compared to general CL scenarios, such as CIL, DIL (domainincremental learning), TIL (task incremental learning), the proposed CMTL setting can indeed be seen as an idealized simplified version. CMTL lets the input data come from the same distribution, which means that all tasks are performed on the same data domain, without considering the case where the data distribution drifts over time. In real-world applications, the data distribution of subsequent tasks may differ from the previous ones. 
2. It is difficult to imagine how this setting could be implemented in reality. In actual scenarios, it might only be achievable by repeatedly labeling the same set of data with new labels. Even updating the data slightly would likely change its domain distribution
3. The methods used for comparison are somehow out of date.
4.The performance of LwP likely depends on careful tuning of the loss weights (λc, λo, λd).

questions:
Please refer to my comments in the 'Weakness' session.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732695428826

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate the feedback provided by the reviewers and the Area Chair, and we are grateful for the opportunity to clarify key aspects of our work. Unfortunately, we haven't received any reviewer's response, thus we are writing to summarize a major clarification and our paper revision.

To start with, we appreciate all the reviewers’ recognition of our work's novelty in new problem formulation and solution. We believe there may be some misunderstandings regarding the scope and novelty of our proposed framework from some of the reviewers, particularly concerning the assumptions of data distribution drift. To clarify, our paper introduces Continual Multitask Learning (CMTL) as a new problem category focusing on label space iteration, where tasks arrive sequentially with distinct labels applied to a consistent input distribution. The misunderstanding appears to stem from the assumption that CMTL must address data distribution drift. However, the proposed CMTL setting deliberately focuses on a scenario where tasks are introduced via label space iteration over a fixed input distribution. Unlike traditional continual learning, which often assumes distribution drift, our framework specifically addresses challenges like catastrophic forgetting and task interference within the label space. This is a realistic assumption in applications like medical imaging or autonomous systems, where the data distribution remains stable while new iterations of data or annotations are incrementally added. Handling data distribution drift falls outside the scope of this work but could be a promising direction for future research.

We emphasize that the reviewer's concern about data distribution drift does not apply to our work, as the proposed setting intentionally focuses on tasks with a fixed input distribution. This simplification enables us to tackle unique challenges in CMTL, such as preserving shared representations across sequential label spaces. Moreover, most reviewers have recognized the novelty of our approach, including the dynamically weighted preservation loss to effectively retains knowledge. The following components are incorporated into the revised manuscript:

 - We have revised the introduction to better articulate the CMTL setting and its practical implications to avoid potential misunderstandings.
 
 - We have revised the caption and description of Figure 2 to provide a clear explanation of its components.
 
- The theoretical perspective on learning problems and the ablation study have been integrated into the main body.

- We have also fixed typos and minor errors that were present in the paper.


Also, we have included following additional experiments in the manuscript:
- We have incorporated recent and relevant continual learning methods into our comparative analysis.
   - Dual View Consistency (DVC) [Gu et al., 2022]
   - Online Bias Correction (OBC) [Chrysakis and Moens, 2023]

- We have conducted experiments using larger models and higher-resolution datasets (see ***Appendix D.6***).

- We have expanded our experiments to assess the applicability of LwP in different task sequence scenarios. The model was tested on scenarios where it continuously learns additional tasks after initially learning five tasks in a multitask learning manner (see ***Appendix D.7***)


  

**References**

Chrysakis, A., & Moens, M.-F. (2023). Online bias correction for task-free continual learning. *International Conference on Learning Representations (ICLR)*.

  
Gu, Y., Wang, Y., Wu, Z., Herrmann, C., & Herrmann, J. M. (2022). Not just selection, but exploration: Online class-incremental continual learning via dual view consistency. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733169332004

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you for reviewing our submission and providing constructive feedback. We have carefully addressed your comments in the rebuttal and believe these revisions help clarify the contributions and resolve potential misunderstandings. If our responses address your concerns, we kindly ask you to reconsider your evaluation. Should there be any remaining questions, we are happy to engage in further discussion and provide clarification before the Dec. 3rd deadline.

Best regards,

Author(s)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received mixed reviews. The reviewers recognized the well designed method for the proposed continual multi-task learning setting, its strong performance in the setting, and extensive experiments. However, they disagreed on the value of the proposed problem setting and benchmarks: Two reviewers appreciated the value of the problem setting (minZ, KdBT), while the others argued that the problem setting as a variation of existing ones and thus pointed out incremental novelty (qi2x, 9JHi). Moreover, three reviewers considered that the benchmarks do not well reflect real-world application scenarios due to due to the lack of distribution shift, small-scale datasets, and outdated model architectures (qi2x, 9JHi, KdBT). Beside these issues, the reviewers raised concerns with lack of comparisons with latest work (qi2x, minZ), potential sensitivity to hyperparameters (qi2x), lack of theoretical foundation of the proposed loss function (minZ), and missing essential ablation study on the dynamic weighting (9JHi).

The authors' rebuttal and subsequent responses in the discussion period address some of these concerns but failed to fully assuage all of them. In particular, after the discussion period, Reviewer qi2x still pointed out the issues on the problem settings and lack of comparisons with latest work. Also, the AC found that the concerns with limitations of the benchmarks have been only partially resolved as the authors did not present additional datasets or incorporate the reviewers' suggestion, although Reviewer 9JHi did not come back and Reviewer KdBT gave a positive score in the end. Further, the AC sees that the rebuttal did not successfully address some of the remaining concerns like sensitivity to hyperparameters (quantitative analysis required) and the lack of theoretical foundation (only empirical analysis results added).

Putting these together, the AC considers that the remaining concerns outweigh the positive comments and the rebuttal, and thus regrets to recommend rejection. The authors are encouraged to revise the paper with the comments by the reviewers and the AC, and submit to an upcoming conference.