Boosting Multiple Views for pretrained-based Continual Learning

Authors: Quyen Tran, Tung Lam Tran, Khanh Doan, Toan Tran, Dinh Phung, Khoat Than, Trung Le
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning, ViT-pretrained continual learning

Abstract:
Recent research has shown that Random Projection (RP) can effectively improve the performance of pre-trained models in Continual learning (CL). The authors hypothesized that using RP to map features onto a higher-dimensional space can make them more linearly separable. In this work, we theoretically analyze the role of RP and present its benefits for improving the model’s generalization ability
in each task and facilitating CL overall. Additionally, we take this result to the next level by proposing a Multi-View Random Projection scheme for a stronger ensemble classifier. In particular, we train a set of linear experts, among which diversity is encouraged based on the principle of AdaBoost, which was initially very challenging to apply to CL. Moreover, we employ a task-based adaptive backbone
with distinct prompts dedicated to each task for better representation learning. To properly select these task-specific components and mitigate potential feature shifts caused by misprediction, we introduce a simple yet effective technique called the self-improvement process. Experimentally, our method consistently outperforms state-of-the-art baselines across a wide range of datasets.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730083797469

thread_type:
official_review

<thread_object>
creator: reviewer_h1kx

<content>
summary:
This paper provides some theoretical analysis on Random Projection (RP) which project features into a higher-dimensional space to improve their linear separability. Furthermore, this paper proposes a Multi-View Random Projection scheme to create a robust ensemble classifier motivated by AdaBoost. Then the proposed method is applied to prompt-based CL methods for a better performance. Experimental results demonstrate its efficacy in various continual learning settings.

strengths:
1.	This paper provides theoretical insights into Random Projection (RP), reinforcing its foundation in an intuitive manner.

2.	The motivation for the proposed Multi-View Random Projection is clear and well-justified, demonstrating its potential to enhance performance.

3.	The paper is well-organized and easy to understand, addressing most of my queries within the main text or the appendix.

weaknesses:
**Major**

1.	The necessity of the huge views should be clarified. As mentioned in the paper, the subsequent atomic views aim to capture the complement knowledge of previous views. However, the final classifier only leverages a part of these views rather than all of them, which may omit some complement knowledge.

2.	The theoretical results are interesting, but the method is heuristic. For example, the design of the voting strategy and the self-improvement are quite complex. It seems that a lot of tricks are introduced directly to improve performance.

3.	The design of the self-improvement process is not intuitive, which may accumulate errors in multiple-step prediction. Specifically, if an incorrect prompt is chosen in the first step, the model may become biased towards predicting the class associated with that wrong prompt, potentially leading to accumulated errors and hindering improvements in prompt prediction accuracy.

4.	The experiments do not include comparisons with simpler ensemble methods, such as randomly sampling views or using all views for the ensemble. Such comparisons can provide more intuitive insights to demonstrate the effectiveness of the proposed method.

**Minor**

1.	There are too many notations which may limit the fluency and readability. For example, in line 177, there is no description for the notation $\gamma (s,g,\mathbb{R}^d)$ and no explanation for the symbolic $g$. So please simplify the notations as much as possible.
2.	Typo in Corollary 4.4 (line 250), “sammple” should be “sample”.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730357537836

thread_type:
official_review

<thread_object>
creator: reviewer_gjf6

<content>
summary:
This manuscript introduces "BoostCL", a novel approach to improve continual learning based on pre-trained models. The manuscript provides theoretical analysis to show that feature separability and model generalization ability benefit from random projection. Then a multi-view random projection is proposed to adapt AdaBoost for CL. A self-improvement process is also proposed for appropriately selecting prompts for each task sample. Empirical validation on multiple CL benchmarks is then conducted.

strengths:
1. The manuscript provides theoretical analysis of Random Projection, demonstrating how RP improves feature separability in high-dimensional space. 
2. The manuscript is clear and well organized. The formulas are presented in an unambiguous manner.

weaknesses:
A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.

1. Although RP and multi-view strategies are theoretically shown to improve feature separability, there is limited discussion on the theoretical upper bound of generalization error.
2. The manuscript does not provide theoretical guidance on choosing the optimal projection dimension in RP, which can greatly impact model performance for different tasks.
3. The proposed self-improvement process for prompt selection could increase the model’s inference complexity, but this potential trade-off is not fully analyzed, especially in scenarios with many tasks.
4. Minor spelling errors, like “sammple” in line 250.
5. More experiment regarding hyper-parameters like number of views, projection dimensions, and threshold settings for the voting mechanism should be conducted.

questions:
1. How to effectively choose the best projection dimension to make appropriate trade-off between performance and computational overhead.
2. How much gpu resource does the proposed method need, will the gpu memory usage increase rapidly by increasing projection dimension.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610956112

thread_type:
official_review

<thread_object>
creator: reviewer_werc

<content>
summary:
This paper presents BoostCL, a method for pretrained-based continual learning that leverages multiple views and random projection (RP) to enhance performance. The authors theoretically analyze the benefits of RP in higher-dimensional spaces and demonstrate its effectiveness through experimental results.

strengths:
1.	The paper provides a novel theoretical analysis of the benefits of random projection in higher-dimensional spaces for continual learning.
2.	The authors conduct a comprehensive set of experiments to evaluate the performance of their proposed method, BoostCL, across various tasks and datasets.
3.	The paper demonstrates the practicality of the proposed method by applying it to real-world continual learning scenarios.

weaknesses:
1. The paper mentions the motivation behind using random projection, but it lacks a comprehensive discussion on why this approach is superior to other potential methods for feature transformation.
2. While the paper proposes a novel method for continual learning, it would be helpful to see a more in-depth comparison with other recent methods in the literature. Specifically, how does BoostCL differ from and improve upon existing approaches?
3. The technical details of the proposed method are somewhat challenging to follow. The paper could benefit from a more clear and concise explanation of the algorithm and its components. 
4. Discussing the scalability of the proposed method to larger datasets or more complex models would be relevant.
5. The author needs to further analyze the consumption of the proposed method compared to other methods in terms of computational resources and training time.
6. The paper could benefit from a more organized and focused presentation of the material. The introduction could be more clearly tailored to motivate the problem and the proposed solution. Additionally, the paper includes several appendices that contain detailed proofs and additional experimental results. While these appendices provide valuable information, they could be more effectively integrated into the main text to enhance the clarity and readability of the paper.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644102842

thread_type:
official_review

<thread_object>
creator: reviewer_htxd

<content>
summary:
This work considers the random projection (RP) strategy for pre-trained models in CL. Motivated by the benefits of RP in high-dimensional space, a multi-view strategy for an efficient classifier is further proposed, in which the principle of Adaboost is adapted to 
overcome inherent obstacles and applied for the first time in CL. In addition, a self-improvement process technique, although simple, also shows significant effectiveness in selecting proper taskspecific prompts. The experimental results demonstrate a positive impact of the proposed method in improving model quality while only applying to linear classifiers.

strengths:
(1)The proposed method BoostCL performs better than existing CL baselines, including Hide-prompt and ranpac.
(2)This work addressed the challenge when applied AdaBoost directly to CL.
(3)A self-improvement process, a simple but effective strategy is designed to help select prompts more accurately when inference.

weaknesses:
As shown in Table 1, the proposed method takes no advantage in anti-forgetting, FFM metric.

questions:
Does some ablation study be conducted on the proposed modules, Prompt selection process and Self-improvement process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732471653564

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to the responses from Reviewers

comment:
Dear Reviewers,

We would like to thank you again for spending your time evaluating our paper.

As the discussion period is expected to conclude shortly, we look forward to hearing your feedback about whether we have addressed your concerns in the rebuttals.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces BoostCL, a method that outperforms existing continual learning (CL) baselines like Hide-prompt and ranpac by addressing the direct application challenges of AdaBoost to CL. On the positive side, the work provides a novel theoretical analysis on the benefits of random projection (RP) in high-dimensional spaces, which improves feature separability and supports continual learning. However, reviewers also raised several major concerns, including the weakness in anti-forgetting performance, technical and writing clarity, model scalability, theoretical analysis (e.g., upper bounds of generalization error), and lack of comparisons with the baseline methods.