Soft-TransFormers for Continual Learning

Authors: Haeyong Kang, Chang D. Yoo
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Soft-Transformers, Continual Learning, Well-initialized Lottery Ticket Hypothesis (WLTH)

Abstract:
Inspired by Well-initialized Lottery Ticket Hypothesis (WLTH), which provides suboptimal fine-tuning solutions, we propose a novel fully fine-tuned continual learning (CL) method referred to as Soft-TransFormers (Soft-TF). Soft-TF sequentially learns and selects an optimal soft-network or subnetwork for each task. During sequential training in CL, Soft-TF jointly optimizes the weights of sparse layers to obtain task-adaptive soft (real-valued) networks or subnetworks (binary masks), while keeping the well-pre-trained layer parameters frozen. In inference, the identified task-adaptive network of Soft-TF masks the parameters of the pre-trained network, mapping to an optimal solution for each task and minimizing Catastrophic Forgetting (CF) - the soft-masking preserves the knowledge of the pre-trained network. Extensive experiments on Vision Transformer (ViT) and CLIP demonstrate the effectiveness of Soft-TF, achieving state-of-the-art performance across various CL scenarios, including Class-Incremental Learning (CIL) and Task-Incremental Learning (TIL), supported by convergence theory.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730082830308

thread_type:
official_review

<thread_object>
creator: reviewer_d7yx

<content>
summary:
This paper presents an approach that combines Prompt techniques with learnable soft masks to address catastrophic forgetting in continual learning.  Inspired by Well-initialized Lottery Ticket Hypothesis (WLTH), which provides suboptimal fine-tuning solutions, this paper proposes a fully fine-tuned continual learning (CL) method referred to as Soft-TransFormers (Soft-TF).

strengths:
1. The paper includes extensive experiments across various datasets and tasks, providing comprehensive evidence for the method’s effectiveness.
2. The paper is well-written, with clear organization and logical flow that effectively communicates complex ideas.

weaknesses:
### 1. Insufficient Technical Explanation

**1.1 Lack of Detailed Specification on Soft Mask \( m \) Initialization, Selection, and Optimization**  
The paper suggests that the soft mask \( m \) is task-specific, adapting the subnetwork structure based on each task's needs. However, it lacks essential details on how these masks are initialized, selected, and optimized for each task. For instance, are there specific initialization strategies, criteria for mask selection, or task-transition strategies? Such missing details make it challenging to understand the core contribution of the paper, especially as the optimization of these masks is crucial for retaining knowledge in continual learning.

**1.2 Potential Growth in Parameters Due to Task-Specific Masks**  
As each task requires an independent soft mask \( m \), the storage cost may grow exponentially with the number of tasks. This is particularly concerning in scenarios with a large number of continual learning tasks, where storage costs could become prohibitive. The paper should discuss how to mitigate this issue.

**1.3 Lack of Clarity on the Setting of \( \alpha_t \)**  
The parameter \( \alpha_t \) is referenced but not clearly explained in terms of how it is set or updated for each task. How does the method ensure that the sum of \( \alpha_t \) across tasks equals 1, and what role does this parameter play in mask composition? Detailed explanations and possible constraints for \( \alpha_t \) are necessary for comprehending how the method balances task adaptation and parameter sharing.

**1.4 Redundant Calculation of \( t_x \) in Algorithm 2**  
Algorithm 2 calculates \( t_x \) twice, raising the question of whether the indices for the E-Prompt and the learned subnetwork are identical. This redundancy lacks justification, and without clear reasoning, readers may question the necessity of these duplicate operations. A clarification is needed regarding the purpose of this repeated calculation and the roles of E-Prompt and learned subnetworks in relation to \( t_x \).

**1.5 Increased Inference Cost Due to Subnetwork Selection**  
According to Algorithm 2, selecting the learned subnetwork for a specific task requires backpropagation for subnetwork selection, which can significantly increase inference costs. This may limit the method's applicability in real-time or resource-constrained environments. 

### 2. Unclear Core Mechanism

**2.1 Ambiguity in the Combination Mechanism of Soft Mask \( m \)**  
The paper does not clarify why and how soft masks \( m \) can be combined based on different tasks. Ensuring each task's mask is unique and effectively isolated from other tasks is vital to avoid task interference and catastrophic forgetting. The paper lacks an explanation of how such task differentiation is achieved in practice. This is a crucial detail for understanding the validity of the proposed method's mechanism.

**2.2 Ensuring Task-Specific Mask Differentiation**  
The authors do not provide details on how task-specific masks are designed to ensure they differ across tasks. If there is overlap or similarity between masks for different tasks, there is a high risk of interference, which could exacerbate forgetting. Further clarification on how each task’s mask maintains uniqueness would strengthen the proposed method.

questions:
I recommend the authors address these concerns by providing detailed descriptions of the soft mask initialization, selection, and optimization processes, as well as methods for managing storage and computational costs with an increasing number of tasks. Clarifying the underlying mechanisms of mask composition and task differentiation would greatly enhance the transparency and reproducibility of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730376926284

thread_type:
official_review

<thread_object>
creator: reviewer_ygek

<content>
summary:
This work proposed sub-network over the prompt tuning technique for continuous learning. 
The overall presentation is clear but the motivation and the novelty is unclear.
Experimental results demonstrate superior performance to other methods.

strengths:
The overall presentation is clear.
The experimental results demonstrate the effectiveness of the proposed method.

weaknesses:
Firstly, the motivation is unclear.  In lines 84-85, the paper stated that " It cannot capture all the nuances of uncorrelated sequential tasks wildly if the task significantly differs from what the CL model was pre-trained initially on". It seems this work aims to address the reliance on a well pretraining, but the proposed method also require a well-initialized network as stated in lines 96-99. 
Besides, in lines 224-225, the task-specific prompts can be regarded as the explicit task-specific fine-tuning.

Secondly, the proposed method seems to be a combination of parameter-efficient fine-tuning method, e.g., prompt + LoRA, prompt + Adapter. How much performance improvement would be caused by adding LoRA or adapters for specific tasks in the baseline method。
The superior performance seems contributed by more learnable parameters.

The experimental results requires more analysis. 
(i) Explain the ``Upper bound''. In fact, since the proposed method modifies the model architecture, the ``Upper bound''  cannot be the same as the other paper.
(ii) Why the proposed method achieves higher performance than the ``Upper bound''? 
(iii) Why the proposed method achieves higher performance when using DualPrompt as baseline while the improvement over the L2P seems smaller. 
(iv). In table 1, why the performance under the  20-Split-CIFAR100 is higher than  10-Split-CIFAR100? In general, the more incremental learning step, the weaker the performance due to forgetting. I suspect that the improved performance is due to learning more parameters. The number of parameters should be reported.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720768781

thread_type:
official_review

<thread_object>
creator: reviewer_rgmo

<content>
summary:
In this paper, the authors attempt to solve the continual learning problem via a soft transformer idea, which adaptively selects subnetworks (specified by learnable masking for pre-trained parameters) and soft network tasked for detailed tasks in CL. During the sequential learning, the model will be trained to achieve a task-specific soft network based on the task-agnostic pre-trained model and sparsely activate some weights in the layers or deactivate someone. Besides, the authors support some theory proofs based on Well-initialized Lottery Ticket Hypothesis to analyze the rationale behind the improvements. Implemented on baselines, like DualPrompt, L2P, and L2p-PGP, they obtain competitive even SoTA performances.

strengths:
1. The presentation sounds well.

2. The final experimental results present competitive even SoTA on some evaluated tasks.

3. Implementing the method on the Prompt Pool, which can divide the prompt pool into several groups and enable different group prompts to acquire different knowledge sounds reasonable while applying soft-network or binary masks on pre-trained weights can further boost the training.

weaknesses:
1. Somehow, I believe such soft-network or subnetwork ideas have been widely studied among the communities, like model compression, model purification, or PEFT. One can think of employing the extra prompt vectors or using a learnable masking strategy, forcing the model to fit the downstream task while the pre-trained weights are fixed to hold the generalizability. Moreover, the prompt pool construction mechanism has been widely studied by DualPrompt, L2P.

2. I am curious about the training efficiency and inference latency since they utilize a large subnetwork or softnework compared with the baseline. 

3. Missing the tunable parameter comparisons with baselines and existing methods.

questions:
Please refer to my above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732171520095

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Our responses to the concerning points

comment:
# Dear All Reviewers and Chairs

Thank you for your constructive feedback and concerns. We have prepared the following responses to address all the points raised by reviewers. We have clarified the proposed **parameter-efficient Soft-TF** through extensive experiments and further emphasized its novel contributions as follows:

### For **Reviewer Rgmo**,

- We have investigated layer-wise training efficiency & inference of Soft-TF (**RTables 1 and 2**) with gradient and prompt ID: only prompt ID-based Soft-TF outperformed strong baselines such as PGP, **Adapter**, and **LoRA**.

- Furthermore, we have observed that the 3-shot Gradient ID inference cost (using samples within a mini-batch) with the last layer (Soft-TF-L[12]) is approximately 1.1 times that of Prompt ID, leading to the most efficient Soft-TF structure, as shown in **RTables 1-1 and 2-1**.

### For **Reviewer ygEK**,
- We have compared Soft-TF with recommended LLMs (**Adapter** and **LoRA**) with the same trainable parameters (**RTables 3 and 4**), showing outperformance (the most parameter-efficient architecture, Soft-TF) regarding accuracy and forgetting.

- We have added the upper bound of Soft-TF for fair comparisons.

- We have explained **why DualPrompt-Soft-TF outperforms  L2P-Soft-TF** with backgrounds of DualPrompt of Global Prompt and Local Prompts (E-Prompts).

- We have revised the Motivation of Soft-TF as the reviewer suggested.

### For **Reviewer d7YX**,

- We have conducted an additional experiment on Random Initialized Performances of Soft-TF to show the importance of well-initialized pre-trained parameters (**see RTable. 5**).

- Particularly in the **Author-Reviewer discussion period**, we have further clarified the difference between our Soft-TF and conventional Transformer fine-tuning baselines such as **Adapter** and **LoRA**, suggesting parameter-wise superior performances (SOTA, **see RTable. 6**) in Accuracy, Forgetting, training, and time efficiency under the comparable trainable parameter settings.

- Furthermore, we have explained more about Algorithm 2 in detail to clarify the Soft-TF algorithm.

Throughout these rebuttals, we have further strengthened our main contribution to **the well-initialized parameter-efficient Soft-TF, outperforming the representative LLM-finetuning methods** (**Adapter** and **LoRA**).

In summary, to our knowledge, **a well (uniformly)-initialized Soft-TF** (mask network) represents **the first approach** to pre-training transformer-based full fine-tuning methods within the Continuous Learning (CL) domain, including applications involving LLM-based techniques.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview: