LW2G: Learning Whether to Grow for Prompt-based Continual Learning

## Abstract

Abstract Continual Learning (CL) aims to learn in non-stationary scenarios, progressively acquiring and maintaining knowledge from sequential tasks. Recent Prompt-based Continual Learning (PCL) has achieved remarkable performance with Pre-Trained Models (PTMs). These approaches grow a prompt sets pool by adding a new set of prompts when learning each new task ( prompt learning ) and adopt a matching mechanism to select the correct set for each testing sample ( prompt retrieval ). Previous studies focus on the latter stage by improving the matching mechanism to enhance Prompt Retrieval Accuracy (PRA). To promote cross-task knowledge facilitation and form an effective and efficient prompt sets pool, we propose a plug-in module in the former stage to Learn Whether to Grow (LW2G) based on the disparities between tasks. Specifically, a shared set of prompts is utilized when several tasks share certain commonalities, and a new set is added when there are significant differences between the new task and previous tasks. Inspired by Gradient Projection Continual Learning, our LW2G develops a metric called Hinder Forward Capability (HFC) to measure the hindrance imposed on learning new tasks by surgically modifying the original gradient onto the orthogonal complement of the old feature space. With HFC, an automated scheme Dynamic Growing Approach adaptively learns whether to grow with a dynamic threshold. Furthermore, we design a gradient-based constraint to ensure the consistency between the updating prompts and pre-trained knowledge, and a prompts weights reusing strategy to enhance forward transfer. Extensive experiments show the effectiveness of our method. The source codes are available at https://github.com/RAIAN08/LW2G .

## 1 Introduction

Compared to learning in stationary scenarios, Continual Learning (CL) equips systems with the ability to learn in non-stationary environments, which is a core step toward achieving human-level intelligence and human-like adaptation. In this learning paradigm, Deep Neural Networks (DNNs) need to learn from a sequential tasks while retaining past knowledge and acquiring novel knowledge. However, simply utilizing standard optimization methods [, ] for training DNNs inevitably erases the parametric representations of old tasks with new input representations during updating. Therefore, a well-known problem Catastrophic Forgetting (CF) arises [, , , , ], where DNNs suffer severe performance degradation on old tasks due to the absence of old data and domain shift in data distributions, making CL an extremely challenging problem.

Recently, Prompt-based Continual Learning (PCL) offers fresh insights into addressing CF [, , , , , , ]. These methods leverage frozen Pre-Trained Models (PTMs) rather than training from scratch and employ Parameter-Efficient Fine-Tuning techniques (PEFTs) [, , , , , ], e.g., prompt. Specifically, PCL involves two stages: (a) *prompt learning*: learning a

Figure: Figure 1: Illustration of HFC. $\mathcal{S}_{i}$ represents the feature space spanned by the old task $i$, while $\mathcal{S}_{i}^{\perp}$ denotes the orthogonal complement to $\mathcal{S}_{i}$. Then, $\text{HFC}(\bm{g},\bm{g}_{i}^{\perp})$ is denoted as $\text{HFC}_{i}$.
Refer to caption: /html/2409.18860/assets/x1.png

task-wised set of prompts to conditionally guide the PTM for the current task, which are stored in an expanding prompt sets pool, and (b) *prompt retrieval*: predicting which task each testing sample belongs to and choosing the corresponding prompt set. Recent studies [, , ] have found that Prompt Retrieval Accuracy (PRA) can significantly influence the performance, since an incorrect set for the testing samples results in a performance decline. Additionally, learning each task individually not only limits the potential for cross-task knowledge facilitation but also leads to parameter redundancy.

One simple solution to this problem is to mimic humans’ integration of information [, , ]. For instance, when several tasks share certain commonalities, they can use a shared set of prompts. However, when tasks differ significantly, a new set should be added. Thus, by adaptively learning whether to grow a new set for PCL, the amount of selectable options is reduced, and the divergence between sets is increased, thereby improving PRA. Furthermore, aggregating multiple tasks’ knowledge into a single set can also facilitate mutual knowledge utilization and promotion among tasks. Nevertheless, establishing suitable metrics to measure this commonality and obtaining task information *a priori* – all of which are challenging in practice. Moreover, gradually integrating knowledge from multiple tasks into a single set also presents an unresolved query, as the knowledge from different tasks can interfere with each other during sequential learning.

Thanks to Gradient Projection-based Continual Learning (GPCL) [, , ], which proposes that learning would not forget if the updated gradient is orthogonal to the feature space spanned by old tasks (denoted as *orthogonal condition*), we propose to use the *orthogonal condition* in GPCL to integrate the knowledge from multiple tasks into a single set of prompts. Specifically, in Figure [1](#S1.F1), the gradient $\bm{g}$ of the new task is modified to its projection $\bm{g}^{\perp}_{1}$ onto $\mathcal{S}_{1}^{\perp}$, and $\bm{g}^{\perp}_{1}$ serves as the real gradient for updating parameters, thereby reducing the forgetting of old knowledge in task $1$. Furthermore, to address the dilemma of whether *to grow* (i.e., initializing a new set of prompts) or *not to grow* (i.e., selecting an old set of prompts from the pool), we introduce a novel metric called Hinder Forward Capability (HFC). *HFC is calculated as the angle $\theta$ between the gradient of the new task $\bm{g}$ and its’ projection $\bm{g}^{\perp}$*. As illustrated in Figure [1](#S1.F1), as $\text{HFC}_{1}$<$\text{HFC}_{2}$ then $\bm{g}_{1}^{\perp}$>$\bm{g}_{2}^{\perp}$, it implies that the hindrance to learning on the set of prompts to task $2$ is larger than that on the set of prompts to task $1$ when updating under the *orthogonal condition*. Thus, when the hindrance on learning a new task is severe, PCL should choose *to grow* a new set; conversely, it tends *not to grow*. Meanwhile, $\bm{g}$ presents a large projection onto $\mathcal{S}_{2}$ indicating higher similarity between the new task and task $2$ than with task $1$.

Based on the analysis, we propose a plug-in module within PCL to Learn Whether to Grow (LW2G), consisting of three components: Dynamic Growing Approach (DGA), Consistency with Pre-trained Knowledge (CPK), and Facilitation for Forward Transfer (FFT). DGA is an automated scheme to learn whether *to grow* (adopt a new set of prompts and store it in the pool) or *not to grow* (utilize an existing set of prompts from the pool) for new tasks based on the introduced HFC metric. Specifically, to incorporate knowledge from multiple tasks into a single set of prompts, we first employ the *orthogonal condition* to learn new tasks without forgetting and calculate the hindrance on learning with each set in the pool through HFC. Meanwhile, we consider an ideal scenario to generate a dynamic threshold, which learn the new task on the pre-trained knowledge feature space $\mathcal{S}^{\text{pre}}$ without any obstacles from old tasks. DGA chooses *to grow* if all HFC values are above this threshold, indicating that learning with each set in the pool encounters excessive hindrance. Conversely, DGA chooses *not to grow* by selecting the old set of prompts with the minimum HFC and learning the new task under the *orthogonal condition*. CPK aims to balance the disruption to pre-trained knowledge caused by continual learning on new tasks and the reduced plasticity brought by strict orthogonality to the entire pre-trained feature space $\mathcal{S}^{\text{pre}}$. Therefore, we propose applying a soft constraint to the gradient when learning new tasks, aiming to align the gradient direction as closely as possible with the feature space of the pre-trained knowledge, ensuring consistency between prompt updates and pre-trained knowledge. Finally, FFT reuses the frozen weights from the existing set of prompts with the maximum HFC to enhance forward transfer.

The contributions of this paper can be summarized as follows: ($1$) We propose an automated learning scheme within PCL, by learning whether to grow or not to grow set of prompts. We aim to form an effective and efficient prompt sets pool where each single set contains knowledge from multiple tasks, thus facilitating cross-task promotion; ($2$) We introduce HFC metric, which not only measures the difference between new and old tasks but also evaluates the hindrance on learning new tasks under the strict *orthogonal condition*. ($3$) LW2G is a plug-in module within existing PCL. Extensive experiments demonstrate its superiority across multiple benchmarks and various CL settings.

## 2 Related Work

#### Continual Learning and Gradient Projection

Numerous efforts have been made to alleviate the core issue of CF [, , ], which can be roughly categorized into three main categories: ($1$) Architecture-based, ($2$) Rehearsal-based, and ($3$) Regularization-based. Architecture-based methods [, , , , , , ] segregate components within the DNNs for each task by expanding the model or constraining the learning rate of part of parameters. However, most of them designed for Task-CL, which is not suitable for challenging Class-CL. Rehearsal-based methods [, , , , , , , , ] mitigate forgetting by replaying real or generated samples of old tasks, which raises concerns about efficiency and privacy. Regularization-based methods [, ] achieve a balance between new and old tasks by designing sophisticated regularization terms. Among them, GPCL methods [, , , , , , , , , , , , , ] focus on the gradient of the parameter. These methods project the gradient orthogonally to the feature space spanned by the old tasks, thereby not affecting the old knowledge.

#### Prompt-based Methods and Transfer Learning

PCL garnered significant attention due to their utilization of PEFT techniques [, , , , , ] to leverage PTMs, achieving rehearsal-free and promising performance [, , , , , , , , , , , , ]. Among them, DualPrompt [] proposed partitioning the knowledge of tasks into general and specific categories, and learns them with g-prompt and e-prompt, respectively. Similarly, S-liPrompt and S-iPrompt [] addressed Domain-CL by leveraging Vision-Language Models (VLMs) to further enhance the learning ability. CODAPrompt [], S-Prompt++ [] and HidePrompt [] improved *prompt retrieval* stage through *attention mechanisms* and auxiliary adapter classifiers. Additionally, recent studies show that fine-tuning downstream tasks or continual learning with PTMs often leads to overfitting due to relatively limited downstream training data, resulting in degradation of pre-trained knowledge [, , , ].

## 3 Preliminaries and Notations

#### Continual Learning

Assume there is a sequence of tasks and their corresponding training datasets $\left\{\mathcal{D}^{i},i=1,2,\ldots\right\}$ without overlapping classes, where $\mathcal{D}^{t}=\left\{\left(\bm{x}_{i,t},\bm{y}_{i,t}\right)\right\}_{i=1}^{n_{t}}$ belongs to the task $t$. We denote the DNN as $\mathcal{W}=\left\{\theta^{l}\right\}_{l=1}^{L}$, where $\theta^{l}$ is the weight of layer $l$. Given a training sample $\bm{x}_{i,t}$, we denote $\bm{x}_{i,t}^{l}$ as the input of layer $l$ and the output is $\bm{x}_{i,t}^{l+1}=f^{l}\left({\theta}^{l},\bm{x}_{i,t}^{l}\right)$, where $f^{l}$ is the operation of layer $l$. We simplify the loss function for learning task $t$ as $\mathcal{L}_{t}(\mathcal{D}^{t})$ and $\mathcal{W}_{t}=\left\{\theta^{l}_{t}\right\}_{l=1}^{L}$ as the DNN after training on task $t$.

#### Gradient Projection Continual Learning

First, for any matrix $\bm{A}$ with suitable dimensions, its projection onto a given space $\mathcal{S}$ is denoted as follows:

$$ $\displaystyle\text{Proj}_{\mathcal{S}}\left(\bm{A}\right)=\bm{A}\bm{B}\left(\bm{B}\right)^{T},$ (1) $$

where $\bm{B}$ is the bases for $\mathcal{S}$ and $\left(\cdot\right)^{T}$ is the matrix transpose.

Then, following [], we briefly introduce how GPCL reduces the interference of old knowledge when learning new tasks. After leaning task $1$, GPCL first constructs a representation matrix for layer $l$ as $\bm{R}_{1}^{l}\in\mathbb{R}^{N\times d}$ from task $1$ only. Next, Singular Value Decomposition (SVD) is performed on $\bm{R}_{1}^{l}$ followed by its $k$-rank approximation $\left(\bm{R}_{1}^{l}\right)_{k}$ with threshold, $\epsilon$. Therefore, the feature space for layer $l$ spanned by task $1$ is built by $\mathcal{S}_{1}^{l}=\text{span}\left\{\bm{B}_{1}^{l}\right\}$, where $\bm{B}_{1}^{l}$ is the bases for $\mathcal{S}_{1}^{l}$. And $\mathcal{S}_{1}^{l}$ is stored in memory $\mathcal{M}=\left\{\mathcal{S}_{1}^{l}\right\}$. When learning task $2$, the gradient of layer $l$ is denoted as $\bm{g}=\nabla_{\theta^{l}}\mathcal{L}_{2}$. As illustrated in Figure [1](#S1.F1), GPCL modify the gradient as follows:

$$ $\displaystyle\bm{g}^{\perp}_{1}=\text{Proj}_{\mathcal{S}_{1}^{\perp}}(\bm{g}),$ (2) $$

where $\mathcal{S}_{1}^{\perp}$ is the orthogonal complement of $\mathcal{S}_{1}^{l}$ and $\bm{g}^{\perp}_{1}$ serves as the real gradient for updating layer $l$. Let $\Delta\theta_{1}^{l}$ denote the change in layer $l$ after learning task $2$. For $\bm{x}_{i,1}\in\mathcal{S}_{1}^{l}$ from task $1$, it follows that $\Delta\theta_{1}^{l}\bm{x}_{i,1}=0$ due to the orthogonality of $\bm{g}^{\perp}_{1}$ with respect to $\mathcal{S}_{1}^{l}$ [, ]. Therefore, we can obtain:

$$ $\displaystyle\theta^{l}_{2}\bm{x}_{i,1}^{l}=(\theta^{l}_{1}+\Delta\theta^{l}_{1})\bm{x}_{i,1}^{l}=\theta^{l}_{1}\bm{x}_{i,1}^{l}.$ (3) $$

It demonstrates that there is no forgetting of knowledge of task $1$, if the gradient for updating parameters is orthogonal to the old feature space. We denote the above condition as the *orthogonal condition*. After learning task $2$, a new representation matrix for layer $l$ denoted as $\bm{R}_{2}^{l}$ is built from task $2$ only. And $\mathcal{S}_{1}^{l}$ in $\mathcal{M}$ needs to be updated by updating $\bm{B}_{1}^{l}$ with unique bases from $\bm{R}_{2}^{l}$. Details are in Appendix [B](#A2).

#### Prompt-based Continual Learning

Recent studies [, , , , ] utilized prompts to leverage the PTMs. Therefore, the DNN is a Vision Transformer (VIT), and the operation of layer $l$, $f^{l}$, is the *attention mechanism* within each transformer block. Hence, the input of VIT after *patch embedding* is $\bm{x}_{e}\in\mathbb{R}^{L_{e}\times d}$, where $L_{e}$ is the token length. Specifically, VPT [, ] prepend a set of learnable tokens $\bm{p}\in\mathbb{R}^{L_{p}\times d}$ to $x_{e}$ and treat $\left[\bm{p},\bm{x}_{e}\right]\in\mathbb{R}^{\left(L_{e}+L_{p}\right)\times d}$ as the input, minimizing $\mathcal{L}$ to encode task-specific knowledge into these prompts while keeping pre-trained weights frozen. PCL involves two stages: *prompt learning* and *prompt retrieval*. In *prompt learning*, PCL grows the prompt sets pool $\mathcal{P}$ by initializing a new set of prompt $(\bm{p}_{i},\bm{k}_{i})$ before learning each new task $i$, where $\bm{p}_{i}$ is combined with the training samples by the *attention mechanism*. Meanwhile, $\bm{k}_{i}$ is optimized by being pulled closer to the vanilla features of the training samples obtained by a VIT without combining with prompts. In *prompt retrieval*, $\bm{k}_{i}$ serves as the query vector for predicting which set of $\bm{p}_{i}$ to choose for each testing sample by a matching mechanism. More details are in Appendix [C](#A3).

## 4 Theory and Method

In this section, we first present a theoretical analysis of GPCL concerning the hindrance on learning new tasks under the *orthogonal condition* (Theorem [1](#Thmtheorem1) and Definition [1](#Thmdefinition1)). Subsequently, as illustrated in Figure [2](#S4.F2), we introduce the plug-in module Learning Whether to Grow (LW2G), which consists of three components: DGA, CPK, and FFT.

### 4.1 Theoretical Analysis on Hindrance in GPCL

Figure: Figure 2: Illustration of three components in LW2G. Before learning task $3$, assume there are two sets in $\mathcal{P}=\left\{(\bm{p}_{1},\bm{k}_{1}),(\bm{p}_{2},\bm{k}_{2})\right\}$. In $\mathcal{P}$, blue represents frozen and unlearnable sets of prompts, whereas red represents learnable sets.
Refer to caption: /html/2409.18860/assets/x2.png

For simplicity, the notation of layer $l$ is omitted in the following analysis. While learning on task $i$, GPCL update the parameters under the *orthogonal condition* to avoid interfering with old knowledge. However, since the gradient represents the direction of local optimal descent for the loss function, modifying it inevitably results in a reduction of local information. To quantify the hindrance under the *orthogonal condition* in GPCL, we first define the following metric.

###### Definition 1 (Hinder Forward Capability, HFC) .

In GPCL, while continually encoding new knowledge into a single model under the orthogonal condition, Hinder Forward Capability (HFC) is defined to evaluate the hindrance on learning new tasks. HFC is the angle between the original gradient obtained through backpropagation $\bm{g}$ and its projection $\bm{g}^{\perp}=\text{Proj}_{\mathcal{S}^{\perp}_{old}}(\bm{g})$ onto $\mathcal{S}^{\perp}_{old}$,

$$ $\text{HFC}(\bm{g},\bm{g}^{\perp})=\arccos\left(\frac{\mathbf{g}\cdot\mathbf{g^{\perp}}}{\|\mathbf{g}\|\|\mathbf{g}^{\perp}\|}\right).$ $$

As illustrated in Figure [1](#S1.F1), a large HFC indicates a significant gap between original gradient $\bm{g}$ and the real gradient $\bm{g}^{\perp}$. Therefore, a large reduction of local information leads to greater hindrance on learning new tasks. Based on this, we formally present the following theorem (see Appendix [A](#A1) for a detailed proof):

###### Theorem 1 .

Given a space $\mathcal{S}_{1}=\text{span}\{\bm{B}_{1}\}$, where $\bm{B}_{1}=[\bm{b}_{1},\dots,\bm{b}_{l}]\in\mathbb{R}^{n\times l}$ is a set of $l$ bases for $\mathcal{S}_{1}$, and a space $\mathcal{S}_{2}=\text{span}\{\bm{B}_{2}\}$, where $\bm{B}_{2}=[\bm{b}_{1},\dots,\bm{b}_{l},\bm{b}_{l+1},\dots,\bm{b}_{l+k}]\in\mathbb{R}^{n\times(l+k)}$ is a set of $l+k$ bases for $\mathcal{S}_{2}$. Then, $\forall\bm{\alpha}$ there always exists:

$$ $\text{HFC}(\bm{\alpha},\text{Proj}_{\mathcal{S}_{1}}(\bm{\alpha}))>\text{HFC}(\bm{\alpha},\text{Proj}_{\mathcal{S}_{2}}(\bm{\alpha})).$ $$

The above Theorem [1](#Thmtheorem1) shows that fewer bases result in a larger HFC. As $\mathcal{S}_{old}$ in $\mathcal{M}$ continues to expand with new bases from each new task, its corresponding orthogonal complement $\mathcal{S}_{old}^{\perp}$ progressively shrinks. Consequently, the bases in $\mathcal{S}_{old}^{\perp}$ steadily decrease, leading to a large HFC and more severe hindrance on learning new tasks.

###### Definition 1 (Hinder Forward Capability, HFC) .

###### Theorem 1 .

### 4.2 Dynamic Growing Approach

Instead of naively growing a new set of prompts for each new task regardless of task dissimilarities, we propose a Dynamic Growing Approach (DGA). DGA involves dynamically learning whether *to grow* (initialize a new set of prompts and store it in the pool) or *not to grow* (utilize an existing set from the pool).

For simplicity, we adopt an example with three tasks to illustrate our method in Figure [2](#S4.F2). A more general description is presented in pseudocode, which can be found in Appendix [G](#A7).

Before learning task $3$, we first qualify the hindrance on each old set in the pool under the *orthogonal condition*. Specifically, we iteratively select an old set $(\bm{p}_{1},\bm{k}_{1})$ from $\mathcal{P}$ and $\mathcal{S}_{1}$ from $\mathcal{M}$, where $\mathcal{S}_{1}$ is the old feature space corresponding to task $1$. We construct a subset of training dataset from task $3$, denoted as $\mathcal{D}^{3}_{\text{sub}}$. For clarity, the gradient to update $(\bm{p}_{1},\bm{k}_{1})$ with $\mathcal{D}^{3}_{\text{sub}}$ is denoted as:

$$ $\displaystyle\bm{g}_{1}=\nabla_{(\bm{p}_{1},\bm{k}_{1})}\mathcal{L}_{3}(\mathcal{D}^{3}_{\text{sub}}).$ (4) $$

To prevent the influence of old knowledge contained in $(\bm{p}_{1},\bm{k}_{1})$ while learning task $3$, the gradient $\bm{g}_{1}$ is required to be modified to $\text{Proj}_{\mathcal{S}^{\perp}_{1}}(\bm{g}_{1})$, where $\mathcal{S}^{\perp}_{1}$ is the orthogonal complement of $\mathcal{S}_{1}$. Then, $\text{Proj}_{\mathcal{S}^{\perp}_{1}}(\bm{g}_{1})$ serves as the real gradient for updating parameters. Based on Theorem [1](#Thmtheorem1), we evaluate the hindrance under the *orthogonal condition* while learning task $3$ on $(\bm{p}_{1},\bm{k}_{1})$ as follows:

$$ $\displaystyle\text{HFC}_{1}=\text{HFC}(\bm{g}_{1},\text{Proj}_{\mathcal{S}^{\perp}_{1}}(\bm{g}_{1})).$ (5) $$

Besides, we define a dynamic threshold based on the task $3$ and the PTM being used. Firstly, we initialize a new set with $(\bm{p}_{1},\bm{k}_{1})$ as follows:

$$ $\displaystyle(\bm{p}_{3},\bm{k}_{3})\Leftarrow(\bm{p}_{1},\bm{k}_{1}).$ (6) $$

Likewise, the gradient to updated $(\bm{p}_{3},\bm{k}_{3})$ is denoted as:

$$ $\displaystyle\bm{g}_{3}=\nabla_{(\bm{p}_{3},\bm{k}_{3})}\mathcal{L}_{3}(\mathcal{D}^{3}_{\text{sub}}).$ (7) $$

Then, by feeding $\mathcal{D}^{3}_{\text{sub}}$ into the VIT without prompts, we can obtain a representation matrix $\bm{R}_{3}^{\text{pre}}$. We can newly build $\mathcal{S}_{3}^{\text{pre}}$ after performing SVD and $k$-rank approximation with pre-trained threshold, $\epsilon_{\text{pre}}$. Then, we can also calculate:

$$ $\displaystyle\text{HFC}_{1}^{\text{pre}}=\text{HFC}(\bm{g}_{3},\text{Proj}_{\mathcal{S}_{3}^{\text{pre},\perp}}(\bm{g}_{3})),$ (8) $$

where $\mathcal{S}_{3}^{\text{pre},\perp}$ is the orthogonal complement of $\mathcal{S}_{3}^{\text{pre}}$.
Here, $\text{HFC}_{1}^{\text{pre}}$ represents the relationship between the gradient of learning task $3$ and the pre-trained knowledge from task $3$. As $(\bm{p}_{3},\bm{k}_{3})$ is newly initialized specifically for training task $3$, it contains no prior knowledge, and thus, there are no obstacles from old tasks. Therefore, $\text{HFC}_{1}^{\text{pre}}$ signifies the ideal scenario when learning new tasks in PCL, which is the *dynamic threshold to evaluate the relative magnitude of hindrance*. Based on this, the gap between learning on old set $(\bm{p}_{1},\bm{k}_{1})$ under the *orthogonal condition* and leaning on new set $(\bm{p}_{3},\bm{k}_{3})$ in an ideal scenario is denoted as follows:

$$ $\displaystyle Z_{1}=\text{HFC}_{1}-\text{HFC}_{1}^{\text{pre}}.$ (9) $$

Thus, if $Z_{1}>0$, it indicates that learning on the old set $(\bm{p}_{1},\bm{k}_{1})$ from $\mathcal{P}$ encounters excessive hindrance.

Likewise, the gap between learning on old set $(\bm{p}_{2},\bm{k}_{2})$ under the *orthogonal condition* and leaning on new set $(\bm{p}_{3},\bm{k}_{3})$ in an ideal scenario can also be calculated as $Z_{2}$, where $(\bm{p}_{3},\bm{k}_{3})$ is a newly initialized set with $(\bm{p}_{2},\bm{k}_{2})$.

#### Opting To Grow or Not To Grow

Based on the analysis, we propose a dynamic growing approach as follows:

$$ $\displaystyle\left\{\begin{array}[]{ll}\emph{ \, \, \, \, To Grow}\,\,\,\,\,\quad\quad\text{if}&\underset{{m\in(1,2)}}{\min}Z_{m}>0\\ \emph{Not To Grow}\quad\quad\quad\text{else}&\underset{{m\in(1,2)}}{\min}Z_{m}\leq 0.\end{array}\right.$ (12) $$

($1$) While chosing To Grow, we initialize a new set $(\bm{p}_{3},\bm{k}_{3})$. Then, update $(\bm{p}_{3},\bm{k}_{3})$ with task $3$ and build a new feature space $\mathcal{S}_{3}$ with threshold, $\epsilon_{\text{task}}$, from task $3$ only and store $\mathcal{S}_{3}$ into $\mathcal{M}$.

($2$) While chosing Not To Grow, we select an old set $(\bm{p}_{t},\bm{k}_{t})$ from $\mathcal{P}$, where $t={\arg\min}_{{m\in(1,2)}}{Z_{m}}$. Then, update $(\bm{p}_{t},\bm{k}_{t})$ with task $3$ under *orthogonal condition* and update the old feature space $\mathcal{S}_{t}$ with threshold, $\epsilon_{\text{task}}$, with new bases from task $3$.

### 4.3 Consistency with Pre-trained Knowledge

Recent studies in transfer learning and domain adaptation revealed that when employing PEFT for fine-tuning PTM, the performance after fine-tuning often falls short of the pre-trained knowledge of PTM itself. However, this aspect has not been extensively studied in PCL.

Therefore, we exploit two distinct level of forgetting issues faced in PCL: ($1$) continuous fine-tuning on downstream tasks leading to the forgetting of pre-trained knowledge, and ($2$) continual learning on new tasks resulting in the forgetting of old tasks.

To tackle the former issue, we adjust the gradient of the new tasks to be orthogonal to the pre-trained feature space. However, due to the domain gap between the incremental task training data and the pre-trained data, a fully orthogonal manner is too stringent and can significantly impact the plasticity. To achieve a balance between maintaining plasticity and fully utilization of the pre-trained knowledge, we propose to apply a soft constraint to the gradient as follows:

$$ $\displaystyle\bm{g}=\bm{g}-(1-\phi)\text{Proj}_{\mathcal{S}^{\text{pre}}_{3}}(\bm{g}),$ (13) $$

where $\phi$ is the coefficient of the soft constraint to control the orthogonality and $\mathcal{S}_{3}^{\text{pre}}$ is the pre-trained feature space for task $3$. When learning on task $3$, the gradient can be obtained from Equation [4](#S4.E4) while DGA chooses to grow, or from Equation [7](#S4.E7) while DGA chooses not to grow. And $\phi$ can flexibly control the real gradient $\bm{g}$, aligning it as closely as possible with the feature space of the pre-trained knowledge, while ensuring the learning ability on new tasks.

### 4.4 Facilitation for Forward Transfer

To facilitate forward knowledge transfer during learning task $3$, we propose a simple yet effective method: *reusing the frozen weights of prompts* from $\mathcal{P}$. Specifically, before learning task $3$, we can characterize the correlation between the new task $3$ and the existing feature space in $\mathcal{M}$ with HFC metric. A larger HFC indicates more projection onto the old feature space $\mathcal{S}_{2}$ than $\mathcal{S}_{1}$, as illustrated in Figure [1](#S1.F1). Therefore, it indicates that task $3$ has higher similarity with task $2$ than task $1$. Consequently, naturally reusing the set of prompts corresponding to task $2$ can effectively facilitate the learning of task $3$.

$$ $\displaystyle\bm{p}_{i}^{*}=\left[\bm{p},\text{stg}(\bm{p}_{\mathcal{K}})\right],$ (14) $$

where $\text{stg}(\cdot)$ means *stop gradient* to frozen the $\bm{p}_{\mathcal{K}}$. Besides, $\bm{p}$ is a newly initialized set of prompts when DGA chooses *to grow* or an old set of prompts from $\mathcal{P}$ when DGA chooses *not to grow*. And $\bm{p}_{\mathcal{K}}$ is obtained as follows:

$$ $\displaystyle\mathcal{K}=\underset{\left\{u_{i}\right\}_{i=1}^{N}\in\left\{1,2\right\}}{\arg\max}{\text{HFC}(\bm{g}_{u_{i}},\text{Proj}_{\mathcal{S}_{u_{i}}}(\bm{g}_{u_{i}}))},$ (15) $$

where $\mathcal{K}$ represents a subset of sets with top-$N$ from $\mathcal{P}$.

**Table 1: Results on three typical settings in Class-CL. We compare LW2G with three baselines: DualPrompt, S-Prompt++, and HidePrompt(^2^22Results are reproduced after fixing a typo in the code. For details, refer to [https://github.com/RAIAN08/LW2G](https://github.com/RAIAN08/LW2G).). All results are the average under three different random seeds. The best results are highlighted in bold.**
| Settings | Methods | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ | SSP $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR_INC10_TASK10 | DualPrompt | 85.94 | 59.44 | 6.38 | 10 |
| DualPrompt [+ LW2G] | 86.86 | 78.33 | 6.03 | 2 |  |
| S-Prompt++ | 89.25 | 99.52 | 4.10 | 10 |  |
| S-Prompt++ [+ LW2G] | 89.32 | 100.0 | 3.46 | 7 |  |
| HidePrompt | 85.77 | 80.78 | 6.19 | 10 |  |
| HidePrompt [+ LW2G] | 87.60 | 95.39 | 4.28 | 2 |  |
| IMR_INC20_TASK10 | DualPrompt | 63.63 | 41.05 | 6.41 | 10 |
| DualPrompt [+ LW2G] | 65.60 | 80.40 | 5.72 | 2 |  |
| S-Prompt++ | 63.26 | 44.31 | 6.22 | 10 |  |
| S-Prompt++ [+ LW2G] | 65.44 | 79.35 | 6.01 | 5 |  |
| HidePrompt | 62.42 | 62.07 | 8.89 | 10 |  |
| HidePrompt [+ LW2G] | 63.23 | 65.13 | 7.19 | 6 |  |
| CUB_INC20_TASK10 | DualPrompt | 82.09 | 66.71 | 6.40 | 10 |
| DualPrompt [+ LW2G] | 82.43 | 70.09 | 5.25 | 7 |  |
| S-Prompt++ | 82.57 | 66.30 | 4.85 | 10 |  |
| S-Prompt++ [+ LW2G] | 82.61 | 87.49 | 4.54 | 3 |  |
| HidePrompt | 85.59 | 88.58 | 3.22 | 10 |  |
| HidePrompt [+ LW2G] | 86.17 | 92.53 | 3.08 | 4 |  |

**Table 2: Results on OMNI benchmark with two extreme settings: 30 tasks and 60 tasks. We present the FFA, PRA and FFM for performance evaluation. Additionally, we provide SSP, FLOPS and Training Time (TT) to measure the computational overhead and methods’ complexity.**
| Settings | Methods | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ | SSP $(\downarrow)$ | FLOPS (G) $(\downarrow)$ | TT (h) $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OMNI_INC10_TASK30 | DualPrompt | 63.36 | 68.47 | 12.92 | 30 | 35.19 | 4.5 |
| DualPrompt [+ LW2G] | 65.12 | 80.95 | 10.75 | 9 | 37.21 | 5.0 |  |
| S-Prompt++ | 64.44 | 55.87 | 9.02 | 30 | 35.17 | 4.5 |  |
| S-Prompt++ [+ LW2G] | 65.90 | 63.86 | 8.50 | 10 | 37.24 | 5.2 |  |
| OMNI_INC5_TASK60 | DualPrompt | 61.85 | 69.94 | 13.50 | 60 | 35.19 | 5.0 |
| DualPrompt [+ LW2G] | 63.17 | 75.31 | 12.01 | 17 | 37.21 | 6.1 |  |
| S-Prompt++ | 62.31 | 54.59 | 10.04 | 60 | 35.17 | 5.1 |  |
| S-Prompt++ [+ LW2G] | 63.70 | 62.60 | 9.90 | 18 | 37.24 | 6.2 |  |

## 5 Experiment

In this section, we first describe the experimental setups, and then present the experimental results.

### 5.1 Experimental Setups

#### Benchmarks

We evaluate our method on multiple datasets against state-of-the-art baselines. Specifically, we use the following datasets: CIFAR100 [] (CIFAR), which contains 100 classes with 100 images per class; CUB200 [] (CUB), which consists of 11,788 images across 200 birds classes; ImageNet-R [] (IMR), which includes 30,000 images from 200 classes that pose challenges for PTMs pre-trained on ImageNet; and Omnibenchmark [] (OMNI), which comprises over 90,000 images from 300 classes. Besides, we denote different experimental settings as ‘Dataset_IncN_TaskM’, e.g., ‘CIFAR_INC10_Task10’, which means learning on CIFAR with 10 tasks and each task contains 10 classes.

#### Baselines

We use DualPrompt [], S-Prompt++ [] and HidePrompt [] as our baselines for Class-CL. Following [], we record the average accuracy of all encountered classes after learning on each task, presenting the last one as the Final Average Accuracy (FAA). We also present the Final Forgetting Measure (FFM) of all tasks and Prompt Retrieval Accuracy (PRA) to measure the accuracy during *prompt retrieval*. Additionally, Selectable Sets of Prompts (SSP) is also provided to demonstrate the amount of sets in $\mathcal{P}$. Please refer to Appendix [E.1](#A5.SS1) for more details.

#### Implementations

Our LW2G needs to set the value of four hyperparameters: $\epsilon_{\text{task}}$, $\epsilon_{\text{pre}}$, $\phi$, and $N$. Details on different benchmarks are provided in Appendix [E.2](#A5.SS2). We use VIT pretrained on ImageNet-21K for all experiments. Furthermore, as the pre-trained feature space is built from PTM, we further validate the effectiveness of LW2G under other PTMs. Results are provided in Appendix [F.6](#A6.SS6).

### 5.2 Main Results

#### Typical Settings

Table [2](#footnote2) presents the results of applying different state-of-the-art PCL methods and incorporating LW2G. We report four metrics FFA, PRA, FFM and SSP, where FFA and FFM are the typical metrics in CL to evaluate the performance. Additionally, PRA and SSP are unique for PCL. LW2G outperforms existing PCL by a large margin in each setting. For IMR, LW2G is better than DualPrompt, S-Prompt++ and Hideprompt by 1.97%, 2.17% and 0.81%, respectively on FFA. For CIFAR, it appears that LW2G brings a significant decent in anti-forgetting, especially comparing with S-Prompt++ and Hideprompt on FFM. As for the PCL unique metrics PRA and SSP, LW2G leads to notable improvements in PRA for all three baselines, with the largest improvement reaching up to 39.35%. Additionally, it also results in a substantial reduction in SSP. For example, DualPrompt combined with LW2G on CIFAR only requires $2$ sets of prompts compared to the original DualPrompt, which utilizes 10 sets. The same reduction in parameters can be observed across multiple settings.

#### Long Task Settings

Learning in the context of long sequential tasks has long been regarded as a more challenging setting in CL. We showcase the performance of DualPrompt and S-Prompt++ on two extreme settings: OMNI_INC10_TASK30 and OMNI_INC5_TASK60 in Table [2](#S4.T2). Existing baselines employ a pool with the size equivalent to the length of tasks, resulting in poor performance on PRA. However, incorporating the LW2G significantly enhances PRA, leading to noticeable improvements in both FFA and FFM. Moreover, we observe that LW2G requires to maitain a memory $\mathcal{M}$ for gradient modification, unavoidably introducing additional computational overhead and lengthening training time. Nevertheless, the results indicate that the extra cost compared to baselines is relatively modest. Additionally, we find that the adoption of LW2G results in a substantial decrease in the total amount of selectable sets, approximately by 70%.

### 5.3 Ablation Study

We conduct an extensive ablation study presented in Table [3](#S5.T3) to validate the effectiveness of the three components in LW2G. Initially, we construct DualPrompt and S-Prompt++ as baselines and progressively incorporate the DGA, CPK, and FFT. Overall, optimizing each component yields clear benefits, with all contributing to the robust gains of LW2G. Interestingly, while CPK and FFT exhibits less pronounced improvements compared to the baseline, the enhancement from DGA is more significant. Besides, the combination of all three components provides the optimal performance, suggesting highly synergistic and complementary effects rather than operating in isolation. Moreover, it is noteworthy that CPK and FFT do not reduce SSP, hence the performance improvement solely stemmed from the enhanced representational capacity of prompts. DGA not only integrates knowledge from multiple tasks into a single set of prompts, thereby enhancing the representational capacity, but importantly, the notable improvement in PRA is attributed to the reduction in the total amount of available sets during *prompt retrieval*, thereby aiding PCL performance.

**Table 3: Ablation study on three components in AutoPrompt. Here we present FFA and PRA for all baselines and variants in LW2G, e.g., “DGA” refers to the use of Dynamic Growing Approach within the baseline methods, DualPrompt and S-Prompt++.**
| Variants | FFA $(\uparrow)$ | PRA $(\uparrow)$ | Variants | FFA $(\uparrow)$ | PRA $(\uparrow)$ |
| --- | --- | --- | --- | --- | --- |
| DualPrompt (baseline) | 63.63 | 41.05 | S-Prompt++ (baseline) | 63.26 | 44.31 |
| DualPrompt [+ DGA] | 65.02 | 77.68 | S-Prompt++ [+ DGA] | 65.18 | 76.35 |
| DualPrompt [+ CPK] | 64.34 | 50.39 | S-Prompt++ [+ CPK] | 63.90 | 52.67 |
| DualPrompt [+ FFT] | 64.08 | 47.17 | S-Prompt++ [+ FFT] | 63.89 | 50.02 |
| DualPrompt [+ LW2G] | 65.60 | 80.40 | S-Prompt++ [+ LW2G] | 65.44 | 79.35 |

Figure: Figure 3: The x-axis denotes the enhancement in PRA with LW2G compared to the baseline. Apart from baseline and LW2G, we also present the results of Task-CL. Task-CL ensures the real upper bound of PCL by providing a correct prompt set for each testing sample through a given task ID.
Refer to caption: /html/2409.18860/assets/x3.png

### 5.4 Detail Analysis

#### Gains on each Task

Figure [3](#S5.F3) presents detailed accuracy on each task. Here, we provide a comparison between DualPrompt and S-Prompt++ on two benchmarks. The x-axis of each plot represents the change from *baseline* to *baseline+LW2G* in terms of PRA. Apart from ($c$), the addition of LW2G all leads to consistent improvements in accuracy on each task, as the PRA of the baseline method in ($c$) has already reached 99.52%. In the other three settings, PRA experiences significant increasment, thereby enhancing classification accuracy. Additionally, we also provide results for *baseline+taskID*, i.e., PCL on Task-CL. In this setting, during inference, taskid is provided to select the correct set for each testing sample, which is considered as the upper bound of PCL. It further demonstrates that our proposed LW2G can effectively reduce the optionality during *prompt retrieval* while ensuring the integration of old and new knowledge, thereby improving performance.

#### Effectiveness of DGA

**Table 4: Different implementations on DGA. Here we present FFA for all variants.**
| DGA Variants | CIFAR | IMR |  |  |
| --- | --- | --- | --- | --- |
| DualPrompt | S-Prompt++ | DualPrompt | S-Prompt++ |  |
| No-DGA (Baseline) | 85.94 | 89.25 | 63.63 | 63.26 |
| DGA-Rand | 85.99 | 88.32 | 64.82 | 64.76 |
| DGA-AG | 84.78 | 85.17 | 63.73 | 63.43 |
| DGA-Max HFC | 86.08 | 86.73 | 64.31 | 63.91 |
| DGA-Min HFC | 86.86 | 89.32 | 65.60 | 65.44 |

While chosing *not to grow*, DGA utilized in LW2G selects the set $(\bm{p}_{*},\bm{k}_{*})$ with the Min-$Z$ from $\mathcal{P}$ when learning task $i$, and learns new knowledge based on this set, adjusting gradient to prevent forgetting of the old knowledge contained in $(\bm{p}_{*},\bm{k}_{*})$. After learning, $(\bm{p},\bm{k})$ encompasses both the new knowledge from task $i$ and the existing old knowledge. Here, we explore the impact of different implementations of DGA on FFA. In Table [4](#S5.T4), No-DGA represents baseline methods, e.g., S-Prompt++ and DualPrompt. DGA-Rand represents randomly selecting an old set of prompts from $\mathcal{P}$. DGA-AG represents that $\mathcal{P}$ consists of only a single set, implying continuous learning of new knowledge on this set of parameters. DGA-Max HFC indicates selecting the set from $\mathcal{P}$ with the maximum HFC value. The results clearly demonstrate the superiority of DGA-Min HFC employed in LW2G over other variants, aligning with the conclusion in Theorem [1](#Thmtheorem1).

## 6 Conclusion

In this paper, we propose a plug-in module within existing Prompt-based Continual Learning (PCL), called Learning Whether To Grow (LW2G). Specifically, LW2G enables PCL to dynamically learn to whether to add a new set of prompts for each task (*to grow*) or to utilize an existing set of prompts (*not to grow*) based on the relationships between tasks. Inspired by Gradient Projection-based Continual Learning (GPCL), we utilize the *orthogonal condition* to form an effective and efficient prompt sets pool. Besides, we also provide a theoretical analysis on hindrance under the *orthogonal condition* in GPCL. Extensive experiments show the effectiveness of our method.

#### Limitations

LW2G needs to construct the feature space of old tasks and store it in memory $\mathcal{M}$ for gradient projection, which results in additional computational overhead. Therefore, exploring alternative methods for constructing the old feature space is crucial for improving the practicality of both LW2G and GPCL.

## Appendix A Proof of Theorem 1

Given a space $\mathcal{S}_{1}=\text{span}\{\bm{B}_{1}\}$, where $\bm{B}_{1}=[\bm{b}_{1},\dots,\bm{b}_{l}]\in\mathbb{R}^{n\times l}$ is a set of $l$ bases for $\mathcal{S}_{1}$, and a space $\mathcal{S}_{2}=\text{span}\{\bm{B}_{2}\}$, where $\bm{B}_{2}=[\bm{b}_{1},\dots,\bm{b}_{l},\bm{b}_{l+1},\dots,\bm{b}_{k}]\in\mathbb{R}^{n\times\left(l+k\right)}$ is a set of $l+k$ bases for $\mathcal{S}_{2}$. $\forall\bm{\alpha}\in\mathbb{R}^{n\times 1}$, denoted $\bm{\alpha}$ on space $\mathcal{S}_{i}$ is $\text{Proj}_{\mathcal{S}_{i}}(\bm{\alpha})$. Following Definition [1](#Thmdefinition1), the ange between $\bm{\alpha}$ and $\text{Proj}_{\mathcal{S}_{i}}(\bm{\alpha})$ is denoted as $\text{HFC}(\bm{\alpha},\text{Proj}_{\mathcal{S}_{i}}(\bm{\alpha}))$. Then there always exists:

$$ $\displaystyle\text{HFC}(\bm{\alpha},\text{Proj}_{\mathcal{S}_{1}}(\bm{\alpha}))\geq\text{HFC}(\bm{\alpha},\text{Proj}_{\mathcal{S}_{2}}(\bm{\alpha})).$ (16) $$

Proof. $\forall\bm{\alpha}\in\mathbb{R}^{n\times 1}$, $\bm{\alpha}=\left[\alpha_{1},\dots,\alpha_{n}\right]^{T}$. Without loss of generality, $\left\{\bm{b}_{i},i=1,\dots,k\right\}$ is a set of *standard orthonormal basis*. As we defined, $\text{Proj}_{\mathcal{S}_{1}}(\bm{\alpha})=\left[g_{1},\dots,g_{l}\right]\in\mathbb{R}^{l\times 1}$ and $\text{Proj}_{\mathcal{S}_{2}}(\bm{\alpha})=\left[g_{1},\dots,g_{l},g_{l+1},\dots,g_{l+k}\right]\in\mathbb{R}^{\left(l+k\right)\times 1}$, where $g_{i}=\langle\bm{\alpha},\bm{b_{i}}\rangle$.

Then, we have

$$ $\displaystyle\begin{split}cos(\bm{\alpha},\text{Proj}_{\mathcal{S}_{1}}(\bm{\alpha}))&=\frac{\bm{\alpha}\cdot\text{Proj}_{\bm{S}_{1}}(\bm{\alpha})}{\|\bm{\alpha}\|\|\text{Proj}_{\bm{S}_{1}}(\bm{\alpha})\|}\\ &=\frac{\sum_{i=1}^{l}\left(g_{i}\right)^{2}}{\sqrt{\sum_{i=1}^{l}\left(g_{i}\right)^{2}}\sqrt{\sum_{i=1}^{n}\left(g_{i}\right)^{2}}}\end{split}$ (17) $$

Likewise, we have

$$ $\displaystyle\begin{split}cos(\bm{\alpha},\text{Proj}_{\bm{S}_{2}}(\bm{\alpha}))&=\frac{\bm{\alpha}\cdot\text{Proj}_{\bm{S}_{2}}(\bm{\alpha})}{\|\bm{\alpha}\|\|\text{Proj}_{\bm{S}_{2}}(\bm{\alpha})\|}\\ &=\frac{\sum_{i=1}^{l+k}\left(g_{i}\right)^{2}}{\sqrt{\sum_{i=1}^{l+k}\left(g_{i}\right)^{2}}\sqrt{\sum_{i=1}^{n}\left(g_{i}\right)^{2}}}\end{split}$ (18) $$

In addition,

$$ $\displaystyle\begin{split}\frac{cos(\bm{\alpha},\text{Proj}_{\bm{S}_{2}}(\bm{\alpha}))}{cos(\bm{\alpha},\text{Proj}_{\bm{S}_{1}}(\bm{\alpha}))}&=\frac{\sum_{i=1}^{l+k}\left(g_{i}\right)^{2}}{\sum_{i=1}^{l}\left(g_{i}\right)^{2}}\frac{\sqrt{\sum_{i=1}^{l}\left(g_{i}\right)^{2}}}{\sqrt{\sum_{i=1}^{l+k}\left(g_{i}\right)^{2}}}\end{split}$ (19) $\displaystyle=\frac{1+C}{\sqrt{(1+C)}}$ (20) $\displaystyle=\sqrt{(1+C)}\geq 1.$ (21) $$

Where $C=\frac{\sum_{i=l+1}^{l+k}\left(g_{i}\right)^{2}}{\sum_{i=1}^{l}\left(g_{i}\right)^{2}}\geq 0$. Thus, $cos(\bm{\alpha},\text{Proj}_{\bm{S}_{2}}(\bm{\alpha}))\geq cos(\bm{\alpha},\text{Proj}_{\bm{S}_{1}}(\bm{\alpha})).$ Thus,
$\text{HFC}(\bm{\alpha},\text{Proj}_{\bm{S}_{1}}(\bm{\alpha}))\geq\text{HFC}(\bm{\alpha},\text{Proj}_{\bm{S}_{2}}(\bm{\alpha}))$.

This finishes the proof.

## Appendix B Building and Updating of Feature Space

In GPCL, a feature space spanned by the old tasks is required during gradient modification, involving two stages: ($1$) Building of the new feature space, and ($2$) Updating of old faeture space. We first introduce the technique used in matrix factorization, Singular Value Decomposition (SVD). Then, details on building or updating of the feature space are also provided.

#### Singular Value Decomposition (SVD)

SVD is a general geometrical tool used in matrix factorization to factorize a given matrix $\bm{A}\in\mathbb{R}^{m\times n}$ into the product of three matrices as follows []:

$$ $\displaystyle\bm{A}=\bm{U}\bm{\Sigma}(\bm{V})^{T},$ (22) $$

where $\bm{U}\in\mathbb{R}^{m\times m}$ and $\bm{V}\in\mathbb{R}^{n\times n}$ are orthogonal. $\bm{\Sigma}\in\mathbb{R}^{m\times n}$ contains the sorted singular values along its main diagonal. Specifically, the diagonal value $\sigma_{i}=\bm{\Sigma}_{ii}$ are the *singular values* of $\bm{A}$ and the number of non-zero $\sigma_{i}$ is equal to $r=\text{rank}({\bm{A}})$. Besides, the columns of $\bm{U}$ and the rows of $(\bm{V})^{T}$ are two sets of orthogonal bases $\left\{\bm{u}_{1},\bm{u}_{2},\dots,\bm{u}_{m}\right\}$ and $\left\{\bm{v}_{1},\bm{v}_{2},\dots,\bm{v}_{n}\right\}$, respectively. As the singular values are sorted in $\bm{\Sigma}$ along its diagonal, the SVD of $\bm{A}$ can be also denoted as follows:

$$ $\displaystyle\bm{A}=\sum\limits_{i=1}^{r}\sigma_{i}\bm{u}_{i}\bm{v}^{\prime}_{i}.$ (23) $$

Therefore, the $k$-rank approximation $(\bm{A})_{k}$ of $\bm{A}$ can be denoted as follows:

$$ $\displaystyle||(\bm{A})_{k}||_{F}^{2}\geq\epsilon||\bm{A}||_{F}^{2},$ (24) $$

where $\epsilon$ is a given error tolerance and $||\cdot||_{F}^{2}$ is the Frobenius norm.

#### Building of the New Feature Space

After training on task $1$, for each layer we construct a representation matrix $\bm{R}_{1}^{l}=\left[\bm{x}^{l}_{1,1},\dots,\bm{x}^{l}_{1,n_{1}}\right]\in\mathbb{R}^{n\times d}$ by concatenating representations of $n$ samples along the columns obtained from sending $n$ samples only from task $1$ into the current DNN, $\mathcal{W}_{1}$. Next, we perform SVD on $\bm{R}_{1}^{l}=\bm{U}_{1}^{l}\bm{\Sigma}_{1}^{l}(\bm{V}_{1}^{l})^{T}$ followed by its $k$-rank approximation $(\bm{R}_{1}^{l})_{k}$ according to the following criteria for the given threshold, $\epsilon_{\text{task}}$:

$$ $\displaystyle||(\bm{R}_{1}^{l})_{k}||_{F}^{2}\geq\epsilon_{\text{task}}||\bm{R}_{1}^{l}||_{F}^{2}.$ (25) $$

Therefore, the feature space for layer $l$ is built by $\mathcal{S}_{1}^{l}=\text{span}\left\{\bm{B}_{1}^{l}\right\}$, where $\bm{B}_{1}^{l}=\left\{\bm{u}_{1}^{l},\dots,\bm{u}_{k}^{l}\right\}$ and $\bm{u}_{i}^{l}$ is the first $k$ vectors in $\bm{U}_{1}^{l}$. And $\mathcal{S}_{1}^{l}$ is stored in memory $\mathcal{M}=\left\{\mathcal{S}_{1}^{l}\right\}$.

#### Updating of the Old Feature Space

After learning task $i$, where $i\geq 2$, $\mathcal{S}_{i-1}^{l}$ in $\mathcal{M}$ needs to be updated to $\mathcal{S}_{i}^{l}$ with new task-specific bases from task $i$. To obtain such bases, for each layer $l$, we utilize the current DNN, $\mathcal{W}_{i}$, to construct a representation matrix $\bm{R}_{i}^{l}=\left[\bm{x}_{1,1}^{l},\dots,\bm{x}_{1,n}^{l}\right]\in\mathbb{R}^{n\times d}$ from task $i$ only. Before performing SVD and subsequent $k$-rank approximation, we first eliminate the common bases that already present in $\mathcal{S}_{i-1}^{l}$ so that newly added bases are unique and orthogonal to the existing bases in $\mathcal{S}_{i-1}^{l}$. To accomplish this, we proceed as follows:

$$ $\displaystyle{\hat{\bm{R}}}_{i}^{l}=\bm{R}_{i}^{l}-\bm{B}_{i-1}^{l}\left(\bm{B}_{i-1}^{l}\right)^{T}\left(\bm{R}_{i}^{l}\right)=\bm{R}_{i}^{l}-\bm{R}_{i,\text{proj}}^{l}.$ (26) $$

Afterwards, SVD is performed on ${\hat{\bm{R}}}_{i}^{l}=\hat{\bm{U}}_{i}^{l}\hat{\bm{\Sigma}}_{i}^{l}(\hat{\bm{V}}_{i}^{l})^{T}$, thus obtaining $h$ new orthogonal bases for minimun value of $h$ statisfying the following criteria for the given threshold, $\epsilon_{\text{task}}$:

$$ $\displaystyle||\bm{R}_{i,\text{proj}}^{l}||_{F}^{2}+||{\hat{\bm{R}}}_{i}^{l}||_{F}^{2}\geq\epsilon_{\text{task}}||\bm{R}_{i}^{l}||_{F}^{2}.$ (27) $$

$\bm{B}_{i-1}^{l}$ is then updated to $\bm{B}_{i}^{l}=\left[\bm{B}_{i-1}^{l},\bm{u}_{1}^{l},\dots,\bm{u}_{h}^{l}\right]$ with $h$ new bases. And $\mathcal{S}_{i-1}^{l}$ is updated to $\mathcal{S}_{i}^{l}=\text{span}\left\{\bm{B}_{i}^{l}\right\}$.

## Appendix C Review of Existing PCL

In this section, we review existing PCL with its pipeline. As illustrated in Figure [4](#A3.F4), existing PCL such as HidePrompt [], S-Prompt++ [], DualPrompt [], L2P [], S-liPrompt, and S-iPrompt [] generally involves two stages: ($1$) *prompt learning*, and ($2$) *prompt retrieval*.

#### Prompt Learning

Given a pre-trained model, such as a Vision Transformer (denoted as VIT), an image after *patch embedding* is denoted as $\bm{x}_{e}\in\mathbb{R}^{\mathcal{L}_{e}\times d}$, where $\mathcal{L}_{e}$ is the length of the patch tokens and $d$ denotes the length of the channels. Before learning task $i$, PCL follows [, ] by utilizing a task-wised set of prompts $\bm{p}_{i}\in\mathbb{R}^{\mathcal{L}_{p}\times\mathcal{L}_{b}\times d}$, where $\mathcal{L}_{p}$ is the length of layer-wised prompts and $\mathcal{L}_{b}$ represents the depth of the blocks into which the prompts is inserted. The new knowledge in task $i$ can be encoded into these newly initialized $\bm{p}_{i}$ as follows:

$$ $\displaystyle\left[\text{cls_token}^{l},\bm{x}_{e}^{l},\bm{p}^{l}\right]$ $\displaystyle=\text{block}^{l}(\left[\text{cls_token}^{l-1},\bm{x}_{e}^{l-1},\bm{p}_{i}^{l-1}\right])\quad\quad l=1,2,\dots,N$ (28) $\displaystyle\bm{y}$ $\displaystyle=\text{Head}^{i}(\text{cls_token}^{N}).$ (29) $$

Here, $\bm{p}_{i}^{l-1}\in\mathbb{R}^{\mathcal{L}_{p}\times d}$ represents the prompts for block $l$. $\bm{x}_{e}^{l-1}$ is the original input of block $l$. Additionally, $\text{Head}^{i}$ represents the classifier head corresponding to task $i$. Since PCL typically considers Class-CL scenarios, a unified classifier head is adopted. This means that while learning task $i$, the weights of the unified classifier head from tasks $1$ to $i-1$ are frozen. Then, $\bm{p}_{i}$ is optimized using the *cross entropy* loss.

Meanwhile, PCL sent $\bm{x}_{e}\in\mathbb{R}^{\mathcal{L}_{e}\times d}$ into the VIT without any prompts as follows:

$$ $\displaystyle\left[\text{cls_token}^{l},\bm{x}_{e}^{l}\right]$ $\displaystyle=\text{block}^{i}(\left[\text{cls_token}^{l-1},\bm{x}_{e}^{l-1}\right])\quad\quad l=1,2,\dots,N.$ (30) $$

Here, we use $\bm{q}=\text{cls_token}^{N}$ from the output of the last block as the valinia feature of the input sample. Then, $\bm{k}_{i}$ is optimized by minimizing the distance between $\bm{q}$ and $\bm{k}_{i}$. There are various methods to measure this distance, such as using cosine similarity as in S-Prompt++ [], DualPrompt [], and L2P []; using KNN in S-liPrompt and S-iPrompt []; or, in the case of HidePrompt [], forgoing
$\bm{k}_{i}$ and instead utilizing an auxiliary classifier head. Overall, the goal is to design a metric that brings $\bm{k}_{i}$ closer to $\bm{q}$, so that during *prompt retrieval*, the correct $\bm{p}_{i}$ can be selected for each testing sample.

After learning task $i$, PCL stores $(\bm{p}_{i},\bm{k}_{i})$ as a pair into the pool $\mathcal{P}=\left\{(\bm{p}_{i},\bm{k}_{i}),i=1,2,\dots\right\}$.

#### Prompt Retrieval

In Class-CL, we do not have access to the task ID. Therefore, given a testing sample, PCL needs to predict which task it belongs to and select the corresponding set from the pool $\mathcal{P}$. Briefly, they first obtain the vanilla feature by sending the testing sample into the VIT without prompts. Then, they use the vanilla feature as a query vector to match $\left\{\bm{k}_{i},i=1,2,\dots\right\}$ in the pool $\mathcal{P}$ through the metric used in *prompt learning*. After selecting the $\bm{k}_{x}$, the $\bm{p}_{x}$ is combined with $\bm{x}_{e}$ for further inference.

Therefore, predicting the *ground truth* set of prompts for each testing sample is a crucial step for PCL, enabling it to achieve appealing performance.

Figure: Figure 4: Pipline of existing PCL. Here, we separate it into two stages: *prompt learning* and *prompt retrieval*. In $\mathcal{P}$, blue represents frozen and unlearnable set of prompts, whereas red represents learnable prompt sets.
Refer to caption: /html/2409.18860/assets/x4.png

## Appendix D A Typo in PyTorch Implementation Code

For compared methods, we adopt the PyTorch implementation (^3^33Pytorch Implementation: https://github.com/JH-LEE-KR/dualprompt-pytorch) of DualPrompt [] and L2P []. However, comparing with the JAX implementation (^4^44Official Implementation: https://github.com/google-research/l2p), we discovered a serious error in PyTorch implementation. Listing [1](#LST1) is a part of the PyTorch reimplementation that initializes the pool. It can be seen that *prefix tuning* is used here, so the size of the pool initializes a new set of prompts for each task, which need to be prepended to the Q value and K value in *attention mechanism*, respectively. Lines 5 and 7 in Listing [2](#LST2) contain errors. Reshaping batched_prompt_raw with a shape of $\left[num_layers,dual,{batch_size},\dots\right]$ to $\left[{num_layers},{batch_size},{dual},\dots\right]$ is incorrect as it causes confusion between the parts that should be prepended to the Q value and K value. This issue also arises during *prompt retrieval*. The correct procedures are provided in lines 5, 7, and 9 of Listing [3](#LST3). Besides, we provide a floatmap to further illustrate the code typo from ‘reshape‘ instead of ‘permute‘ in Figure [5](#A4.F5).

Figure: Listing 1: 1

Figure: Listing 2: 1

Figure: Listing 3: 1

Figure: Figure 5: A floatmap shows the difference between the original code and the corrected code.
Refer to caption: /html/2409.18860/assets/x5.png

## Appendix E Implementation Details

In this section, we provide the implementation details of all experiments.

### E.1 Evaluation Metrics

We utilize four evaluation metrics for PCL, including the Final Average Accuracy (FAA), Final Forgetting Measure (FFM), Prompt Retrieval Accuracy (PRA) and Selectable Sets of Prompts (SSP).

FAA and FFM are common evaluation metrics in Continual Learning and are formally defined as follows:

$$ FAA $\displaystyle=\frac{1}{T}\sum_{i=1}^{T}A_{i,T},$ (31) FFM $\displaystyle=\frac{1}{T-1}\sum_{i=1}^{T-1}\max_{t\in\left\{1,\dots,T-1\right\}}(A_{i,t}-A_{i,T}),$ (32) $$

where $T$ is the length of the sequential tasks, $A_{i,T}$ is the classification accuracy on the task $i$ after learning the last task $T$.

As analyzed in Appendix [C](#A3), predicting the *ground truth* set of prompts for each testing sample is a crucial step in PCL. Therefore, we adopt a unique evaluation metric, Prompt Retrieval Accuracy (PRA), for PCL, which is formally defined as follows:

$$ PRA $\displaystyle=\frac{1}{T}\sum_{i=1}^{T}R_{i,T},$ (33) $$

where $R_{i,T}$ is the accuracy of predicting the set of prompts for each testing sample on task $i$ after learning the last task $T$. Besides, we also use Selectable Sets of Prompt (SSP) to represent the total amount of selectable sets of prompts in the pool $\mathcal{P}$. SSP is not only positively correlated with the number of learnable parameters, but it also effectively reflects how the LW2G proposed in this paper can significantly reduce the selectable amount in baseline methods, thereby benefiting PRA.

### E.2 Training Regime and Hyperparameters

Following the implementations of previous work [], we train DualPrompt on CIFAR, IMR and CUB with 40, 50, and 50 epochs, respectively; Hideprompt on CIFAR, IMR and CUB with 50, 150, and 50 epochs, respectively; S-Prompt++ on CIFAR, IMR and CUB with 40, 120, and 40 epochs, respectively. The length of prompts $\mathcal{L}_{e}$ is 20 for all settings. Depth of prompts are as follows: In DualPrompt: g-prompts are inserted in the block $0-1$ and e-prompts are inserted in the block $2-4$. In HidePrompt and S-Prompt++ prompts are inserted in the block $0-4$. All the experimental results in this paper are averaged over five trials with five different random seeds. We use 1 4090 GPU for experiments in typical setting and 1 A800 GPU for experiments in long task settings.

For LW2G, the detailed settings for $\epsilon_{\text{task}}$, $\epsilon_{\text{pre}}$, $\phi$, and $N$ are illustrated in Table [5](#A5.T5).

**Table 5: Hyperparameters of $\epsilon_{\text{task}}$, $\epsilon_{\text{pre}}$, $\phi$, and $N$ in typical settings.**
| Settings | Methods | $\epsilon_{\text{task}}$ | $\epsilon_{\text{pre}}$ | $\phi$ | $N$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR_INC10_TASK10 | DualPrompt | 0.95 | 0.95 | 0.5 | 1 |
| S-Prompt++ | 0.95 | 0.95 | 1.0 | 1 |  |
| HidePrompt | 0.99 | 0.99 | 0.5 | 1 |  |
| IMR_INC20_TASK10 | DualPrompt | 0.99 | 0.99 | 0.6 | 1 |
| S-Prompt++ | 0.99 | 0.99 | 0.4 | 1 |  |
| HidePrompt | 0.90 | 0.90 | 0.2 | 1 |  |
| CUB_INC20_TASK10 | DualPrompt | 0.90 | 0.90 | 0.3 | 1 |
| S-Prompt++ | 0.99 | 0.99 | 0.9 | 1 |  |
| HidePrompt | 0.95 | 0.95 | 0.7 | 1 |  |

## Appendix F Further Results

### F.1 Ablation studies on four hyperparameters

#### ϵ task subscript italic-ϵ task \epsilon_{\text{task}} , ϵ pre subscript italic-ϵ pre \epsilon_{\text{pre}} :

In Gradient Projection Continual Learning (GPCL), $\epsilon$ is usually used to construct the feature space in the SVD. Previous works set it between 0.9 and 0.99. In LW2G, $\epsilon_{task}$ and $\epsilon_{pre}$ are also used for feature space constiuction (old knowledge and pre-trained knowledge feature space). Thus, we follow the value in [, , ] and set these two parameters with the same value. We performed a grid search for appropriate values under different settings. As shown in Table [6](#A6.T6), LW2G consistently bring performance improvement for any of the aforementioned values.

#### ϕ italic-ϕ \phi :

$\phi$ controls the pre-trained knowledge and the acquisition of new task knowledge. We performed a grid search for $\phi$ and the results are shown in Table [7](#A6.T7).

#### N 𝑁 N :

Experiments showed significant improvement at $N=1$ compared to $N=0$, with no added benefit and increased computational overhead at higher values. Table [2](#footnote2) in the main paper indicates that SSP remains small when combined with LW2G. Thus, for efficiency and generality, we chosed $N=1$ as the default.

**Table 6: Impact of Distinct Threshold of $\epsilon_{\text{task}}$, $\epsilon_{\text{pre}}$ on CIFAR_INC10_TASK10**
| Settings | $\epsilon_{\text{task}}$ | $\epsilon_{\text{pre}}$ | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- |
| DualPrompt | Na | Na | 85.94 | 59.44 | 6.38 |
| DualPrompt [+ LW2G] | 0.50 | 0.50 | 86.89 | 60.67 | 5.44 |
| 0.90 | 0.90 | 87.03 | 65.57 | 5.77 |  |
| 0.95 | 0.95 | 86.86 | 78.33 | 6.03 |  |
| 0.99 | 0.99 | 86.48 | 100.0 | 7.12 |  |
| S-Prompt++ | Na | Na | 89.25 | 99.52 | 4.10 |
| S-Prompt++ [+ LW2G] | 0.50 | 0.50 | 89.28 | 99.76 | 4.33 |
| 0.90 | 0.90 | 88.54 | 100.0 | 4.48 |  |
| 0.95 | 0.95 | 89.32 | 100.0 | 3.46 |  |
| 0.99 | 0.99 | 89.25 | 92.32 | 6.00 |  |
| HidePrompt | Na | Na | 85.77 | 80.78 | 6.19 |
| HidePrompt [+ LW2G] | 0.50 | 0.50 | 86.85 | 81.70 | 5.78 |
| 0.90 | 0.90 | 86.57 | 84.93 | 5.14 |  |
| 0.95 | 0.95 | 86.93 | 90.10 | 5.02 |  |
| 0.99 | 0.99 | 87.60 | 95.39 | 4.28 |  |

**Table 7: Impact of Distinct Threshold of $\phi$ in DualPrompt [+ LW2G] on three typical settings**
| $\phi$ | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | Baseline |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FFA | 78.33 | 78.33 | 78.33 | 74.03 | 78.33 | 72.66 | 74.03 | 72.66 | 72.66 | 64.81 | 59.44 |
| PRA | 86.42 | 86.61 | 86.52 | 86.18 | 86.86 | 86.38 | 86.82 | 86.39 | 86.49 | 86.68 | 85.94 |
| FFM | 6.25 | 6.15 | 6.04 | 6.04 | 6.03 | 5.74 | 6.48 | 5.73 | 5.50 | 5.70 | 6.38 |
| SSP | 2 | 2 | 2 | 3 | 2 | 3 | 3 | 3 | 3 | 5 | 10 |

### F.2 Ablation studies on three modules in LW2G

In this section, we provide all experiments of any combination of proposed modules and the results are shown in Table [8](#A6.T8). The performance of any combimation can consistently outperform that of the baseline, illustrating the effectiveness of these modules.

**Table 8: Ablation studies**
| Variants | FFA | PRA | SSP |
| --- | --- | --- | --- |
| DualPrompt | 63.63 | 41.05 | 10 |
| DualPrompt [+ DGA] | 65.02 | 77.68 | 2 |
| DualPrompt [+ CPK] | 64.34 | 50.39 | 10 |
| DualPrompt [+ FFT] | 64.08 | 47.17 | 10 |
| DualPrompt [+ DGA, CPK] | 65.37 | 78.13 | 2 |
| DualPrompt [+ DGA, FFT] | 65.12 | 77.90 | 2 |
| DualPrompt [+ CPK, FFT] | 64.49 | 51.20 | 10 |
| DualPrompt [+ LW2G] | 65.60 | 80.40 | 2 |

### F.3 Overhead about calculation burden and time cost

First, we need to clarify that LW2G only requires selecting prompt sets from the pool to calculate gradients and HFC before learning each new task. The purpose is to decide whether to learn on a newly initialized set of prompts or reuse an existing set from the pool when learning a new task. After this, if opting to grow, the parameter update process does not introduce additional computation compared to the baseline. If opting not to grow, gradient projection is used during parameter updates to minimize the impact on old tasks. The computational overhead introduced by this step is a common issue in Gradient Projection Continual Learning (GPCL). This is detailed in Table [2](#S4.T2) of the main paper, where both FLOPS and TT (Training Time) are shown to increase.

Additionally, we further analyze the memory cost. In LW2G, the extra memory is divided into two parts: a set of bases for the pre-trained knowledge space and a set of bases for the old task feature space. The size of these two sets depends on the choice of during the SVD. In the following Table [9](#A6.T9), we analyze the memory introduced by Gradient Projection as varies. The "Bases" indicates the total number of bases for the two sets; "Extra Memory" represents the additional memory required. Specifically, we calculate the memory by considering each base as a tensor of length 768, stored as float32.

It is also worth reiterating that the proposed LW2G, inspired by gradient projection methods, introduces a novel and dynamic prompt growing strategy for prompt continual learning. The calculation burden and time cost are common issues with gradient projection methods, which we explicitly mention in the limitations section. Although addressing this problem is beyond the scope of this study, we will consider it as a direction for future research.

**Table 9: Discussion of the effects of memory on IMR_INC20_TASK10**
|  | $\epsilon$ | FFA | Bases | Extra Memory |
| --- | --- | --- | --- | --- |
| HidePrompt | / | 85.77 | 0 | 0 |
| HidePrompt [+ LW2G] | 0.90 | 86.57 | 429 | $\leq$ 5 MB |
|  | 0.95 | 86.93 | 509 | $\leq$ 5 MB |
|  | 0.99 | 87.60 | 640 | $\leq$ 5 MB |

### F.4 Visiliztions of Dynamic Process of LW2G with PCL

In the proposed LW2G method, the DGA module determines whether to grow a new set of prompts or reuse an existing set from the prompt sets pool based on the hindrance on learning new tasks while maintaining old knowledge under orthogonal condition. This impact is measured with the HFC metric proposed in the main paper. We provide a detail dynamic process in the following Table [10](#A6.T10) and Table [11](#A6.T11).

**Table 10: Variation process of HidePrompt [+ LW2G] on IMR.**
| Task | Calculation Process | Minimal Z | Option | Prompt sets pool |
| --- | --- | --- | --- | --- |
| 1 | / | / | To Grow a new ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1 |
| 2 | $\text{HFC}_{1}$=8.81, $\text{HFC}_{1}^{\text{pre}}$=7.17 | $Z_{1}$=1.64>0 | To Grow a new ($\bm{p}_{2},\bm{k}_{2}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2 |
| 3 | $\text{HFC}_{1}$=8.83, $\text{HFC}_{1}^{\text{pre}}$=7.22<br>$\text{HFC}_{2}$=9.24, $\text{HFC}_{2}^{\text{pre}}$=8.03 | $Z_{2}$=1.21>0 | To Grow a new ($\bm{p}_{3},\bm{k}_{3}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3 |
| 4 | $\text{HFC}_{1}$=7.34, $\text{HFC}_{1}^{\text{pre}}$=8.82<br>$\text{HFC}_{2}$=9.26, $\text{HFC}_{2}^{\text{pre}}$=8.00<br>$\text{HFC}_{3}$=9.15, $\text{HFC}_{3}^{\text{pre}}$=8.97 | $Z_{1}$=-1.48<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3 |
| 5 | $\text{HFC}_{1}$=9.24, $\text{HFC}_{1}^{\text{pre}}$=8.12<br>$\text{HFC}_{2}$=9.11, $\text{HFC}_{2}^{\text{pre}}$=9.07<br>$\text{HFC}_{3}$=12.95, $\text{HFC}_{3}^{\text{pre}}$=7.24 | $Z_{2}$=0.04>0 | To Grow a new ($\bm{p}_{4},\bm{k}_{4}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3<br>($\bm{p}_{4},\bm{k}_{4}$)$\rightarrow$ Task 5 |
| 6 | $\text{HFC}_{1}$=9.23, $\text{HFC}_{1}^{\text{pre}}$=8.02<br>$\text{HFC}_{2}$=9.29, $\text{HFC}_{2}^{\text{pre}}$=9.23<br>$\text{HFC}_{3}$=12.94, $\text{HFC}_{3}^{\text{pre}}$=7.29<br>$\text{HFC}_{4}$=9.03, $\text{HFC}_{4}^{\text{pre}}$=9.14 | $Z_{4}$=-0.11<0 | Not To Grow with ($\bm{p}_{4},\bm{k}_{4}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3<br>($\bm{p}_{4},\bm{k}_{4}$)$\rightarrow$ Task 5,6 |
| 7 | $\text{HFC}_{1}$=9.23, $\text{HFC}_{1}^{\text{pre}}$=8.08<br>$\text{HFC}_{2}$=12.96, $\text{HFC}_{2}^{\text{pre}}$=7.33<br>$\text{HFC}_{3}$=9.14, $\text{HFC}_{3}^{\text{pre}}$=9.25<br>$\text{HFC}_{4}$=12.84, $\text{HFC}_{4}^{\text{pre}}$=9.16 | $Z_{3}$=-0.11<0 | Not To Grow with ($\bm{p}_{3},\bm{k}_{3}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3,7<br>($\bm{p}_{4},\bm{k}_{4}$)$\rightarrow$ Task 5,6 |
| 8 | $\text{HFC}_{1}$=9.21, $\text{HFC}_{1}^{\text{pre}}$=8.19<br>$\text{HFC}_{2}$=12.94, $\text{HFC}_{2}^{\text{pre}}$=7.50<br>$\text{HFC}_{3}$=12.86, $\text{HFC}_{3}^{\text{pre}}$=9.23<br>$\text{HFC}_{4}$=12.60, $\text{HFC}_{4}^{\text{pre}}$=9.02 | $Z_{1}$=1.02>0 | To Grow a new ($\bm{p}_{5},\bm{k}_{5}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3,7<br>($\bm{p}_{4},\bm{k}_{4}$)$\rightarrow$ Task 5,6<br>($\bm{p}_{5},\bm{k}_{5}$)$\rightarrow$ Task 8 |
| 9 | $\text{HFC}_{1}$=9.41, $\text{HFC}_{1}^{\text{pre}}$=8.08<br>$\text{HFC}_{2}$=12.95, $\text{HFC}_{2}^{\text{pre}}$=7.26<br>$\text{HFC}_{3}$=12.83, $\text{HFC}_{3}^{\text{pre}}$=9.26<br>$\text{HFC}_{4}$=12.61, $\text{HFC}_{4}^{\text{pre}}$=9.17<br>$\text{HFC}_{5}$=7.98, $\text{HFC}_{5}^{\text{pre}}$=7.50 | $Z_{5}$=0.48>0 | To Grow a new ($\bm{p}_{6},\bm{k}_{6}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3,7<br>($\bm{p}_{4},\bm{k}_{4}$)$\rightarrow$ Task 5,6<br>($\bm{p}_{5},\bm{k}_{5}$)$\rightarrow$ Task 8<br>($\bm{p}_{6},\bm{k}_{6}$)$\rightarrow$ Task 9 |
| 10 | $\text{HFC}_{1}$=9.24, $\text{HFC}_{1}^{\text{pre}}$=7.99<br>$\text{HFC}_{2}$=12.97, $\text{HFC}_{2}^{\text{pre}}$=7.29<br>$\text{HFC}_{3}$=12.84, $\text{HFC}_{3}^{\text{pre}}$=9.10<br>$\text{HFC}_{4}$=12.59, $\text{HFC}_{4}^{\text{pre}}$=9.03<br>$\text{HFC}_{5}$=7.98, $\text{HFC}_{5}^{\text{pre}}$=8.99<br>$\text{HFC}_{6}$=6.99, $\text{HFC}_{6}^{\text{pre}}$=7.53 | $Z_{5}$=-1.01<0 | Not To Grow with ($\bm{p}_{5},\bm{k}_{5}$) | ($\bm{p}_{1},\bm{k}_{1}$)$\rightarrow$ Task 1,4<br>($\bm{p}_{2},\bm{k}_{2}$)$\rightarrow$ Task 2<br>($\bm{p}_{3},\bm{k}_{3}$)$\rightarrow$ Task 3,7<br>($\bm{p}_{4},\bm{k}_{4}$)$\rightarrow$ Task 5,6<br>($\bm{p}_{5},\bm{k}_{5}$)$\rightarrow$ Task 8,10<br>($\bm{p}_{6},\bm{k}_{6}$) $\rightarrow$ Task 9 |

**Table 11: Variation process of DualPrompt [+ LW2G] on IMR.**
| Task | Calculation Process | Minimal Z | Option | Prompt sets pool |
| --- | --- | --- | --- | --- |
| 1 | / | / | To Grow a new ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1 |
| 2 | $\text{HFC}_{1}$=13.90, $\text{HFC}_{1}^{\text{pre}}$=40.23 | $Z_{1}$=-26.33<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2 |
| 3 | $\text{HFC}_{1}$=20.22, $\text{HFC}_{1}^{\text{pre}}$=40.80 | $Z_{1}$=-20.58<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3 |
| 4 | $\text{HFC}_{1}$=25.09, $\text{HFC}_{1}^{\text{pre}}$=41.50 | $Z_{1}$=-16.41<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4 |
| 5 | $\text{HFC}_{1}$=29.15, $\text{HFC}_{1}^{\text{pre}}$=42.92 | $Z_{1}$=-13.77<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4,5 |
| 6 | $\text{HFC}_{1}$=32.85, $\text{HFC}_{1}^{\text{pre}}$=42.78 | $Z_{1}$=-9.33<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4,5,6 |
| 7 | $\text{HFC}_{1}$=36.35, $\text{HFC}_{1}^{\text{pre}}$=41.85 | $Z_{1}$=-5.5<0 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4,5,6,7 |
| 8 | $\text{HFC}_{1}$=39.39, $\text{HFC}_{1}^{\text{pre}}$=42.42 | $Z_{1}$=-3.03 | Not To Grow with ($\bm{p}_{1},\bm{k}_{1}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4,5,6,7,8 |
| 9 | $\text{HFC}_{1}$=42.54, $\text{HFC}_{1}^{\text{pre}}$=41.37 | $Z_{1}$=1.17>0 | To Grow a new ($\bm{p}_{2},\bm{k}_{2}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4,5,6,7,8<br>($\bm{p}_{2},\bm{k}_{2}$) $\rightarrow$ Task 9 |
| 10 | $\text{HFC}_{1}$=42.54, $\text{HFC}_{1}^{\text{pre}}$=40.92<br>$\text{HFC}_{2}$=13.81, $\text{HFC}_{2}^{\text{pre}}$=41.81 | $Z_{2}$=-28.00<0 | Not To Grow with ($\bm{p}_{2},\bm{k}_{2}$) | ($\bm{p}_{1},\bm{k}_{1}$) $\rightarrow$ Task 1,2,3,4,5,6,7,8<br>($\bm{p}_{2},\bm{k}_{2}$) $\rightarrow$ Task 9,10 |

### F.5 Comparison with Two Concurrent Works

We note that two concurrent works, SEED [] and PGP [], are closely related to our motivation and methodology, respectively. In this section, we compare our proposed LW2G with these approaches.

PGP first introduced Gradient Projection-based Continual Learning (GPCL) in the context of PCL, leveraging GPCL to ensure that old knowledge is not forgotten. They demonstrated that in the scenario of PCL, the construction of the feature space could be translated into the prompt space and input space. However, unlike PGP, LW2G aims to dynamically learn whether *to grow* (initialize a new set of prompts) or *not to grow* (reuse prompts in pool) for each new task based on specific commonalities between tasks. To achieve this, LW2G adopts the idea of the *orthogonal condition* in GPCL to integrate knowledge from multiple tasks into a single set of prompts while preserving old knowledge. Additionally, we analyze the hindrance on learning new tasks caused by the *orthogonal condition* and use the degree of inhibition under this condition as an adaptive criterion for our Dynamic Growing Approach. Furthermore, in Table [12](#A6.T12), we compare the results of the Baseline, Baseline + PGP, and Baseline + LW2G. In both typical and long task settings, Baseline + LW2G consistently outperforms Baseline + PGP. Moreover, LW2G significantly outperforms PGP in PRA and SSP, further highlighting our approach’s focus on the amount of selectable sets during the *prompt retrieval* stage in PCL.

Meanwhile, SEED proposed a continual learning method based on Mixture-of-Experts (MoE). Specifically, SEED maintains multiple sets of experts and dynamically determines which expert should be used to learn new tasks with minimal impact on old tasks. However, SEED fixes the total number of experts at the start of training, which inevitably reduces plasticity as the amount of tasks increases. In contrast, LW2G achieves complete dynamic expansion of ’experts’ (which are sets of prompts in PCL) by assessing the degree of inhibition on new tasks under the *orthogonal condition*, thus eliminating the need to predefine the amount of experts.

**Table 12: Results on typical and long task settings. Here, we present DualPrompt as the baseline, with PGP and LW2G added to the baseline respectively. The best results are highlighted in bold.**
| Settings | Methods | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ | SSP $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR_INC10_TASK10 | DualPrompt | 85.94 | 59.44 | 6.38 | 10 |
| DualPrompt [+ PGP] | 86.72 | 59.15 | 6.01 | 10 |  |
| DualPrompt [+ LW2G] | 86.86 | 78.33 | 6.03 | 2 |  |
| IMR_INC20_TASK10 | DualPrompt | 63.63 | 41.05 | 6.41 | 10 |
| DualPrompt [+ PGP] | 63.82 | 41.18 | 5.65 | 10 |  |
| DualPrompt [+ LW2G] | 65.60 | 80.40 | 5.72 | 2 |  |
| CUB_INC20_TASK10 | DualPrompt | 82.09 | 66.71 | 6.40 | 10 |
| DualPrompt [+ PGP] | 81.58 | 66.88 | 7.01 | 10 |  |
| DualPrompt [+ LW2G] | 82.43 | 70.09 | 5.25 | 7 |  |
| OMNI_INC10_TASK30 | DualPrompt | 63.36 | 68.47 | 12.92 | 30 |
| DualPrompt [+ PGP] | 63.74 | 67.95 | 12.97 | 30 |  |
| DualPrompt [+ LW2G] | 65.12 | 80.95 | 10.75 | 9 |  |
| OMNI_INC5_TASK60 | DualPrompt | 61.85 | 69.94 | 13.50 | 60 |
| DualPrompt [+ PGP] | 62.24 | 68.68 | 14.64 | 60 |  |
| DualPrompt [+ LW2G] | 63.17 | 75.31 | 12.01 | 17 |  |

### F.6 Performance Under Other PTMs

To show the efficacy of proposed method under different PTMs, we evaluate our method by extending three distinct PTMs, namely IBOT1k [], IBOT21k [] and DINO []. The results are shown in the Table [13](#A6.T13), Table [14](#A6.T14) and Table [15](#A6.T15).

**Table 13: Results under IBOT21k when comparing LW2G with three baselines. The best results are highlighted in bold.**
| Settings | Methods | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ | SSP $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR_INC10_TASK10 | DualPrompt | 74.03 | 72.16 | 15.93 | 10 |
| DualPrompt [+ LW2G] | 74.76 | 78.33 | 13.92 | 3 |  |
| S-Prompt++ | 78.37 | 78.83 | 9.00 | 10 |  |
| S-Prompt++ [+ LW2G] | 78.83 | 75.20 | 8.69 | 3 |  |
| HidePrompt | 86.12 | 85.02 | 5.98 | 10 |  |
| HidePrompt [+ LW2G] | 86.40 | 92.06 | 5.84 | 2 |  |
| IMR_INC20_TASK10 | DualPrompt | 47.96 | 38.62 | 5.36 | 10 |
| DualPrompt [+ LW2G] | 49.13 | 64.05 | 5.33 | 3 |  |
| S-Prompt++ | 46.20 | 37.77 | 7.01 | 10 |  |
| S-Prompt++ [+ LW2G] | 48.97 | 71.04 | 6.30 | 3 |  |
| HidePrompt | 62.00 | 67.28 | 5.63 | 10 |  |
| HidePrompt [+ LW2G] | 63.67 | 82.18 | 5.80 | 3 |  |

**Table 14: Results under IBOT1k when comparing LW2G with three baselines. The best results are highlighted in bold.**
| Settings | Methods | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ | SSP $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR_INC10_TASK10 | DualPrompt | 71.58 | 84.72 | 19.41 | 10 |
| DualPrompt [+ LW2G] | 71.79 | 84.90 | 18.99 | 3 |  |
| S-Prompt++ | 75.70 | 83.76 | 9.46 | 10 |  |
| S-Prompt++ [+ LW2G] | 76.01 | 84.37 | 8.91 | 3 |  |
| HidePrompt | 84.83 | 83.50 | 6.48 | 10 |  |
| HidePrompt [+ LW2G] | 85.54 | 88.02 | 5.75 | 3 |  |
| IMR_INC20_TASK10 | DualPrompt | 56.68 | 38.15 | 5.18 | 10 |
| DualPrompt [+ LW2G] | 56.89 | 57.57 | 5.04 | 3 |  |
| S-Prompt++ | 52.38 | 39.78 | 7.18 | 10 |  |
| S-Prompt++ [+ LW2G] | 55.82 | 55.90 | 7.13 | 3 |  |
| HidePrompt | 64.77 | 67.94 | 6.90 | 10 |  |
| HidePrompt [+ LW2G] | 65.15 | 78.27 | 4.86 | 3 |  |

**Table 15: Results under DINO when comparing LW2G with three baselines. The best results are highlighted in bold.**
| Settings | Methods | FFA $(\uparrow)$ | PRA $(\uparrow)$ | FFM $(\downarrow)$ | SSP $(\downarrow)$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR_INC10_TASK10 | DualPrompt | 69.46 | 88.80 | 18.96 | 10 |
| DualPrompt [+ LW2G] | 70.13 | 89.01 | 18.03 | 3 |  |
| S-Prompt++ | 74.62 | 87.60 | 10.71 | 10 |  |
| S-Prompt++ [+ LW2G] | 71.36 | 89.30 | 12.38 | 2 |  |
| HidePrompt | 82.89 | 82.05 | 7.45 | 10 |  |
| HidePrompt [+ LW2G] | 83.58 | 88.57 | 7.08 | 3 |  |
| IMR_INC20_TASK10 | DualPrompt | 52.41 | 38.74 | 5.93 | 10 |
| DualPrompt [+ LW2G] | 54.22 | 75.75 | 5.77 | 2 |  |
| S-Prompt++ | 50.00 | 37.72 | 6.75 | 10 |  |
| S-Prompt++ [+ LW2G] | 65.44 | 79.35 | 6.01 | 5 |  |
| HidePrompt | 62.42 | 62.07 | 8.89 | 10 |  |
| HidePrompt [+ LW2G] | 64.04 | 86.43 | 4.82 | 2 |  |

## Appendix G Algorithm

Figure: Algorithm 1 Learning Whether to Grow

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the weaknesses.
soundness: 2
strengths: The motivation of the proposed method is clear. There are theoretical supports. The experimental superiority over baselines can be seen.
summary: This work proposes a strategy for determining the dynamic expansion of the prompt pool over time for prompt-based Continual Learning methods, based on techniques inspired by Gradient Projection Memory for Continual Learning.
weaknesses: 1. It seems to me that this paper is the application of the technique in [1] to the existing methods with the criteria to expand the prompt pool. However, this technique is quite similar to those in [2], [3]. The authors should highlight the novelty and innovative contribution of this work. 

2. It is not clear how the performance on CUB of HiDE is has minimal change, while those on CIFAR100 and ImageNet-R is decrease significantly. If the codebase of HiDE has some problem, why didn't the authors consider other codebases of other baselines?

3. The authors claim their method can promote positive knowledge transfer, however relevant experimental results seem to be lacking.

4. The method saves the number of parameters to learn, but storing the basis vectors of the subspaces is also expensive.

5. This work is close to [1], thus, it is essential to have the comparisons between them.

[1] Prompt Gradient Projection for Continual Learning, ICLR 2024.

[2] Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer, NeurIPS 2022.

[3] TRGP: Trust Region Gradient Projection for Continual Learning, ICLR 2022.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: see above
soundness: 4
strengths: - LW2G is an approach that actively decides whether to grow new prompts or reuse existing ones for each new task. This aspect distinguishes LW2G from previous methods.

- Introduces a new Hinder Forward Capability (HFC) metric to quantify the hindrance of learning a new task on old prompts under orthogonality constraints. 

- The three main components of LW2G - DGA, CPK, and FFT - provide a complete implementation for reusing existing prompts in continual learning tasks. However, to some extent, the approach is quite redundant, as it requires repeatedly training all the prompts to determine the optimal reuse strategy.

- The authors have shared their code and implementation, which, although I haven't personally tested it, lends some credibility to the soundness of their method. At least, from the README.md file, it appears they have provided a full implementation of the Hide-Prompt version.

- Overall, the proposed method is sound as there are so comprehensive experiments.
summary: The paper proposes a new method called LW2G (Learning Whether to Grow) for prompt-based continual learning.
LW2G decides whether to grow the prompt pool or reuse existing prompts when learning a new task, based on measuring the hindrance of learning the new task on old prompts compared to an ideal hindrance-free scenario. I think the idea of Hinder Forward Capability (HFC) is inspiring, that measures the difference between new and old tasks as well as the hindrance imposed on learning new tasks under a strict orthogonality constraint to old task feature spaces.
Extensive experiments demonstrating the effectiveness of adding LW2G to existing prompt-based continual learning methods across multiple datasets and settings.
weaknesses: - One important aspect missing from the experiments in this paper is the average results from multiple runs. This is particularly evident in the paper, as the experimental results in Tables 1, 2, and 3 show that the proposed method has only a slight difference compared to the baselines. It's quite possible that running the baselines multiple times could yield a better result. I'm not claiming that this paper has done so. But providing the mean and standard deviation would better demonstrate statistical robustness.

- LW2G  needs for significant manual intervention and careful hyperparameter selection to ensure the prompt resule/increase strategy works appropriately. For example, HFC thresholds, CPK's soft constraint coefficient and FFT's top-N selection.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: (1)Does LW2G still work in the context of various pre-trained models?
(2)Is it still effective in more prompt-based CL methods, such as coda-prompt?
(3)As more recent studies show, lora-based CL methods are more effective than prompt-based in CIL tasks. So, does the proposed strategy have more advantages?
(4)When does the Theorem 1 hold? Any assumptions?
soundness: 2
strengths: (1)By learning whether to grow or not to grow set of prompts, this work forms an effective and efficient prompt sets pool where each single set contains knowledge from multiple tasks, thus facilitating cross-task promotion. 
(2)LW2G is a plug-in and effective module within existing PCL.
summary: This paper proposes a plug-in module within existing Prompt-based Continual Learning (PCL), 
called Learning Whether To Grow (LW2G). LW2G enables PCL to dynamically learn to whether to add a new set of prompts for each task (to grow) or to utilize an existing set of prompts (not to grow) based on the relationships between tasks. LW2G consists of three components: Dynamic Growing Approach (DGA), Consistency with Pre-trained Knowledge (CPK), and Facilitation for Forward Transfer (FFT). Its superiority across multiple benchmarks and various CL settings is also verified.
weaknesses: I have two main concerns: 
(1)The proposed LW2G is mainly designed for prompt-based CL methods, and further improves their performances. However, the improvements with respect to the mainstream CL metrics, FAA and FMM, as shown in Table 1, are not significant. In particular, for some stronger baselines, S-prompt++, the performance with LW2G makes no obvious performance. 
(2)It seems that the performance reported on the Hide-prompt paper is better than that in this work, in which Hide-prompt even performs better than S-prompt++.
(3) FFA or FAA? It is a little confused.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: see the weakness.

I'd like to raise my score if all my concerns are addressed.
soundness: 3
strengths: The experiments and analysis are very thorough and well-written, and the proposed method has some novelty
summary: In this paper, the authors propose a method namely Learn Whether to Grow (LW2G), wherein a shared set of prompts is utilized when several tasks share certain commonalities, and a new set is added when there are significant differences between the new task and previous tasks. The proposed  LW2G develops a metric named Hinder Forward Capability (HFC) to measure the hindrance. With HFC, an automated scheme Dynamic Growing Approach adaptively learns whether to grow with a dynamic threshold. Furthermore, a gradient-based constraint is introduced to ensure consistency between the updating prompts and pre-trained knowledge. Extensive experiments are provided.
weaknesses: Weakness：

1: I agree that Prompt Retrieval Accuracy (PRA) has a certain impact on performance and is an urgent problem to be solved before 2023. However, some of the lasted prompt-based methods ([1], [2], etc) and some of the lasted Adapter-Based methods ([3], [4], [5], etc.) do not need to consider PRA issues and surpass the classical methods using the matching mechanism. I have doubts whether the PRA problem should be the main motivation for building continuous learning solutions using pre-trained models.

2: There are also some methods for optimizing the prompt selection (matching) mechanism. For example, [6], [7], [8] all try to improve the matching mechanism. Compared with the above methods, the improvement of the projection method seems to be relatively limited. it may not prove that your method is superior to them.

3: The comparison is not sufficient, lacking performance comparison of some of the latest methods (as above) and the display of plug-in effects in the latest methods, etc. 

4: Gradient projection and whether to choose growth are not novel in the field of CL, but they are somewhat novel in the PTM-based CIL task.

[1]: CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
[2]: Generating Instance-level Prompts for Rehearsal-free Continual Learning
[3]: InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning
[4]: Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
[5]: Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need
[6]: Consistent Prompting for Rehearsal-Free Continual Learning
[7]: One-stage Prompt-based Continual Learning
[8]: Evolving Parameterized Prompt Memory for Continual Learning

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Does the proposed method require computing the representation matrix for every layer after learning each task? Additionally, what specific samples are used to calculate the representation matrix?

2. What does the variable $N$ in line 165 refer to? 

3. How is $\mathbf{B}_t^l$ obtained? While it appears to be introduced in the supplementary material, it is recommended to briefly explain this process in the main paper as it is essential for understanding the overall method.
soundness: 3
strengths: 1. Dynamically growing the prompt pool for PCL methods aligns well with the continual learning paradigm.

2. Using a gradient-projection-based metric to assess task similarity looks reasonable.
summary: This paper introduces Learning Whether to Grow (LW2G), a plug-in module for Prompt-based Continual Learning (PCL). LW2G determines whether to expand the prompt pool by adding new prompts or reusing existing ones based on the similarity between tasks, aiming to improve efficiency and knowledge transfer. Inspired by Gradient Projection Continual Learning (GPCL), the method leverages orthogonal projections to measure task interference using a new metric called Hinder Forward Capability (HFC). LW2G also introduces strategies for gradient-based prompt consistency and prompt weight reuse to further optimize learning. Experimental results show that LW2G improved the performances of three baseline PCL models.
weaknesses: While the idea of leveraging orthogonal conditions to mitigate forgetting is reasonable, the paper does not sufficiently formulate this method within the context of PCL. Section 3 introduces the formulation for layer $l$, but it lacks clarity on how this formulation applies specifically for PCL, where prompts serve as the primary trainable parameters. A detailed and precise formulation is essential to fully understand the proposed method.

The reported performance of baseline models appears lower than those reported in prior work. For instance, HiDe-Prompt, when initialized with ImageNet-21K pre-trained weights, achieves FAA scores of 92.61 on CIFAR-100, 75.06 on ImageNet-R, and 86.56 on CUB, which are much higher than the ones presented in Table 1 under the same task settings. These discrepancies raise concerns about the reliability of the reported results and warrant further explanation.

The differences between the proposed LW2G and other orthogonal continual learning methods are not sufficiently highlighted. A more explicit comparison would help better position LW2G and clarify its unique contributions.


Some terminologies require clearer definitions. For example, the term “matrix with suitable dimensions” is vague. A more precise description is needed to avoid confusion.

Some minors. 1) There seem to be some messy codes in line 082. 2) Use “ViT” instead of “VIT” to align with established conventions. 3) The notion $l$ is slightly reused for the layer index and the number of bases for space $\mathcal{S}_1$.

Meta-review: This paper aims to improve the efficiency and knowledge transfer for prompt-based continual learning. The authors propose a method called Learning Whether to Grow (LW2G) to decide whether to grow the prompt pool or reuse existing prompts when learning a new task. Extensive experiments are conducted to evaluate the performance of the proposed method. Several reviewers raise their concerns about the novelty, rationality, definition of concepts, experimental support, comparison results and manual intervention, some key concerns remain after the rebuttal. Based on the above considerations, I think the current manuscript does not match the ICLR’s requirement and I do not recommend to accept this manuscript.

Decision: Reject