Title: Visual Prompt Tuning in Null Space for Continual Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Continual Learning

Visual prompt tuning

Gradient orthogonal projection

Null space
Abstract: Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL), by selecting and updating relevant prompts in the vision-transformer models. On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference on tasks that have been learned to overcome catastrophic forgetting in CL. However, different from the orthogonal projection in the traditional CNN architecture, the prompt gradient orthogonal projection in the ViT architecture shows completely different and greater challenges, i.e., 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. Theoretically, we have finally deduced two consistency conditions to achieve the prompt gradient orthogonal projection, which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient orthogonal projection. Extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods. Our code is available at https://github.com/zugexiaodui/VPTinNSforCL

Visual Prompt Tuning in Null Space for Continual Learning

Abstract

Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL), by selecting and updating relevant prompts in the visiontransformer models. On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference on tasks that have been learned to overcome catastrophic forgetting in CL. However, different from the orthogonal projection in the traditional CNN architecture, the prompt gradient orthogonal projection in the ViT architecture shows completely different and greater challenges, i.e. , 1) the highorder and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. Theoretically, we have finally deduced two consistency conditions to achieve the prompt gradient orthogonal projection , which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient orthogonal projection . Extensive experimental results demonstrate the effectiveness of anti-forgetting on four classincremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods. Our code is available at https://github.com/zugexiaodui/VPTinNSforCL .

1 Introduction

Continual learning (CL) is crucial for AI models to adapt to the ever-changing environment by learning sequentially arrived data, where the catastrophic forgetting is the key challenge [21, 28]. Recently, prompt tuning-based continual learning methods [40, 32, 34, 44, 10, 22, 38, 46, 20, 12, 18] have been attracting increasing attention due to their impressive performances in the CL field. Existing prompt tuning-based works tackle the downstream continual learning problem by selecting and updating relevant prompts, which is encoded with full task-specific knowledge while exploiting the general knowledge of the pre-trained ViTs [40, 39].

On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference with tasks that have been learned to overcome catastrophic forgetting in CL. It is worth noting that forgetting can be theoretically resolved by gradient orthogonal projection methods [43, 31, 36, 45], which have

been extensively explored especially when adapting CNN models. Nevertheless, it remains a huge gap to introduce the orthogonal projection-based methods of CNNs to visual prompt tuning due to the following challenges: 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. For the linear operation in convolution or fully-connected layers, the output features of old tasks can remain unchanged by updating the weights in the orthogonal subspace of previous input features. While for self-attention, three linear transformations are employed on input tokens, followed by high-order and non-linear operations for the self-attention interaction of tokens. It makes the relationship between the update of prompts and the output image tokens much more complex, far exceeding mere linearity.

In this work, we theoretically deduced two consistency conditions to achieve the prompt gradient orthogonal projection , which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. To be concrete, we firstly take the full self-attention and LayerNorm into consideration and derive a strict condition for eliminating the interference through a comprehensive analysis of the forward propagation of the ViT layer. Then we further propose to convert the condition of self-attention into its two sufficient conditions, which enables us to address the challenge of high order and nonlinearity. Thirdly, we propose a constraint of invariant prompt distribution that removes the obstacle to the final simplification of the conditions brought by the LayerNorm. The consistency conditions reveal that if the prompt update can be orthogonal to (1) the normalized previous input image tokens projected with the second-order qkv-transformation matrices of the pre-trained model, and (2) the activated attention map generated by image queries and prompt keys, the interference in visual prompt tuning can be eliminated theoretically.

In practice, based on the proposed consistency conditions, an effective null-space-based approximation solution [36] has been proposed to implement the prompt gradient orthogonal projection , while the invariant prompt distribution constraint is implemented by incorporating a loss function which penalizes the drifting of prompt distribution over sequential tasks. We validate our Null-Space Projection for Prompts (NSP 2 ) approach on extensive class-incremental benchmarks: 10- and 20-split CIFAR-100, 10-split ImageNet-R [39] and 10-split DomainNet [38], with the sequential fine-tuning VPT and CLIP models as baselines. Our approach brings 4% ∼ 10% improvements in accuracy, and reduces 9% ∼ 17% forgetting, which is superior to state-of-the-art methods.

Our contributions are summarized as follows: (1) We introduce the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. (2) Two sufficient consistency conditions for the self-attention and an invariant prompt distribution constraint for LayerNorm are theoretically deduced, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. (3) Extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods.

2 Related Work

Prompting-Based Approaches: Most of the prompting-based approaches adopt a two-stage framework [37, 39, 14, 15, 32, 42, 34, 35, 11, 18, 19]: querying a group of prompts for an individual sample and using them to prompt the pre-trained models. For example, L2P [40] first selects a group of prompts from a prompt pool and then feeds them into the ViT. CPrompt [11] proposes to mitigate the gap between training and testing stages to enhance prediction robustness and boost prompt selection accuracy. These approaches essentially focus on acquisition of task-specific prompts tailored to individual samples. There are also several one-stage methods [2, 22, 38, 44, 20] based on prompt tuning. (1) Slowly updating trainable parameters [10, 44]: e.g. , LAE [10] updates an offline expert with a large momentum to reduce the change of features. (2) Expandable backbones [46, 20]: e.g. , EASE [46] trains a distinct lightweight adapter module for each new task, and designs a semantic mapping to complement the drift of old class prototypes. (3) Enhancing classifiers rather than focusing on learning features [38, 22, 12]: e.g. , ESN [38] proposes an anchor-based classifier alignment approach based on energy-based models. As introduced above, these works still lack of a theoretical solution to the interference problem for visual prompt tuning. In our work, we conduct a deep analysis of this problem and provide a theoretical guidance on eliminating the interference.

Orthogonal Projection-Based Approaches: Orthogonal projection-based approaches [43, 4, 8, 31, 36, 17, 45] can theoretically eliminate the interference of new tasks on old tasks for linear layers. OWM[43] constructs a projector to find the direction orthogonal to the input space. GPM [31] first projects new gradients to the subspace important to the old tasks and then subtracts the projected components for updating parameters. Adam-NSCL [36] projects the parameter updates to the approximate null space of previous inputs. However, due to the different relationships between parameter updates and outputs in the linear operation and self-attention, the consistency condition used in CNNs is not directly applicable to the prompt tuning in ViTs. In our work, we derive the consistency conditions for the visual prompt tuning, enabling the application of orthogonal projectionbased approaches to it, where the null-space projection [36] is adopted in our approach to get an approximate solution efficiently. We notice that a recently emerged work PGP [26] implements GPM [31] to prompt-based frameworks. However, it obtains the same conclusion as that of the linear operation under a simplified attention, which limits its application and performance as compared in the appendix D .

3 Preliminaries

Continual Learning: In the setting of continual learning, a network f ( ·| Θ ) with parameters Θ is sequentially trained on a stream of disjoint tasks {T 1 , T 2 , · · · , T T } , where task T t is associated with paired data { ( X <i> t , y <i> t ) |T t | i =1 } of size |T t | . When a task T t arrives, the model f ( ·| Θ ) would be trained for the current task, while the data from previous tasks is unreachable.

Forward Propagation of Visual Prompt Tuning in ViT Layers: We describe the forward propagation process of the ViT layer for visual prompt tuning, as illustrated in Figure 1 . Let X ∈ R N × D and P ∈ R M × D denote the N input image tokens of a sample (including the pre-trained class token if available) and M prompts, respectively, where D is the dimension of each token. In the ViT layer, only the prompts P are trainable parameters. The remaining parameters in LayerNorm, qkv-transformations and subsequent MLP introduced below are pre-trained and kept frozen. We use Z = [ X ; P ] ∈ R ( N + M ) × D to denote the concatenated input tokens. First, they undergo the LayerNorm [1] operation LN( · ) :

where µ Z , σ Z ∈ R N + M , α , β ∈ R D . The ⊙ and division here denote the element-wise (Hadamard) product and division, respectively. Note that the vectors µ Z , σ Z , α and β are broadcasted to match the matrices of dimensions ( N + M ) × D , enabling them to carry out operations with Z . Then the normalized tokens are fed into the qkv-transformations:

where W { q,k,v } ∈ R D × D . The vector b { q,k,v } ∈ R D is broadcasted to a matrix of dimensions ( N + M ) × D to facilitate the addition operation. Next is the self-attention:

where Q X denotes the image tokens serving as queries. Eq. (3) can be expanded as Affinity, softmax (on rows) and Aggregation operations:

It is worth noting that the rows of the attention map where the prompts serve as queries ( i.e. , Q P ) do not need to be computed, as formulated in Eq. (4) and illustrated in Figure 1 . The reason is that in VPT-Deep [13], the output prompts of this ViT layer will be replaced with new trainable prompts in the subsequent layer. Omitting Q P has no impact on the output image tokens of the ViT layer, as

Image tokens

Self-attention

Outputtokens

Qkv-transformation

Affinity

Aggregation

Qx

Kx

LayerNorm

Kx

LayerNorm

LN(X)

AX

Ap

Softmax

Sx

Sp

×

Fz

ML

P

LN(P)

QP

Kp

Vp

QP

X

P

Attentionmap

Prompts

(Neglected)

the subsequent Aggregation, LayerNorm and MLP operations are performed independently for each token. If no new prompts are added in the next layer, the output prompts can be just discarded as well.

After the self-attention, operations consist of another LayerNorm and the MLP layer are applied individually to each token, without any interaction among the tokens. Finally, the output fine-tuned image tokens are fed into the next ViT layer.

Orthogonal Projection in Convolutional Layers: A convolutional operation is actually a linear operation. For a convolutional layer f conv ( ·| Θ t ) in task T t , we use Θ t ∈ R D in × D out to denote its unrolled convolutional kernel matrix [5]. Here, D in represents the number of pixels within a kernel, and D out corresponds to the number of kernels. Each convolutional patch from the input feature map is flattened into a row vector with a dimension of D in . These row vectors of totaling n p patches compose the input feature matrix X t ∈ R n p × D in . The output feature for X t in task T t is expected to remain unchanged (referred to as consistent) in the next task T t +1 to prevent forgetting:

̸

By substituting Θ t +1 = Θ t +∆ Θ , with ∆ Θ = 0 denoting the weight update in T t +1 , the consistency condition for the convolutional layer is established as follows:

Eq. (9) suggests that if the weight update ∆ Θ is orthogonal to the previous input feature X t during training in the new task, the corresponding output feature will remain unchanged. Thereby, the interference of the new task on the old task is eliminated. This can be realized by projecting the candidate weight update Θ G into the orthogonal subspace of X t : ∆ Θ = P Θ G , where P ∈ R D in × D in is an orthogonal projection matrix [43, 36, 31].

Similarly, for the prompt tuning which fine-tunes the prompts P t in a ViT layer f vit ( X t | P t ) , we also aim to satisfy the following consistency objective for the purpose of anti-forgetting:

̸

However, the consistency condition in Eq. (9) does not hold for Eq. (10), since f vit ( X t | P t ) = X t P t in prompt tuning. Instead, all the tokens X t and P t first undergo a LayerNorm and then interact via the self-attention mechanism, as previously described. The complicated forward propagation within the ViT layer brings huge challenge to analyzing the consistency conditions in relation to the prompt update ∆ P . In the next section, we will tackle this challenge and derive the consistency conditions for visual prompt tuning.

4 Method

We use Z t = [ X t ; P t ] and Z t +1 = [ X t ; P t +1 ] to denote the input tokens before and after updating the prompts, respectively, where P t +1 = P t +∆ P , ∆ P = 0 . Our goal is to analyze how to satisfy Eq. (10) and derive one or more conditions expressed in terms of the prompt update ∆ P . These conditions will subsequently guide the application of orthogonal projection to ∆ P .

which can be further simplified as:

̸

4.1 Analysis of Consistency Conditions

As can be seen in Figure 1, those outputs of LayerNorm and qkv-transformations corresponding to the image tokens remain unaffected by the updates to the prompts. Hence, the essence of attaining the consistency objective Eq. (10) can be turned into analyzing how to keep the output of self-attention in Eq. (3) unchanged as the prompts are updated, i.e. , satisfying:

However, the nonlinear operation ( i.e. , softmax ) and the potential higher-order term W ⊤ k Z ⊤ ZW v arising from K ⊤ Z V Z in Eq. (3) complicate the direct resolution of this objective. Specifically, the non-injection property of the softmax function causes non-unique solutions. The multiplication between K Z ⊤ t +1 V Z t +1 derives a quadratic term LN( P t + ∆ P ) ⊤ LN( P t + ∆ P ) , which result in difficult optimization for ∆ P .

To address this issue, we propose two sufficient conditions consisting solely of linear operations. Specifically, we split the process of self-attention into two primary stages, i.e. , the Affinity described by Eq. (4) and the Aggregation outlined in Eq. (6). We can achieve Eq. (11) by ensuring the consistency of each stage:

We first analyze the consistency objective of Affinity, i.e. , Eq. (12), for Z t and Z t +1 :

where √ D is omitted for simplicity. Upon fulfilling Eq. (12), we can obtain S Z t = S Z t +1 , corresponding to the output of Eq. (5). Subsequently, we analyze the consistency objective of Aggregation in Eq. (13), yielding results for Z t and Z t +1 as:

Based on Eq. (12 -17), we are able to derive the following two equations, respectively:

Note that we expect to further deduce Eq. (18) and Eq. (19) to obtain equations among LN( P t ) , LN( P t +∆ P ) and ∆ P . However, due to the square root and quadratic terms in the expressions of the standard deviations σ P t and σ P t +∆ P , it is difficult to express σ P t +∆ P in terms of σ P t and σ ∆ P . Consequently, it is challenging to derive a straightforward equation that relates LN( P t ) and LN( P t +∆ P ) through ∆ P .

To simplify the problem, we introduce an additional constraint on the distribution of prompts. Concretely, we require that the updated prompts P t +∆ P retain the same distribution as P t , i.e. , meeting the following assumption:

In this way, we can establish a straightforward mathematical relationship connecting LN( P t +∆ P ) , LN( P t ) and ∆ P :

Consequently, we can apply Eq. (21) to simplify Eq. (18) and (19) as:

It should be noted that in Eq. 22 and Eq. 23 , W k , W v and α are pre-trained parameters kept frozen throughout the continual learning process. Q X t and S P t are two matrices derived from the input X t . As our objective is to ensure that the above two equations remain valid for the variables Q X t and S P t , it is sufficient to meet the following conditions, in which W v can be ignored whereas W k remains crucial:

Now we have obtained the simplified formulas expressed by ∆ P in Eq. (24) and (25).

To sum up, we convert the overall consistency equation Eq. (11) into two sufficient conditions Eq. (12) and (13) for Affinity and Aggregation, respectively. Consequently, we derive two corresponding consistency conditions Eq. (24) and (25) expressed by the prompt update ∆ P , under the constraint of invariant prompt distribution formulated in Eq. (20). The deduced conditions can satisfy the consistency objective in Eq. (10), thereby achieving the goal of eliminating the interference of the new task on the old task for visual prompt tuning.

As Q X t = LN( X t ) W q + b q , Eq. (24) implies that if the (transposed) prompt update can be orthogonal to the normalized previous input image tokens X t projected with a second-order transformation matrices W q W ⊤ k of the pre-trained ViT, the consistency for Affinity can be guaranteed. When we ignore the normalization and the bias term in Q X t , Eq. (24) can be simplified as X t W q W ⊤ k ∆ P ⊤ = 0 . The simplified condition is still essentially different from the consistency condition of linear layers ( i.e. , Eq. (9)) and that deduced in [26] ( i.e. , X t ∆ P ⊤ = 0 ). It indicates the interaction between the image tokens and prompts within ViT layers is fundamentally distinct, leading to a unique consistency condition related to the second-order transformation matrices W q W ⊤ k of the pre-trained model. Moreover, Eq. (25) is also an essential condition served as one of the sufficient conditions for the consistency of the whole ViT layer. It implies that if the prompt update can be orthogonal to the activated attention map generated by the image queries ( Q X ) and prompt keys ( K P ), the consistency of Aggregation can be achieved.

4.2 Optimization of Consistency Conditions

To jointly optimize Eq. (24) and (25), we need to solve ∆ P that can meet both equations concurrently. Here, we employ a separate optimization approach to get an approximate solution efficiently. Initially, it ensures ∆ P ⊤ is orthogonal to the subspace spanned by Q X t W ⊤ k to satisfy Eq. (24). Subsequently, it makes ∆ P orthogonal to the subspace spanned by S P t to satisfy Eq. (25).

Specifically, we use P G to denote the candidate parameter update generated by the optimizer for the prompts. We aim to obtain a projection matrix B such that ∆ P = B P G . Following the previously mentioned separate optimization strategy, we first ensure ∆ P ⊤ is orthogonal to Q X t W ⊤ k by the projection matrix B 1 : ∆ P ⊤ = B 1 P ⊤ G . Then ∆ P is made orthogonal to S P t by another projection matrix B 2 : ∆ P = B 2 P G . Therefore, the objective of the optimization turns into obtaining the two projection matrices B 1 and B 2 to satisfy Eq. (24) and (25). Inspired by the null-space projection method [36], the bases of B 1 and B 2 correspond to the null-space bases of Q X t W ⊤ k and S P t , respectively. We use U 1 , 0 ∈ R D × R 1 and U 2 , 0 × R M × R 2 to denote the bases of the null spaces for Q X t W ⊤ k and S P t , where R 1 and R 2 indicate their nullities. U 1 , 0 and U 2 , 0 can be obtained from the right singular vectors associated with the zero singular values, through the process of singular value decomposition (SVD) applied by SVD(( Q X t W ⊤ k ) ⊤ Q X t W ⊤ k ) and SVD( S ⊤ P t S P t ) , respectively. In this way, we get the projection matrices B 1 = U 1 , 0 U ⊤ 1 , 0 ∈ R D × D and B 2 = U 2 , 0 U ⊤ 2 , 0 ∈ R M × M , which are the solutions enabling ∆ P to jointly satisfy Eq. (24) and (25):

For the constraint Eq. (20), we incorporate an additional loss function aimed at penalizing the drift of prompt distribution, hence realizing a relaxed version of this constraint:

In Eq. (27), µ P t and σ P t represent the target prompt distribution obtained in task T t , while µ P t +1 and σ P t +1 denote the distribution to be optimized in task T t +1 .

To sum up, we use Eq. (26) to realize Eq. (24) and (25), and use Eq. (27) to meet Eq. (20), thereby achieving the consistency objective Eq. (10) for anti-forgetting. We provide a full algorithm of our approach in the appendix A .

4.3 Extension to Multi-Heads

We further extend the consistency conditions Eq. (24) and (25) to multi-head self-attention, a common feature in current transformer-based models. Suppose there are H heads and d = D/H represents the dimension of each token in a head. We use Q X t .h ∈ R N × d , W k.h ∈ R D × d and S P t .h ∈ R N × M to denote the corresponding matrices in Eq. (24) and (25) for the h -th head, respectively. The objective is to ensure these conditions are met across all heads, i.e. , Q X t .h W ⊤ k.h ∆ P ⊤ = 0 and S P t .h ∆ P = 0 , ∀ h ∈ { 1 , 2 , · · · , H } . Let Ω 1 ,t = [ Q X t . 1 W ⊤ k. 1 ; · · · ; Q X t .H W ⊤ k.H ] ∈ R HN × D and Ω 2 ,t = [ S P t . 1 ; · · · ; S P t .H ] ∈ R HN × M represent the concatenated matrices from all the heads, respectively. Based on block matrix properties, those two sets of conditions can be formulated as Ω 1 ,t ∆ P ⊤ = 0 and Ω 2 ,t ∆ P = 0 . To sum up, The main difference between single-head and multiheads is that the parameter update should be orthogonal to the subspace spanned by the concatenation matrices from all heads for multi-heads self-attention. Therefore, for the multi-heads variant, only an additional step of concatenation of the matrices from all heads is required in our algorithm.

5 Experiments

5.1 Experimental Setups

In our experiments, we mainly utilize the VPT [13] with a ViT-B/16 backbone [9] pre-trained on ImageNet-21k. Additionally, we validate the effectiveness on the CLIP [27] model, wherein the visual prompts are inserted into the image encoder. Our experiments are conducted across 4 classincremental benchmarks: 10- and 20-split CIFAR-100, 10-split ImageNet-R and 10-split DomainNet. We report the mean values of the final average accuracy and final average forgetting over 3 runs with different random seeds. Given that the null spaces of Q X t W ⊤ k and S P t may not always exist in practice, we compute the approximate null spaces and determine the nullities R 1 and R 2 in an adaptive manner, rather than the way suggested in [36]. For more detailed information regarding the experimental setups, please refer to Appendix B .

5.2 Main Results

Validation of Effectiveness: The comparison between our approach and the sequential fine-tuning VPT and CLIP baselines is shown in Table 1 . For the VPT model, the proposed NSP 2 achieves 4.47% ∼ 10.26% improvements in accuracy on the 4 benchmarks. Meanwhile, it reduces the forgetting by 9.05% ∼ 17.11%. As to the CLIP model, the NSP 2 improves the accuracy by 6.48% ∼ 9.31%, and reduces the forgetting by 2.68% ∼ 17.27%. We calculate the accuracy across all previously encountered tasks after completing training on each task. The accuracy curves of VPT-Seq and VPT-

AcC.

100

99.67 99.56 98.41

97.65

96.88

VPT-Seq

VPT-NSP2

AcC.

广

94.43

90.63

VPT-Seq

VPT-NSP2

95

90

97.43

97.0396.40

93.38

95.43

92.28

94.59

91.60

90.46

93.86

88.23

92.36

91.74

87.27

90

80

87.62

86.82

85.78

85.67

82.37

79.72

84.52

78.46

83.50

82.56

76.93

81.05

76.38

80.38

73.86

78.88

72.46

85

Task

70

Task

2

3

4

5

7

8

9

10

2

3

4

5

6

7

8

9

10

(a)10-split CIFAR-100

(b)10-split ImageNet-R

90

75

79.03

↑

48.74

82.78

72.55

↑

25.96

73.05

28.50

81.06

Forgetting

50.84

45

30

85

70

75.05

50.98

69.65

77.27

↑

28,43

85.35

71.43

29.12

74.65

55.68

Forgetting

45

30

60

15

55

15

45

ImageNet-1K

53.81

10.37

ImageNet-1K

6.12

ImageNet21K-P

5

52.36

LAION-2B

8.50

0

40

50.26

ImageNet-1K

ImageNet-1K

8.24

ImageNet21K-P

7.41

45.31

LAION-2B

10.28

0

(Naive Classification)

(DINO)

(MIIL)

(CLIP)

(Naive Classification)

(DINO)

(MIL)

(CLIP)

(a)10-split CIFAR-100

(b)20-split CIFAR-100

80

37.04

82.76

Forgetting

45

80

72.97

61.83

77.71

↑

79.96

49.24

84.92

Forgetting

60

70

66.81

69.42

↑

19.23

72.67

1

71.86

25.20

F

30

65

69.13

28.55

62.90

39.41

40

60

50

ImageNet-1K

51.68

9.59

ImageNet-1K

65.22

7.15

ImageNet21K-P

56.99

7.89

LAION-2B

4.88

15

0

35

50

6tot

ImageNet-1K

ImageNet-1K

8.65

ImageNet21K-P

52.83

10.17

LAION-2B

9.29

20

0

(Naive Classification)

(DINO)

(MIIL)

(CLIP)

(Naive Classification)

(DINO)

(MIIL)

(Image-Text)

(c)10-split ImageNet-R

(d)10-split DomainNet

NSP 2 on 10-split CIFAR-100 and 10-split ImageNet-R are displayed in Figure 2. They demonstrate our approach consistently outperforms the baseline throughout the sequential learning of tasks.

We conduct additional experiments with the VPT model, utilizing the weights pre-trained on different datasets as well as different paradigms, as shown in Figure 3 . The pre-training paradigms and datasets include: naive classification on ImageNet-1k [30], DINO [3] on ImageNet-1k, MIIL [29] on ImageNet21k-P and CLIP on LAION-2B [6] (we only use its image encoder). As can be seen from the figure, our approach not only significantly enhances accuracy but also markedly mitigates forgetting. These results further demonstrate the generalizability of the proposed approach.

Comparison with Existing Methods: We compare our method with existing methods in Table 2, where the competitors include many recent works. The proposed VPT-NSP 2 achieves state-of-the-art performance on the four benchmarks, with surpassing the second best approach by an average of 1.49% in accuracy. The forgetting of our approach is not the lowest, which is reasonable since our approach sacrifices some stability for a better trade-off between stability and plasticity. The outperforming accuracy can demonstrate the superiority of our method.

Ablation Study: The two consistency conditions Eq. (24) and (25), along with the constraint Eq. (20), constitute the main components of our approach. They correspond to B 1 , B 2 in Eq. (26), and L LN in Eq. (27). We study their effects on the four benchmarks using VPT-NSP 2 , with results presented in Table 3. We can see that the projection for Affinity ( B 1 ) plays a crucial role, which brings 3.31% ∼ 9.03% improvement in accuracy and 5.42% ∼ 14.76% decline in forgetting. Furthermore, the projection for Aggregation ( B 2 ) and the loss L LN for invariant prompt distribution are indispensable as well for minimizing forgetting. Optimal accuracy is achieved when all three conditions are applied.

Model Analysis: We analyze the evolution of training losses on the 10-split CIFAR-100 and 10-split ImageNet-R benchmarks, as shown in Figure 4 . Each point on the curve represents the training loss of the data in T 1 / T 2 after the model has been trained on subsequent tasks. As can be seen, the losses of VPT-NSP 2 on previous tasks can be almost retained, confirming that our approach can effectively mitigate the interference of new tasks on old tasks.

Trade-off between Stability and Plasticity: We first adaptively determine the nullities R 1 and R 2 for B 1 and B 2 to achieve near-minimum forgetting. Based on this, we assign two weights η 1 and η 2 to the projection matrices to control the trade-off between stability and plasticity: ∆ P =

VPT-NSP²-T

×VPT-NSP²-J2

VPT-Seq-T

×VPT-Seq-J2

VPT-NSP²-T

XVPT-NSP²-J2

VPT-Seq-T

×VPT-Seq-J2

Loss

Loss

1.5

0.10

1.0

0.05

0.5

**x

0.00

0.0

Task

2

3

4

S

7

8

9

10

2

3

4

5

6

7

8

9

10

(a)10-split CIFAR-100

(b)10-split ImageNet-R

[ η 2 B 2 +(1 -η 2 ) I ] P G [ η 1 B 1 +(1 -η 1 ) I ] , where I denotes the identity matrix. The effects of η 1 and η 2 which are set to a same value ¯ η is shown in Figure 5 . As the weight decreases, the accuracy increases first owing to better plasticity, and then decreases due to worse stability caused by the forgetting. It implies that a trade-off can be achieved by the two weights of projections.

Long-sequence Continual Learning Weexperiment on 5 benchmarks under the protocols of 50 tasks and 100 tasks to validate that our approach remains effective even within the context of long-sequence continual learning. The results are presented in Table 4. Despite lacking plasticity enhancement, VPT-NSP 2 can outperform existing state-of-the-art approaches and especially surpasses L2P by a large margin. This demonstrates that forgetting is still the predominant factor affecting performance in long sequence of tasks. With the plasticity enhancement, VPT-NSP 2 achieves significant increase in accuracy (by 1.1% ∼ 2.9%). This demonstrates that our plasticity enhancement is effective in learning new knowledge in long-sequence continual learning.

Acc.

Acc.

Acc.

Acc.

91.56

91.57

91.25

90.82

90.75

89.65

89.89

89.20

88.62

87.58

78.45

78.78

78.50

78.34

78.21

83.32

83.54

82.97

82.03

81.69

1.0

0.95

0.9

0.85

0.8

1.0

0.95

0.9

0.85

0.8

1.0

0.95

0.9

0.85

0.8

1.0

0.95

0.9

0.85

0.8

n

Forgetting

3.09

3.38

4.20

5.24

5.99

√Forgetting

4.59

4.91

5.54

6.76

8.78

√Forgetting

4.10

5.04

5.94

6.77

7.50

√Forgetting

5.92

8.54

9.65

10.93

12.51

(a)10-split CIFAR-100

(b)20-split CIFAR-100

(c)10-splitImageNet-R

(d)10-splitDomainNet

6 Conclusion

In this paper, we study the interference problem of visual prompt tuning in ViTs, and propose two consistency conditions which can eliminate the interference in theory under the constraint of invariant prompt distribution. They guarantee the consistency of Affinity, Aggregation and distribution of prompts in LayerNorm, respectively, which jointly achieve the consistency objective of the whole ViT layer. We adopt the null-space projection to implement the two conditions and utilize an extra loss to satisfy the constraint. Our experiments on various benchmarks demonstrate the effectiveness of the proposed conditions for anti-forgetting, and our approach achieves state-of-the-art performances.

Limitation Discussion: To simplify the derivation of our consistency conditions, we introduce a constraint of invariant prompt distribution. Although the superior results show that it may not be a very strong assumption, it is not an exact solution.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62101453, 62176198 and 62201467; in part by the Project funded by China Postdoctoral Science Foundation under Grant 2022TQ0260 and Grant 2023M742842, in part by the Young Talent Fund of Xi'an Association for Science and Technology under Grant 959202313088, in part by Innovation Capability Support Program of Shaanxi (Program No. 2024ZC-KJXX-043) and in part by the Natural Science Basic Research Program of Shaanxi Province (No. 2022JC-DW-08).

References

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. CoRR , abs/1607.06450, 2016.

Benjamin Bowman, Alessandro Achille, Luca Zancato, Matthew Trager, Pramuditha Perera, Giovanni Paolini, and Stefano Soatto. À-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14984-14993, 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In IEEE/CVF International Conference on Computer Vision , pages 9630-9640, 2021.

Arslan Chaudhry, Naeemullah Khan, Puneet K. Dokania, and Philip H. S. Torr. Continual Learning in Low-rank Orthogonal Subspaces. In Advances in Neural Information Processing Systems , 2020.

Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers in handwriting recognition . Suvisoft, 2006.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible Scaling Laws for Contrastive Language-Image Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2818-2829, 2023.

Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning Augmentation Strategies From Data. In IEEE/CVF International Conference on Computer Vision , pages 113-123, 2019.

Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng. Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning. In Advances in Neural Information Processing Systems , volume 34, pages 18710-18721, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations , 2021.

Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A Unified Continual Learning Framework with General Parameter-Efficient Tuning. In IEEE/CVF International Conference on Computer Vision , pages 11449-11459, 2023.

Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent Prompting for Rehearsal-Free Continual Learning. CoRR , abs/2403.08568, 2024.

Wei-Cheng Huang, Chun-Fu Chen, and Hsiang Hsu. OVOR: OnePrompt with virtual outlier regularization for rehearsal-free class-incremental learning. In International Conference on Learning Representations , 2024.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In European Conference on Computer Vision , volume 13693, pages 709-727, 2022.

Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In IEEE/CVF International Conference on Computer Vision , pages 11813-11823, October 2023.

Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-based continual learning. In IEEE/CVF International Conference on Computer Vision , pages 11429-11439, October 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , 2015.

Yajing Kong, Liu Liu, Zhen Wang, and Dacheng Tao. Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning. In European Conference on Computer Vision , volume 13686, pages 219-236, 2022.

Muhammad Rifki Kurniawan, Xiang Song, Zhiheng Ma, Yuhang He, Yihong Gong, Yang Qi, and Xing Wei. Evolving Parameterized Prompt Memory for Continual Learning. In AAAI Conference on Artificial Intelligence , pages 13301-13309, 2024.

Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N. Metaxas. Steering Prototypes with Prompt-Tuning for Rehearsal-Free Continual Learning. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2523-2533, 2024.

Yan-Shuo Liang and Wu-Jun Li. InfLoRA: Interference-free low-rank adaptation for continual learning. arXiv preprint arXiv:2404.00228 , 2024.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation , volume 24, pages 109-165. Elsevier, 1989.

Mark D. McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random Projections and Pre-trained Models for Continual Learning. In Advances in Neural Information Processing Systems , 2023.

Thomas De Min, Massimiliano Mancini, Karteek Alahari, Xavier Alameda-Pineda, and Elisa Ricci. On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformers. In IEEE/CVF International Conference on Computer Vision Workshops , pages 3577-3586, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems , pages 8024-8035, 2019.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment Matching for Multi-Source Domain Adaptation. In IEEE/CVF International Conference on Computer Vision , pages 1406-1415, 2019.

Jingyang Qiao, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, and Yuan Xie. Prompt Gradient Projection for Continual Learning. In International Conference on Learning Representations , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning , volume 139, pages 8748-8763, 2021.

Roger Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological review , 97(2):285, 1990.

Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. ImageNet-21K Pretraining for the Masses. In NeurIPS Datasets and Benchmarks , 2021.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision , 115(3):211-252, 2015.

Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient Projection Memory for Continual Learning. In International Conference on Learning Representations , 2021.

James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11909-11919, June 2023.

Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When Prompt-based Incremental Learning Does Not Meet Strong Pretraining. In IEEE/CVF International Conference on Computer Vision , pages 1706-1716, 2023.

Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality. In Advances in Neural Information Processing Systems , 2023.

Runqi Wang, Xiaoyue Duan, Guoliang Kang, Jianzhuang Liu, Shaohui Lin, Songcen Xu, Jinhu Lv, and Baochang Zhang. AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3654-3663, June 2023.

Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training Networks in Null Space of Feature Covariance for Continual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 184-193, June 2021.

Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning. In Advances in Neural Information Processing Systems , 2022.

Yabin Wang, Zhiheng Ma, Zhiwu Huang, Yaowei Wang, Zhou Su, and Xiaopeng Hong. Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference. In AAAI Conference on Artificial Intelligence , pages 10209-10217, 2023.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning. In European Conference on Computer Vision , volume 13686, pages 631-648, 2022.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to Prompt for Continual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 139-149, June 2022.

Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models , 2019.

Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, Peng Wang, and Yanning Zhang. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions on Multimedia , 2023.

Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual Learning of Context-Dependent Processing in Neural Networks. Nature Machine Intelligence , 1(8):364-372, August 2019.

Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model. In IEEE/CVF International Conference on Computer Vision , pages 19091-19101, October 2023.

Zhen Zhao, Zhizhong Zhang, Xin Tan, Jun Liu, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Rethinking Gradient Projection Continual Learning: Stability/Plasticity Feature Space Decoupling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3718-3727, June 2023.

Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CoRR , abs/2403.12030, 2024.

Appendix: Visual Prompt Tuning in Null Space for Continual Learning

A Algorithm

An overview and algorithm of our approach are provided in Figure 6 and Algorithm 1 , respectively. We first initialize the overall uncentered covariance matrices [36] C 1 and C 2 , as well as the null-space projection matrices B 1 and B 2 . During training, the cross-entropy loss for classification and the loss of prompt distribution L LN are jointly utilized for optimization. Subsequently, we get the candidate prompt updates P G computed by the optimizer. Then P G is projected by the null-space projection matrices B 1 and B 2 for updating the prompts. After the convergence, we obtain the matrices J 1 and J 2 to temporarily store Q X t W ⊤ k and S P t for the data of the current task. Then they are used to update the uncentered covariance matrices C 1 and C 2 by addition. Finally, we update the null-space projection matrices using the uncentered covariance matrices, which will be used in the next task.

Algorithm 2 shows the process of computing a null-space projection matrix. First, an input uncentered covariance matrix C is decomposed by SVD, from which we can get the singular values and right singular vectors. Next, we determine the nullity R ( i.e. , the dimension of null space) of C according to the maximum second derivative, which is introduced in Section C . Then we select R right singular vectors corresponding to the R smallest singular values considered close to 0 as the bases of null space. Finally, we compute the normalized projection matrix, which provides an upper bound for the scale of the projected gradients and prevents excessive gradient magnitudes. In our implementation, the null-space projection matrix is added by an identity matrix with a weight η (specifically η 1 for B 1 and η 2 for B 2 ). η is a hyper-parameter for the trade-off between stability and plasticity, which is also introduced in Section C

B Experimental Setups and Implementation Details

Models: We validate our approach on the Vision Transformer (ViT) [9] and CLIP [27] models in the experiments, whose backbones are both ViT-Base/16 [9]. The ViT is pre-trained on ImageNet-21k, and we insert 4 prompts into each of the 12 layers for fine-tuning, which is referred to as "VPT" [13]. The classifiers are dependently trained in each task and the orthogonal projection is not applicable to them. All the classifiers from the available tasks are concatenated to make prediction during inference. For the CLIP model pre-trained on the WebImageText, we insert 4 prompts into each of the first 3 layers of the image encoder, while the text encoder is kept frozen. The logit scale that serves as a learnable scalar parameter to scale the cosine similarities between image features and text features is also set to trainable. We observed a serious cross-task confusion among the tasks in the CLIP model. Hence, we follow [44] to utilize the class-wise mean and covariance of previous features extracted before the embedding projection head ( i.e. , the last linear layer of the image encoder) to refine the projection head, after the prompt tuning stage in each task.

Input image

tokens X,

Output image

tokens Y,

Forward propagation

Forwardpropagation

;Orthogonal projection

ViT

Layer

Back propagation

Update

△P

Gradient G

Backpropagation

Back propagation

Prompt update

△P

Prompt gradient

G

Subspace

of J1

Prompts P,

Algorithm 1 NSP 2 for Visual Prompt Tuning

Inputs: Datasets D t = { ( X <i> t , y <i> t ) } |T t | i =1 for task T t ∈ {T 1 , T 2 , · · · } , ViT model f model ( ·| P t ) with the prompts P t to be optimized (the classifier is omitted for simplicity), uncentered covariance matrices C 1 and C 2 , projection matrices B 1 and B 2

Outputs: The optimized prompts P

Algorithm 2 Computing Null-Space Projection Matrix

Inputs: Uncentered covariance matrix C , hyper-parameter η ∈ [0 , 1] for the trade-off between stability and plasticity (mentioned in Section C)

Outputs: Null-space projection matrix B

1: Get the singular values Λ in descending order and the corresponding right singular vectors U by singular value decomposition Λ, U ⊤ ← SVD( C ) , where the left singular vectors are omitted

2: Calculate the nullity R by the maximum second derivative as introduced in Eq. (28)

3: Select the right singular vectors of the R smallest singular values in U as U 0 ← U [ D -R : D ]

4: Compute the projection matrix B ← U 0 U ⊤ 0 ∥ U 0 U ⊤ 0 ∥ F

5: Update B with the weight η by B ← η B +(1 -η ) I (corresponding to Eq. (29))

Benchmarks: We conduct experiments under the class-incremental learning protocol, where the classes in each task are disjoint, and task identity is unknown during inference. Four class-incremental benchmarks with three widely used datasets are adopted: 10- and 20-split CIFAR-100, 10-split ImageNet-R [39] and 10-split DomainNet [25, 38]. For the CIFAR-100 dataset, the total of 100 classes are randomly split into 10 or 20 tasks, which can evaluate the ability to handle different numbers of tasks. We follow [39] to randomly split the 200 classes in ImageNet-R into 10 tasks, which forms the 10-split ImageNet-R benchmark. For the 10-split DomainNet, we follow the same dataset protocol adopted in [38] and [11] to select the top 200 classes with the most images from the original DomainNet [25], and randomly split them into 10 tasks with 20 classes per task. 25% samples of the training data in each dataset are picked as a validation set for searching optimal hyper-parameters.

Metrics: Formally, the final average accuracy and final average forgetting are defined as:

where T is the number of tasks, a T,i is the accuracy of the T -th model on the i -th task samples, and a j,i is the accuracy of the j -th model on the i -th task samples.

Higher accuracy means the model performs better, while lower forgetting means stronger stability ( i.e. , the ability to retain old knowledge). However, lower forgetting does not always generate higher accuracy since the accuracy is also affected by plasticity ( i.e. , the ability to learn new knowledge). The accuracy is the main metric we should focus on as it reflects the precision of classification in practice.

Implementations Details: For all the datasets and models, the images fed into the models are resized to 224 × 224 pixels and augmented by AutoAugment [7] during training. For the VPT-based models, we use the Adam optimizer [16] with β 1 = 0 . 9 , β 2 = 0 . 999 and a weight decay of 5 × 10 -5 to train 100 epochs with an initial learning rate of 0.01 and a batch size of 256 on all benchmarks. The learning rate is scaled by a factor of 0.1 at the 50-th and 80-th epoch. Our training losses consist of the cross-entropy loss for classification and the loss L LN in Eq. (27) whose coefficient is set to 1. Through cross validation on the validation set, we set the temperatures in the cross-entropy loss to 28, 25, 30 and 30 for the 10-split CIFAR100, 20-split CIFAR100, 10-split ImageNet-R and 10-split DomainNet benchmarks. There are two hyper-parameters η 1 and η 2 used for the trade-off between stability and plasticity in null-space projection as introduced in Section C , and we set both of them to be 0.97, 0.95, 0.94 and 0.95 for the four benchmarks by cross validation.

As to the CLIP-based models, the differences in training settings are as follows. We train them for 20 epochs with the batch size of 220 and the learning rate 0.001 which decays at the 10-th and 16-th epoch. The temperatures are all set to 1 since the logit scale is trainable. η 1 and η 2 are set to 0.98 which is a proper value for all the benchmarks. We refine the embedding projection head for 50 epochs using the SGD optimizer with a learning rate of 0.001, a momentum of 0.9 and a weight decay of 1 × 10 -4 .

We implement our approach in PyTorch [24] with the timm library [41]. The experiments are performed on a server with 128 GB RAM and four NVIDIA RTX 4090 GPUs. Each of the experiment can be finished in three hours.

C Trade-off between Stability and Plasticity

Given that the null space of covariance matrix does not always exist in practice, Wang et al. [36] suggest approximating it by selecting the bases whose associated singular values approach zero, where the singular values smaller than a specified multiple (denoted as γ in our paper) of the smallest one are selected. However, we experimentally find this strategy and the experience for selecting γ are not suitable for prompt tuning in ViTs to determine the nullities R 1 and R 2 for the uncentered covariance matrices C 1 and C 2 in Algorithm 1 , which will be introduced afterwards. To solve this problem, we propose an adaptive nullity strategy to determine the nullities in an adaptive manner. Utilizing the characteristic that the curve of descending singular values forms an "L" shape, we divide the curve into two parts by the point where the gradient changes fastest to cover most of the small singular values. It is realized by calculating the maximum second derivative of the points:

where λ j denotes the j -th singular value. We find it reaches near-minimum forgetting in our experiments which also means reaching near-optimal stability. Furthermore, to enhance the plasticity, we fuse the projection matrices with identity matrices by the weights η 1 ∈ [0 , 1] and η 2 ∈ [0 , 1] which should be close to 1:

In this way, we can make a trade-off between stability and plasticity by enhancing the plasticity based on near-optimal stability, and η 1 and η 2 are the hyper-parameters to control the trade-off.

D Comparison with PGP

D.1 Difference in Methods

The main difference between our method and PGP [26] are summarized as follows. (1) We derive a different consistency condition for Affinity even if we ignore the LayerNorm operation and the bias terms in the qkv-transformation. Specifically, our simplified consistency condition for Affinity is X t W q W ⊤ k ∆ P ⊤ = 0 , contrasted with X t ∆ P ⊤ = 0 in PGP. (2) We analyze the consistency conditions for the complete self-attention, i.e. , softmax( Q X K ⊤ Z √ D ) V Z which contains the Aggregation operation. However, PGP does not account for the Aggregation. (3) We take the LayerNorm before self-attention into consideration and propose an invariant prompt distribution constraint, while it is ignored in PGP.

In conclusion, we conduct a comprehensive analysis of prompt tuning for the consistency objective, which provides a complete guarantee to eliminate the interference of new tasks on previous tasks. As demonstrated in our ablation study in the Experiment section, the consistency of Aggregation and LayerNorm also contribute to reducing forgetting, and thereby they should not be ignored. We make a comparison of the performance between PGP and our approach in the next subsection.

D.2 Performance Comparison

We compare with PGP [26] using the VPT-Seq and L2P [40] baselines on the four benchmarks in our experiments. The results are shown in Table 5 . We implement PGP to VPT ( i.e. VPT-PGP) under the same training settings as VPT-NSP 2 for a fair comparison. For the L2P-based methods, we insert prompts into the first three layers instead of only the first layer in the original implementation [40]. An orthogonal projection is also applied to the prompt pool which is essentially a linear layer in L2P-based models. We follow the training setting of PGP to train the L2P-based methods. The results in Table 5 demonstrate that our full approach can achieve more improvements in accuracy and reduce more forgetting than PGP. Even when applying only the projection matrix B 1 for the Affinity operation, our approach also performs better than PGP, demonstrating the effectiveness of our proposed method for mitigating the interference problem.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state the claims in our abstract: we derive two consistency conditions of eliminating the interference problem under the invariant prompt distribution assumption for visual prompt tuning in the filed of continual learning. We implement them by the null-space projection method, and we validate the effectiveness and generalizability of our method. Our contributions are elaborated in the last paragraph of the Introduction section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation of our approach is discussed in the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The formulas used to derive our proposed conditions are numbered or crossreferenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the algorithm of our approach, experimental settings and hyperparameters adopted in our experiments in the Experimental Setups and Implementation Details section of the appendix. Our code is also available in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is available in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the detailed experimental settings, including the data splits, hyperparameters, optimizer and other settings in the experimental setups of appendix. We also provide the code in the supplemental material for a thorough reference.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the mean results over three runs in our experiments, and we report the standard deviations in the subsection of comparison with existing methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information of our compute resources in the experimental setups of appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and make sure our research conforms the ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: As a fundamental research in machine learning, the potential societal impact is not obvious at this stage.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No new pre-trained models or datasets are released in this work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We explicitly cite the used assets in our papers, including the ViT model, timm library, DomainNet dataset, etc. , and respect their license and terms during usage.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is provided in the supplemental material. A documentation for running the experiments is contained in the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. Moreover, two sufficient consistency conditions for self-attention and an invariant prompt distribution constraint for LayerNorm are theoretically deduced, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. Finally, extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods.

STRENGTHS:
1)The research motivation for algorithms is reasonable, and the theoretical proof is solid. Constraining the new learnable parameter orthogonal to the previous weight is a reasonable motivation to prevent historical knowledge forgetting.

2)Extensive evaluation and amazing performance show the superiority of the proposed method.

WEAKNESSES:
1)Eq.(8) seems be error.

2)L139: Eq.(8) suggests that if the weight update ∆Θ is orthogonal to the previous input feature Xt during training in the new task, the corresponding output feature will remain unchanged. Moreover, will this degrade the model’s discriminative ability for the current task when the weight update is orthogonal to the previous feature?

3)What is the model’s complexity and running time compared to the baseline VPT?

4)Suggest using a more accurate diagram to describe the algorithm to help understand it more clearly.

QUESTIONS:
Please see “Weaknesses”

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. They propose two sufficient consistency conditions for the self-attention and an invariant prompt distribution constraint for LayerNorm, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. Extensive experimental results show the superiority of the proposed method.\n\nSTRENGTHS:\n1)\tThe idea in this paper is novel and interesting. They introduce the orthogonal projection into the visual prompting tuning for continual learning, which comprehensively considers the full operations in the Transformer Layer.\n2)\tThe experimental results are sufficient and superior to the SOTA methods.\n3)\tOverall, this paper is well organized and well-written, which is easy for the readers to follow.\n\nWEAKNESSES:\n1)\tIn figure 1 of the overall framework, it is unclear why the Qp term in the affinity matrix can be neglected. The authors should provide clear reasons for such conclusion.\n2)\tIn the method part of Eq.(10), In order to satisfy F_{Z_t}=F_{Z_t-1}, the authors transform Eq.(10) into the following Eq.(11) and Eq.(12), which are two sufficient conditions. Why not optimizing Eq.(10) directly? Please give detailed reasons.\n3)\tWhat’s the main differences between the single-head and multi-head self-attention for introducing the proposed orthogonal projection into the visual prompting tuning for continual learning?\n4)\tIn the method optimization part, the authors propose the approximation method as illustrated in Eq.(25). Why not using the optimization method proposed in PGP[26]? What are the main differences ?\n\nQUESTIONS:\nSee the weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to eliminate the interference on previously learned knowledge for visual prompt tuning in the field of continual learning, so that catastrophic forgetting can be mitigated. To this end, it analyzes the conditions for keeping the output features unchanged in the transformer block that features the self-attention mechanism. Two consistency conditions for the self-attention are derived by deducing the proposed two sufficient conditions for the consistency objective. Moreover, a constraint on the distribution of the prompts is proposed to further simplify the LayerNorm operation. Consequently, the interference can be eliminated in theory by achieving the proposed two consistency conditions and the prompt-invariance constraint. The proposed approach implements the two consistency conditions by performing two null-space projections on the prompt gradients during training a new task; the constraint is implemented by an additional loss function that penalizes the drifting of prompt distribution across sequential tasks. Substantial experiments demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\nThis work comprehensively analyzes the conditions for learning without interference in the transformer-based visual prompt tuning. The proposed two consistency conditions with the constraint provide a theoretical guarantee on eliminating the interference problem. With this solid mathematical support, the proposed approach, which performs null-space projection on prompt gradients, shows significant improvements in reducing forgetting and increasing accuracy. The effectiveness of the approach is validated on extensive experiments, involving four class-incremental benchmarks and various pre-training datasets and paradigms. By visualizing the evolution of training losses, the effectiveness in mitigating interference is demonstrated as well. The proposed approach also achieves state-of-the-art performance on the four benchmarks. Besides, the adaptive nullity and plasticity enhancement strategy is also well-motivated and validated to be an effective way of balancing stability and plasticity. The paper is clearly structured and easy to follow. The figures in the paper are clear and well-designed, enhancing the overall readability and comprehension of the paper.\n\nWEAKNESSES:\n1. The consistency objective Eq. (10) is decomposed into two sufficient conditions, namely Eq. (11) and Eq. (12). A detailed explanation is necessary to understand why direct analysis and simplification of Eq. (10) is not pursued.\n2. The variable Q_p is omitted during the deduction of consistency conditions. Further explanation is needed to clarify why it can be disregarded.\n3. In the Adam-NSCL [36] method, the trade-off between stability and plasticity is controlled solely by the nullity. In contrast, this approach involves adding weighted identity matrices to the projection matrices. How do these two methods differ in balancing stability and plasticity? A deeper discussion is expected to highlight the advantage of the proposed trade-off strategy.\n4. The number of training epochs in this approach (100) is greater than that in other approaches (e.g., 50 epochs in DualPrompt for ImageNet-R). It would be better to provide a justification for this training setting.\n5. As described in line 4 of Algorithm 2, the projection matrix is normalized by a Frobenius norm. There is a lack of explanation for this operation.\n\nQUESTIONS:\nIn addition to the parameters of the prompts, the classifier contains parameters that require updates (i.e. the weights and biases) as well. Why not perform an orthogonal projection on the parameters of the classifier to reduce forgetting?\n\nLIMITATIONS:\nReasonable discussion on limitations is included in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel paradigm for continual learning based on prompt tuning. By deriving the constraints for orthogonal projection of prompt gradients in ViTs, the method aims to minimize forgetting during the learning process. Experiments on four benchmarks show that NSP² achieves superior performance by avoiding forgetting.\n\nSTRENGTHS:\n1. The paper focuses on orthogonal projection methods in continual learning, extending CNN-based methods to the ViT architecture, resulting in performance improvements.\n2. The method is supported by mathematical derivations of the derived constraints, providing strong theoretical guarantees.\n3. The paper addresses continual learning based on pre-trained models, which is a valuable topic.\n\nWEAKNESSES:\n1. While orthogonal gradient updates help prevent forgetting in continual learning, they also hinder positive knowledge transfer between tasks. In methods like L2P, similar tasks might select the same prompts, potentially improving performance on previous tasks after learning new ones.\n2. The derivation process in the methods section is overly lengthy, which leads to a very brief description of the experimental section. The comparison with existing methods is reduced to just three lines of text. Consider moving some derivations to the appendix.\n3. The experimental tables contain many blanks, which is not conducive to a systematic comparison of different methods' performance and weakens the contribution of this method.\n4. The paper omits comparisons with some leading methods, such as DAP[1], which is also based on prompt tuning.\n\n[1] Jung D, Han D, Bang J, et al. Generating instance-level prompts for rehearsal-free continual learning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11847-11857.\n\nQUESTIONS:\nAccording to my understanding, as the number of tasks increases, the shrinking gradient subspace will make gradient updates increasingly difficult. Therefore, I would like to see the performance of NSP² on longer sequences of continual learning tasks.\n\nLIMITATIONS:\nAs mentioned in the paper, additional constraints are introduced to simplify the consistency conditions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}