Visual Prompt Tuning in Null Space for Continual Learning

Authors: yue  lu
shizhou  zhang
  de cheng
yinghui  xing
nannan  wang
peng  wang
yanning  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 09:22:11

Abstract: Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL), by selecting and updating relevant prompts in the vision-transformer models. On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference on tasks that have been learned to overcome catastrophic forgetting in CL. However, different from the orthogonal projection in the traditional CNN architecture, the prompt gradient orthogonal projection in the ViT architecture shows completely different and greater challenges, i.e., 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. Theoretically, we have finally deduced two consistency conditions to achieve the prompt gradient orthogonal projection, which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient orthogonal projection. Extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods. Our code is available in the supplemental material.

Main paper: 
 introduction: 
 Continual learning (CL) is crucial for AI models to adapt to the ever-changing environment by learning sequentially arrived data, where the catastrophic forgetting is the key challenge [21, 28] . Recently, prompt tuning-based continual learning methods [40, 32, 34, 43, 10, 22, 38, 45, 20, 12, 18] have been attracting increasing attention due to their impressive performances in the CL field. Existing prompt tuning-based works tackle the downstream continual learning problem by selecting and updating relevant prompts, which is encoded with full task-specific knowledge while exploiting the general knowledge of the pre-trained ViTs [40, 39] .On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference with tasks that have been learned to overcome catastrophic forgetting in CL. It is worth noting that forgetting can be theoretically resolved by gradient orthogonal projection methods [42, 31, 36, 44] , which have been extensively explored especially when adapting CNN models. Nevertheless, it remains a huge gap to introduce the orthogonal projection-based methods of CNNs to visual prompt tuning due to the following challenges: 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. For the linear operation in convolution or fully-connected layers, the output features of old tasks can remain unchanged by updating the weights in the orthogonal subspace of previous input features. While for self-attention, three linear transformations are employed on input tokens, followed by high-order and non-linear operations for the self-attention interaction of tokens. It makes the relationship between the update of prompts and the output image tokens much more complex, far exceeding mere linearity.In this work, we theoretically deduced two consistency conditions to achieve the prompt gradient orthogonal projection, which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. To be concrete, we firstly take the full self-attention and LayerNorm into consideration and derive a strict condition for eliminating the interference through a comprehensive analysis of the forward propagation of the ViT layer. Then we further propose to convert the condition of self-attention into its two sufficient conditions, which enables us to address the challenge of high order and nonlinearity. Thirdly, we propose a constraint of invariant prompt distribution that removes the obstacle to the final simplification of the conditions brought by the LayerNorm. The consistency conditions reveal that if the prompt update can be orthogonal to (1) the normalized previous input image tokens projected with the second-order qkv-transformation matrices of the pre-trained model, and (2) the activated attention map generated by image queries and prompt keys, the interference in visual prompt tuning can be eliminated theoretically.In practice, based on the proposed consistency conditions, an effective null-space-based approximation solution [36] has been proposed to implement the prompt gradient orthogonal projection, while the invariant prompt distribution constraint is implemented by incorporating a loss function which penalizes the drifting of prompt distribution over sequential tasks. We validate our Null-Space Projection for Prompts (NSP 2 ) approach on extensive class-incremental benchmarks: 10-and 20-split CIFAR-100, 10-split ImageNet-R [39] and 10-split DomainNet [38] , with the sequential fine-tuning VPT and CLIP models as baselines. Our approach brings 4%∼10% improvements in accuracy, and reduces 9%∼17% forgetting, which is superior to state-of-the-art methods.Our contributions are summarized as follows: (1) We introduce the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. (2) Two sufficient consistency conditions for the self-attention and an invariant prompt distribution constraint for LayerNorm are theoretically deduced, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. (3) Extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods. 
related work: 
 Prompting-Based Approaches: Most of the prompting-based approaches adopt a two-stage framework [37, 39, 14, 15, 32, 34, 35, 11, 18, 19] : querying a group of prompts for an individual sample and using them to prompt the pre-trained models. For example, L2P [40] first selects a group of prompts from a prompt pool and then feeds them into the ViT. CPrompt [11] proposes to mitigate the gap between training and testing stages to enhance prediction robustness and boost prompt selection accuracy. These approaches essentially focus on acquisition of task-specific prompts tailored to individual samples. There are also several one-stage methods [2, 22, 38, 43, 20] based on prompt tuning. (1) Slowly updating trainable parameters [10, 43] : e.g., LAE [10] updates an offline expert with a large momentum to reduce the change of features. (2) Expandable backbones [45, 20] : e.g., EASE [45] trains a distinct lightweight adapter module for each new task, and designs a semantic mapping to complement the drift of old class prototypes. (3) Enhancing classifiers rather than focusing on learning features [38, 22, 12] : e.g., ESN [38] proposes an anchor-based classifier alignment approach based on energy-based models. As introduced above, these works still lack of a theoretical solution to the interference problem for visual prompt tuning. In our work, we conduct a deep analysis of this problem and provide a theoretical guidance on eliminating the interference.Orthogonal Projection-Based Approaches: Orthogonal projection-based approaches [42, 4, 8, 31, 36, 17, 44] can theoretically eliminate the interference of new tasks on old tasks for linear layers. OWM [42] constructs a projector to find the direction orthogonal to the input space. GPM [31] first projects new gradients to the subspace important to the old tasks and then subtracts the projected components for updating parameters. Adam-NSCL [36] projects the parameter updates to the approximate null space of previous inputs. However, due to the different relationships between parameter updates and outputs in the linear operation and self-attention, the consistency condition used in CNNs is not directly applicable to the prompt tuning in ViTs. In our work, we derive the consistency conditions for the visual prompt tuning, enabling the application of orthogonal projectionbased approaches to it, where the null-space projection [36] is adopted in our approach to get an approximate solution efficiently. We notice that a recently emerged work PGP [26] implements GPM [31] to prompt-based frameworks. However, it obtains the same conclusion as that of the linear operation under a simplified attention, which limits its application and performance as compared in the appendix D . 
preliminaries: 
 Continual Learning: In the setting of continual learning, a network f (•|Θ) with parameters Θ is sequentially trained on a stream of disjoint tasks{T 1 , T 2 , • • • , T T }, where task T t is associated with paired data {(X <i> t , y <i> t ) |Tt| i=1 } of size |T t |.When a task T t arrives, the model f (•|Θ) would be trained for the current task, while the data from previous tasks is unreachable.Forward Propagation of Visual Prompt Tuning in ViT Layers: We describe the forward propagation process of the ViT layer for visual prompt tuning, as illustrated in Figure 1 . Let X ∈ R N ×D and P ∈ R M ×D denote the N input image tokens of a sample (including the pre-trained class token if available) and M prompts, respectively, where D is the dimension of each token. In the ViT layer, only the prompts P are trainable parameters. The remaining parameters in LayerNorm, qkv-transformations and subsequent MLP introduced below are pre-trained and kept frozen. We use Z = [X; P] ∈ R (N +M )×D to denote the concatenated input tokens. First, they undergo the LayerNorm [1] operation LN(•):EQUATIONwhere µ Z , σ Z ∈ R N +M , α, β ∈ R D . The ⊙ and division here denote the element-wise (Hadamard) product and division, respectively. Note that the vectors µ Z , σ Z , α and β are broadcasted to match the matrices of dimensions (N + M ) × D, enabling them to carry out operations with Z. Then the normalized tokens are fed into the qkv-transformations:EQUATIONwhere W {q,k,v} ∈ R D×D . The vector b {q,k,v} ∈ R D is broadcasted to a matrix of dimensions (N + M ) × D to facilitate the addition operation. Next is the self-attention:EQUATIONwhere Q X denotes the image tokens serving as queries. Eq. ( 3) can be expanded as Affinity, softmax (on rows) and Aggregation operations:EQUATIONEQUATIONIt is worth noting that the rows of the attention map where the prompts serve as queries (i.e., Q P ) do not need to be computed, as formulated in Eq. ( 4) and illustrated in Figure 1 . The reason is that in VPT-Deep [13] , the output prompts of this ViT layer will be replaced with new trainable prompts in the subsequent layer. Omitting Q P has no impact on the output image tokens of the ViT layer, as the subsequent Aggregation, LayerNorm and MLP operations are performed independently for each token. If no new prompts are added in the next layer, the output prompts can be just discarded as well.After the self-attention, operations consist of another LayerNorm and the MLP layer are applied individually to each token, without any interaction among the tokens. Finally, the output fine-tuned image tokens are fed into the next ViT layer.Orthogonal Projection in Convolutional Layers: A convolutional operation is actually a linear operation. For a convolutional layer f conv (•|Θ t ) in task T t , we use Θ t ∈ R Din×Dout to denote its unrolled convolutional kernel matrix [5] . Here, D in represents the number of pixels within a kernel, and D out corresponds to the number of kernels. Each convolutional patch from the input feature map is flattened into a row vector with a dimension of D in . These row vectors of totaling n p patches compose the input feature matrix X t ∈ R np×Din . The output feature for X t in task T t is expected to remain unchanged (referred to as consistent) in the next task T t+1 to prevent forgetting:EQUATIONBy substituting Θ t+1 = Θ t +∆Θ, with ∆Θ ̸ = 0 denoting the weight update in T t+1 , the consistency condition for the convolutional layer is established as follows:EQUATIONwhich can be further simplified as:EQUATIONEq. ( 9) suggests that if the weight update ∆Θ is orthogonal to the previous input feature X t during training in the new task, the corresponding output feature will remain unchanged. Thereby, the interference of the new task on the old task is eliminated. This can be realized by projecting the candidate weight update Θ G into the orthogonal subspace of X t : ∆Θ = PΘ G , where P ∈ R Din×Din is an orthogonal projection matrix [42, 36, 31] .Similarly, for the prompt tuning which fine-tunes the prompts P t in a ViT layer f vit (X t |P t ), we also aim to satisfy the following consistency objective for the purpose of anti-forgetting:EQUATIONHowever, the consistency condition in Eq. ( 9) does not hold for Eq. ( 10), since f vit (X t |P t ) ̸ = X t P t in prompt tuning. Instead, all the tokens X t and P t first undergo a LayerNorm and then interact via the self-attention mechanism, as previously described. The complicated forward propagation within the ViT layer brings huge challenge to analyzing the consistency conditions in relation to the prompt update ∆P. In the next section, we will tackle this challenge and derive the consistency conditions for visual prompt tuning. 
method: 
 We use Z t = [X t ; P t ] and Z t+1 = [X t ; P t+1 ] to denote the input tokens before and after updating the prompts, respectively, where P t+1 = P t + ∆P, ∆P ̸ = 0. Our goal is to analyze how to satisfy Eq. ( 10) and derive one or more conditions expressed in terms of the prompt update ∆P. These conditions will subsequently guide the application of orthogonal projection to ∆P.As can be seen in Figure 1 , those outputs of LayerNorm and qkv-transformations corresponding to the image tokens remain unaffected by the updates to the prompts. Hence, the essence of attaining the consistency objective Eq. ( 10) can be turned into analyzing how to keep the output of self-attention in Eq. (3) unchanged as the prompts are updated, i.e., satisfying:EQUATIONHowever, the nonlinear operation (i.e., softmax) and the potential higher-order termW ⊤ k Z ⊤ ZW v arising from K ⊤ Z V Z in Eq.(3) complicate the direct resolution of this objective. Specifically, the non-injection property of the softmax function causes non-unique solutions. The multiplication between K Z ⊤ t+1 V Zt+1 derives a quadratic term LN(P t + ∆P) ⊤ LN(P t + ∆P), which result in difficult optimization for ∆P.To address this issue, we propose two sufficient conditions consisting solely of linear operations. Specifically, we split the process of self-attention into two primary stages, i.e., the Affinity described by Eq. ( 4) and the Aggregation outlined in Eq. ( 6). We can achieve Eq. ( 11) by ensuring the consistency of each stage:f aff (Q Xt , K Zt ) = f aff (Q Xt , K Zt+1 ), f agg (S Zt , V Zt ) = f agg (S Zt+1 , V Zt+1 ). (12) (13)We first analyze the consistency objective of Affinity, i.e., Eq. ( 12), for Z t and Z t+1 :EQUATIONwhere √ D is omitted for simplicity. Upon fulfilling Eq. ( 12), we can obtain S Zt = S Zt+1 , corresponding to the output of Eq. ( 5). Subsequently, we analyze the consistency objective of Aggregation in Eq. ( 13), yielding results for Z t and Z t+1 as:EQUATIONBased on Eq. (12) (13) (14) (15) (16) (17) , we are able to derive the following two equations, respectively:EQUATIONNote that we expect to further deduce Eq. ( 18) and Eq. ( 19) to obtain equations among LN(P t ), LN(P t + ∆P) and ∆P. However, due to the square root and quadratic terms in the expressions of the standard deviations σ Pt and σ Pt+∆P , it is difficult to express σ Pt+∆P in terms of σ Pt and σ ∆P . Consequently, it is challenging to derive a straightforward equation that relates LN(P t ) and LN(P t + ∆P) through ∆P.To simplify the problem, we introduce an additional constraint on the distribution of prompts. Concretely, we require that the updated prompts P t + ∆P retain the same distribution as P t , i.e., meeting the following assumption:EQUATIONIn this way, we can establish a straightforward mathematical relationship connecting LN(P t + ∆P), LN(P t ) and ∆P:EQUATIONConsequently, we can apply Eq. ( 21) to simplify Eq. ( 18) and ( 19) as:Q Xt W ⊤ k LN(P t ) ⊤ = Q Xt W ⊤ k LN(P t ) ⊤ + Q Xt W ⊤ k ∆P ⊤ /σ ⊤ Pt ⊙ α ⊤ , S Pt LN(P t )W v = S Pt LN(P t )W v + S Pt ∆PW v /σ Pt ⊙ α. (22) (23)It should be noted that in Eq. 22 and Eq. 23 , W k , W v and α are pre-trained parameters kept frozen throughout the continual learning process. Q Xt and S Pt are two matrices derived from the input X t . As our objective is to ensure that the above two equations remain valid for the variables Q Xt and S Pt , it is sufficient to meet the following conditions, in which W v can be ignored whereas W k remains crucial:Q Xt W ⊤ k ∆P ⊤ = 0 S Pt ∆P = 0 (24) (25)Now we have obtained the simplified formulas expressed by ∆P in Eq. ( 24) and (25) .To sum up, we convert the overall consistency equation Eq. ( 11) into two sufficient conditions Eq. ( 12) and ( 13) for Affinity and Aggregation, respectively. Consequently, we derive two corresponding consistency conditions Eq. ( 24) and ( 25) expressed by the prompt update ∆P, under the constraint of invariant prompt distribution formulated in Eq. ( 20). The deduced conditions can satisfy the consistency objective in Eq. ( 10), thereby achieving the goal of eliminating the interference of the new task on the old task for visual prompt tuning.As Q Xt = LN(X t )W q + b q , Eq. ( 24) implies that if the (transposed) prompt update can be orthogonal to the normalized previous input image tokens X t projected with a second-order transformation matrices W q W ⊤ k of the pre-trained ViT, the consistency for Affinity can be guaranteed. When we ignore the normalization and the bias term in Q Xt , Eq. ( 24) can be simplified as X t W q W ⊤ k ∆P ⊤ = 0. The simplified condition is still essentially different from the consistency condition of linear layers (i.e., Eq. ( 9)) and that deduced in [26] (i.e., X t ∆P ⊤ = 0). It indicates the interaction between the image tokens and prompts within ViT layers is fundamentally distinct, leading to a unique consistency condition related to the second-order transformation matrices W q W ⊤ k of the pre-trained model. Moreover, Eq. ( 25) is also an essential condition served as one of the sufficient conditions for the consistency of the whole ViT layer. It implies that if the prompt update can be orthogonal to the activated attention map generated by the image queries (Q X ) and prompt keys (K P ), the consistency of Aggregation can be achieved.To jointly optimize Eq. ( 24) and ( 25), we need to solve ∆P that can meet both equations concurrently. Here, we employ a separate optimization approach to get an approximate solution efficiently. Initially, it ensures ∆P ⊤ is orthogonal to the subspace spanned by Q Xt W ⊤ k to satisfy Eq. ( 24). Subsequently, it makes ∆P orthogonal to the subspace spanned by S Pt to satisfy Eq. ( 25). Specifically, we use P G to denote the candidate parameter update generated by the optimizer for the prompts. We aim to obtain a projection matrix B such that ∆P = BP G . Following the previously mentioned separate optimization strategy, we first ensure ∆P ⊤ is orthogonal toQ Xt W ⊤ k by the projection matrix B 1 : ∆P ⊤ = B 1 P ⊤ G .Then ∆P is made orthogonal to S Pt by another projection matrix B 2 : ∆P = B 2 P G . Therefore, the objective of the optimization turns into obtaining the two projection matrices B 1 and B 2 to satisfy Eq. ( 24) and (25) . Inspired by the null-space projection method [36] , the bases of B 1 and B 2 correspond to the null-space bases of Q Xt W ⊤ k and S Pt , respectively. We use U 1,0 ∈ R D×R1 and U 2,0 × R M ×R2 to denote the bases of the null spaces for Q Xt W ⊤ k and S Pt , where R 1 and R 2 indicate their nullities. U 1,0 and U 2,0 can be obtained from the right singular vectors associated with the zero singular values, through the process of singular value decomposition (SVD) applied by SVD((Q Xt W ⊤ k ) ⊤ Q Xt W ⊤ k )and SVD(S ⊤ Pt S Pt ), respectively. In this way, we get the projection matricesB 1 = U 1,0 U ⊤ 1,0 ∈ R D×D and B 2 = U 2,0 U ⊤ 2,0 ∈ R M ×M, which are the solutions enabling ∆P to jointly satisfy Eq. ( 24) and ( 25):EQUATION)For the constraint Eq. ( 20), we incorporate an additional loss function aimed at penalizing the drift of prompt distribution, hence realizing a relaxed version of this constraint: To sum up, we use Eq. ( 26) to realize Eq. ( 24) and ( 25), and use Eq. ( 27) to meet Eq. ( 20), thereby achieving the consistency objective Eq. ( 10) for anti-forgetting. We provide a full algorithm of our approach in the appendix A .EQUATIONWe further extend the consistency conditions Eq. ( 24) and ( 25) to multi-head self-attention, a common feature in current transformer-based models. Suppose there are H heads and d = D/H represents the dimension of each token in a head. We useQ Xt.h ∈ R N ×d , W k.h ∈ R D×d and S Pt.h ∈ R N ×Mto denote the corresponding matrices in Eq. ( 24) and ( 25) for the h-th head, respectively. The objective is to ensure these conditions are met across all heads, i.e.,Q Xt.h W ⊤ k.h ∆P ⊤ = 0 and S Pt.h ∆P = 0, ∀h ∈ {1, 2, • • • , H}. Let Ω 1,t = [Q Xt.1 W ⊤ k.1 ; • • • ; Q Xt.H W ⊤ k.H ] ∈ R HN ×D and Ω 2,t = [S Pt.1 ; • • • ; S Pt.H ] ∈ R HN ×M representthe concatenated matrices from all the heads, respectively. Based on block matrix properties, those two sets of conditions can be formulated as Ω 1,t ∆P ⊤ = 0 and Ω 2,t ∆P = 0. To sum up, The main difference between single-head and multiheads is that the parameter update should be orthogonal to the subspace spanned by the concatenation matrices from all heads for multi-heads self-attention. Therefore, for the multi-heads variant, only an additional step of concatenation of the matrices from all heads is required in our algorithm. 
experimental setups: 
 In our experiments, we mainly utilize the VPT [13] with a ViT-B/16 backbone [9] pre-trained on ImageNet-21k. Additionally, we validate the effectiveness on the CLIP [27] model, wherein the visual prompts are inserted into the image encoder. Our experiments are conducted across 4 classincremental benchmarks: 10-and 20-split CIFAR-100, 10-split ImageNet-R and 10-split DomainNet. We report the mean values of the final average accuracy and final average forgetting over 3 runs with different random seeds. Given that the null spaces of Q Xt W ⊤ k and S Pt may not always exist in practice, we compute the approximate null spaces and determine the nullities R 1 and R 2 in an adaptive manner, rather than the way suggested in [36] . For more detailed information regarding the experimental setups, please refer to Appendix B .Validation of Effectiveness: The comparison between our approach and the sequential fine-tuning VPT and CLIP baselines is shown in Table 1 . For the VPT model, the proposed NSP 2 achieves 4.47%∼10.26% improvements in accuracy on the 4 benchmarks. Meanwhile, it reduces the forgetting by 9.05%∼17.11%. As to the CLIP model, the NSP 2 improves the accuracy by 6.48%∼9.31%, and reduces the forgetting by 2.68%∼17.27%. We calculate the accuracy across all previously encountered tasks after completing training on each task. The accuracy curves of VPT-Seq and VPT- NSP 2 on 10-split CIFAR-100 and 10-split ImageNet-R are displayed in Figure 2 . They demonstrate our approach consistently outperforms the baseline throughout the sequential learning of tasks.We conduct additional experiments with the VPT model, utilizing the weights pre-trained on different datasets as well as different paradigms, as shown in Figure 3 . The pre-training paradigms and datasets include: naive classification on ImageNet-1k [30] , DINO [3] on ImageNet-1k, MIIL [29] on ImageNet21k-P and CLIP on LAION-2B [6] (we only use its image encoder). As can be seen from the figure, our approach not only significantly enhances accuracy but also markedly mitigates forgetting. These results further demonstrate the generalizability of the proposed approach.Comparison with Existing Methods: We compare our method with existing methods in Table 2 , where the competitors include many recent works. The proposed VPT-NSP 2 achieves state-of-the-art performance on the four benchmarks, with surpassing the second best approach by an average of 1.49% in accuracy. The forgetting of our approach is not the lowest, which is reasonable since our approach sacrifices some stability for a better trade-off between stability and plasticity. The outperforming accuracy can demonstrate the superiority of our method.Ablation Study: The two consistency conditions Eq. ( 24) and ( 25), along with the constraint Eq. ( 20), constitute the main components of our approach. They correspond to B 1 , B 2 in Eq. ( 26), and L LN in Eq. ( 27). We study their effects on the four benchmarks using VPT-NSP 2 , with results presented in Table 3 . We can see that the projection for Affinity (B 1 ) plays a crucial role, which brings 3.31%∼9.03% improvement in accuracy and 5.42%∼14.76% decline in forgetting. Furthermore, the projection for Aggregation (B 2 ) and the loss L LN for invariant prompt distribution are indispensable as well for minimizing forgetting. Optimal accuracy is achieved when all three conditions are applied.We analyze the evolution of training losses on the 10-split CIFAR-100 and 10-split ImageNet-R benchmarks, as shown in Figure 4 . Each point on the curve represents the training loss of the data in T 1 /T 2 after the model has been trained on subsequent tasks. As can be seen, the losses of VPT-NSP 2 on previous tasks can be almost retained, confirming that our approach can effectively mitigate the interference of new tasks on old tasks.We first adaptively determine the nullities R 1 and R 2 for B 1 and B 2 to achieve near-minimum forgetting. Based on this, we assign two weights η 1 and η 2 to the projection matrices to control the trade-off between stability and plasticity: ∆P = Table 2: Comparison with existing methods that use the pre-trained ViT-B/16 on ImageNet-21k. The standard deviations are also reported if available. Missing results in the corresponding papers are denoted as "-". The results marked with † and ‡ are implemented by [11] and [10] , respectively. The highest accuracies are in bold, and the second highest accuracies are underlined. [η 2 B 2 + (1 -η 2 )I] P G [η 1 B 1 + (1 -η 1 )I],where I denotes the identity matrix. The effects of η 1 and η 2 which are set to a same value η is shown in Figure 5 . As the weight decreases, the accuracy increases first owing to better plasticity, and then decreases due to worse stability caused by the forgetting. It implies that a trade-off can be achieved by the two weights of projections.Long-sequence Continual Learning We experiment on 5 benchmarks under the protocols of 50 tasks and 100 tasks to validate that our approach remains effective even within the context of long-sequence continual learning. The results are presented in Table 4 . Despite lacking plasticity enhancement, VPT-NSP 2 can outperform existing state-of-the-art approaches and especially surpasses L2P by a large margin. This demonstrates that forgetting is still the predominant factor affecting performance in long sequence of tasks. With the plasticity enhancement, VPT-NSP 2 achieves significant increase in accuracy (by 1.1%∼2.9%). This demonstrates that our plasticity enhancement is effective in learning new knowledge in long-sequence continual learning.  
conclusion: 
 In this paper, we study the interference problem of visual prompt tuning in ViTs, and propose two consistency conditions which can eliminate the interference in theory under the constraint of invariant prompt distribution. They guarantee the consistency of Affinity, Aggregation and distribution of prompts in LayerNorm, respectively, which jointly achieve the consistency objective of the whole ViT layer. We adopt the null-space projection to implement the two conditions and utilize an extra loss to satisfy the constraint. Our experiments on various benchmarks demonstrate the effectiveness of the proposed conditions for anti-forgetting, and our approach achieves state-of-the-art performances.Limitation Discussion: To simplify the derivation of our consistency conditions, we introduce a constraint of invariant prompt distribution. Although the superior results show that it may not be a very strong assumption, it is not an exact solution.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide the detailed experimental settings, including the data splits, hyperparameters, optimizer and other settings in the experimental setups of appendix. We also provide the code in the supplemental material for a thorough reference.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report the mean results over three runs in our experiments, and we report the standard deviations in the subsection of comparison with existing methods.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide the information of our compute resources in the experimental setups of appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: As a fundamental research in machine learning, the potential societal impact is not obvious at this stage.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No new pre-trained models or datasets are released in this work.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We explicitly cite the used assets in our papers, including the ViT model, timm library, DomainNet dataset, etc., and respect their license and terms during usage.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our code is provided in the supplemental material. A documentation for running the experiments is contained in the code. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: An overview and algorithm of our approach are provided in Figure 6 and Algorithm 1 , respectively. We first initialize the overall uncentered covariance matrices [36] C 1 and C 2 , as well as the null-space projection matrices B 1 and B 2 . During training, the cross-entropy loss for classification and the loss of prompt distribution L LN are jointly utilized for optimization. Subsequently, we get the candidate prompt updates P G computed by the optimizer. Then P G is projected by the null-space projection matrices B 1 and B 2 for updating the prompts. After the convergence, we obtain the matrices J 1 and J 2 to temporarily store Q Xt W ⊤ k and S Pt for the data of the current task. Then they are used to update the uncentered covariance matrices C 1 and C 2 by addition. Finally, we update the null-space projection matrices using the uncentered covariance matrices, which will be used in the next task.Algorithm 2 shows the process of computing a null-space projection matrix. First, an input uncentered covariance matrix C is decomposed by SVD, from which we can get the singular values and right singular vectors. Next, we determine the nullity R (i.e., the dimension of null space) of C according to the maximum second derivative, which is introduced in Section C . Then we select R right singular vectors corresponding to the R smallest singular values considered close to 0 as the bases of null space. Finally, we compute the normalized projection matrix, which provides an upper bound for the scale of the projected gradients and prevents excessive gradient magnitudes. In our implementation, the null-space projection matrix is added by an identity matrix with a weight η (specifically η 1 for B 1 and η 2 for B 2 ). η is a hyper-parameter for the trade-off between stability and plasticity, which is also introduced in Section CModels: We validate our approach on the Vision Transformer (ViT) [9] and CLIP [27] models in the experiments, whose backbones are both ViT-Base/16 [9] . The ViT is pre-trained on ImageNet-21k, and we insert 4 prompts into each of the 12 layers for fine-tuning, which is referred to as "VPT" [13] . The classifiers are dependently trained in each task and the orthogonal projection is not applicable to them. All the classifiers from the available tasks are concatenated to make prediction during inference. For the CLIP model pre-trained on the WebImageText, we insert 4 prompts into each of the first 3 layers of the image encoder, while the text encoder is kept frozen. The logit scale that serves as a learnable scalar parameter to scale the cosine similarities between image features and text features is also set to trainable. We observed a serious cross-task confusion among the tasks in the CLIP model. Hence, we follow [43] to utilize the class-wise mean and covariance of previous features extracted before the embedding projection head (i.e., the last linear layer of the image encoder) to refine the projection head, after the prompt tuning stage in each task.Figure 6 : Illustration of our algorithm. The input image tokens with prompts are fed into the ViT layer for forward propagation. During optimization, the gradients of the prompts will be projected into the orthogonal direction to the subspace of the previous task T t-1 . The projected prompt update will be used to update the prompts for anti-forgetting.Inputs:Datasets D t = {(X <i> t , y <i> t )} |Tt| i=1 for task T t ∈ {T 1 , T 2 , • • • }, ViT model f model (•|P t )with the prompts P t to be optimized (the classifier is omitted for simplicity), uncentered covariance matrices C 1 and C 2 , projection matrices B 1 and B 2 Outputs: The optimized prompts P t 1: Initialization:Randomly initialize P t ; C 1 = 0, C 2 = 0, B 1 = I, B 2 = I 2: for task T t ∈ {T 1 , T 2 , • • • } do 3: repeat 4:Sample a mini-batch X t , y t ∼ D tObtain prediction by ŷt ← f model (X t |P t )6:Compute the classification loss L total ← CrossEntropy(ŷ t , y t )7: if t > 1 then 8:Compute the loss of prompt distribution L LN by Eq. ( 27) Update the uncentered covariance matricesC 1 ← C 1 + J ⊤ 1 J 1 and C 2 ← C 2 + J ⊤ 2 J 2 25:Compute the null-space projection matrices B 29))Benchmarks: We conduct experiments under the class-incremental learning protocol, where the classes in each task are disjoint, and task identity is unknown during inference. Four class-incremental benchmarks with three widely used datasets are adopted: 10-and 20-split CIFAR-100, 10-split ImageNet-R [39] and 10-split DomainNet [25, 38] . For the CIFAR-100 dataset, the total of 100 classes are randomly split into 10 or 20 tasks, which can evaluate the ability to handle different numbers of tasks. We follow [39] to randomly split the 200 classes in ImageNet-R into 10 tasks, which forms the 10-split ImageNet-R benchmark. For the 10-split DomainNet, we follow the same dataset protocol adopted in [38] and [11] to select the top 200 classes with the most images from the original DomainNet [25] , and randomly split them into 10 tasks with 20 classes per task. 25% samples of the training data in each dataset are picked as a validation set for searching optimal hyper-parameters.Metrics: Formally, the final average accuracy and final average forgetting are defined as:Final average accuracy = 1 T T i=1 a T,i , Final average forgetting = 1 T -1 T -1 i=1 max j∈{1,2,••• ,T -1} (a j,i -a T,i ),where T is the number of tasks, a T,i is the accuracy of the T -th model on the i-th task samples, and a j,i is the accuracy of the j-th model on the i-th task samples.Higher accuracy means the model performs better, while lower forgetting means stronger stability (i.e., the ability to retain old knowledge). However, lower forgetting does not always generate higher accuracy since the accuracy is also affected by plasticity (i.e., the ability to learn new knowledge). The accuracy is the main metric we should focus on as it reflects the precision of classification in practice.Implementations Details: For all the datasets and models, the images fed into the models are resized to 224 × 224 pixels and augmented by AutoAugment [7] during training. For the VPT-based models, we use the Adam optimizer [16] with β 1 = 0.9, β 2 = 0.999 and a weight decay of 5 × 10 -5 to train 100 epochs with an initial learning rate of 0.01 and a batch size of 256 on all benchmarks. The learning rate is scaled by a factor of 0.1 at the 50-th and 80-th epoch. Our training losses consist of the cross-entropy loss for classification and the loss L LN in Eq. ( 27) whose coefficient is set to 1. Through cross validation on the validation set, we set the temperatures in the cross-entropy loss to 28, 25, 30 and 30 for the 10-split CIFAR100, 20-split CIFAR100, 10-split ImageNet-R and 10-split DomainNet benchmarks. There are two hyper-parameters η 1 and η 2 used for the trade-off between stability and plasticity in null-space projection as introduced in Section C , and we set both of them to be 0.97, 0.95, 0.94 and 0.95 for the four benchmarks by cross validation.As to the CLIP-based models, the differences in training settings are as follows. We train them for 20 epochs with the batch size of 220 and the learning rate 0.001 which decays at the 10-th and 16-th epoch. The temperatures are all set to 1 since the logit scale is trainable. η 1 and η 2 are set to 0.98 which is a proper value for all the benchmarks. We refine the embedding projection head for 50 epochs using the SGD optimizer with a learning rate of 0.001, a momentum of 0.9 and a weight decay of 1 × 10 -4 . We implement our approach in PyTorch [24] with the timm library [41] . The experiments are performed on a server with 128 GB RAM and four NVIDIA RTX 4090 GPUs. Each of the experiment can be finished in three hours.Given that the null space of covariance matrix does not always exist in practice, Wang et al. [36] suggest approximating it by selecting the bases whose associated singular values approach zero, where the singular values smaller than a specified multiple (denoted as γ in our paper) of the smallest one are selected. However, we experimentally find this strategy and the experience for selecting γ are not suitable for prompt tuning in ViTs to determine the nullities R 1 and R 2 for the uncentered covariance matrices C 1 and C 2 in Algorithm 1 , which will be introduced afterwards. To solve this problem, we propose an adaptive nullity strategy to determine the nullities in an adaptive manner. Utilizing the characteristic that the curve of descending singular values forms an "L" shape, we divide the curve into two parts by the point where the gradient changes fastest to cover most of the small singular values. It is realized by calculating the maximum second derivative of the points:EQUATIONwhere λ j denotes the j-th singular value. We find it reaches near-minimum forgetting in our experiments which also means reaching near-optimal stability. Furthermore, to enhance the plasticity, we fuse the projection matrices with identity matrices by the weights η 1 ∈ [0, 1] and η 2 ∈ [0, 1] which should be close to 1:EQUATIONIn this way, we can make a trade-off between stability and plasticity by enhancing the plasticity based on near-optimal stability, and η 1 and η 2 are the hyper-parameters to control the trade-off.The main difference between our method and PGP [26] are summarized as follows.(1) We derive a different consistency condition for Affinity even if we ignore the LayerNorm operation and the bias terms in the qkv-transformation. Specifically, our simplified consistency condition for Affinity isX t W q W ⊤ k ∆P ⊤ = 0, contrasted with X t ∆P ⊤ = 0 in PGP. (2)We analyze the consistency conditions for the complete self-attention, i.e., softmax(Q X K ⊤ Z √ D )V Zwhich contains the Aggregation operation. However, PGP does not account for the Aggregation. (3) We take the LayerNorm before self-attention into consideration and propose an invariant prompt distribution constraint, while it is ignored in PGP.In conclusion, we conduct a comprehensive analysis of prompt tuning for the consistency objective, which provides a complete guarantee to eliminate the interference of new tasks on previous tasks. As demonstrated in our ablation study in the Experiment section, the consistency of Aggregation and LayerNorm also contribute to reducing forgetting, and thereby they should not be ignored. We make a comparison of the performance between PGP and our approach in the next subsection.We compare with PGP [26] using the VPT-Seq and L2P [40] baselines on the four benchmarks in our experiments. The results are shown in Table 5 . We implement PGP to VPT (i.e. VPT-PGP) under the same training settings as VPT-NSP 2 for a fair comparison. For the L2P-based methods, we insert prompts into the first three layers instead of only the first layer in the original implementation [40] . An orthogonal projection is also applied to the prompt pool which is essentially a linear layer in L2P-based models. We follow the training setting of PGP to train the L2P-based methods. The results in Table 5 demonstrate that our full approach can achieve more improvements in accuracy and reduce more forgetting than PGP. Even when applying only the projection matrix B 1 for the Affinity operation, our approach also performs better than PGP, demonstrating the effectiveness of our proposed method for mitigating the interference problem. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: We have reviewed the NeurIPS Code of Ethics and make sure our research conforms the ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper receives unanimous accept recommendations. After a careful check of paper, comments, and rebuttal, the AC agrees with the reviewers and think the authors has well addressed the reviewer concern. The idea of this paper is novel and the result is good for continual learning. Thus, the AC makes a accept recommendation.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel paradigm for continual learning based on prompt tuning. By deriving the constraints for orthogonal projection of prompt gradients in ViTs, the method aims to minimize forgetting during the learning process. Experiments on four benchmarks show that NSP² achieves superior performance by avoiding forgetting.
strength_weakness: strength_weakness: 1. While orthogonal gradient updates help prevent forgetting in continual learning, they also hinder positive knowledge transfer between tasks. In methods like L2P, similar tasks might select the same prompts, potentially improving performance on previous tasks after learning new ones.
2. The derivation process in the methods section is overly lengthy, which leads to a very brief description of the experimental section. The comparison with existing methods is reduced to just three lines of text. Consider moving some derivations to the appendix.
3. The experimental tables contain many blanks, which is not conducive to a systematic comparison of different methods' performance and weakens the contribution of this method.
4. The paper omits comparisons with some leading methods, such as DAP[1], which is also based on prompt tuning.

[1] Jung D, Han D, Bang J, et al. Generating instance-level prompts for rehearsal-free continual learning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11847-11857.
questions: questions: According to my understanding, as the number of tasks increases, the shrinking gradient subspace will make gradient updates increasingly difficult. Therefore, I would like to see the performance of NSP² on longer sequences of continual learning tasks.
limitations: limitations: As mentioned in the paper, additional constraints are introduced to simplify the consistency conditions.
paper_summary: paper_summary: This work aims to eliminate the interference on previously learned knowledge for visual prompt tuning in the field of continual learning, so that catastrophic forgetting can be mitigated. To this end, it analyzes the conditions for keeping the output features unchanged in the transformer block that features the self-attention mechanism. Two consistency conditions for the self-attention are derived by deducing the proposed two sufficient conditions for the consistency objective. Moreover, a constraint on the distribution of the prompts is proposed to further simplify the LayerNorm operation. Consequently, the interference can be eliminated in theory by achieving the proposed two consistency conditions and the prompt-invariance constraint. The proposed approach implements the two consistency conditions by performing two null-space projections on the prompt gradients during training a new task; the constraint is implemented by an additional loss function that penalizes the drifting of prompt distribution across sequential tasks. Substantial experiments demonstrate the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. The consistency objective Eq. (10) is decomposed into two sufficient conditions, namely Eq. (11) and Eq. (12). A detailed explanation is necessary to understand why direct analysis and simplification of Eq. (10) is not pursued.
2. The variable Q_p is omitted during the deduction of consistency conditions. Further explanation is needed to clarify why it can be disregarded.
3. In the Adam-NSCL [36] method, the trade-off between stability and plasticity is controlled solely by the nullity. In contrast, this approach involves adding weighted identity matrices to the projection matrices. How do these two methods differ in balancing stability and plasticity? A deeper discussion is expected to highlight the advantage of the proposed trade-off strategy.
4. The number of training epochs in this approach (100) is greater than that in other approaches (e.g., 50 epochs in DualPrompt for ImageNet-R). It would be better to provide a justification for this training setting.
5. As described in line 4 of Algorithm 2, the projection matrix is normalized by a Frobenius norm. There is a lack of explanation for this operation.
questions: questions: In addition to the parameters of the prompts, the classifier contains parameters that require updates (i.e. the weights and biases) as well. Why not perform an orthogonal projection on the parameters of the classifier to reduce forgetting?
limitations: limitations: Reasonable discussion on limitations is included in the paper.
paper_summary: paper_summary: This paper introduces the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. They propose two sufficient consistency conditions for the self-attention and an invariant prompt distribution constraint for LayerNorm, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. Extensive experimental results show the superiority of the proposed method.
strength_weakness: strength_weakness: 1)	In figure 1 of the overall framework, it is unclear why the Qp term in the affinity matrix can be neglected. The authors should provide clear reasons for such conclusion.
2)	In the method part of Eq.(10), In order to satisfy F_{Z_t}=F_{Z_t-1}, the authors transform Eq.(10) into the following Eq.(11) and Eq.(12), which are two sufficient conditions. Why not optimizing Eq.(10) directly? Please give detailed reasons.
3)	What’s the main differences between the single-head and multi-head self-attention for introducing the proposed orthogonal projection into the visual prompting tuning for continual learning?
4)	In the method optimization part, the authors propose the approximation method as illustrated in Eq.(25). Why not using the optimization method proposed in PGP[26]? What are the main differences ?
questions: questions: See the weaknesses above.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. Moreover, two sufficient consistency conditions for self-attention and an invariant prompt distribution constraint for LayerNorm are theoretically deduced, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. Finally, extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods.
strength_weakness: strength_weakness: 1)Eq.(8) seems be error.

2)L139: Eq.(8) suggests that if the weight update ∆Θ is orthogonal to the previous input feature Xt during training in the new task, the corresponding output feature will remain unchanged. Moreover, will this degrade the model’s discriminative ability for the current task when the weight update is orthogonal to the previous feature?

3)What is the model’s complexity and running time compared to the baseline VPT?

4)Suggest using a more accurate diagram to describe the algorithm to help understand it more clearly.
questions: questions: Please see “Weaknesses”
limitations: limitations: Yes

Comments: title: Rebuttal Response
Concerns addressed
Concerns addressed
Concerns partially addressed
Please have a discussion
Please have a discussion
Please have a discussion
Concerns addressed
comment: I have read the rebuttals from authors and discussions from others. And most of my concerns have been addressed. Thus, I maintain my initial score. Thanks.
Thanks for your detailed explanation. Overall, NSP² performs very well, and it would be even better if further exploration could be made in promoting knowledge transfer between tasks. The authors have addressed almost all of my concerns, so I have raised my score to 7.
Dear Reviewer rV5K,

We sincerely thank you very much for the meaningful comments and providing the valuable feedback. We really hope that our responses can address all the remaining concerns. Thank you again for your great help and many good questions and suggestions, which largely help improve the quality of our paper. 

We would like to clarify if you have further concerns. Thanks very much.
Thanks for the authors' response, that have addressed most of my concerns.
Thank you for your kind reply and support for our work. We are glad that we have addressed most of your concerns. Below is our response to the remaining concerns in your comments.

**Reply to Q1:** This is a valuable viewpoint on continual learning. It has always been our pursuit to enable deep neural networks to learn new knowledge better by utilizing the experience of previously learned knowledge like human beings. In our future work, we will consider exploring mining the parameters of different importance on learning old and new tasks, and combining this scheme with anti-forgetting techniques such as orthogonal projection to enhance knowledge transfer capabilities without forgetting.

Specifically, the learnable parameters in a network can be divided into three parts according to the contribution to learning new knowledge and preserving old knowledge by the following process. After finishing training on an old task and before learning on a new task, we can use a criterion (e.g., gradients) to **measure the importance of different parameters on the old task and the new task**, respectively. Then we divide those parameters into three sets by: 1) **high importance on the old task**, 2) **high importance on the new task**, and 3) **moderate importance on both old and new tasks**. During training in the new task, different strategies are adopted for these three sets of parameters. 1) For the parameters important to the old task, we use **a completely strict orthogonal constraint** on them to reduce forgetting and keep stability. 2) For the parameters important to the new task, we **do not perform any constraint** on them to maximize the stability. 3) For the parameters of moderate importance on both old and new task, we use **an appropriate relaxed orthogonal constraint** on them. In this way, the whole model can be trained with an enhanced knowledge transfer capability and retain a good ability of anti-forgetting.

**Reply to Q2:** Following L2P, the accuracies reported in Table III of our PDF for **ALL the methods** are the "**final average accuracy**" (*i.e.*, 85.80 instead of 91.51 for EASE). Therefore, we would like to emphasize that **the comparison in our table is fair.**

Specifically, **EASE reports two different metrics** for accuracy in the original paper, which are referred to as "final average accuracy (FAA)" and "mean average accuracy (MAA)" here. They correspond to $\bar{\mathcal{A}}$ and $\mathcal{A}_B$ in the Table 1 of EASE, respectively.

The finally average accuracy represents the accuracy over all learned tasks when the model finishes continual training on the last task. It is defined as:

$\mathcal{A} _B = \frac{1}{B} \sum _{i=1} ^{B} a _{B,i}$

where $B$ is used to denote the number of total tasks to correspond to the symbols used in EASE, and $a_{B,i}$ is the accuracy of the $B$-th model (*i.e.*, the model after training on the last task) on the $i$-th task’s test data.

The mean average accuracy is the mean value of $\mathcal{A} _1, \mathcal{A} _2, \cdots, \mathcal{A} _B$:

$\bar{\mathcal{A}}=\frac{1}{B}\sum_{i=1}^{B}\mathcal{A}_i$

Since the accuracy in early learning tasks are usually high under the class-incremental learning protocol, *i.e.*, $\mathcal{A}_1 > \mathcal{A}_2 > \cdots > \mathcal{A}_B$ usually holds, MAA is almost always higher than FAA. This explains why the results in the column $\bar{\mathcal{A}}$ are higher than those in the column $\mathcal{A}_B$ in the Table 1 of EASE. For a fair comparison with other methods, we report FAA in Table III which are widely adopted in existing prompt-tuning-based papers.

For a clearer comparison **under these two metrics**, we select those methods which report both FAA and MAA from Table III of the PDF, including: InfLoRA [20], EASE [45] and CPrompt [11]. The results are shown in **Table VII**. **The columns of FAA have been reported in Table III (corresponding to "Acc."), and the columns of MAA are newly added for a comparison under the mean average accuracy metric.** It can be seen that our approach surpasses other approaches in both metrics. Especially, **VPT-NSP2 outperforms EASE by an average of 3.86% under the FAA metric and an average of 3.35% under the MAA metric.**

**Table VII**: Comparison with the methods that report both finally average accuracy (FAA) and mean average accuracy (MAA). **The italic values are produced by us due to lack of official results, while the others are from their corresponding original papers.** Best results are highlighted in bold.
||20S-CIFAR-100|20S-CIFAR-100|10S-CIFAR-100|10S-CIFAR-100|10S-ImageNet-R|10S-ImageNet-R|10S-DomainNet|10S-DomainNet|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|-:|:-:|
|Method|FAA|MAA|FAA|MAA|FAA|MAA|FAA|MAA|
|EASE|85.80|91.51|87.76|92.35|76.17|81.73|*78.89*|*84.58*|
|InfLoRA|*81.42*|*87.42*|87.06|91.59|75.65|80.82|*81.45*|*88.75*|
|CPrompt|*83.97*|*90.08*|87.82|92.53|77.14|82.92|82.97|88.54|
|VPT-NSP2|**89.89**|**93.75**|**91.74**|**96.02**|**78.88**|**84.84**|**83.54**|**88.94**|
I appreciate the detailed response from the authors and the efforts they made over the past week. The authors have partially addressed my concerns as follows:

1. The proposed method NSP² performs well on long-sequence CL tasks, and by adjusting the hyperparameter $\bar{\eta}$, it can further relax the orthogonality constraint to facilitate learning more new knowledge.
2. More results from comparison methods have been presented, which enhances the credibility and contribution of the paper.

However, I still have some concerns:

1. Continual learning should not just involve learning each task separately to prevent forgetting. Like human learning mechanisms, algorithms should perhaps focus more on how to leverage existing knowledge to quickly and accurately master a new skill, as well as how to use new knowledge to better solve previous tasks, rather than learning entirely separately, like separate multi-task learning. Nevertheless, the orthogonal prompt tuning method proposed by the authors for Vision Transformers also provides a solid theoretical foundation for applying ViTs to continual learning, which is a significant contribution.
2. The results provided in Table III of the PDF seem to be lower for the comparison methods than those reported in the original papers, and the differences are substantial. For example, for 20S-CIFAR100, EASE is reported to achieve 91.51 in the original paper, while the authors report only 85.80. Similar discrepancies are observed for 10S-CIFAR100. I would like to understand if there are any differences in implementation or experimental settings.

For these reasons, I have increased my score to 5. If the authors can further address my above concerns, I will consider raising the score further.
This paper receives mixed reviews. The authors have provided a detailed response. Please give your reply and check whether there is still unclear point for authors to clarify.
This paper receives mixed reviews. The authors have provided a detailed response. Please give your reply and check whether there is still unclear point for authors to clarify.
This paper receives mixed reviews. The authors have provided a detailed response. Please give your reply and check whether there is still unclear point for authors to clarify.
I appreciate your detailed response. The authors have addressed all the concerns I raised in my initial review, and after considering the other feedback, I kept my score.
Dear reviewer rV5K,

Thank you very much for reviewing our paper and giving us some good questions. We have tried our best to answer all the questions according to the comments. Especially, we conduct experiments on **long-term continual tasks** across 5 benchmarks, and compare our approach with 6 existing SOTA methods. **We eagerly expect the reviewer to examine the PDF for the results and comparison.** The experimental results in Table I show that our approach remains effective and outperforms SOTA methods including L2P under the long-term CL protocol. It demonstrates that our approach can mitigate the shrinking gradient subspace and learn new knowledge effectively by the proposed relaxion for orthogonal projections, even under the setting of longer sequences of continual learning tasks.

We sincerely hope that our responses can address all your concerns. Is there anything that needs us to further clarify for the given concerns?

Thank you again for your hard work.