Visual Prompt Tuning in Null Space for Continual Learning
Authors: Yue Lu
Shizhou Zhang
De Cheng
Yinghui Xing
Nannan Wang
PENG WANG
Yanning Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual Learning
Visual prompt tuning
Gradient orthogonal projection
Null space
Primary area: machine_vision

Abstract: Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL), by selecting and updating relevant prompts in the vision-transformer models. On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference on tasks that have been learned to overcome catastrophic forgetting in CL. However, different from the orthogonal projection in the traditional CNN architecture, the prompt gradient orthogonal projection in the ViT architecture shows completely different and greater challenges, i.e., 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. Theoretically, we have finally deduced two consistency conditions to achieve the prompt gradient orthogonal projection, which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient orthogonal projection. Extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods. Our code is available at https://github.com/zugexiaodui/VPTinNSforCL

Visual Prompt Tuning in Null Space for Continual Learning
Yue Lu', Shizhou Zhang'; De Cheng'; Yinghui Xing', Nannan Wang', Peng Wang', Yanning Zhang'
School of Computer Science, Northwestern Polytechnical University. China School of Telecommunications Engineering, Xidian University, China zgxd@mail 1. nwpu. edu. en, szzhang@nvpu. wew..dduu...m cn, dcheng@xi dian. edu. en, eyh_7491000 1wwpu...wp...... edu. en, invangōxidian edu. cn, peng wangwwwgggnup.ccommm edu. en, ynzhang@nupu. edu.cn
Abstract
Existing prompt-tuning methods have demonstrated impressive performances continual learning (CL), by selecting and updating relevant prompts in the vision- transformer models. On the contrary, this paper aims learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as ensure no interference on tasks that have been learned overcome catastrophic forgetting in CL. However, different from the orthogonal projection in the traditional CNN architecture, the prompl gradient orthogonal projection in the VIT architecture shows completely different and greater challenges, i.e., 1) the high- order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. Theoretically. deduced two consistency conditions to achieve the prompt gradient orthogonal projection, which provide theoretical guarantee of climinating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. In practice, effective null-space- based approximation solution has been proposed to implement the prompt gradient orthogonal projection. Extensive experimental results demonstrate the effectiveness of anti-forgetting four class- incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances state-of-the-art methods. Our code is available at /tttps://githu.... http::///E.hh..oocoooneeeeeoo///
have finally
Introduction
Continual learning (CL) is crucial for Al models to adapt to the ever-changing environment by learning
sequentially arrived data, where the catastrophic forgetting is the key challenge 22 28 Recently,
prompt tuming-basse continual learning methods [40] 32 34 [0] 38
46 20
12
18 have been
attracting inereasing attention due to their impressive performances the CL field. Existing prompt
juning-based works tackle the downstream continual learning problem by selecting and updating relevant prompts, which is encoded with full task-specific knowledge while exploiting the general
knowledge of the pre-trained ViTs 40 39
On the contrary, this paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features, so as to ensure no interference with tasks that have been learned to overcome catastrophic forgetting in CL. It is worth noting that forgetting can be theoretically resolved by gradient orthogonal projection methods 43
36 45), which have
"Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

been extensively explored especially when adapting CNN models. Nevertheless, it remains huge gap to introduce the orthogonal projection- based methods of CNNs to visual prompt tuning due to the following challenges: 1) the high-order and non-linear self-attention operation; 2) the drift of prompt distribution brought by the LayerNorm in the transformer block. For the linear operation in convolution or fully-connected layers, the output features of old tasks can remain unchanged by updating the weights in the orthogonal subspace of previous input features. While for self-attention, three linear transformations are employed on input tokens, followed by high-order and non-linear operations for the self-attention interaction of tokens. It makes the relationship between the update of prompts and the output image tokens much more complex, far exceeding mere linearity.
In this work, we theoretically deduced two consistency conditions achieve the prompt gradient orthogonal projection, which provide a theoretical guarantee of eliminating interference on previously learned knowledge via the self-attention mechanism in visual prompt tuning. To be concrete, we firstly take the full self-attention and LayerNorm into consideration and derive a strict condition for eliminating the interference through a comprehensive analysis of the forward propagation of the VIT layer. Then we further propose to convert the condition of self-attention into its two sufficient conditions, which enables us to address the challenge of high order and nonlinearity, Thirdly, we propose constraint of invariant prompt distribution that removes the obstacle to the final simplification of the conditions brought by the LayerNorm. The consistency conditions reveal that the prompt update can be orthogonal to (1) the normalized previous input image tokens projected matrices of the pre-trained model, and (2) the activated
with the second-ordes gkv-transformatte attention map generated by image queries and prompt keys, the interference in visual prompt tuning can be eliminated theoretically.
In practice, based on the proposed consistency conditions, an effective spass-paaceseee based approxi- mation solution (36) has been proposed to implement the prompt gradient orthogonal projection. while the invariant prompt distribution constraint is implemented by incorporating a loss function which penalizes the drifting of prompt distribution over sequential tasks. We validate our Null-Space Projection for Prompts (NSP²) approach on extensive class-incremental benchmarks: 10- and 20-split SEFAR-100, 10-spli ImageNet-R [39] and 10-split DomainNet (38], with the sequential fine-tuning VPT and CLIP models as baselines. Our approach brings 4%~ improvements in accuracy, and reduces 9%~17%% forgetting, which is superior to statc-of-the the-art methods.
Our contributions are summarized as follows: (1) We introduce the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of transformer layer on the interference problem. (2) Two sufficient consistency conditions for the self-attention and an invariant prompt distribution constraint for LayerNorm are theoretically deduced, based on which effective null-space-baset approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. (3) Extensive experimental results demonstrate the effectiveness of anti- forgettggg on four class-increment benchmarks with diverse pre- pre-trained baseline models, and our approach achieves superior performances to state-of sfate-of-the-at tf-thh-art methods.
2 Related Work
Prompting-B Brosetin--Based Approaches: Most of the prompting-basee based approaches adopt a two-stage frame-
18 19:: querying a group of prompts for individual sample and using them to prompt the pre-trained models. For example, L.P 40 first selects a group of prompts from a prompt pool and then feeds them into the VIT. CPrompt to mitigate the gap between training and testing stages to enhance prediction robustness and boost prompt selection accuracy. These approaches essentially focus on acquisition of task-specific prompts tailored to individual samples. There are also several one-stage methods (2 [2 (8 [4 (0) based on prompt tuning. (I) Slowly updating trainable parameters 10 44 e.g., LAE I0 updates an offline expert with large momentum to reduce the change of features. (2) Expandable backbones [46]] 20
work [37 39
14
15
32
42
34
35
proposes
e.g., EASE [46] trains distinct lightweight adapter module for each new task, and designs se- mantic mapping to complement the drift of old class prototypes. (3) Enhancing classifiers rather than focusing on learning features [38 22 12 : e.g., ESN (88 proposes an anchor-based classifier alignment approach based on energy-based models. As introduced above, these works still lack of a theoretical solution to the interference problem for visual prompt tuning. In our work, we conduct a deep analysis of this problem and provide theoretical guidance climinating the interference.

---

Orthogonal Projection-Based Approaches: Orthogonal projection based approaches
43 4 8
31
36
can theoretically eliminate the interference new tasks on old tasks for linear layers. constructs a projector to find the direction orthogonal to the input space. GPM 31 first projects new gradients to the subspace important to the old tasks and then subtracts the projected components for updating parameters. Adam-NSCL. [36] projects the parameter updates to the approximate null space of previous inputs. However, due to the different relationships between parameter updates and outputs in the linear operation and self-attention the consistency condition used in CNNs is not directly applicable to the prompt tuning in ViTs. In our work, we derive the consistency conditions for the visual prompt tuning, enabling the application of orthogonal projection- based approaches to it, where the null-space projection [36] is adopted in our approach to get an approximate solution efficiently. We notice that recently emerged work PGP [26] implements GPM [31] to prompt-based frameworks. However, it obtains the same conclusion as that of the linear operation under simplified attention, which limits its application and performance compared in the appendix D.
17
45
OWM
4
3 Preliminaries
Continual Learning: the setting of continual learning, a network f(- with parameters + is sequentially trained on a stream of disjoint tasks {71, T2, ... Tr), where task Te is associated with paired data {xx, Vt <>>>)) ) of size [Tgl- When task T; arrives, the model f ((-(@) would be trained for the current task, while the data from previous tasks is unreachable.
Forward Propagation of Visual Prompt Tuning in VIT Layers: We describe the forward propaga- tion process of the ViT layer for visual prompt tuning, as illustrated in Figure 1 Let X E N and P Mx denote the N input image tokens of sample (including the pre-trained class token if available) and M prompts, respectively, where D is the dimension of each token. In the ViT layer, only the prompts P are trainable parameters. The remaining parameters in LayerNorm, qkv-transformations and subsequent MLP introduced below are pre-trained and kept frozen. We use Z = P E ( to denote the concatenated input tokens. First, they undergo the LayerNorm [ operation LN(-): LN
uz 3 a c
where Mz, oz E N+M az E DRD. The @ and division here denote the element-wise (Hadamard) product and division, respectively. Note that the vectors μz, dz a and B3 are broadcasted to match the matrices of dimensions (N + .) x D, enabling them to carry out operations with Z. Then the normalized tokens are fed into the qlv-transformation::
Qz = LN(Z) + by. Kz = LN(Z) + by, Vz = WL, + b..
(2)
where W (g.k.o) E DDD, The vector b blg.k.03 E RD is broadcasted to a matrix of dimensions (N + M) x facilitate the addition operation. Next is the self-attention:
Qx Vz.
Fz = SSA softmax(
VD
where Qx denotes the image tokens serving as queries. Eq. can be expanded as Affinity, softmax (on rows) and Aggregation operations:
QxKI Qx
K
Kp
Az Tar(Qx Kz)
R NX(++M)
(4)
VD
VD
Sz softmax (Az) softmax(
|Ax
RNNN
Ap E ERNXM]] = -Sx Sp], (5)
Vx
Fz fagg (zz Vz) S2 = [Sx Sp)
NXD. VP
(6)
is worth noting that the rows of the attention map where the prompts serve as queries (i.e., Qp) do not need to be computed, as formulated in Eq. (4) and illustrated in Fi The reason is that in VPT-Deep [13), the output prompts of this VIT layer will be replaced with new trainable prompts in the subsequent layer. Omitting Qp has no impact the output image tokens of the ViT layer, as
3

---

Image tokens p...
Self-artention
Output tokens
Affinity
Aggregation
LayerNor
Okv-nrassoomaa www.ssfooonaaaa
Qx
Kx
Ki
Kp
LayerNorm
LN(X)
Qx
Ap
Softma
Sx
Sp
Vx
MLP
LN(P)
Or
Kr
Vi
Prompis
map
(Neglected)
Figure 1: Illustration of the forward propagation in VIT layer. Residual connections omitted. The red crosses indicate the rows of attention map or the output prompts can be neglected.
the subsequent Aggregation, LayerNorm and MLP operations are performed independently for each token. no new prompts are added in the next layer, the output prompts can be just discarded as well.
After the self-attention, operations consist of another LayerNorm and the MLP layer are applied individually to each token. without any interaction among the tokens. Finally, the output fine-tuned image tokens are fed into the next ViT layer.
Orthogonal Projection in Convolutional Layers: A convolutional operation is actually linear operation. For a convolutional layer feonv (-)O:) ) in task Tr, we use θ E Pia P to denote its unrolled convolutional kernel matrix (5). Here, Din represents the number of pixels within kernel, and Dout corresponds to the number of kernels. Each convolutional patch from the input feature map is flattened into row vector with dimension of Din. These vectors of totaling np patches compose the input feature matrix X, R The output feature for X, in task Ti is expected to remain unchanged (referred to as consistent) in the next task T++1 to prevent forgetting:
feonv (X, θ fcomv X @++1).
By substituting 0++1 A0, with A * denoting the weight update in Te+1, 1, the consistency condition for the convolutional layer is established as follows:
X, = x,, + A0),
(8)
which can be further simplified as:
X
Eq. (9) suggests that the weight update A orthogonal the previous input feature X, during training in the new task, the corresponding output feature will remain unchanged. Thereby, the interference of the new task on the old task is eliminated. This can be realized by projecting the candidate weight update θg into the orthogonal subspace of X,; AO = pO where P E R. Piu D.,. an orthogonal projection matrix
31.
Similarly, for the prompt tuning which fine-tunes the prompts P, a VIT layer fvin (t)),, we also aim satisfy the following consistency objective for the purpose of anti-forgetting:
fvil Px) fvi (X,
(10)
However, the consistency condition in Eq. 9 does not hold for Eq. 10 since fvie * X,P in prompt tuning. Instead, all the tokens X; and P. first undergo a LayerNorm and then interact via the self-attention mechanism, previously described. The complicated forward propagation within the ViT layer brings huge challenge to analyzing the consistency conditions relation to the prompt update AP. In the next section, will tackle this challenge and derive the consistency conditions for visual prompt tuning.
Method
use Z. = Xx P.) and Z,+1 = [X,; Pt+1] denote the input tokens before and after updating the prompts, respectively, where P P, AP, AP & O. Our goal is to analyze how to satisfy Eq. (10) and derive one or more conditions expressed in terms of the prompt update AP. These conditions will subsequently guide the application of orthogonal projection to AP.

---

4.1 Analysis of Consistency Conditions
As can seen in Figure | those outputs of LayerNorm and gkv-transformations corresponding to the image tokens remain unaffected by the updates to the prompts. Hence, the essence of attaining the consistency objective Eq. (TO) can be turned into analyzing how to keep the output of self-attention in Eq. 33 unchanged as the prompts are updated, i.e, satisfying:
Fz.- 1Z.
(II)
However, the nonlinear operation (ie., softmax) and the potential higher-order term W Z zw, arising from K2 Vz in Eq. (3) complicate the direct resolution of this objective. Specifically, the non-injection property of the softmax function causes non-unique solutions. The multiplication between Kz Vze+1 derives a quadratic term LN(P, AP) LN(P,, + AP), which result in difficult optimization for AP.
To address this issue, propose two sufficient conditions consisting solely of linear operations. Specifically, we split the process of self-attention into two primary stages, i.e,, the Affinity described by Eq. ( and the Aggregation outlined in Eq. (6). We can achieve Eq. (II) by ensuring the consistency of each stage:
(Qx. Kz,, = fan Qxx: KZ.r1) Sugs(Su (Sz,, Vz,) = Sagg (Sz..1 Vz.1))
(12)
(13)
We first analyze the consistency objective of Affinity, i.e. Eq.
for Z₁ and Z
Satt( Jx (Qx, Kz,) = Qx: KK
KP.l
Qx Qx. [LN((,) W bb] be]]
(14)
fall (Qx, Kz.t,)
Qxx KX
Qx. (LN(P+1I) + bn] br]]
(15)
where vD is omitted for simplicity. Upon fulfilling Eq. (12). obtain Sz, Sz corre- sponding to the output of Eq. (5). Subsequently, we analyze the consistency objective of Aggregation Eq. (13. yielding results for Z, and Z as:
{ JNEE (Sz., Vz.) Sx, Vx, + Sp. Vp. = Sx. Vx + Sp, [N(P.) + by), fagE (Sz...: Vz...) = fagg (Sz., Vz.41) = Sx, Vx, + Sp. [LN(P W. + bbl.
(16)
(17)
Based on Eq.
12
L7p. we are able to derive the following two equations, respectively:
Qx. W L.(P)) T Qx, W LI) Qx. W. + AP) LN,,N)(.)).. = Sp, LN(P Sp, LN,P, + AP)W,.
(18)
(19)
Note that we expect further deduce Eq. (18) and Eq. (19) obtain equations among LN(P:). LN(P, + AP) and AP. However, due to the square root and quadratic terms in the expressions of the standard deviations OP. and PPPAAP. is difficult express OP.+AP in terms of OP. and ΦAP. Consequently, it is challenging to derive a straightforward equation that relates LN(P, and LN(P, + AP) through AP.
To simplify the problem, we introduce an additional constraint on the distribution of prompts. Concretely, we require that the updated prompts P, AP retain the same distribution Pt, iee., meeting the following assumption:
MP:+AP = 1'p.
(20)
OP.+AP = OP.
In this way, we can establish a straightffrwwad mathematical relationship connecting LN(P, + AP). LN(P,) and AP:
LN(P++PP) P.+AP MP.+AP
P Mp, AP
AP Da.
Oa+B
Oa+B= LN(P,)
OPPAAP
dP,
dP, (21)
5

---

Consequently, we can apply Eq. 21 to simplify Eq. 18 and 19 as:
[Qx. W. Qx. (.P)) T Qx. W, AP /op, Oa (p,WN(P.)W. = SP,, + Sp, APW. oP../ Oa.
(22)
(23)
It should be noted that in Eq. and Eq. 23 WA. W., and a are pre-trained parameters kept frozen throughout the continual learning process. Qx, and Sp, are two matrices derived from the input
X, As our objective is to ensure that the above two equations remain valid for the variables Qx
and Sp., it is sufficient to meet the following conditions, in which W. can be ignored whereas Wa remains crucial:
Qx. WT AP =0
(24)
SP, =
(25)
Now we have obtained the simplified formulas expressed by AP in Eq. and
To sum up, we convert the overall consistency equation Eq. (11) into two sufficient conditions Eq. [2
and B for Affinity and Aggregation, respectively. Consequently, we derive two corresponding consistency conditions Eq. (2 and (23) expressed by the prompt update AP, under the constraint of invariant prompt distribution formulated in Eq. (20). The deduced conditions can satisfy the
consistency objective in Eq. (1O). thereby achieving the goal of eliminating the interference of the new task on the old task for visual prompt tuning.
As Qx, = (X. W, g + by. Eq. (24) implies that if the (transposed) prompt update can be orthogo- nal to the normalized previous input image tokens X, projected with second-order transformation matrices W, W of the pre-trained ViT, the consistency for Affinity can be guaranteed. When we ig- nore the normalization and the bias term in Qx, Eq. (24) can be simplified as X, W q W AP = 0. The simplified condition is still essentially different from the consistency condition of linear layers (i.e., Eq. (9) and that deduced in [26] (i.e., X,AP = O). It indicates the interaction between the image tokens and prompts within VIT layers is fundamentally distinct, leading
unique consistency condition related to the second-order transformation matrices W, q W. of the pre-trained model. Moreover, Eq. 25 is also an essential condition served as one of the sufficient conditions for the consistency of the whole ViT layer. It implies that if the prompt update can be orthogonal to the activated attention map generated by the image queries (Qx) and prompt keys (Kp), the consistency of Aggregation can be achieved.
4.2 Optimization of Consistency Conditions
To jointly optimize Eq. (24) and (25). we need to solve AP that can meet both equations concurrently.
Here, we employ separate optimization approach to get an approximate solution efficiently. Initially, it ensures AP is orthogonal to the subspace spanned by Qx. W to satisfy Eq. (24). Subsequently,
it makes AP orthogonal to the subspace spanned by Sp. to satisfy Eq. 25
Specifically, we use Pg to denote the candidate parameter update generated by the optimizer for the prompts. We aim to obtain a projection matrix B such that AP = BP Following the previously mentioned separate optimization strategy, we first ensure AP is orthogonal to Qx, W by the projection matrix By: APT = B, P. Then AP is made orthogonal to Sp, by another projection matrix By: AP B.P.. g. Therefore, the objective of the optimization turns into obtaining the two projection matrices B, and B, to satisfy Eq. (4) and (25). Inspired by the null-space projection method (36), the bases of B, and B2 correspond to the null-space bases of Qx, WT and Sp., respectively. We use UI.0 E DDRR and U2 x MRa Ra to denote the bases of the null spaces for Qx. WT and Sp., where Rj and R2 indicate their nullities. U1.0 and U2 can be obtained from the right singular vectors associated with the zero singular values, through the process of singular value decomposition (SVD) applied by SVD((Qx, W Qx. W and SVD Sp.), respectively. In
this way, we get the projection matrices B, = U U.
1.0 E DxD and B3 = U2 2,0 E MRM
which are the solutions enabling AP to jointly satisfy Eq. 24 and 25
A B1 = (U2.0 U,2,0U.TO P
(26)
For the constraint Eq. (2)p. incorporate additional loss function aimed at penalizing the drift of prompt distribution, hence realizing a relaxed version of this constraint:
LI ||M -Mp, l1a + | -P:ll1.
(27)

---

Table 1: Comparison with the baselines ("-Seq") on four benchmarks using two types of models. The upper-bound means jointly training all the classes in the dataset.
10S-CIFAR-I00
20S-CIFAR-100
10S-ImageNet-R
I0S-DomainNet
Method
Acc.
Forgetting L Acc. Forgetting + Acc. Forgetting
Acc. 1 Forgetting
VPT-Seq
87.27
12.33
82.36
17.36
72.46
19.41
73.28
25.65
VPT-NSP2
91.74
3.28
89.89
4.91
78.88
5.06
83.54
8.54
Upper-bound
93.87
93.87
84.60
89.25
CLIP-Seq 72.91
15.13
71.37
17.89
75.69
19.21
67.73
35.60
CLIP-NSP2 80.96
12.45
79.83
13.77
82.17
6.42
77.04
18.33
Upper-bound 84.52
84.52
84.86
81.65
In Eq. (77 Mp, and dP, represent the target prompt distribution obtained in task Te, while MPa11 and P denote the distribution to be optimized in task T6+1-
To sum up, we use Eq. (26) to realize Eq. (24) and 25 and use Eq. (27) to meet Eq. (00 thereby
achieving the consistency objective Eq. 10 for anti-forgetting We provide a full algorithm of our
approach in the appendix A
4.3 Extension Multi-Heads
We further extend the consistency conditions Eq. 24 and 25 to multi-head self-attention, a common
feature current transformer-basee models. Suppose there are H heads and d = D/H represents the dimension of each token in head. We use Qx E IR Nxd Wa.h Dxd and Sp, E N
to denote the corresponding matrices in Eq. 22 and (25) for the h-th head, respectively. The objective to ensure these conditions are met across all heads, i.e., Qx W.
and Qx W.. k. HN and H represent the concatenated matrices from all the heads, respectively. Based on block matrix properties, those two sets of conditions can be formulated as and AP 0. To sum up. The main difference between single-head and multi- heads is that the parameter update should be orthogonal the subspace spanned by the concatenation matrices from heads for multi-heads self-attention. Therefore, for the multi heads variant, only an additional step of concatenation of the matrices from all heads is required in our algorithm.
A 0, Ah E 21, .... H). Let Q2 [Qx,. WT.: 02 = 1 :S...H] ..... ...
5 Experiments
5.1
Experimental Setups
In our experiments, we mainly utilize the VPT [3] with a ViT-B/16 backbone [9] pre-trained on ImageNet-2Ik. Additionally, we validate the effectiveness the CLIP (27] model, wherein the visual prompts are inserted into the image encoder. Our experiments are conducted across 4 class- incremental benchmarks: 10- and 20-split CIFAR-100, 10-split ImageNet-R and 10-split DomainNet. We report the mean values of the final average accuracy and final average forgetting over 3 runs with different random seeds. Given that the null spaces of Qx, W and Sp, may not always exist in practice, we compute the approximate null spaces and determine the nullities R, and R2 in an adaptive manner, rather than the way suggested in [36]. For more detailed information regarding the experimental setups, please refer to Appendix B
5.2 Main Results
The comparison between our approach and the sequential fine-tuning VPT and CLIP baselines is shown in Table [. For the VPT model, the proposed NSP2 achieves 4.47% 10.26% improvements accuracy on the benchmarks. Meanwhile, reduces the forgetting by 9.05%~17.11 17.11%. As to the CLIP model, the NSP² improves the accuracy by 6.48%~9.3 %, and reduces the forgetting by 17.2%% We calculate the accuracy across all previously encountered tasks after completing training on each task. The accuracy curves of VPT-Seg and VPT.
Validation of Effectiveness:
7

---

100
98
90.00
VPT-Scc
VPT-NSP
Adc.
$4.43
VPT-NSP
90
95
97.00
93.38
92.28
05.43
91.60
95.86
91.2
$7.02
$2.57
$5.67
$1.05
80
79.88
83
76.93
76
Task
Task
10-plit CIFAR-100
10-split Imagehet-
Figure 2: Task-by-task accuracy changing curves of VPT-Seg and VPT-NSP on two benchmarks.
Accemses
Forgcling
50.44
Accimi
Torpating
75
70JTT
45
75.5
25:95
30
8.50
DSST LAKIN-S
SmageNee- (Nirce Clawelliaiion
Inugsher-1
IregeNee21--P (MIL)
LAKON-BB (CLIP)
IDING
Conoticannn)
(a) 10-splix CIFAR-100
(b) 20-split CIFAR-100
Forsetia 45
Accimo
80
39.41
$1.83
710
66.51
25
$3.8k
7.49
10
InepeNc22K.FF
LAION-JB
Chane Chestication)
(DINO)
(MILa
(CLIP))
(Narve Classiiiaton
(DIN3)
(MIIL)
(Inage-Tast)
10-splic ImageNettR
10-split DomainNet
Figure 3: Results of utilizing diffferent pre-training datasets and paradigms. The blue and yellow bars represent accuracy and forgetting, respectively. The upward arrows indicate the accuracy increasing from VPT-Seq to VPT-NSP", whereas the downward arrows denote the reduction in forgetting.
NSP² on 10-split CIFAR-100 and 10-split ImageNet-R are displayed in Figure They demonstrate our approach consistently outperforms the baseline throughout the sequential learning of tasks.
We conduct additional experiments with the VPT model, utilizing the weights pre-trained on different
datasets as well as different paradigms, as shown in Figure 3 The pre-training paradigms and
datasets include: naive classification on ImageNet- 1k [30], DINO [3] ImageNet-Ik, Ik, MIIL [29
on ImageNet2 k-P and CLIP on LAION-2B 6 (we only use its image encoder). As can be seen from the figure, our approach not only significantly enhances accuracy but also markedly mitigates forgetting. These results further demonstrate the generalizability of the proposed approach.
We compare our method with existing methods in Table 2 where the competitors include many recent works. The proposed VPT-NSP² achieves state-of-the-art performance the four benchmarks, with surpassing the second best approach by average of 1.49% in accuracy. The forgetting of our approach is not the lowest, which is reasonable since our approach sacrifices some stability for better trade-off between stability and plasticity. The outperforming accuracy can demonstrate the superiority of our method.
Comparison with Existing Methods:
The two consistency conditions Eq. (24g and (25). along with the constraint Eq. 20 constitute the main components of our approach. They correspond to B1, B2 in Eq. (26), and Ln in Eq. 27 We study their effects the four benchmarks using VPT-NSP2. with results presented in Table We can see that the projection for Affinity (Bi) plays a crucial role, which brings 3.31 9.03% improvement in accuracy and 5.42% 4.76% decline in forgetting. Furthermore, the projection for Aggregation (B2) and the loss LIN for invariant prompt distribution are indispensable as well for minimizing forgetting. Optimal accuracy is achieved when all three conditions are applied.
Ablation Study:
We analyze the evolution training losses on the 10-split CIFAR- 100 and 10-split ImageNet-R benchmarks, shown in Figure 4 Each point on the curve represents the training loss of the data in Ti/T2 after the model has been trained on subsequent tasks. As can be seen, the losses of VPT-NSP2 previous tasks be almost retained, confirming that approach can effectively mitigate the interference of new tasks on old tasks.
Model Analysis:
Trade-off between Stability and Plasticity: We first adaptively determine the nullities R₁ and Ra for B, and 13 to achieve near-minimum forgetting. Based on this, we assign two weights m and n2 to the projection matrices to control the trade-off between stability and plasticity: AP

---

Table 2: Comparison with existing methods that use the pre-trained ViT-B/16 on ImageNet-2Ik., The standard deviations are also reported if available. Missing results in the corresponding papers are denoted as "-". The results marked with f and are implemented by [1] and [10], respectively. The highest accuracies are in bold, and the second highest accuracies are underlined.
IOS CIFAR-100
COS-CIFAR-IOR
I0S ImageNet-R
UlS-DomainNet
Method
Venue
Acc.
Forgetting
Acc.
Forgetting
Acc.
Forgetting
Acc.
Forgetting
L2P 40
CVPR'22 83.83 1.
20.57
9.73
81. 81.17
8.98
DualPrompt 39
ECCV'22 4 5, 3,16=009
82.02
68. 68.134.49
4.68
81. 81.70
8, 8.04
CODA-P
32
CVPR'23 86.25 10.74 1.27
75. 75,4510.56
1.64
80.04.070
10.16
ESN
38
AAAI'23 86.34 4.76
80.56
62.61 +0.%
79.22+20z
10.62
APG
33
ICCV'23 89.35
6.01
88.64
6.51
73.27
8.59
LAE
10
ICCV'23 85.5910.46
83,93
72.66
DualP-LGCL
15
ICCV'23 10.2 5.10
69 09.46..00
4,20
C-LN
23
ICCVW"23 2WW."233669995 86.95 6.98=048
76.3640.51
8.31
EvoPrompt
AAAT'24 87.97 87.97 2.60=04z
84.64
3.98
76.83
2.78+006
29.5
3 2036
OVOR-Deep 12
ICLR'24 85.99 6.42
14.15
6.8 1077
76. 1021
7.16
79.61 1036
4.77
DualP-PGP 26
ICLR'24 86, 96.92.00s 5.
83. 2001
7.91
69.34
4.53g004
80.41
8.
InfLoRA 20
CVPR'24 87.0610.25 6.22
81 81.42.054
6.42.003
15.65
5.731044
81.45
5.35
EASE (46
CVPR'24 87.76
5,94
85,80
7.19
76.17
7.82
78.89
7,89
CPrompe III
CVPR'24 87 80.2 5.
33
6.85
17.14
5.97
82.97
7.45
VPT-NSP-
This work 91.74 2.28 89.89 20.22
4,91 10.59
78,88
20.50
5.064026
83.54
2.54
Table 3: Ablation studies of each component in our approach on the four benchmarks.
10S-CIFAR-100
20S-CIFAR-100
10S- 10S-ImageNer-R
10S-Domain
B1 B2 LN
Acc. t Forgetting | Acc. Forgetting
Acc.
Forgetting
Acc. 1 Forgetting
87.27
12.33
82.36
17.36
72,46
19.41
73.28
25.65
90.58
6.91
88.13
10.27
78.05
8.14
82.31
10.89
88.74
10.85
83.32
16.48
74.71
14.69
78.87
17.81
91.33
4.22
88.96
6.42
78.37
6.25
83.17
8.95
91.42
3.94
88.46
8.64
78.30
6.31
83.13
9.32
V
89.36
9.32
86.67
11.59
75.27
13.35
79.45
16.50
V
91.74
3.28
89.89
4.91
78.88
5.06
83.54
8.54
VPTAISP-5
VPT-SoppT
- VPT-Sau3;
VPT-NSP-7 - VPTNSP-,
VPT - VPT-Sop-3
100
0.10
1.0
0.05
0.5
0.00
Tiak
0.0
Task
10
2
10
(n) 10-pliit CIFAR-100
(b) 10-plii IntgeNNt-RR
Figure 4: Training loss curves of VPT-NSP² and VPT-Seq on tasks Ti and T2 when the models are trained on sequential tasks.
?) )I] where I denotes the identity matrix. The effects of m and 7/2 which are set to same value is shown in Figure As the weight decreases, the accuracy increases first owing to better plasticity, and then decreases due to worse stability caused by the forgetting. It implics that trade-off can be achieved by the two weights of projections.
[12B2 + (1 - 1 Bn (1-1))
Long-sequence Continual Learning We experiment on 5 benchmarks under the protocols of 50 tasks and 100 tasks to validate that our approach remains effective even within the context of long- long-sequence continual learning. The results are presented in Table 4 Despite lacking plasticity enhancement, VPT-NSP2 can outperform existing state-of of-the-art approaches and especially surpasses L2P by large margin. This demonstrates that forgetting still the predominant factor affecting performance in long sequence of tasks. With the plasticity enhancement, VPT-NSP" achieves significant increase in accuracy (by 1.1 2.9%%. This demonstrates that our plasticity enhancement effective in learning new knowledge in long- soggseeuennce continual learning.

---

Acc
91.96
#1.57
$1.25
90.75
$8.62
7K.M
78.21
$2.05
$1.80
7
10:
1185
1.0
3.55
0.85
ON
1.
3.34
4.20
4.59
5.54
6.76
4.10
5:94
5-92
9:65
Forgeting
7.50
Forgeting
(a) I0lsppir CIFAR CIFAR-100
(b) 20-splir CIFA A---OO
(c) Inl-pplic TmageNNetRR
(d) 10-spiit Domain Net
Figure 5: Effect of the projection matrix weight r) on the accuracy and forgetting for the stability- plasticity trade-off the four benchmarks.
Table 4: Results for 50 tasks and 100 tasks on CIFAR-100, ImageNet-R and DomainNet datasets. i indicates no plasticity enhancement, and & indicates using the balanced plasticity enhancement where is the default value less than 1. Our approach still outperforms other methods in long sequences of tasks.
50S AAS-CIFARIIN 00 50S S0S-ImaggNNer 100S 100-- O00-DomainnNet
Method
Acc. Forgetting Acc. Forgetting Acc. Forgetting Acc.
Forgetting
Acc.
Forgetting
L2P
76.19
12.06
48.53
12.99
59.45
11.53
38.87
15.26
50.52
17.66
EvoPrompt
76.60
13.86
68.53
10.03
67.68
10.41
61.84
15.84
56.35
21.39
OVOR
65.69
14.28
60.08
5.86
66.27
7.43
40.49
8.12
47.65
8.91
InfLoRA
61.49
13.68
59.02
11.02
69.96
9.51
38.16
15.11
44.32
17.85
EASE
74.47
9.31
68.17
7.76
61.20
10.01
47.55
8.22
33.08
32.14
CPrompt
74.97
7.45
68.47
8.16
67.87
9.36
56.95
10.20
53.73
12.14
VPT-Seq 70.47
29.21
56.38
37.91
58.39
44.79
49.72
45.53
46.39
49.34
VPT-NSP21
81.92
6.56
67.32
6.35
70.13
9.92
59.97
10.07
54.44
11.04
VPT-NSP2 82.98
6.66
69.48
6.51
71.28
11.36
62.23
12.13
57.35
13.82
Conclusion
In this paper, study the interference problem of visual prompt tuning in ViTs, and propose two consistency conditions which can eliminate the interference in theory under the constraint of invariant prompt distribution. They guarantee the consistency of Affinity, Aggregation and distribution of prompts in LayerNorm, respectively, which jointly achieve the consistency objective the whole ViT layer. We adopt the null-space projection implement the two conditions and utilize an extra loss to satisfy the constraint. Our experiments on various benchmarks demonstrate the effectiveness of the proposed conditions for anti-forgetting, and our approach achieves state-of-the-ar performances.
Limitation Discussion:
To simplify the derivation of our consistency conditions, we introduce a constraint of invariant prompt distribution. Although the superior results show that it may not be very strong assumption, it is not an exact solution.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62101453, 62176198 and 62201467; in part by the Project funded by China Postdoctoral Science Foundation under Grant 2022TQ0260 and Grant 2023M742842, in part by the Young Talent Fund of Xi' an Association for Science and Technology under Grant 9592023 1088, in part by Innovation Capability Support Program Shaanxi (Program No. 20247C-KIXX and part by the Natural Science Basic Research Program of Shaanxi Province (No. 2022JC-DW-08).
10

---

References
[1] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. CoRR, abs/ 605.04500 2016.
Benjamin Bowman, Alessandro Achille, Luca Zancato, Matthew Trager, Pramuditha Perera, Giovanni Paolini, and Stefano Soatto. A-la-carte Prompt Tuning (APT); Combining Distinct Data Via Composable Prompting. In IEEE/CVF Conference Computer Vision and Pattem Recognition, pages 2023.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging Properties in Self-Supervised Vision Transformers. IEEE/CVF International Conference on Computer Vision, pages 9630-9640, 2021.
[4] Arslan Chaudhry, Nacemullah Khan, Puncet K. Dokania, and Philip H. S. Torr. Continual Learning in Low-rank Orthogonal Subspaces. In Advances in Neural Information Procesingg Systems, 2020.
[5] Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers in handwriting recognition. Suvisoft, 2006.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman. Mitchell Wortsman, Gabriel Ilharco, Cade Gordon. Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible Scaling Laws for Contrastive Language-Image Learning. JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2818-2829, 2023.
[7] Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning Augmentation Strategies From Data. In IEEF/CVF International Conference on Computer Vision, pages 113-123, 2019.
[8] Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng. Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning. In Advances in Neural Information Processing Systems, volume 34, pages 18710-1878 18721, 2021.
[9] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
[10] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang, A Unified Continual Learning Framework with General Parameter-Efffiien Tuning. In IEEE/CVF International Conference on Computer Vision, pages 11449 1 11459, 2023.
[11] Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent Prompting for Rehearsal-Free Continual Learning. CoRR, abs/2403.08568, 2024.
[12] Wei-Cheng Huang, Chun-Fu Chen, and Hsiang Hsu. OVOR: OnePrompt with virtual outlier regularization rehearsal-free maas-inncremetal learning. International Conference on Learning Representations. 2024.
[13] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardic, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In European Conference on Computer Vision, volume 1 13693, pages 709-727, 2022.
[14] Dahuin Jung. Dongyoon Han. Jihwan Bang. and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In /EEE/CVP International Conference Computer Vision, pages 11813-11823
October 2023.
[15] Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt- based continual learning. In IEEE/CVF International Conference on Computer Vision, pages 1 11429 11439, October 2023.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
[17] Yajing Kong. Liu Liu, Zhen Wang, and Dacheng Tao. Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning. In European Conference on Computer Vision, volume 13686, pages 219-236 2022.

---

[18] Muhammad Rifki Kurniawan. Xiang Song, Zhiheng Ma, Yuhang He. Yihong Gong, Yang Qi. and Xing Wei. Evolving Parameterized Prompt Memory for Continual Learning. In AAAI Conference on Artificial Intelligence, pages 13301-13309, 2024.
[19] Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N. Metaxas. Steering Prototypes with Prompt- Tuning for Rehearsal- Free Continual Learning. In /EEE/CVF Winter Conference on Applications of Computer Vision. pages 2523 2533, 2024.
[20] Yan-Shuo Liang and Wu-Jun Li. InfLoRA: Interference- free low-rank adaptation for continual learning. arXiv preprint arXiv:2404.00 00228, 2024.
[21] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-166 165. Elsevier, 1989.
[22] Mark D. McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad. and Anton van den Hengel. RanPAC: Random Projections and Pre-trained Models for Continual Learning, In Advances in Neural Information Processing Systems, 2023.
[23] Thomas De Min, Massimiliano Mancini, Karteek Alahari, Xavier Alameda-Pineee and Elisa Ricci. On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformers. In /EEE/CVF International Conference on Computer Vision Workshops, pages 3577-3586, 2023.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
[25] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment Matching for Multi-Source Domain Adaptation. In IEEF/CVF International Conference on Computer Vision, pages 1406-1415, 2019.
[26] Jingyang Qiao, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, and Yuan Xic. Prompt Gradient Projection for Continual Learning. In International Conference on Learning Representations, 2024.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, volume 139, pages 7743-8763, 2021.
[28] Roger Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
[29] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. ImageNet-21K Pretraining for the Masses. In NeurlPS Datasets and Benchmarks, 2021.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-253) 2015.
[31] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient Projection Memory International Conference on Learning Representations. 2021.
for Continual Learning. In
[32] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonille Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: COntinual Decomposed Attention- Based Prompting for Rehearsal- Continual Learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19099 11919, June 2023.
[33] Yu-Ming Tang. Yi-Xing Peng, and Wei-Shi Zheng. When Prompt-based Incremental Learning Does Not Meet Strong Pretraining. In IEEE/CVF International Conference on Computer Vision, pages 1706-1716. 2023.
[34] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang. Hang Su, and Jun Zhu. Hierarchical Decompo- sition of Prompt- Based Continual Learning: Rethinking Obscured bub-optimality. In Advances in Neural Information Processing Systems, 2023.
12

---

[35] Runqi Wang, Xiaoyue Duan, Guoliang Kang, Jianzhuang Liu, Shaohui Lin, Songcen Xu. Jinhu Lv, and Baochang Zhang. AtrrCLIP: A non-mmncellntal Learner for Incremental Knowledge Learning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3654-3663, June 2023.
[36] Shipeng Wang. Xiaorong Li, Jian Sun, and Zongben Xu. Training Networks in Null Space of Feature Covariance for Continual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 184-193, June 2021.
[37] Yabin Wang. Zhiwu Huang, and Xiaopeng Hong. S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning. In Advances in Neural Information Processing Systems, 2022.
[38] Yabin Wang, Zhiheng Ma, Zhiwu Huang, Yaowei Wang, Zhou Su, and Xiaopeng Hong. Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference. In AAAI Conference on 2023.
Artificial Intelligence, pages
[39] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi. Ruoxi Sun. Han Zhang, Chea-Yo Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. DualPrompt: Complementary Prompting for Rehearsal-Frre Continual Learning. In European Conference Computer Vision, volume 13686, pages 631-648, 2022.
[40] Zifeng Wang. Zizhao Zhang, Chen Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to Prompt for Continual Learning. /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, June 2022.
[41] Ross Wightman. Pytorch image models. 2019.
http::/tttoorccimmeeeeeeee models
[42] Yinghui Xing, Qirui Wu, Cheng, Shizhou 7hang, Guoqiang Liang, Peng Wang, and Yanning Zhang. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions on Multimedia, 2023.
[43] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual Learning of Context-Dependeet Processing in Neural Networks. Nature Machine Intelligence, 18):3644-37, August 2019.
[44] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SLCA: Slow Learner with Classifier Alignment for Continual Learning on Pre-trained Model. In JEEE/CVF International Conference on Computer Vision, pages 19091-19101, October 2023.
[45] Zhen Zhao, Zhizhong Zhang, Xin Tan, Jun Liu, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Rethinking Gradient Projection Continual I Learning: Stability/Plasticity Feature Space Decoupling. /EEE/VV Conference on Computer Vision and Puttern Recognition, pages 3718-3727, June 2023.
[46] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable Subspace Ensemble for Pre- Trained Model-Basee Class- Incremental Learning. CoRR, abs/2403. 12030, 2024.
13

---

Appendix:
Visual Prompt Tuning in Null Space for Continual Learning
A
Algorithm
An overview and algorithm of our approach are provided in Figure 0 and Algorithm I. respectively. We first initialize the overall uncentered covariance matrices [36 C and C2, as well as the null-space projection matrices B1 and Bg. During training, the cross-entropy, loss for classification and the loss of prompt distribution LIN are jointly utilized for optimization. Subsequently, we get the candidate prompt updates Pg computed by the optimizer. Then Pş is projected by the null-space projection matrices B1 and B3 for updating the prompts. After the convergence, we obtain the matrices JI and J2 to temporarily store Qx, W and Sp, for the data of the current task. Then they are used to update the uncentered covariance matrices C₁ and C2 by addition. Finally, we update the null-space projection matrices using the uncentered covariance matrices, which will be used in the next task.
Algorithm 2 shows the process of computing a null-space projection matrix. First, an imput uncentered covariance matrix C is decomposed by SVD. from which we can get the singular values and right singular vectors. Next, we determine the nullity (i.e., the dimension of null space) of C according to the maximum second derivative, which is introduced in Section C. Then we select R right singular vectors corresponding
the R smallest singular values considered close to the bases of null space. Finally, we compute the normalized projection matrix, which provides an upper bound for the scale of the projected gradients and prevents excessive gradient magnitudes. the null-space projection matrix added by an identity matrix with weight 7) (specifically m for B1 and 1/2 for B2). is a introduced in Section C
our implementation,
for the trade-off between stability and plasticity, which also
B Experimental Setups and Implementation Details
We validate our approach on the Vision Transformer (VIT) 9 and CLIP 27 models the experiments, whose backbones are both ViT-Base/16 (9]. The ViT is pre-trained on ImageNet-21k, and we insert 4 prompts into each the 12 layers for fine-tuning, which is referred to as "VPT" [33. The classifiers are dependently trained in each task and the orthogonal projection is not applicable to them. All the classifiers from the available tasks are concatenated make prediction during inference. For the CLIP model pre-trained on the Weblmage'Text, we insert prompts into each of the first 3 layers the image encoder, while the text encoder kept frozen. The logit scale that serves as a learnable scalar parameter to scale the cosine similarities between image features and text features is also set to trainable. We observed a serious cross-tate -toss confusion among the tasks in the CLIP model. Hence, follow [44] utilize the class-wise mean and covariance of previous features extracted before the embedding projection head (i.e., the last linear layer of the image encoder) to refine the projection head, after the prompt tuning stage in each task.
Models:
Input image tokens X,
Outpul imaaee tokens Y.
Forward propagation
Forward propagation
Orthogonal projection
VIT Layer
Back propagation
Update
Gradient G.
Back propagation Prompt gradient G,
Back prapagation
Prompt update
Subspace of
Prompts
Figure 6: Illustration of our algorithm. The input image tokens with prompts are fed into the ViT layer for forward propagation. During optimization, the gradients of the prompts will be projected into the orthogonal direction to the subspace of the previous task T.-1- The projected prompt update will be used to update the prompts for moggetorg.......
14

---

Algorithm 1 NSP2 for Visual Prompt Tuning
Inputs:
... ". ViT model [ (P;) Datasets D; { Dr yr ) for task T; E (Ti, T2, with the prompts P. to be optimized (the classifier omitted for simplicity), uncentered covari- ance matrices C₁ and C2, projection matrices B₁ and B2
Outputs: The optimized prompts P.
1: Initialization: Randomly initialize Pt: C1 - 0, C2 - B1 I, -1 -1
2: for task T T2. do
3: repeat
4: Sample mini-batch 2; ~D; Dr
5: Obtain prediction by D. <- mundel (P))
6:
Compute the classification loss Littal t- CrossEEtropy yc) if then
7:
8:
Compute the loss of prompt distribution LLN by Eq. (27
9:
Accumulate the losses Ciotel &- Liotat + LLN
10:
end if
11:
Get the candidate prompt update Pg from the optimizer by the loss Ltotal
12:
if then
13:
Compute the prompt update AP + ByPgB1 by the null-space projection Eq. 26
14:
else
15:
Directly adopt the candidate prompt update AP
Pg
16:
end if
Update the prompts by P. P. learning_rate AP until convergence
18:
19:
Initialize two temporary matrices J1 and J2 []
20:
for X
E DD do Get the matrices (Qx, )
21:
)x and Sp by the forward propagation fmodel (2, P.,) Update J, and J2 by concatenating (Qx, W.) <<>> and J,, sp: and J2, respectively
22:
23:
end for
24:
Update the uncentered covariance matrices C₁ f- C₁ + J. and C2 t C2 + J J2
25:
Compute the null-space projection matrices BJ and B3 by Algorithm 2 using C₁ and C2
26:
end for
Algorithm 2 Computing Null-Space Projection Matrix
Inputs:
Uncentered covariance matrix C, stability and plasticity (mentioned in Section C
77 E [0, 1] for the trade-off between
Outputs: Null-space projection matrix B
1: Get the singular values 1 in descending order and the corresponding right singular vectors U by singular value decomposition A, U SVD(C), where the left singular vectors omitted
2: Calculate the nullity Rt by the maximum second derivative as introduced in Eq. (28)
3: Select the right singular vectors of the R smallest singular values U Uo DID-10 R:D1
U.
4: Compute the projection matrix <-
TU.U IP
5: Update B with the weight y) by B +- nBB (1 I)) (corresponding to Eq. 29
Benchmarks: We conduct experiments under the class-incremental learning protocol, where the
classes in each task are disjoint, and task identity is unknown during inference. Four class-inceemental benchmarks with three widely used datasets are adopted: 10- and 20-split CIFAR-100, 10-split ImageNet-R (39 and 10-split DomainNet [25 38. For the CIFAR- 100 dataset, the total of 100 classes are randomly split into 10 or 20 tasks, which can evaluate the ability to handle different numbers of tasks. We follow [39 to randomly split the 200 classes in ImageNet-R into 10 tasks, which forms the 10-split ImageNet-R benchmark. For the 10-split DomainNet, we follow the same dataset protocol adopted in (38 and [T] to select the top 200 classes with the most images from the original DomainNet [25]. and randomly split them into 10 tasks with 20 classes per task. 25% samples of the training data in each dataset are picked as a validation set for searching optimal hyper-parameters.
15

---

Metrics: Formally, the final average accuracy and final average forgetting are defined as:
T
Final average accuracy =
Σ
T
i=1
Final average forgetting
1
Σ
(aj, ar,s); -E
max
T-1
1=1
where T is the number of tasks, aT is the accuracy of the T-th model on the i-th task samples, and aj is the accuracy of the j-th model on the i-th task samples.
Higher accuracy means the model performs better, while lower forgetting means stronger stability (i.e.. the ability to retain old knowledge). However, lower forgetting does not always generate higher accuracy since the accuracy is also affected by plasticity (i.e., the ability to learn new knowledge). The accuracy is the main metric we should focus on as it reflects the precision of classification in practice.
Implementations Details: For all the datasets and models, the images fed into the models are resized to 224 224 pixels and augmented by Augment 77] during training. For the VPT-based models, we use the Adam optimizer [16) with 31 = 0.9, 82 0.999 and a weight decay of 5 x 10-9 to train 100 epochs with initial learning rate of 0.01 and a batch size of 256 on all benchmarks. The learning rate is scaled by a factor of 0.1 at the 50-th and 80-th epoch. Our training losses consist of the cross-entropy loss for classification and the loss LLN Eq. (27 whose coefficient set to 1. Through cross validation on the validation set, we set the temperatures in the cross-entropy loss to 28, 25, 30 and 30 for the 10-split CIFAR 100, 20-split CIFAR100, 10-split ImageNet-R and 10-split DomainNet benchmarks. There are two saperr-arameterss stability and plasticity in null-space projection as introduced Section and we set both of them to be 0.97, 0.95, 0.94 and 0.95 for the four benchmarks by cross validation.
and 7/2 used for the trade-off between
As to the CLiP-based models, the differences in training settings are as follows. We train them for 20 epochs with the batch size of 220 and the learning rate 0.001 which decays at the 10-th and 16-th epoch. The temperatures are all set to since the logit scale is trainable. 1/1 and 7/2 are set 0.98 which is a proper value for all the benchmarks. We refine the embedding projection head for 50 epochs using the SGD optimizer with a learning rate of 0.001, a momentum of 0.9 and weight decay of x 10 10-4.
We implement our approach in PyTorch (24] with the timm library 4
The experiments
performed on a server with 128 GB RAM and four NVIDIA RTX 4090 GPUs Each of the experiment can be finished in three hours.
C Trade-off between Stability and Plasticity
Given that the null space of covariance matrix does not always exist in practice, Wang et al. (36 suggest approximating it by selecting the bases whose associated singular values approach zero, where the singular values smaller than a specified multiple (denoted as y in our paper) of the smallest one are selected. However, we experimentally find this strategy and the experience for selecting are not suitable for prompt tuning in ViTs to determine the nullities R, and R₂ for the uncentered covariance matrices C₁ and C2 in Algorithm D. which will be introduced afterwards. To solve this problem, we propose an adaptive nullity strategy to determine the nullities an adaptive manner. Utilizing the characteristic that the curve of descending singular values forms an "I." shape, we divide the curve into two parts by the point where the gradient changes fastest to cover most of the small singular values. It is realized by calculating the maximum second derivative of the points:
R1 = D arg max max(Ayy1 22g + } D-1
(28)
Ro = M - arg max (A - 2Xg- + ) M-1
where ' denotes the j-th singular value. We find it reaches near-minimun forgetting in our experiments which also means reaching near-optimal stability. Furthermore, to enhance the plasticity,
we fuse the projection matrices with identity matrices by the weights 7h E [0, 1] and n22 E [0, 1]
which should be close to I:
AP = 1m2 + (1 17) P2 B1 + (1 m]]1].
(29)
16

---

In this way, we can make trade-off between stability and plasticity by enhancing the plasticity based on near-optimal stability, and 77 and 7/2 are the hyper-parameters to control the trade-off.
D Comparison with PGP
D.1 Difference in Methods
The main difference between our method and PGP [26] are summarized as follows, (1) We derive different consistency condition for Affinity even if we ignore the LayerNorm operation and the bias terms the gkv-transformattin. Specifically, our simplified consistency condition for Affinity is X, W. W AP
0, contrasted with X,AP
0 in PGP. (2) We analyze the consistency
conditions for the complete self-attention, i..., softmax Qx operation. However, PGP does not account for the Aggregation. (3) We take the LayerNorm before self-attention into consideration and propose an invariant prompt distribution constraint, while is ignored in PGP.
VD Vz which contains the Aggregation
In conclusion, we conduct a comprehensive analysis of prompt tuning for the consistency objective, which provides complete guarantee eliminate the interference of tasks previous tasks. As demonstrated in our ablation study in the Experiment section, the consistency of Aggregation and LayerNorm also contribute to reducing forgetting, and thereby they should not be ignored. We make a comparison of the performance between PGP and our approach in the next subsection.
D.2 Performance Comparison
We compare with PGP [26] using the VPT-Seq and L2P [40] basclines the four benchmarks in our experiments. The results are shown in Table 5). We implement PGP to VPT (ie. VPT-PGP) under the same training settings VPT-NSP2 for fair comparison. For the I.2P-based methods, insert prompts into the first three layers instead of only the first layer the original implementation 40 An orthogonal projection is also applied to the prompt pool which is essentially linear layer in L2P-based models. We follow the training setting of PGP to train the L2P-based methods. The results in Table 5 demonstrate that our full approach and reduce more forgetting than PGP. Even when applying only the projection matrix B, for the Affinity operation, approach also performs better than PGP, demonstrating the effectiveness of our proposed method for mitigating the interference problem.
achieve more improvements in accuracy
Table 5: Comparison with PGP on four benchmarks and two continual learning baselines. "-B1" indicates only the projection matrix B₁ is used in approach
IOS -IS-CIFAR-100 20S-CIFAR-IMP 100
10S- Ims-ImageNet-R
10S-DomainNet
Method
Acc.t
Forgetting
Acc.t Forgetting
Acc.t
Forgetting
Acc.1 Forgetting
VPT-Seq
87.27
12.33
82.36
17.36
72.46
19.41
73.28
25.65
VPT-PGP
87.76
11.98
82.71
16.85
73.12
18.92
73.98
25.15
VPT-NSP"-E
90.58
6,91
88.13
10.27
78.05
8.14
82.31
10.89
VPT-NSP2
91.74
3.28
89.89
4.91
78.88
5.06
83.54
8.54
L2P
84.12
6.36
81.46
8.69
61.25
9.32
65.73
10.19
L2P-PGP
84.70
5.96
82.04
8.11
62.01
8.55
66.31
9.63
L2P-NSP2-B B
86.39
4.60
82.99
7.34
64.10
7.17
67.48
8.21
L2P-NSP2
86.78
4.22
83.37
6.93
64.66
6.84
68.14
7.79
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly state the claims in our abstract: we derive two consistency con- ditions of eliminating the interference problem under the invariant prompt distribution assumption for visual prompt tuning in the filed of continual learning, We implement them by the null-space projection method, and we validate the effectiveness and generalizability of our method. Our contributions are elaborated in the last paragraph of the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of our approach is discussed in the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The formulas used to derive our proposed conditions are numbered or cross- referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the algorithm of our approach, experimental settings and hyper- parameters adopted in our experiments in the Experimental Setups and Implementation Details section of the appendix. Our code is also available in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
19

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our code is available the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed experimental settings, including the data splits, hyper- parameters, optimizer and other settings in the experimental setups of appendix. We also provide the code in the supplemental material for a thorough reference.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the mean results over three runs in our experiments, and we report the standard deviations in the subsection of comparison with existing methods.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information of our compute resources in the experimental setups of appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and make sure our research conforms the ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
21

---

Justification: As a fundamental research in machine learning, the potential societal impact is not obvious at this stage.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No new pre-trained models or datasets are released in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We explicitly cite the used assets our papers, including the ViT model, timm library, DomainNet dataset, etc., and respect their license and terms during usage.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
22

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is provided in the supplemental material. A documentation for running the experiments is contained in the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "VUuIwSNNuV", "reviewer": "Reviewer_rV5K", "summary": "This paper proposes a novel paradigm for continual learning based on prompt tuning. By deriving the constraints for orthogonal projection of prompt gradients in ViTs, the method aims to minimize forgetting during the learning process. Experiments on four benchmarks show that NSP² achieves superior performance by avoiding forgetting.", "questions": "According to my understanding, as the number of tasks increases, the shrinking gradient subspace will make gradient updates increasingly difficult. Therefore, I would like to see the performance of NSP² on longer sequences of continual learning tasks.", "limitations": "As mentioned in the paper, additional constraints are introduced to simplify the consistency conditions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper focuses on orthogonal projection methods in continual learning, extending CNN-based methods to the ViT architecture, resulting in performance improvements.\n2. The method is supported by mathematical derivations of the derived constraints, providing strong theoretical guarantees.\n3. The paper addresses continual learning based on pre-trained models, which is a valuable topic.", "weaknesses": "1. While orthogonal gradient updates help prevent forgetting in continual learning, they also hinder positive knowledge transfer between tasks. In methods like L2P, similar tasks might select the same prompts, potentially improving performance on previous tasks after learning new ones.\n2. The derivation process in the methods section is overly lengthy, which leads to a very brief description of the experimental section. The comparison with existing methods is reduced to just three lines of text. Consider moving some derivations to the appendix.\n3. The experimental tables contain many blanks, which is not conducive to a systematic comparison of different methods' performance and weakens the contribution of this method.\n4. The paper omits comparisons with some leading methods, such as DAP[1], which is also based on prompt tuning.\n\n[1] Jung D, Han D, Bang J, et al. Generating instance-level prompts for rehearsal-free continual learning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11847-11857.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1:** In theory, it seems that orthogonal projection methods do not have the merits of backward knowledge transfer. However, this problem can be alleviated by two techniques in our approach. **(1)** We adopt a plasticity enhancement strategy which employs a hyper-parameter $\\bar{\\eta}$ to control the weight of each projection matrix: $\\Delta P=[\\bar{\\eta}B_2+(1-\\bar{\\eta})I]P_ \\mathcal{G}[\\bar{\\eta}B_1+(1-\\bar{\\eta})I]$. By incorporating weighted identity matrices, the strict orthogonal update direction can be relaxed to some extent. This relaxation enhances the model’s ability to integrate new knowledge when learning a new task. **(2)** The orthogonal projection matrix is constructed from an approximated null space as suggested in [36], since an exact null space may not exist in practice. The orthogonal projection can also be relaxed by the approximation.\n\nWe utilize the experiments on both **long-term** and **regular** CL settings to demonstrate the effectiveness of our approah.\n\n**(1)** We experiment on 5 benchmarks under the protocols of 50 tasks and 100 tasks to validate that our approach remains effective even within the context of **long-term CL**. The results are presented in **Table I** in the PDF and are strongly recommended for review by the reviewer. Despite lacking plasticity enhancement, VPT-NSP2 can outperform existing SOTA approaches and especially surpasses L2P by a large margin. This demonstrates that **forgetting is still the predominant factor affecting performance in long sequence of tasks**.\nWith the plasticity enhancement, VPT-NSP2 achieves significant increase in accuracy (by 1.1%\\~2.9%). **This demonstrates that our plasticity enhancement is effective in learning new knowledge in long-term CL.**\n\n**(2)** The experimental results across the four **regular CL** benchmarks are shown in Table II. NSP2 also outperforms L2P significantly even without plasticity enhancement. It achieves higher accuracy on all the benchmarks when using plasticity enhancement. Figure A shows the effects of the orthogonal projection weight $\\bar{\\eta}$. Accuracy can be improved with the decrease of $\\bar{\\eta}$, validating that **the proposed relaxed orthogonal constraints on gradients can promote learning new knowledge to achieve better performance**. The steady decrease in forgetting also verifies that **orthogonal constraints can be relaxed by decreasing $\\bar{\\eta}$**.\n\nIn the above experiments, our approach focusing on anti-forgetting outperforms L2P and our baseline (i.e. VPT-Seq) significantly, implying that **forgetting has a greater impact on the performance of continual learning than backward knowledge transfer.**\nIn L2P, prompts from old tasks may be selected and trained without constraints in new tasks, causing interference between tasks and leads to forgetting. Subsequent improvements, such as CODA-Prompt [32] and CPrompt [11], mitigate this issue by freezing prompts from old tasks and incrementally training new prompts for new tasks, thereby preventing the interference on old tasks and achieving better performance. The development of technical approaches also reflects that **catastrophic forgetting remains a predominant challenge to be addressed in continual learning**.\n\nOverall, the backward knowledge transfer ability is influenced by the plasticity of models. Nevertheless, the stability-plasticity dilemma remains **a fundamental challenge** in the field of CL. Achieving a better trade-off in this dilemma can enhance both anti-forgetting and backward knowledge transfer abilities. **In future work, we will focus on improving the backward transfer ability of our approach while maintaining its anti-forgetting capability to achieve better performance.**\n\n**2:** The detailed derivation process in the method section is crucial for explaining the derivation of the two projection matrices used in our approach. However, we understand the need for a more detailed experimental section. We appreciate the reviewer's suggestion and will consider moving some derivations to the appendix to provide more detailed experimental conclusions.\n\n**3:** We have made extensive efforts over the past week to reproduce 6 approaches for a more systematic comparison, including EvoPrompt [18] (AAAI’24), OVOR-Deep [12] (ICLR’24), DualP-PGP [26] (ICLR’24), InfLoRA [20] (CVPR’24), EASE [45] (CVPR’24) and CPrompt [11] (CVPR’24). The results are highlighted in blue in Table III. VPT-NSP2 achieves the highest accuracy, surpassing the second-best method by 0.6%\\~2.4% across the four benchmarks.\n\n**4:** In Table III and Table 2 in our paper, we compare with 12 prompt-tuning-based methods proposed in 2023 and 2024. DAP has the problem of **batch information leakage which results in comparison unfairness.** As stated by Zhou et al. [46] in the discussions on comparison fairness, \"during inference,... it is equal to directly annotating the task identity. When removing the batch information... a drastic degradation in the performance ...\"\n\nThe reproduced results reported in [46] are shown in Table IV. The accuracy declines by 4.8\\~42.6% with an average of 27.3% when eliminating the batch information leakage. Besides, we reproduce DAP on four benchmarks used in our experiments. We also observe a drastic degradation (by 16.42%\\~23.24%) in accuracy, as shown in Table V.\n\nThe official code of DAP, line 210-236 in file vit.py, assumes that test samples are batched and all of them come from the same task, which is an unreasonable assumption.\n\n>[46] Zhou D W, et al. Continual learning with pre-trained models: A survey. preprint arXiv:2401.16386, 2024.\n\n**5:** Response to the question: The experimental results for 50 and 100 tasks across 5 benchmarks are shown in Table I. Our approach surpasses the second-best competitor by 0.4%~6.4% with an average improvement of 2.0%, demonstrating that **our approach has the ability to handle longer sequences of continual learning tasks**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V0r9dElYOv", "reviewer": "Reviewer_4B73", "summary": "This work aims to eliminate the interference on previously learned knowledge for visual prompt tuning in the field of continual learning, so that catastrophic forgetting can be mitigated. To this end, it analyzes the conditions for keeping the output features unchanged in the transformer block that features the self-attention mechanism. Two consistency conditions for the self-attention are derived by deducing the proposed two sufficient conditions for the consistency objective. Moreover, a constraint on the distribution of the prompts is proposed to further simplify the LayerNorm operation. Consequently, the interference can be eliminated in theory by achieving the proposed two consistency conditions and the prompt-invariance constraint. The proposed approach implements the two consistency conditions by performing two null-space projections on the prompt gradients during training a new task; the constraint is implemented by an additional loss function that penalizes the drifting of prompt distribution across sequential tasks. Substantial experiments demonstrate the effectiveness of the proposed approach.", "questions": "In addition to the parameters of the prompts, the classifier contains parameters that require updates (i.e. the weights and biases) as well. Why not perform an orthogonal projection on the parameters of the classifier to reduce forgetting?", "limitations": "Reasonable discussion on limitations is included in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work comprehensively analyzes the conditions for learning without interference in the transformer-based visual prompt tuning. The proposed two consistency conditions with the constraint provide a theoretical guarantee on eliminating the interference problem. With this solid mathematical support, the proposed approach, which performs null-space projection on prompt gradients, shows significant improvements in reducing forgetting and increasing accuracy. The effectiveness of the approach is validated on extensive experiments, involving four class-incremental benchmarks and various pre-training datasets and paradigms. By visualizing the evolution of training losses, the effectiveness in mitigating interference is demonstrated as well. The proposed approach also achieves state-of-the-art performance on the four benchmarks. Besides, the adaptive nullity and plasticity enhancement strategy is also well-motivated and validated to be an effective way of balancing stability and plasticity. The paper is clearly structured and easy to follow. The figures in the paper are clear and well-designed, enhancing the overall readability and comprehension of the paper.", "weaknesses": "1. The consistency objective Eq. (10) is decomposed into two sufficient conditions, namely Eq. (11) and Eq. (12). A detailed explanation is necessary to understand why direct analysis and simplification of Eq. (10) is not pursued.\n2. The variable Q_p is omitted during the deduction of consistency conditions. Further explanation is needed to clarify why it can be disregarded.\n3. In the Adam-NSCL [36] method, the trade-off between stability and plasticity is controlled solely by the nullity. In contrast, this approach involves adding weighted identity matrices to the projection matrices. How do these two methods differ in balancing stability and plasticity? A deeper discussion is expected to highlight the advantage of the proposed trade-off strategy.\n4. The number of training epochs in this approach (100) is greater than that in other approaches (e.g., 50 epochs in DualPrompt for ImageNet-R). It would be better to provide a justification for this training setting.\n5. As described in line 4 of Algorithm 2, the projection matrix is normalized by a Frobenius norm. There is a lack of explanation for this operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1:** We do not simplify Eq. (10) directly because it leads to difficulty in deriving the solution expressed in terms of $\\Delta\\mathbf{P}$. Eq. (10) introduces non-unique solutions and a quadratic term of $\\Delta\\mathbf{P}^2$, which is explained in detail as follows.\n\nAccording to Eq. (3), we derive the following equation when directly simplifying Eq. (10):\n$softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}K_{X_t}^{\\top}\\end{bmatrix} }{\\sqrt{D}}\\right)\\begin{bmatrix}V_{X_t}\\\\\\\\V_{X_t}\\end{bmatrix}\n=softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}K_{X_{t+1}}^{\\top}\\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix} V_{X_t}\\\\\\\\ V_{X_{t+1}}\\end{bmatrix} $\\\nIt can be further expanded as:\\\n$softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}[LN(P_t)W_k+b_k]^{\\top} \\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix}V_{X_t}\\\\\\\\ LN(P_t)W_v+b_v\\end{bmatrix}\n=softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}[LN(P_t+\\Delta P)W_k+b_k]^{\\top}\\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix}V_{X_t}\\\\\\\\ LN(P_t+\\Delta P)W_v+b_v\\end{bmatrix}$\n\nHowever, it is hard to simplify the above equation to derive a solution expressed in terms of $\\Delta\\mathbf{P}$. First, the non-injection property of the softmax function causes non-unique solutions. That is to say, we cannot derive $\\mathbf{a}=\\mathbf{b}$ from $softmax\\left(\\mathbf{a}\\right)=softmax\\left(\\mathbf{b}\\right)$. Second, when we omit the softmax operation, the multiplication between $\\mathbf{Q}_{X_t}\\left[LN\\left(\\mathbf{P}_t+\\Delta\\mathbf{P}\\right)\\mathbf{W}_k\\right]^\\top$ and $LN\\left(\\mathbf{P}_t+\\Delta\\mathbf{P}\\right)W_v$ derives a quadratic term $LN\\left(\\mathbf{P}_t+\\Delta\\mathbf{P}\\right)^\\top LN\\left(\\mathbf{P}_t+\\Delta \\mathbf{P}\\right)$, which results in difficult simplification. Due to the above obstacles, we propose two sufficient conditions (*i.e.*, Eq. (11) and Eq. (12)), rather than simplifying Eq. (10) directly.\n\n**2:** In VPT-Deep [13], the output tokens corresponding to the input prompts of the current ViT layer will be replaced by new trainable prompts in the next ViT layer. Therefore, we do not need to compute the output tokens corresponding to the input prompts. According to the forward propagation of ViT, the output prompt tokens are derived from $\\mathbf{Q}_P$. Consequently, the variable $\\mathbf{Q}_P$ can be omitted.\n\nSpecifically, $\\mathbf{Q}_P$ represents the prompt queries in the attention map. After the self-attention operation, the corresponding derived tokens of $\\mathbf{Q}_P$ are denoted as $\\mathbf{F} _{Q_P}$.  Then $\\mathbf{F} _{Q_P}$ will undergo another LayerNorm and an MLP to derive the output prompt tokens $\\mathbf{Y} _{Q_P}$. $\\mathbf{Y} _{Q_P}$ can be neglected due to the replacement of new trainable prompts. To reduce computation overhead, $\\mathbf{Q}_P$ can be just neglected in the previous affinity operation in self-attention. Note that omitting $\\mathbf{Q}_P$ has no impact on the output image tokens of the ViT layer, as the subsequent Aggregation, LayerNorm and MLP operations are performed independently for each token.\n\n**3:** The difference lies in that Adam-NSCL balances stability and plasticity solely by controlling the nullity, while our method first achieves a near-optimal stability point and then enhances plasticity by weakening the orthogonal constraints. Adding a weighted identity matrix to the gradients in our approach is more flexible for enhancing plasticity. In our method, the orthogonal constraint is strict when $\\bar{\\eta}=1$, and the model's stability reaches a near-optimal level. As $\\bar{\\eta}$ gradually decreases to 0, the orthogonal constraints are progressively relaxed until completely eliminated. This enables the model to achieve maximum plasticity for learning new tasks. Therefore, incorporating a weighted identity matrix into orthogonal constraints facilitates the model's efficient and effective acquisition of new knowledge.\n\n**4:** Orthogonal projection-based methods constrain the update direction of parameters during training. To fully optimize the model parameters and converge to a (local) optimal point, the model usually requires adequate training with more epochs. Therefore, we adopt a larger number of training epochs.\n\n**5:** The projection matrix is normalized by a Frobenius norm to provide an upper bound for the scale of gradients after projection. Specifically, the Frobenius norm is sub-multiplicative [47]. For two matrices $\\mathbf{A}$ and $\\mathbf{B}$, the inequality $||\\mathbf{AB}||_F\\le||\\mathbf{A}||_F||\\mathbf{B}||_F$ holds. For convenience, we use $\\tilde{\\mathbf{U}}_0$ to denote $\\mathbf{U}_0 \\mathbf{U}_0^\\top$ in line 4 of Algorithm 2. Then the projection matrix is denoted as $\\mathcal{B}=\\frac{\\tilde{\\mathbf{U}}_0}{||\\tilde{\\mathbf{U}}_0||_F}$.  Consequently, when the gradient matrix $\\mathbf{P} _\\mathcal{G}$ is multiplied by the projection matrix $\\mathcal{B}$, we have $ ||\\mathcal{B}\\mathbf{P} _\\mathcal{G}||_F \\le ||\\mathcal{B}||_F ||\\mathbf{P} _\\mathcal{G}||_F=|\\mathbf{P} _\\mathcal{G}||_F$. This demonstrates that using the Frobenius norm can provide an upper bound for the scale of the projected gradients, thereby preventing excessive gradient magnitudes.\n\n> [47] Meyer C D. Matrix analysis and applied linear algebra Society for Industrial and Applied Mathematics, 2023.\n\n**6:** Response to the question: The reason is that the classifier in each task is trained independently rather than continuously. The classifiers of previously learned tasks remain fixed during training on new tasks, eliminating the need for gradient calculations. Moreover, the classifier of the current task is only updated in this task and is unaffected by the classes of previous tasks, rendering orthogonal constraints unnecessary. As a result, orthogonal projections are not required for classifiers in any task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H562M9iY8D", "reviewer": "Reviewer_W3C1", "summary": "This paper introduces the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. They propose two sufficient consistency conditions for the self-attention and an invariant prompt distribution constraint for LayerNorm, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. Extensive experimental results show the superiority of the proposed method.", "questions": "See the weaknesses above.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1)\tThe idea in this paper is novel and interesting. They introduce the orthogonal projection into the visual prompting tuning for continual learning, which comprehensively considers the full operations in the Transformer Layer.\n2)\tThe experimental results are sufficient and superior to the SOTA methods.\n3)\tOverall, this paper is well organized and well-written, which is easy for the readers to follow.", "weaknesses": "1)\tIn figure 1 of the overall framework, it is unclear why the Qp term in the affinity matrix can be neglected. The authors should provide clear reasons for such conclusion.\n2)\tIn the method part of Eq.(10), In order to satisfy F_{Z_t}=F_{Z_t-1}, the authors transform Eq.(10) into the following Eq.(11) and Eq.(12), which are two sufficient conditions. Why not optimizing Eq.(10) directly? Please give detailed reasons.\n3)\tWhat’s the main differences between the single-head and multi-head self-attention for introducing the proposed orthogonal projection into the visual prompting tuning for continual learning?\n4)\tIn the method optimization part, the authors propose the approximation method as illustrated in Eq.(25). Why not using the optimization method proposed in PGP[26]? What are the main differences ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1:** In VPT-Deep [13], the output tokens corresponding to the input prompts of the current ViT layer will be replaced by new trainable prompts in the next ViT layer. Therefore, we do not need to compute the output tokens corresponding to the input prompts. According to the forward propagation of ViT, the output prompt tokens are derived from $\\mathrm{Q}_P$. Consequently, the $\\mathrm{Q}_P$ term can be neglected.\n\nSpecifically, $\\mathrm{Q}_P$ represents the prompt queries in the attention map. After the self-attention operation, the corresponding derived tokens of $\\mathrm{Q}_P$ are denoted as $\\mathrm{F} _{Q_P}$. Then $\\mathrm{F} _{Q_P}$ will undergo another LayerNorm and an MLP to derive the output prompt tokens $\\mathrm{Y} _{Q_P}$. $\\mathrm{Y} _{Q_P}$ can be neglected due to the replacement of new trainable prompts. To reduce computation overhead, $\\mathrm{Q}_P$ can be just neglected in the previous affinity matrix in self-attention. Note that omitting $\\mathrm{Q}_P$ has no impact on the output image tokens of the ViT layer, as the subsequent Aggregation, LayerNorm and MLP operations are performed independently for each token.\n\n**2:** The reason is that optimizing Eq. (10) directly leads to difficulty in deriving the solution expressed in terms of $\\Delta\\mathbf{P}$. Eq. (10) introduces non-unique solutions and a quadratic term of $\\Delta\\mathbf{P}^2$, which is explained in detail as follows.\n\nAccording to Eq. (3), we derive the following equation when directly optimizing Eq. (10):\n$softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}K_{X_t}^{\\top}\\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix}V_{X_t}\\\\\\\\ V_{X_t}\\end{bmatrix}\n=softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}K_{X_{t+1}}^{\\top}\\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix} V_{X_t}\\\\\\\\ V_{X_{t+1}}\\end{bmatrix} $\\\nIt can be further expanded as:\\\n$softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}[LN(P_t)W_k+b_k]^{\\top} \\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix}V_{X_t}\\\\\\\\ LN(P_t)W_v+b_v\\end{bmatrix}\n=softmax\\left(\\frac{\\begin{bmatrix}Q_{X_t}K_{X_t}^{\\top}&Q_{X_t}[LN(P_t+\\Delta P)W_k+b_k]^{\\top}\\end{bmatrix}}{\\sqrt{D}}\\right)\\begin{bmatrix}V_{X_t}\\\\\\\\ LN(P_t+\\Delta P)W_v+b_v\\end{bmatrix}$\n\nHowever, it is hard to simplify the above equation to derive a solution expressed in terms of $\\Delta\\mathbf{P}$. First, the non-injection property of the softmax function causes non-unique solutions. That is to say, we cannot derive $\\mathbf{a}=\\mathbf{b}$ from $softmax\\left(\\mathbf{a}\\right)=softmax\\left(\\mathbf{b}\\right)$. Second, when we omit the softmax operation, the multiplication between $\\mathbf{Q}_{X_t}\\left[LN\\left(\\mathbf{P}_t+\\Delta\\mathbf{P}\\right)\\mathbf{W}_k\\right]^\\top$ and $LN\\left(\\mathbf{P}_t+\\Delta\\mathbf{P}\\right)W_v$ derives a quadratic term $LN\\left(\\mathbf{P}_t+\\Delta\\mathbf{P}\\right)^\\top LN\\left(\\mathbf{P}_t+\\Delta \\mathbf{P}\\right)$, which results in difficult optimization. Due to the above obstacles, we transform Eq. (10) into Eq. (11) and Eq. (12), rather than optimizing Eq. (10) directly.\n\n**3:** The main difference is that the parameter update should be orthogonal to the subspace spanned by the concatenation matrices from all heads for multi-head self-attention. Specifically, for the singe-head self-attention, the parameter update should be orthogonal to the subspaces spanned be $\\mathbf{Q} _{X_t} \\mathbf{W}_k^\\top$ and $\\mathbf{S} _{P_t}$ according to Eq. (23) and Eq. (24). When considering the multi-head self-attention, the subspaces should be spanned by $[\\mathbf{Q} _{X_t.1} \\mathbf{W}_k^\\top; \\mathbf{Q} _{X_t.2} \\mathbf{W}_k^\\top; \\cdots; \\mathbf{Q} _{X_t.H} \\mathbf{W}_k^\\top]$ and $[\\mathbf{S} _{P_t.1}; \\mathbf{S} _{P_t.2}; \\cdots; \\mathbf{S} _{P_t.H}]$, where $\\mathbf{Q} _{X_t.h}$ and $\\mathbf{S} _{P_t.h}$ denote the corresponding intermediate activations in the $h$-th head ($h\\in\\\\{1,2,⋯,H\\\\}$), respectively. $H$ is the number of heads, and \"$[;]$\" represents the concatenation of matrices along the first dimension. Therefore, only an additional step of concatenation of the corresponding matrices from all heads is required for introducing the proposed orthogonal projection into the multi-head self-attention.\n\n**4:** In PGP [26], the matrices to be optimized in their two conditions are element-wise summed. Then, SVD is applied to the summed matrix to obtain the orthogonal projection matrix. To enable the addition of the two matrices, PCA dimensionality reduction is also used on one of the matrices to align the dimensions.\n\nHowever, this summing approach is not applicable in our method, since the two matrices $\\mathcal{B}_1$ and $\\mathcal{B}_2$ are multiplied on the right and the left, respectively. Even with dimensionality reduction, they cannot be summed and merged into a single matrix. In fact, the two projection matrices have different meanings: $\\mathcal{B}_1$ is a constraint on individual tokens, while $\\mathcal{B}_2$ is a constraint on a specific dimension of all tokens at the dimension level. Therefore, it is not suitable to add them directly in terms of their respective meanings. Consequently, the optimization method used in PGP [26] cannot be applied in our case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VKXJYMSWc", "reviewer": "Reviewer_2TYW", "summary": "This paper introduces the orthogonal projection into the visual prompt tuning for continual learning, which comprehensively considers the full operations of a transformer layer on the interference problem. Moreover, two sufficient consistency conditions for self-attention and an invariant prompt distribution constraint for LayerNorm are theoretically deduced, based on which an effective null-space-based approximation solution is introduced to implement the prompt gradient orthogonal projection for visual prompt tuning. Finally, extensive experimental results demonstrate the effectiveness of anti-forgetting on four class-incremental benchmarks with diverse pre-trained baseline models, and our approach achieves superior performances to state-of-the-art methods.", "questions": "Please see “Weaknesses”", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1)The research motivation for algorithms is reasonable, and the theoretical proof is solid. Constraining the new learnable parameter orthogonal to the previous weight is a reasonable motivation to prevent historical knowledge forgetting.\n\n2)Extensive evaluation and amazing performance show the superiority of the proposed method.", "weaknesses": "1)Eq.(8) seems be error.\n\n2)L139: Eq.(8) suggests that if the weight update ∆Θ is orthogonal to the previous input feature Xt during training in the new task, the corresponding output feature will remain unchanged. Moreover, will this degrade the model’s discriminative ability for the current task when the weight update is orthogonal to the previous feature?\n\n3)What is the model’s complexity and running time compared to the baseline VPT?\n\n4)Suggest using a more accurate diagram to describe the algorithm to help understand it more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1:** The symbol \"$\\Rightarrow$\"  may cause confusion in Eq. (8). Our intention was to convey that the left equation can be *simplified to yield* the right equation, rather than the left equation *being a sufficient condition* for the right one. We will correct this to ensure a more precise expression.\n\n**2:** The orthogonal projection will degrade the model’s discriminative ability for the current task, since the update direction is constrained into a subspace smaller than the original gradient space. This is a fundamental challenge known as stability-plasticity dilemma [48] in continual learning. The stability indicates the model’s discriminative ability for the old tasks, while the plasticity indicates the model’s discriminative ability for the current (new) task. A stronger stability usually causes a weaker plasticity and vice versa. **The plasticity-stability dilemma objectively exists in continual learning. We cannot completely eliminate this dilemma yet we can improve the overall accuracy by carefully balancing the stability and plasticity.**\n\nIn our approach, two techniques help to enhance the plasticity of our model. **(1)** As introduced in the \"Trade-off between Stability and Plasticity\" section, we employ a hyper-parameter $\\bar{\\eta}$ to control the weight of each projection matrix: $\\Delta \\mathbf{P}=[\\bar{\\eta}\\mathcal{B}_2 + (1-\\bar{\\eta})\\mathbf{I}] \\mathbf{P} _{\\mathcal{G}} [\\bar{\\eta}\\mathcal{B}_1 + (1-\\bar{\\eta})\\mathbf{I}]$. When $\\bar{\\eta}$ is less than 1, the update direction of parameters is not strictly constrained to the orthogonal direction. Instead, the parameters can update in the original direction of the gradients. This relaxation enhances the model’s discriminative ability when learning a new task. As demonstrated in the experimental results of Figure 5, the model can achieve higher accuracy with higher (worse) forgetting as $\\bar{\\eta}$ decreases. It indicates the importance of the trade-off between stability and plasticity, and our approach can make a good trade-off. **(2)** As suggested in [36], the orthogonal projection matrix is constructed from an approximated null space, since an exact null space may not always exist in practice. The orthogonal projection can also be relaxed by the approximation to encourage the model to acquire new knowledge.\n\n> [48] Mermillod M, Bugaiska A, Bonin P. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 2013, 4: 504.\n\n**3:** (1) Complexity\n\nCompared to the baseline “VPT-Seq”, our approach introduces the following additional computation: 1) The null-space projections in each optimization step, *i.e.*, Eq. (25). For a group of prompts $\\mathbf{P}$ of shape $M \\times D$, the complexity of the two projections is $\\mathcal{O}\\left(DM(D+M)\\right)$. Suppose there are $L$ ViT layers equipped with prompts. The batch size and epochs are represented as $n_{batch}$ and $n_{epoch}$, respectively. We denote the total number of samples in all $T$ tasks as $n_{total}$. After training all the tasks, the complexity introduced by the null-space projections is $\\mathcal{O}\\left( \\frac{n_{total} n_{epoch} LDM(D+M)}{n_{batch}} \\right)$. 2) The loss of prompt distribution. $i.e.$, Eq. (26). We use $E_{dist}$ to represent the computational overhead of the distribution loss between two scalar elements. Considering all the layers and optimization iterations during training, the complexity introduced by the prompt distribution loss is $\\mathcal{O}\\left( \\frac{n_{total} n_{epoch} LDM E_{dist}}{n_{batch}} \\right)$. 3) The forward process to obtain $\\mathbf{Q} _{X_t} \\mathbf{W}_k^\\top$ and $\\mathbf{S} _{P_t}$ for each sample after the training stage of a task, *i.e.*, line 21 of Algorithm 1 in the appendix. We denote the computational cost of the model’s forward propagation as $E _{model}$. Thereby, the introduced additional computation is $n _{total} E _{model}$. 4) Computation of uncentered covariance matrices, *i.e.*, line 24 of Algorithm 1. The complexity of computing the two uncentered covariance matrices is $\\mathcal{O}\\left( n _{total}^2 N^2 (D+M) \\right)$ , where $N$=197 is the number of image tokens in the ViT-B/16 model. 5) Computation of the null-space projection matrices, *i.e.*, line 25 of Algorithm 1. In each task, the two projection matrices are only need to be computed and updated once. We use $E _{matrices}$ to denote the computational cost in this process. The total additional computation in $T$ tasks is $T E _{matrices}$.\n\nOverall, the complexity compared to the baseline VPT is $\\mathcal{O}\\left( \\frac{n _{total} n _{epoch} LDM(D+M+E _{dist})}{n _{batch}} + n _{total} E _{model} + n _{total}^2 N^2 (D+M) + T E _{matrices} \\right)$.\n\n(2) Running time\n\nWe report the average running time over three runs for the baseline model and our approach on the four benchmarks in Table VI. Compared to the baseline VPT-Seq, the running time increases by 2\\~9 minutes (2.38%\\~6.43%) across these benchmarks, with an average increase of 4.5 minutes (3.84%). The additional running time introduced by our approach is acceptable as it constitutes only a small portion of the overall running time.\n\n**4:** Thank you for the valuable suggestion. We will add the diagram to describe our algorithm visually.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback, with three reviewers (W3C1, 2TYW and 4B73) strongly supporting our work. We are encouraged that reviews think our paper:
- **the idea in this paper is novel and interesting** (by Reviewer W3C1);
- **the theoretical proof is solid** (by Reviewer 2TYW);
- **providing strong theoretical guarantees** (by Reviewer rV5K);
- **substantial experiments demonstrate the effectiveness of the proposed approach** (by Reviewer 4B73).

The main concern of reviewer rV5K is that orthogonal constraints may hinder learning new knowledge and degrade the performance on long-term continual learning. We address this issue by **conducting experiments with respect to 50 and 100 tasks across 5 benchmarks**. The results demonstrate that our approach remains **effective and superior** even within the context of long-term continual learning. Moreover, our experimental analysis on regular CL benchmarks also verifies that orthogonal constraints **can be relaxed by the proposed plasticity enhancement**, which **promotes the model to learn new knowledge and achieve better performance**.

All questions are addressed in reviewer-specific responses. Additionally, please find the PDF attached with helper tables and figures. These are referenced and described in our individual responses to reviewers.

Meta-review: This paper receives unanimous accept recommendations. After a careful check of paper, comments, and rebuttal, the AC agrees with the reviewers and think the authors has well addressed the reviewer concern. The idea of this paper is novel and the result is good for continual learning. Thus, the AC makes a accept recommendation.
Decision: Accept (poster)