AdaMSS: Adaptive Multi-Subspace Approach for Parameter-Efficient Fine-Tuning
Authors: Jingjing Zheng
Wanglong Lu
Yiming Dong
Chaojie Ji
Yankai Cao
Zhouchen Lin
Venue: NeurIPS 2025 poster ()
Keywords: Parameter-Efficient Fine-Tuning
Low-Rank Adaptation
Low-Rank Representation
Primary area: deep_learning

Abstract: In this paper, we propose AdaMSS, an adaptive multi-subspace approach for parameter-efficient fine-tuning  of large models. Unlike traditional parameter-efficient fine-tuning methods that operate within a large single subspace of the network weights, AdaMSS leverages subspace segmentation to obtain multiple smaller subspaces and adaptively reduces the number of trainable parameters during training, ultimately updating only those associated with a small subset of subspaces most relevant to the target downstream task. By using  the lowest-rank representation, AdaMSS achieves more compact expressiveness and finer tuning of the model parameters. Theoretical analyses demonstrate that AdaMSS has better generalization  guarantee than LoRA, PiSSA, and other single-subspace low-rank-based methods. Extensive experiments across image classification, natural language understanding, and natural language generation tasks show that AdaMSS achieves comparable performance to full fine-tuning  and outperforms other parameter-efficient fine-tuning   methods in most cases, all while requiring fewer trainable parameters. Notably, on the ViT-Large model, AdaMSS achieves 4.7\% higher average accuracy than LoRA across seven tasks, using just 15.4\% of the trainable parameters. On RoBERTa-Large, AdaMSS outperforms PiSSA by 7\% in average accuracy across six tasks while reducing the number of trainable parameters by approximately 94.4\%. These results demonstrate the effectiveness of AdaMSS in parameter-efficient fine-tuning. The code for AdaMSS is available at https://github.com/jzheng20/AdaMSS.

AdaMSS: Adaptive Multi-Subspace Approach for Parameter-Effiieet Fine-Tuning
Jingjing Zheng 12.36 j /heeg2330gmmil....... com
Wanglong Lu Wanglong. Luðnasdaq com
Yiming Dong' yimingdong wttp://onnnmm@uuoooooooooo com
Chaojie Ji236 jej @math. ubc.ca
Yankai Ca0
Zhouchen L7n1 zlin@pku. edu. cn
yankai. caolubo.ca
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University IInstitute of Applied Mathematics, The University of British Columbia
Centre for AI Docision-Making and Action, The University of British Columbia AI Analytics Team, Nasdaq, St. John's. Sohn's,,n NI., Canada Department of Chemical and Biological Engineering, The University of British Columbia
DDeaarmeent of Mathematics, The University of British Columbia
Institute for Artificial Intelligence, Peking University
SPazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
Abstract
this paper, we propose AdaMSS, an adaptive multi-subspace approach for paramcter-efffcient fine-tuning of large models. Unlike traditional parameter- efficient fine-tune tne-ttuing methods that operate within a large single subspace the network weights, AdaMSS leverages subspace segmentation obtain multiple smaller subspaces and adaptively reduces the number of trainable parameters during training, ultimately updating only those associated with small subset of subspaces most relevant to the target downstream task. By using the lowest-rank representation, AdaMSS achieves the model parameters. Theoretical analyses demonstrate that AdaMSS has better generalization guarantee than LoRA, PiSSA, and other single-subspace low-rank- based methods. Extensive experiments across image classification. natural language understanding, and natural language generation tasks show that AdaMSS achieves comparable performance to full fine-tuning and outperforms other parameter- efficient fine-tuning methods most cases, all while requiring fewer trainable parameters. Notably, on the ViT-Large model, AdaMSS achieves 4.7% higher average accuracy than LoRA across seven tasks, using just 15. 55.4% of the trainable parameters. On RoBERTa-Large AdaMSS outperforms PiSSA by 7% in average accuracy across six tasks while reducing the number of trainable parameters by approximately 94.4%. These results demonstrate the effectiveness of AdaMSS parameter-efficient fine-tuning. The code for AdaMSS available at https: //github. com/ jzheng20/Adatte AdaMSS.
compact expressiveness and finer tuning of
Introduction
With the successful application of large pre-trained models in natural language processing and computer vision, the sarameter-efficien fine-tuning (PEFT) methods have gradually become key
39th Conference Neural Information Processing Systems (NeurIPS 2025).
Cooresponding authors

---

z
S2
z
S3
2
(b)
(c)
Figure I: (a) Weight matrix W; (b) Multi-subssaces structure of the columns of W (i.e. [W].)) (c) The lowest-rank representation of W. where each block Z (R) serves representation for the columns lying in the subspace Sk-
strategies for efficiently adapting these models. Among these PEFT methods, LoRA (Low-Rank Adaptation) [1] has gained particular attention, as it adopts learnable low-rank structures represent weight changes, significantly reducing memory requirements. Following LoRA, numerous low-rank adaptation methods have since been developed [2. 3, 4, 5, 7, 8, 9, 10, 11].
Despite sharing common goal of PEFT, these methods primarily differ in how they impose low-rank assumptions. For instance, LoRA [1] and AdaLoRA [3] assume that weight changes AW& E R=x" are low-rank and represent them using the product two smaller matrices, A E "xx and B E Rr where < Alternatively, PiSSA [2] assumes that the original weight matrix W approximately low-rank and achieves efficient fine-tuning by training its principal components. A further line of research is represented by methods such LoRA-GA [5]. which hypothesize that the gradient of the loss function with respect to the weights-Ea ie.. (E(W)-xxhibis low-rank structure.
These approaches typical single-subspace methods, they rely on the assumption that weight changes, weights, or gradients lie in, or approximately lie in, a single low-dimensional subspace. However, this assumption inevitably leads to fundamental trade-off between limited expressiveness and parameter efficiency: approximating the weights with a single fow-dimensional subspace with small limits the model's capacity for adaptation, while increasing the subspace dimension improves expressiveness but results in a substantial rise in the number of trainable parameters and memory usage.
To address the dilemma between limited expressiveness and parameter efficiency, we propose an Adaptive Multi Multi-Subsparn Subspace approach (AdaMSS) that enables finer tuning of model parameters and has stronger expressiveness. As we will illustrate this work, the columns of the weight matrix are approximately distributed across multiple linear subspaces (Figure I (a)-(b) provides conceptual illustration of multi-subspaces structure of the network weights).
The key features of this work are summarized as follows:
Compact Expressiveness: By leveraging the observation that the network weights approximately located in multiple subspaces, AdaMSS uses a lowest-rank representation for the network weights, which exhibit an approximate block-diagonal structure (see Figure I (c)). This design leads to a more compact representation while preserving both global and local low-rank structure from the original weights (see Property 1), which is essential for ensuring effective adaptation across multiple subspaces.
Generalization Guarantee: We theoretically show that, under the same rank assumption т. the proposed AdaMSS achieves lower Gaussian complexity bound and therefore expected to exhibit a stronger generalization capability compared to single-subspace-bart based methods. Theorem formalizes an upper bound on the expected loss of AdaMSS, providing its generalization guarantee.
Multi-Subs Multi-SubspacceBasee Adaptive Budget Allocation: Thanks to the multi-subspaac struc- ture of network weights, we can adaptively freeze the parameters associated with subspaces of lower importance, and eventually updates only those associated with small subset of subspaces relevant to the target downstream task, where the importance of each subspace is evaluated during training.

---

Experimental results demonstrate the effectiveness of AdaMSS for PEFT across diverse tasks and show that outperforms existing methods in most cases while significantly reducing the number of trainable parameters. For example, compared with LoRA, AdaMSS achieves 4.7% higher average accuracy on ViT-Large [12] and 8.5% higher accuracy on GSM8K [13] with LLaMA 2-7B [14], using only 13.63% and 1.25% of the trainable parameters, respectively. On RoBeRTa-Large [15], AdaMSS achieves comparable performance to LoRA while using just 5.62% of the trainable param- eters. These results suggest that multi-subspace-baseee fine-tuning offers promising direction for improving the trade-off between expressiveness and parameter efficiency.
Preliminaries
In this section, we introduce the notations used throughout the paper, and provide preliminaries and a brief review of existing literature relevant to our study. Unless otherwise specified, we use a, a, and to denote scalars, vectors, and matrices, respectively. The transpose of A is denoted by T We use j.j. [A]... and [A]:,] to represent the (f, ji-th element, the i-th row vector, and the j-th column vector of A, respectively.
2.1 Low-Rank Representation for Subspace Segmentation
To get the lowest-rank presentation for given samples [M] .2. [M]. R., Liu al. proposed the Low-Rank Representation (LRR) model [16] expressed as follows:
I E*] min rank Z) + X|| lo s.t. M.MM- MZ E,
(1)
Z.E
where Z known as the coefficient matrix, rank (Z) denotes the rank of matrix E Ruax Le., the number of singular values of Z, M Rdx= represents the matrix composed of the samples, E represents the residual component (i.e., noise and redundancy in the samples). The norm | l is used quantify the magnitude of the nonzero entries of E, and its specific definition depends on the prior knowledge about E. The problem (1) can be approximated by convex formulation and solved iteratively [16]. and it computationally expensive. Fortunately, when E O, problem (1) has a dlosed-form solution, given by Z VV where I x is obtained through the skinny SVD of M, i.e., M USV
Assuming the samples [M] [M] [M]. drawn from a union of linear subspaces {S
where the subspace number K and {S* are unknown, the goal of subspace segmentation is to simultancously assign the samples into their respective subspaces. When the subspaces {S₂ H are independent of each other, samples from different subspaces cannot be expressed as linear combinations of each other. Consequently, Z" becomes a block-diagonal matrix. To solve the subspace segmentation problem, Liu al. determine the subspace number K and to obtain the segmentation results based on Z* [16]. This LLR- subspace segmentation method is widely applied in clustering t1, 17, 18, 19].
3 Multi-Subspace-Bactt Adaptation
Let Wo E R denote the pretrained weight matrix of a network layer, where d and represent the input and output dimensions of the layer, respectively. The full fine-tuning (FF) of the network is formalized as
AW* arg
min AW ERixn
L(W. + AW),
where denotes the loss function, AW pdxn is the update on Wo. o. and = W, AW is the corresponding updated weights. In this section, we aim to find a compact representation of W via subspace segmentation [16]. which has both low-rank and strictly block- Bock-diagonall structure. By doing so, instead of directly updating W. FF, we perform incremental updates within this compact representation.
PI this work, the samples here does refer layer input
3

---

-00
(a)
(c)
Figure 2: Illustration multiple subspaces structure pretrained network weights: an approximate block-diagonal structure of Z from the first layer of the pretrained (a) ViT-Large (query), (b) Roberta-Large (query), and (c) LLaMA 2-7B (query).
3.1 Motivation: Multi-Subspace Structure of the Network Weights
Motivated by PiSSA [2], we focus on updating the principle components of Wo (denoted as Wo). and decompose the pretrained weight Wo Wo Wo + Wre.. where W res Wo R..
for given R, and Wo [ 00 is
the truncated SVD of Wo- We apply the LRR to obtain a lowest-rank representation of Wo- Considering computational complexity, we adapt the LRR with E 0:
Z = arg min rank Zo)
s.t. W WoZo.
(3)
This formulation uses the coefficient matrix Zo to linearly represent the columns of Wo Rdxn By analyzing Zü we observe that exhibits an approximate block-diagonal structure (see Figure 2 for R = 100), indicating that the columns of Wo WoZ are grouped into different clusters, where columns within the same cluster are approximately linear combinations of each other. In other words, the row space of Wo can be approximated by multiple smaller low-rank subspaces. By leveraging the approximate block dickonia structure and low-rank properties of Zü, drive more compact representation of Wo than Wo- 0-
3.2 A Compact Representation By Subspace Segmentation
In the previous subsection, we analyzed the multi-subspace structure of Wa and obtained a compact representation Z which exhibits approximate block-diagonal structure. We derive well- organized and strictly block-diagonal representation of Wo by subspace segmentation.
Leveraging subspace segmentation and the multi-subspace structure, the row space of W can be approximately segmented into K smaller low-dimensional subspaces. We assume that the columns in Wo = [W(") w ,.... ,..., w are well-organized and partitioned based the segmentation results, where each block wi(k) Rdxa= contains the columns assigned to the k-th subspace. Accordingly, W., Zg and W res are also divided into K blocks, leading to the following decomposition
( w
1
- W
w w!! (K) (diag (Zi) (2)) (Z))(())) + (E()) EE!"
k)
+ [L) rest (2)
w(k)
reat
where Zi [Z (1) [2]
[Z [K) is decomposed into a sum strictly block block-diagonal
matrix diag (Z%) (1), (Z (2) .... (k)) and noise matrix [E(" E(() ... , [=[ and each
block (Z)) E R xn, is the k-th diagonal block of Z
Define W = W ((k) + (k)
ress we have
(((k)
(k)
W. (1) (k)
(k) rea
(5)

---

Algorithm Initialization of AdaMSS
Input: Wo. R. and Fk-
Output:
Estimated K, ( 1== and the initialized values of {B k
k= and {C
k==
Step 1:
Compute the skinny SVD of W o: Wo V
Step Obtain Wa a by W. 1U. Sul 1. Ul., for given R:
b Steps 1-3 are derived from Section 3.1. Step 4: Determine (or estimate) the number of subspaces K, and apply clustering to assign the the
Step 3: Obtain Z by Z = ol.... V 1.:1:
columns of Wa a to K subspaces [16], as detailed in Algorithm (see Appendix C);
Step 5: Obtain the matrix blocks (W)) TRmb (
(h)
(k)
1 and ((Z))())
K
Tkmi by the segmentation
results;
Steps 4-5 are derived from Section 3.2.
Step For each k, compute the skinny SVD of (A W( ((( ((k)
(k)
(v
W. )T:
Wo W.
Step 7: Perform the QR decomposition of s
(k) Wo W.
(V (k)
((Z)) (k)
s(k)
(v
T (Z))(A)
Wo W
Q (()) (())
Step Set (k) U() Section 3.3. Wo"
initialize ((*) Q (R) and ( = 0.
Steps 6-8 are derived from
Sulti-Subspace-Bae Based Incremental Update
Pretrained Weight
Matrix W
BeD
yER"
Rd
gih-1i
Hulti-Subspace -Based Incremental Update
m
Aggression
Rearrange
Figure 3: Illustration of the proposed Multi-Subspace-Basee Adaptation framework.
for each k where (Z)) E ma xna is used approximately represents W (k) We thus obtain
(W (1) ( w w
(2)
diag (Z)) (Z)) (k)
(
(1) 1
W (K) res
(6)
reat
and new representation of Wo:
diag (2))
(Z)
The following properties hold for this representation.
Property 1. If the subspaces are independent (i.e., rank = EK (ank((Z)((k)) then
(1) (Global dructure-preserving Σ rank( WW(h (h)) ΣK rank (( (Zjj) (k))
rank Wo))
(2) (Local structure- preserrr-pressrvvnngg) (ank((Z)))(5) rank (W for N: 1,2,
Proof. The corresponding proof is given in the Appendix A.
Property I ensures both global and local low-rank structure preservation by the representation of W Specifically, the total rank across all subspaces remains consistent with that the original weight matrix (Property 1 (1)), while each subspace individually preserves its intrinsic low-rank structure (Property (2)). These properties are eritical for maintaining original model abilities and enabling reliable adaptation across multiple subspaces.
5

---

3.3 Multi-Subspace-Baseee Incremental Update
In the previous subsection, we presented a representation of Wa based on subspace segmentation, as shown in (7). In this subsection, we propose a multi subsi-- based adaptation strategy. Rather than updating Wo directly, the proposed adaptation performs incremental updates for each (z))
We introduce an incremental update A (k) (with rank r2) for each (Zi) (A) in (6), and compute the updated weight block W (R) as
(k) W = W o ((Z))()) + AZ
w + W (k) AZ (k)
(8)
0
where the second equality follows from (5). Since the computation involves multiplication with the large matrix w calculating the gradient with respect (Z can be computationally expensive.
To address this, we decompose W (k) by the skinny SVD: w/ (k) U ((k) (v
W Let
Wo Wo Wo
R == rank (w) and A (k) = U RddRB, Then, the update becomes:
Wo
W (8) w (b) A (8)
(kk) Wo
(k) W.
) (Z(
(9)
Since rank (s(k) (v AT ()) I rk. we introduce two trainable matrices B ((k) E W. W.
R RR xrx and
((()) E R and reparameterize the k-th updated weight block as:
(( w!kh) (k) + (A) ((k) (k())
(10)
This formulation yields a low-rank update (i.e., A (k) (k)) ((k)) within each subspace. The overall framework is illustrated in Figure
To ensure that the initial value of (+) remains consistent with w we initialize (* in (10) as O. The matrix (A) E R Rx xrk is initialized either as an orthogonal matrix, as described Steps 7-8 of the Algorithm The complete initialization procedure of AdaMSS is presented in Algorithm 1. The parameter count and computational complexity analysis of the proposed method, along with comparisons to LoRA and PiSSA, are provided in the Appendices and E, respectively.
4 Multi-Subspace-Base Adaptive Budget Allocation
In this section, we introduce an adaptive multi-subspacc--asel budget allocation mechanism. By integrating this strategy with the Multi-Subsspce-Baa Based Adaptation framework proposed in the previous section, we derive the final method, AdaMSS.
Thanks to the multi-subspaac structure of the weights, during training by calculating the importance score [20] for each (*)) [B( (C(1))] R(Re+ne)xre directly. Following [20], the importance score s(t)() defined as
perform adaptive budget allocation
(6) H (H))) Tk (R na)
Σ 7()) (H(())) 7 (H 155):
(11)
1
where
(1)) (1H 81 (17 (1H (+)))) + (1 (1- (1)())
7 (H (cs) = 32U (1-1 (HH ((n) + - 32) ( (() 7 ((()(
(12)
= |HH 4l.
and denotes the training step. This importance score measures how consistently and significantly each (k) contributes to loss reduction during training.
At each update step, we rank all H (k) by their importance scores (t) [ (k)), retain the top-K, for training, and freeze the rest. The value of K2 is gradually reduced from K to Kearget following
smooth cubic decay schedule (i.e., with exponent p 3) over training steps. The adaptive budget
allocation mechanism is stopped when the number of trainable H (k) for k
KK falls below

---

A T'Triinall Parameters
x100*
Trainable Parameters
0
2/100
Trainabl Paramneters
StanfordCars
FGVC
CIFARIOO
500
1000
200
400
800
5000
10000
Training Steps
Training
Training Steps
(a)
(b)
(c)
Figure 4: The adaptive change in the number of trainable parameters of AdaMSS during the training process.
Ktarget- Figure presents examples the StanfordCars,, FGVC', and CIFAR100' tasks, illustrating the proposed adaptive budget allocation strategy, which gradually reduces the number of trainable parameters during the training process.
Analysis of AdaMSS
In the previous section, we introduced AdaMSS that is based the following reparametriization:
(13) (R0] Rd B diag B ( ( ,B(K) ER TM, ((K)
W = W ABC,
where A (A C' diag (1)), ( k 1,2, K.
RE Txxn (A
(k) and rank(C(k)) S Tk for
We establish the upper bound of expected loss by AdaMSS Lemma 1 and Theorem 1.
As our discussions given the Appendix B.2, the upbound of the Gaussian complexity for LoRA and its single subspace variants (such AdaLoRA, PiSSA and LoRA-GA) is given by (()) RB W for |AW l2 B. Ek=1 rk rank(AW]) applying Cauchy-Schwarz K
m incquality and using Σ Ex=1 T2k = n,, we have Σk=1 TTRT S. v rank(AW)rz Therefore, AdaMSS attains lower Gaussian complexity bound when Bg S B, and can be expected to have stronger generalization capability than single subspace-base
tine-tuning methods, as indicated by Theorem 1. These results be further extended deep Lipschitz neural networks by applying Maurer's Gaussian complexity chain rule [21].
Lemma 1. [Gaussian complexity of the AdaMSS for shallow Lipschitz neural network] For the
class of spectrally bounded shallow Lipschitz network
FAdaMSS = =[S 0 ())
W Wo ABC,
( ( = I, ( I B&),
(14)
the Gaussian complexity of this function class is upper bounded as follows:
Gs(FAdaMss) S c())
TkTTk
m
where -EK nly ns denotes the width of the weight matrix ((k), L()) is Lipschitz constant for ||æe|| S R.
function 1), X [r] ......,...... Th E Rdxm for the samples {m and max;=
Proof. The corresponding proof is given in the Appendix B.1.
Theorem Let g(-) be ((g)-Lipschitz loss function from (fw(x),)) to [0, 11, where fw FAdaMSS and (ær, y) E X V, C Rd and Y are feature space and output space, respectively. For any 8 > O, the following holds with probability at least for randomly chosen i.i.d. samples V:)
m m Σ ((f(())))) + (((gT((())
K Σ k=1
9 log 2
(x)
RB
TRTLk m
2m
"https://huggiggfee htt.:///w....tttttttee//ailiaaoa Fatiia.
7

---

Proof. Combining Lemma I with Vector-valued Gaussian complexity Generalization Bound Theorem [22], we obtain the generalization bound for AdaMSS.
Experiments
In this section, we present comparative experiments to evaluate the effectiveness of AdaMSS across tasks in image classification (IC), natural language understanding (NLU), and natural language generation (NLG). To better understand the contribution of each component, define variant of AdaMSS without the proposed adaptive budget allocation mechanism, referred to as AdaMSS base. for clarity of comparison. We report the average number of trainable parameters per epoch in our experimental results, as the number of trainable parameters in AdaMSS is adaptively reduced during the training. Since our focus is bow-rank-based adaptation, we compare AdaMSS not only with baseline methods such as full fine-tuning (FF) and linear probing (LP), which only tune the classification head, but also with several low-rank-based PEFT methods, including LoRA []],
DyLoRA [23], A [3], PiSSA [2]. LORETTA [10], and LoRA-PRO [6]. For each method, we prioritize using hyperparameters (e.g.. rank r) recommended by the authors. For a fair comparison. we replicate the experimental setups in [1. 2, 24]. A complete list of all hyperparameters and settings provided in the Appendix H. Ablation studies are given in Appendix G. All experiments were performed with Python version 3. The best results are highlighted in bold, and the top three results are underlined.
6.1 Image Classification
Table The performance of different fine-tuning methods on various image classification datasets for the ViT-Bass model and ViT-Large model, averaged over 5 random seeds, where the hyperpa-
rameter rąK in AdaMSS base and AdaMSS corresponds to the assumed rank of AZ(A) and is set the
same value for all k.
Model
Method
Tainnbbee
UifondPPess
Stanforot ars
CIEARIO
EuroSAI
Accuracy (S)
FOVL
RESISC
CIFARTON
Avg.
LP FF
90.28.0.4
25.76.0.1
96.41an..
88.72a6.
17.64104
74.22
$4.28
68.15
85.M
95.14gu44
79.78
98.92
99055
$4.84
96.13_..1
22.38
LaRA Er
16
SHIK
98.19.64
45.39.0.4
98.78 10.3
9844
25.16102
92:00 6.2
920224.
(e 8)
113K
93.84
78.41
98.4
0867
$1,56
93.81
93.3
86.00
PISSA (F
35K
9383
60.29.5.
98.7
9847
29.81.02
92.87 6.2
91.9
8032
VLT-Bas
LORA-PRO
313K
94.03
72.12
98.77
98.65.6.
43.99
93.66
22.54e...
84.74
WeGeFT
49K
92.71
76.18
98 98.46.03
96.226.
51.8200
90000
91.46.0.1
85.51
LoRFTTA (r
57K
93. gm
74.15
98. g1
98.67
48 8
91. 16.
91.87g1
85.57
AdaMSS
125K
94.02.0
81.55
98.8
98,72008
56,95104
94.18
92.13
85,05
94.33.03
80.4d-0.3
98. 49
54.45.0a
04.01
91.91
AdaMSS 122
42K
139
18.7
18.4
33.2
13.0210.0
91.90...
86.99
LF
91.1 9l.llana
37.9
07.78
9261a01
2182
R100
81.28
72.91
HF
30
88.90
19
9904
96.43
137M
94.22..3
99.13.00
98.316.1
42.32
94.71
PISSA (p 8)
34
9879
59.8120
94.99
92.42
PiSSA (r
147K
93
83.04
g4
98.7
56. 72
94.64-03
91. 91.25
88.48
ViT-Large
LRO
835K
94 16.3
83.57
99.30 ipa
9881
57.71
95.12 6.1
91 01.58
88.04
W
2011
94.19.03
63.96.01
98.3
60.82.0.
94.19
92.69
89.12
LoRETTA (e-
132K
78224
6844
88 80a
98.0
58.4408
94.53
98.28
82.86
AdaMSS
3)
483K
94.74
85 85
99 llane
98 mm
65. 65.30
92.12
91.5 91.51
90.33
85.4455
90
98
64.31
178K
94.58.03
83.1
19.08
98.51001
39,27.04
94.68 6.3
93.43
We evaluate all methods on IC using the widely adopted Vision Transformer (ViT) [12], prevalent foundation model in computer vision, across seven public datasets: OxfordPets*. StanfordCars CIFAR10', EuroSATS, FGVC*, RESISC45", and CIFAR100' The number of training epochs is set as 10. As shown in Table 1, AdaMSS achieves higher average accuracy than other PEFT methods, including LP. LORA, PiSSA, LORA-PRO, and LORETTA, and performs comparable accuracy with FF on the seven image classification datasets while using significantly fewer trainable parameters. More specifically, with the ViT-Base model, AdaMSS (59K) achieves average accuracy comparable to PiSSA (r = 8), using only 19.5% of its tranable parameters, and achieves 6.5% higher accuracy than PiSSA (r 1) at a similar parameter budget. This is because that AdaMSS's multi-subspaace design allows it to capture richer features with fewer parameters. On the ViT-large model, AdaMSS achieves 4.7% higher average accuracy than LoRA, while requiring only with 15.4% of the trainable
8

---

parameters, demonstrating AdaMSS's strong parameter efficiency and transfer performance in image classification tasks.
6.2 Natural Language Understanding
Table 2: The performance of different fine-tuning methods six datasets of the GLUE benchmark
for the RoBERTa- model and RoBgRTa-Large model, averaged over 5 random seeds.
Model
Method
# Trainable Purameters
SST-2
MRPC
Apc.
CoLA MCC
QNLI
Apc.
RTE
STS-B PCC
Avg.
FF
125M
94.8
90.2
63.6
928
78.7
91.2
85.2
LoRA
0.3M
95.1
89.7
63.4
93.3
78.4.a
91.5
85.2
AdaLoRA
0.3M
94.5-n.2
88.7+0.5
62.0-a.
93.1=02
81.0-a
90.5
85.0
DyLoRA
0.3M
94.3 __0.5
19.5
61.1_oa
92.2.0.
78.7.07
10.1
84.5
PISSA
93.9-a.1
89.3
62.1-2.9
91.3
77.3-11
90.5
LoRA-PRO
0.3M
94.2=0.5
90.1
64.3+0.72
92.0e0.2
80.2-1.
90.9
85.3
LoRA (r
0055M
93.7-0.6
89.2a0.8
62.3_s.
90.620.4
29.5.04
80.8.000
PISSA (r= 1)
0.05MM
93.3-n.2
89.3
62.6-1.4
90.6e0.4
74,9-1.2
90.0+0.3
83.4
LoRETTA
0.57M
94.6-0.a
88.3
61.8-
92.7
75.1
90.5
83.8
WeGeFT
94. 14.5
89.5
63.5
91.220.4
78.6114
90.5g0
AdaMSS (fu
D.02MM
94.6
89.2.1..
64.3
92.4.0.
77.2,007
90.610.1
84.7
AdaMSS
0.032M
94.6_0.a
88 88.8
64.5
77.3g0
90.410
84.7
90.9
92.4
LoRA
0.8
96.2-n.s
90.2+1.0
68.2-1.
94.8
85.2-1.1
92,3+0
87.8
PISSA (F 8)
0.8M
95.5-mm2
86.9.2.
61.
91.1
56.8...2
91.8104
80.7
RoBERTs- Large
LeRA-PRO
0&M
95.9-n.2
90.9
66.7-2.0
93.0en.s
60.5-115.
92.0++.1
83.2
LoRA (r
0.147M
95.7=0.4
38.
62.2=2.4
98.920.2
82.2
78.2
83.4
PISSA (p=
95.2_0.a
84.9
93420
65.9.na
91.320
LoRETTA
0.132M
96.2
90.5 10.4
69.5....
94.1 10.
$3.0
92.0
82.6
WeGeFT
0.66M
95.0_0.8
75.7
64.0_..
93.720s
$3.211
A1
78.9
AdaMSS (FL 1)
0.097M
96.3 0.2
90.510.3
68.0 n.9
94.610.
87.3 1.0
92.01.0
88.
AdaMSS -rr=)) 1)
0.045M
96.1
90.3
67.2.
94.5
87.1-21
91.9.00
$7.9
We evaluate all methods on the General Language Understanding Evaluation (GLUE) benchmark [25] using the robustly optimized BERT models, i.e., ROBERTa- and RoBERTa-Large [15], for the evaluation. We evaluate the performance of the fine-tuned models using three key metrics: Matthew's correlation coefficient (MCC) for CoLA, Pearson correlation coefficient (PCC) for STS-B, and accuracy (Acc.) for all other tasks. For all methods, the maximum number of training epochs is set to 100 and select the best epoch for each run. Table 2 presents the results of all methods. As shown in the Table 2, on RoBERTa-Base -Base AdaMSS achieves average accuracy comparable to other low-rank-basee PEFT methods while using the fewest trainable parameters. For RoBERTa Large, AdaMss outperforms PiSSA, LoRA-PRO, and LoRETTA by around 5% in average accuracy while using fewer parameters, and achieves performance comparable both LoRA and FF.
6.3 Natural Language Generation
In this subsection, we compare AdaMSS with other methods using range of models, including LLaMA 2-7B [14], Mistral-77 [26], and Gemma-7B [27], on natural language generation (NLG) tasks. We adopt the fixed value K 10 for AdaMSS. All comparison results accuracy are summarized in Table As the results demonstrate, AdaMSS and AdaMSS; pase consistently outperforms existing PEFT methods in most cases, while using only small number of trainable parameters. Notably, AdaMSS achieves oftte-of-thh-art performance on both GSM8K [13], MATH [28] for LLaMA 2-7B and Gemma-7B. with clear accuracy margin over all other baselines and using less than 1% of the parameters required for Full FF. For Mistral-7B, AdaMSS remains highly competitive and yields the best result on MATH using only 4M trainable parameters.
Table further compares the initialization cost of various PEFT methods on LLaMA 2-7B. As shown in the Table 4, AdaMSS incurs moderate setup cost, significantly lower than LoRETTA but higher than the PiSSA and LoRA-PRO. This reflects the structural complexity both AdaMSS and LORETTA, which aim construct compact representations of network weights: AdaMSS through multi-subspace segmentation, and LoRETTA via tensor decomposition.
7 Conclusions and Future Works
In this work, inspired by our observation of the multi-subspace structure in network weights, we proposed AdaMSS, novel PEFT approach, to address the limitations of single-subspace methods that

---

Table 3: Comparing different methods NLG tasks. PEFT methods marked with denote tuning of all major projection modules, including Aproj Kproj, Yprojs Oproj. UPproj. dowNproj. and gete whereas the default setting only updates Qprojs and Yproj-
Model
Method
Trainable Parameters
GSMBK
MATH
Full FT
673M
49,05
7.22
PSSSA (r
19M
44.11
5.84
LLaMA 2-78
PISSA Ir = 8)
4M
36.39
5.35
LoRETTA
0.3M
37.86
4.6
AdaMSS luse F2
4M
$1.10
7.57
AdaMSS (72
4M
50.80
7.22
AdaMSS 02
0.M
44.41
6.05
Foll FT
7242M
67.02
LoRA"
168M
67.70
19:68
PISSAT (r=8
20.40
LoRA-PRO" (r
=
20M
69.59
19.17
Mistral-78
PISSA
3M
64.26
16.87
LoRETTA (r
0.3M
62.6
15.66
AdaMSS
4M
70.71
20.44
AdaMSS'
2M
70.74
19:47
AdaMSS
U.5M
65.43
17.74
Full FT
8538M
71.34
22.74
LoRA'
200M
74.90
31.28
PISSA (r
25M
75.48
29.59
LoRA-PRO"
25M
75.90
29.25
Genma-
PISSA Ir
3M
71.52
27.53
LoRETTA (r
0.2M
70.23
26.28
AdaMSS bus (7%
6M
75.33
29.73
AdaMSS Ir =
4M
76.41
28.64
AdaMSS (7) 3)
Table 4: Comparing Initialization Costs of Different PEFT Methods LLama 2-7B.
Method
PiSSA LoRA-Pro
LORETTA
AdaMSS
Avg. Time (s)
2.278
2.006
12,878
8.777
often struggle with the dilemma between limited expressiveness and parameter efficiency. Compared to low-rank-basee fine-tuning methods, AdaMSS enables more expressive and adaptive parameter updates by leveraging subspace segmentation, thereby achieving both parameter efficiency and strong generalization capability. Comprehensive theoretical analysis and extensive empirical evaluations demonstrate the advantages of AdaMSS. In future work, we aim to extend the multi-subspace perspective beyond fine-tuning to broader areas such network compression and pruning. We believe that subspace segmentation offers promising direction for learning compact, disentangled, and efficient representations for network weights, which could benefit model compression and pruning deep neural networks.
Acknowledgements
Zhouchen Lin was supported by National Key R&D Program of China (2022ZD0160300) the NSF China (No. 62276004) and the State Key Laboratory of General Artificial Intelligence. Yankai Cao acknowledges funding from discovery program of the Natural Science and Engineering Research Council of Canada (RGPIN-0019004999) and New Frontiers in Research Fund The authors also gratefully acknowledge the computing resources and services provided by Digital Research Alliance of Canada www.alliancecan.coo and Advanced Research Computing at the University of British Columbia.
References
[1] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: low-rank adaptation of large language models. arXiv preprint 1rXiv:2/06. (00.0.5..000085, 2021.
[2] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: principal singular values and singular vectors adaptation of large language models. In Neural Information Processing Systems, 2024.
10

---

[3] Qingru Zhang. Minshuo Chen. Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Reprexentations, 2023.
[4] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized lmss. Neural Information Processing Systems, 36, 2024.
[5] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: ga: Low-rank adaptation with gradient approximation. In Neural Information Processing Systems, 2024.
[6] Zhengbo Wang. Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? 2025.
[7] Mahdi Nikdan, Soroush Tabesh, Elvir Crnčević, and Dan Alistarh. Rosa: accurate parameter-efficient fine-tuning via robust adaptation. In International Conference on Machine Learning, pages 38187-38206, 2024.
[8] Shubhankar Borse, Shreya Kadambi, Nilesh Pandey, Kartikeya Bhardwaj. Viswanath Ganapathy, Sweta Priyadarshi. Risheek Garrepalli, Rafael Esteves, Munawar Hayat, and Fatih Porikli. Foura: Fourier low-rank adaptation. In Neural Information Processing Systems, 2024.
[9] Yang Li, Shaobo Han, and Shihao Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks. In Neural Information P Processing Systems, 2024.
[10] Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor-trac train adaptation for satra-low-prrmette fine-tuning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3161-3176, 2024.
[11] Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, and Ivan Oseledets. Lotr: low tensor rank weight adaptation. arXiv preprint 2rXiv:2402.01376 2024.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: transformers for image recognition scale. arXiv preprint 201.1:2010.11929, 2020.
[13] Karl Cobbe, Vineet Kosaraju. Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[14] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Manie-Anne Lachaux, Thibaut I Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molyhog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Fric Michael Smith, Ranjan Subramanian. Xiaoqing Ellen Tan, Binh Tang, Ross Taylor. Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models, 2023.
[15] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.1169 2019.
[16] Guangean Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):171-184, 2012.
[17] Pan Zhou, Canyi Lu, and Jiashi Feng. Tensor low-rank representation for data recovery and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020.
[18] Shiqiang Du, Baokai Liu, Guangrong Shan, Yuqing Shi, and Weilan Wang. Enhanced tensor low-rank representation for clustering and denoising. Knowledge-Baud Based Systems, 243:108468. 2022.
[19] Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao. Robust latent low rank representation for subspace clustering. Neurocomputing, 145:369-33 0733699333360 2014.

---

[20] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin. Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International conference on machine learning, pages 26809-26823. PMLR, 2022.
[21] Andreas Maurer. A chain rule for the expected suprema of gaussian processes. Theoretical Computer Science, 650:109-1221 1229-12.22000 2016.
[22] Andrea Pinto, Akshay Rangamani, and Tomaso Poggio. On generalization bounds for neural networks with low rank layers. arXiv preprint arXiv:2411.13755 2024.
[23] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: parameter efficient ing pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.
[24] Ziqi Gao, Qichao Wang. Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform. In International Conference on Machine Learning. 2024.
[25] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07661 2018.
[26] Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed, Mistral 7b, 2023.
[27] Gemma Team, Thomas Mesnard. Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology- arXiv preprint S4kiv:24033082900 2024.
[28] Longhui Yu. Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024.
[29] Roman Vershynin. High-dimensional probability. 2020.
[30] Jianbo Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888-905, 2000.
[31] Xiang Lisa Li and Percy Liang. Prefix-tuning: optimizing continuous prompts for generation. arXiv preprint axXiv:2101.001 2021.
[32] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang. and Jie Tang. P-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks, 2022.
[33] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Seaameter-efficien transfer learning for nlp, 2019.
[34] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: simple parameter-efficient fine-tuning for transformer- masked models, 2022.
[35] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang. Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Neural Information Processing Systems, 2024.
[36] Massimo Bini, Karsten Roth, Zeynep Akata, and Anna Khoreva. Ether: efficient finetuning of large-scale models with hyperplane reflections. In International Conference on Machine Learning, pages 4007-4026. 2024.
[37] Frank Lauren Hitchcock. The expression of tensor polyadic of products. Journal of Mathematical Physics, (6):164-189, 1927.
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly articulate the core contributions of the paper, including the proposed methodology, the scope of experiments, and key findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
13

---

Justification:
The authors mention several shortcomings and directions for improvement in the "Future Work" section, reflecting an awareness of the method's current boundaries.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All thcoretical results in the paper are supported by clearly stated assumptions. The proof' are included in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details of the experiments are given in the Appendix.
Guidelines:
14

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets employed in the experiments are publicly accessible and appropri- ately referenced. The code of the proposed method is included in the supplemental material, with sufficient instructions
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / wttpcc/cccpclcpuuiiceeeeeeoo
/eceee/mddSSubbiac iismiission aassonPollt onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
15

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details are given in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports experimental results with multiple independent runs, and the variance across these runs is provided to capture statistical variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The details are given in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/meuuipsscc// ctpppblcc/ehiiiicicu...........
Answer: [Yes]
The research fully conforms to the NeurIPS Code of Ethics. All data used are publicly available and properly cited, no human subjects or private user information were involved, and no actions were taken that could lead to harm or discrimination.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper presents foundational research. As such, it does not have an immediate societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML.)
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
17

---

Answer: [Yes]
Justification: The paper does not release any models or datasets that pose a high risk for misuse. All datasets used publicly available and widely adopted benchmarks, and new data is scraped or generated for release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and code packages used in the paper are publicly available and properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswitheeee thoooee com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper releases a new code and provides detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
18

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The study does not involve any crowdsourcing or human subjects. All data used are collected from publicly available sources generated synthetically.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve any experiments with human subjects crowd- sourcing. Therefore, IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were used in the development of the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https http://riip..cc//eeeeeeeeeeceeee /cCe////eeeeeeec//20055,,, for what should or should not be described.
19

---

A
The Proof of Property
Proof. From Z = [Vl...aa V Vol.1. 1:1. we know rank Zū) = rank W o).
Since
W (R)
w 2)) (6), it follows that rank rank(W
(M) ( s rank rank((Z))(())) Thus, we have
rank
rank (Znk((Z))(A))
rank rank((Zg)(()) [Z)](t) rank Z rank rank(W.))
k=1
k=1
On the other hand, we know that rank (W)) S Σx k
k= rank (W (K) Therefore, we can conclude
rank W (k)
rank((Z))())) rank(W Wo). o).
Since rank (W)))) must hold that
(k) S rank( (k)) for all and Ex=rank rank rank( = Σ (an)(
rank ran ) = rank raak((Zi)) (1))
B
The Proofs of the Results in Section 5
B.1 The Proof of Lemma
Proof. By Talagrand's 's contraction lemma [29]. we have
1
Gs(FAdaMss L())
E. sup W, XT),
(15)
m
w
where each entry in I E Buxn ie., [ follows standard Gaussian distribution.
Since
E (sup W, XT) KW XT)] + E, /sup (ABC, XT)
B.C
[sup (Bsup(BC. (ACC, XT),,
(16)
B.C
we only need to study the upper bound of (sup (ACC, XT)).
We divide into K blocks according {n i.e. = [r(), r
[(K] for (())
pmxax (k- 1,2, K). From the Cauchy-Schware inequality, we have
K
K
sup sup(ABC (ABC, = Σ B,C
(A
(k) (b)) (Kk) ((k) xI(()) <
B (k) (1)) |
B C())
k=
k=
Vra Be (
(17)
Using the fact that E| S 2 for any random variable Y, we obtain
E, [|X (k) [r] v E, [IXT(* = n
/n(xxT xx
(T( S Ry/num
(18)
Combining (17) and (18), obtain
E., W, XT = E., sup (ACC, XT) < Σ
Vr Br E. (1 Σ
Vr By m.m.
Pymm
w
B.C
k=1
1=1
(19)
20

---

Algorithm 2 Estimation of K and Subspace Segmentation [16]
Input: Vo. R, Ko, and 0.
Step 1: Construct the affinity matrix [F). = (LO'") 1st), where U is formed by [Vo].::R with
normalized rows:
Step 2: Estimate the number of subspaces K by K = max(Ko, n - int (E= (a,
[16), where (o.(L)] are the singular values of the Laplacian matrix I D
where D diag Σ Flyy.... EFFinn)). int(-) the function of the nearest integer, and
1,
i
fa(
Tog2 (1+
otherwise.
Step Construct an undirected graph by using the affinity matrix F; Step 4: Apply the NCut [30] to segment the vertices into K clusters;
Substituting (19) into (15) leads
K E, su, rx) < (()) RBs
Gs(FAdaMss) & (()) m
Tk
m
B.2 Gaussian Complexity Analysis for LoRA and Its Single Subspace Variants
By the Vector-valued Gaussian complexity Generalization Bound Theorem [22], we can directly derive the Gaussian complexity of LoRA, stated in Lemma 2.
Lemma 2. [Gaussian complexity of the LoRA for a shallow Lipschitz neural network] For the class of spectrally bounded shallow Lipschitz network
Fiora {Jw = ((xW) W Wo AW, |AW s B}
(20)
the Gaussian complexity of this function class is upper bounded as follows:
Gs(Fiora) < RB
r
(21)
where r rank( AW W), denotes the width of the weight matrix AW, [(Ø) is Lipschitz constant for function d. X [ ................................ T E pdxm for the samples {æ; ) and max 1,2,.. ,m ||>:|| < ft
Similarly, by [22], we can obtain the same upper bound in (21) for the Gaussian complexity of PiSSA and other single subspace variants under the same assumptions.
C Subspace Segmentation (Algorithm 2)
As Step 2 in Algorithm 2 incurs significant computational and memory costs, practical alternative for large models is to fix K = Ko for large models.
D Parameter Count
Table 5: Comparison of parameter counts among different low-rank-based methods for d = n: represents the rank of the incremental update used in LoRA and PiSSA.
LoRA
PiSSA
AdaMSS
# Trainable Parameters
2rn
2rn
Σ
Σ (runa + ru Rx.)
Given (A) = w (k) + A (b) (1) (A) for 1,2, K, where (k) E R Ra xrx and ((k) e
X are trainable, the total trainable parameter count in AdaMSS is given by E (ran&
21

---

which is always less than 2( 2 k=1 Ta m n& for d = n. Compared to the trainable parameter count in LoRA and PiSSA, shown in Table 5, AdaMSS introduces significantly fewer parameters for Ek=1 Tke owing to the fact that ry S Rg I nu and Ek=1 Tkk
E Computation Complexity
By the proposed adaptation, we have Y X + ABC), where E Rach sizexn is the input. In table 6, compare the computational complexity of gradient computation for LoRA, PiSSA, and AdaMSS. The table shows that AdaMSS's computational complexity is on the same order as LoRA and PiSSA when E 1i=1 =.
Table 6: Comparison of LoRA, PiSSA and AdaMSS for d = n. where underline denotes the trainable parameters, and A E Ruxr and B E Rxx for LoRA and PiSSA.
LoRA
PiSSA
AdaMSS
Forward
Y
XW
XAB
Y XW,
XAB
Y XW, 0 XABC
res
Gradient
0C = X
0c
B
ac
XT
oc
B
AB = (XA) 0C OY
ac
CT
0A
OY
0A
OY
8C A
X
BC
8C
BC
BC B (X
ac
8B
DY
JB
DY
OC
OY
Cost
Orr?² + bach size n²)
Orn + bach size x n²)
O O K
i== runn 2 bach size
n²
F
Comparison of Optimizer Memory Efficiency and Training Speed
Since method uses fewer trainable parameters, it exhibits superior memory efficiency, shown in Tables and 8.
Table 7: Optimizer memory consumption (MB) of different PEFT methods ViT-Large using fp32 precision.
Method
LoRA = 16)
PiSSA (r 8)
LoRETTA (r 5)
MdMMSSpas (rk =
Memory (MB)
18.84
9.6
1.584
2.136
Table 8: Optimizer memory consumption (MB) of different PEFT methods RoBERTa-Larg using fp32 precision.
Method
LoRA
PiSSA =) 8)
LoRETTA 5)
AdaMSS (rx 1)
Memory (MB)
9.6
9.6
1.58
0.54
Regarding training speed, as discussed in the main text, the computational complexity of our gradient updates remains on the same order as LoRA and PiSSA when r = Σ Tke where r is the rank hyperparameter used in LoRA and PiSSA. However, in practice, the multi-subspace adaptive budget allocation in our method accelerates training by selectively updating only the most important subspaces (see Table 9). Moreover, this adaptive budget allocation offers greater flexibility in balancing training efficiency and model performance. Specifically, the number of trainable subspaces gradually reduced to target value Kurrget according exponent 3. A larger p results in faster reduction in the number of trainable subspaces. Table 10 provides an ablation study demonstrating the impact different p values on both training speed and final performance.
smooth decay schedule with decay
22

---

Table 9:
Average training time (in seconds) of different PEFT methods on RoBERTa-Large for natural language understanding task (STS-B).
Method
LoRA (r = 8) PiSSA (r = 8)
LoRETTA (r
AdaMSS (rx = 1, E. rk 2 10)
Avg. Time (s)
5325.77
5174.04
5441.47
4972.03
Table 10: Training time (in seconds) and performance (PCC) of AdaMSS with varying p on RoBgRTa-Large for natural language understanding task (STS-B).
10
15
Avg. Time (s) PCC
4913.63
4780.97
4667.35
91.6610.03
91.5210.02
91.35
G
Ablation Study
G.1 Comparison of Different Adaptive Budget Allocation Strategies
We compare our importance-score- based adaptive budget allocation strategy (described in the main text) with the following two alternative approaches:
Random adaptive budget allocation: At each training step, K, subspaces are randomly selected, and only the corresponding parameters updated.
f -nnorm-based adaptive budget allocation: At each training step, the fn-nrrm of parameters associated with each subspace is computed, and the top- K2 subspaces with the largest f-normms are selected for updating.
As shown in Table 11, both random and & f1-nomm-bat based adaptive allocation strategies lead to consider- able performance degradation. In contrast, our importance-sone weweee-cco-resseeeeeeeeeee allocation method achieves the best performance for all cases.
Table 11: Performance comparison of different adaptive budget allocation strategies on the ViT-Lerge model.
Methods Dataset
StanfordCars
CIFAR100
FGVC
peporrtaccesccre-bbaee adaptive budget allocation
85.2440.3
93.22+0.01
64.31
Random adaptive budget allocation
83.6340.30
93.1140.10
59. 59.3640.72
---orrm-base based adaptive budget allocation
79.89+0.41
92.00+0.21
40.77
G.2 Performance of AdaMSS With Varying
This subsection explores how the performance of AdaMSS varies with different values of rx and Ktarget using ViT-Large, where Tı E (1, 3) and Ktarget E {100, 200, 300, 400,500) 500). Each configuration is evaluated the StanfordCars, FGVC datasets, and CIFAR100. All results presented in Figures 5. As shown the results, increasing т can lead to significant improvements in accuracy.
G.3 Sensitivity Analysis of and Ktarget in Adaptive Budget Allocation
We evaluate the sensitivity of AdaMSS to different values of p (100, 200, 300, 400) using the -Lrge, as shown in Tables 15-17. The results demonstrate that AdaMSS maintains robust performance across wide range of and Kaaget settings.
and Kurget
23

---

Table 12: Results of AdaMSS base under varying 7 and Ko values for StanfordCars.
Aoo
5
10
15
20
0.001
84.25
85.40.0.19
85.40.0.22
40.28
85.67 :0.42
0.01
84. 84.1240.24
85.20
85.31
85,41 10.13
85.50
0.05
84.77+0.16
85.02
85.23+0.28
85.59
85.70=0.21
0.10
84.58
84.93
85.60+0.26
85.3640.29
85,42.+0.23
0.15
84.72+0.39
85, 0..12
85.38+0.33
85.61
85.63+0.45
0.20
84.57 10.09
84.86 10.17
85.53
85.39 10.34
85.65 1..19
Table 13: Results of AdaMSS base under varying T and Ko values for CIFAR100.
</Ko
10
15
20
0.001
93.27+0.09
93.44
93.47+0.06
03.57+0.112
93.48 +0.15
0.01
93.38
10.13
93.50+0.10
93.53
93.57
0.05
93.4210.12
93.43 10.12
93.51.o.00
93.45 10.30
93.5610.00
0.10
93.40 10.12
93.47
10009
93.58 10.13
93.45
93.64 10.10
0.15
93.43
93.43
93.52.0.06
93.55
93.6010.11
0.20
93.43
93.54 10.05
93.4840.00
93.53
93.51 10.11
Table 14: Results of AdaMSS base under varying 7 and Ko values for FGVC.
Aoo
10
15
20
0.001
61.36
65.33
66. 10.53
66.62.10.7
66.98 10.73
0.01
61.90
64.58 10069
65.27 10.64
66.10 10.69
66.60
0.05
61.64
64.63
04.63 1.99
65.51 11.05
66.63 10.48
67.44_ 10.68
0.10
62.42+0.33
40.49
65,8% 65,88
66. +0.21
67. 67.1640.82
0.15
62.5640.73
64.90=1.00
65.72+
66.86+0.30
66.44
0.20
62.00
64.76
65.68 +0.93
66.65
66.8640.60
Table 15: Results of AdaMSS under varying and Karget values for StanfordCars.
p) Karget
100
200
300
400
500
84.78-0.32
85.0640.16
85.34
85.35
85. 85.16400.12
84.88
$5.11+0.11
85.21
85.2640.20
85.2 85.21
85.03_0.29
84.91
$5.21+0.a8
85,0640.20
85,02-10.18
84.69 =0.20
84.95+u0.13
85.03 20229
85.02+40.21
85.19 15.19
5
84.89
-0.27
84.67
85.04
85.16 85.1640.26
00.37
Table 16: Results of AdaMSS under varying P and Karget values for CIFAR100.
p) Karget
100
200
300
400
500
93.47-0.10
93.49+0.11
93.41
93.56+0.10
93.34
93.41
93.4440.07
93.54+0.08
93.42+0.13
93.47=0.07
93.42_0.14
93.38
93.56+0.09
93.46+0.14
93.55
93.25
93.53
93.40+0.10
93.4840.15
93.50
93.35
93.45-0011
93.37
93.41
93.41
Table 17: Results of AdaMSS under varying and Kinget values for FGVC.
p) argett
100
200
300
400
500
64.76
65.5644.60
64.98 40.81
65.73 64.90
64.55 20.73
64.38
65.14-40.89
65.1 65.11
64.99-10.7
63.73=o.31
64.16 64.1641.22
64.77
65.33
65.77
63.02 0.89
64.35 40.62
64.64
10.81
64.54
65.13 10.61
62.74 1.53
63.43
65. 65.12.4..83
64.28
65.32_1073

---

94
83.0
93.6
A00
60
Nec.
93.2
6
6
100
200
300
500
100
200
400
100
400
Karger
"iarger
(a) StanfordCars
(h) FGVC
(c) CIFAR100
Figure 5: The performance of AdaMSS with different values of T2 and K Krrrger
66
85.5
99.8
AC5.
84.5
Acc
62
83.5
-0-A0 oifi diTreet K.
A0 With cflerer
90.2
Astdlia WITh CEENNNE
03
10
2
10
K
(a) StanfordCars
(b) FGVC
(c)
CIFARIO0
Figure 6: The performance of AdaMSS with different values of K for rk 3.
G.4 Sensitivity Analysis the Subspace Number K
G.4.1 K estimated by Algorithm 2 for given and Ko
In Algorithm 2, the threshold is used to detect near-zero singular values in the normalized Laplacian matrix and is set to small value. The parameter Ko serves a lower bound on the estimated number of subspaces. We evaluate different settings of & (0.001, 0.01, 0.05, 0.10, 0.15, 0.20] and Ko E 5, 10, 15, 20} using ViT-Large with AdaMSS as reported in Tables 12-14. The results demonstrate that AdaMSSpase maintains robust performance across all three datasets (StanfordCars, CIFAR100, and FGVC) when Ko 2 10, regardless of the specific choice of
G.4.2 K given manually.
This subsection investigate the effect of varying on the performance of AdaMSS base at F comparing estimated K (shown as red dashed lines) with given values of K. As shown in Figures increasing can also lead to significant improvements accuracy. Notably, the improvement associated with increasing K highlights the advantage of the multi-subspace approach.
G.5 Initialization of AdaMSS
In this subsection, we compare two initialization strategies for B (k), the proposed orthogonal initialization (as described in Steps 6-8 of Algorithm 1) and a random initialization following the LoRA strategy. While the latter avoids the computations in Steps 6-7 and simplifies the initialization procedure, we do not recommend this approach for AdaMSS. This is because randomly initializing
((k) disrupts the preservation of structural information contained W (k) As reparameterization of
(8), the decomposition A (k) (kk (10) is designed to remain as close as possible to the subspace
spanned by W (k)
which is crucial for AdaMSS a method derived from a subspace segmentation
perspective.
Figure analyzes the impact the two initialization strategies on ViT-Large using the StanfordCars and FGVC datasets, under varying values of K and rk. The results show that orthogonal initialization
25

---

Orthogonal Intialization intalication
64
62
Abc.
Orthoggaa cotion (Estiaated
56
Ontrogonal Intaalcationn (K-10)
Randum iniisication
2
(a) FGVC
(b) FGVC
86
Unthogoraa Intillzaaoo
85
Acc.
82
Orthagonal Intialzzaion (Estmatedd
Orthogonal Intialzation (K-10)
Randim Initalzation
82
2
10
2
(c) StanfordCars
(d) StanfordCars
Figure 7: The performance AdaMSS with different initialization strategies.
consistently yields superior performance across all configurations compared to random initialization, highlighting the effectiveness of orthogonal initialization in preserving subspace structure.
H Experimental Setup and Hyper-parameter Configuration
For fair comparison, replicate the experimental setups in [1, 2, 24]:
The reported results are averaged over five random seeds. All models are fine-tuned by updating only the query and value projection matrices, while the classification head is updated for every method.
IC and NLU:
NLG: Following [2], we use the datasets listed in Table 21, and all experiments are conducted 100K-example subsets and trained for single epoch. The results are averaged over three runs.
A detailed list of all hyper-parameters and settings can be found in the Tables 18-20.
In our experiments, we set 100, p and r 0.01. If not specified, Ko to 10.
Table 18: Hyper-Paramedt Configuration for AdaMSS base and AdaMSS on IC.
Model
Hyperparameter
OxfordPets
StanfordCars
CIFARIO
EuroSAT
FOVC
RESISC4S CIFARIO0
Optimizer
AdamW
Batch Size
Both
Epochs
Seeds
17.77.777.7777777777
Learning Rate
0.005
0.01
0.01
0.01
(101
0.01
0.01
Learning Rate (Head)
0.005
0.005
0.005
0.000S
0.005
0.005
0.005
ViT-Bare Base
Weight Decay
0.0005
0.0
0.05
0.05
0.0005
0.0005
0.05
Learning Rate
0.001
0.01
0.01
0.01
0.01
0.01
0.01
ViT-Large
Learning Rate (Head)
0.0005
0.0005
0.0005
Weight Decay
0.0005
0.1
0.1
0,01
0.0005
0.1
0.05
26

---

Table 19: Hyper-Parament Configuration for AdaMSS base and AdaMSS NLU.
Model
Hyperparameter
SST-
MRIC
CoLA
QNLI
RIE
STS-B
Optimizer
Adam
Batch Size
32
Both
Epochs
100
Seeds
10,11111
1222222,33333,,4444444
Learning Rate
0.001
0.01
0.001
0.001
0.0005
0.001
Learning Rate (Head)
0.005
0,0005
0.005
01.055
0.005
0.005
RoBERTa-Base Base
Weight Decay
0.0005
0,0
0.005
0.005
0.005
0.005
Learning Rate
0.0005
0.005
0.001
Learning Rate (Head)
0.0005
5c-05
0.0005
0.05
0.005
0.0005
RoBERTa-Larg.
Weight Decay
0.0
0.005
0.1
0.005
0.5
0.0005
Table 20: Plyper-Paramees Configuration for AdaMSS and AdaMSS on NL.G.
Model
Hynerparameter
GSMSK MATH
Optimizer
Adam
Both
Batch Size
Epochs
Learning Rate
6c-4
Genna
Learnnng Rate
2e-4
More Evidence for Multiple Subspaces Structure in Pretrained Network Weights
In this section, we provide additional numerical evidence for the presence of multi-subspace structures in pretrained models across layers and tasks.
Beyond the approximate block-diagonal patterns shown in Figure 2, we also examine the distribution of weight column vectors through the singular values of a Laplacian matrix constructed from the principal components. Tables 22-25 report the singular value distributions of these Laplacian matrices, computed from the principal components of the query weight matrices at each layer of several pretrained models, including ViT-Large, LLaMA 2-7B, Mistral-7B, and Gemma-7B. The results show that, for most layers, the dominant singular values remain substantially large, while the trailing ones are close to zero (e.g., helow 0.01).
The number of near-zero singular values can be interpreted estimate of the number of disjoint subspaces in the weight space. This offers further numerical evidence for the existence multi- subspace structures within the principal components of pretrained weights.
Related Works
J.1 Parameter-Efficien Fine-Tuning (PEFT)
In recent years, a variety of PEFT methods have been proposed to adapt large pre- pre-trained models downstream tasks while minimizing the number of trainable parameters. Broadly, current PEFT methods include prefix-tuning [31, 32), adapter-based methods [33], sparse fine-tuning [34, 24], orthogonal fine-tuning [35, 36). and low-rank-based adaptation [1, 2, 4, 5, 6, 7, 9]. Specifically, prefix-tuning methods introduce small task-specific trainable vectors, known as prefixes, while keeping the original model parameters frozen [31, 32]. In contrast, adapter-based methods [33] insert small trainable modules within each layer of the model, allowing efficient task adaptation by updating only these modules. Meanwhile, sparse fine-tuning, such as BitFit [34] and FourierFT [24], update only small subset of parameters in bias terms or transformed weights. On another front, orthogonal fine-tuning introduces a learnable orthogonal transformation applied the pre-trained traitre weights, with the goal of preserving the model's 's original capabilities during adaptation [35, 36). In addition to the above, low-rank-based adaptation methods, for example LoRA [1], PiSSA [2], LoRA-GA [5], Foura
Table 21: The experimental setup
Fine-tuned on MetaMathQA [28]
Evaluated Datasets
GSM8K [13], MATH [28]
27

---

Table 22: Singular value distribution of the Laplacian matrices constructed from the principal components of the query weight matrices at each layer of the pretrained ViT-Large model.
Layer
01
0101
2201
0301
4011
Su1
0601
0701
0801
0901
01001
0u024
1
1.00
0.87
0.78
0.59
0.38
0.21
0.09
0.02
0.00
0.00
0.00
0.00
3
1.00
0.83
0.78
0.72
0.65
0.57
0.47
0.37
0.24
0.10
0.00
0.00
5
1.00
0.83
0.81
0.79
0.77
0.74
0.70
0.64
0.54
0.38
0.05
0.00
7
1.00
0.82
0.81
0.79
0.77
0.74
0.71
0.67
0.60
0.47
0.01
0.00
9
1.00
0.82
0.81
0.79
0.78
0.76
0.74
0.71
0.67
0.56
0.00
0.00
11
1.00
0.83
0.81
0.80
0.79
0.77
0.75
0.73
0.68
0.55
0.00
0.00
13
1.00
0.83
0.81
0.80
0.78
0.76
0.74
0.71
0.66
0.52
0.00
0.00
15
0.99
0.81
0.79
0.77
0.76
0.73
0.71
0.67
0.63
0.53
0.00
0.00
17
1.00
0.82
0.80
0.79
0.77
0.76
0.74
0.71
0.68
0.60
0.00
0.00
19
1.00
0.82
0.81
0.80
0.78
0.77
0.76
0.74
0.71
0.66
0.00
0.00
21 1.00
0.82
0.81
0.80
0.80
0.79
0.78
0.77
0.75
0.72
0.00
0.00
23
1.00
0.83
0.82
0.81
0.80
0.80
0.79
0.78
0.77
0.75
0.12
0.00
Table 23: Singular value distribution of the Laplacian matrices constructed from the principal
components of the query weight matrices at each layer of the pretrained LLaMA 2-7B model.
Layer
d1
J501
Jo00
11501
d2001
Faom
03501
07001
d'0022
Cassa
Carst
daoos
1.00
0.31
0.19
0.13
0.08
0.04
0.02
0.00
0.00
0.00
0.00
0.00
0.00
0.00
3 1.00
0.51
0.48
0.43
0.36
0.27
0.17
0.08
0.01
0.00
0.00
0.00
0.00
0.00
5
0.50
0.47
0.45
0.42
0.38
0.31
0.19
0.03
0.00
0.00
0.00
0.00
7
0.99
0.46
0.41
0.37
0.34
0.30
0.24
0.03
0.00
0.00
0.00
9
0.43
0.38
0.35
0.32
0.28
0.22
0.13
0.02
0.00
0.00
0.00
11
1,00
0.50
0.46
0.43
0.38
0.31
0.24
0.15
0.03
0.01
0.00
0.00
0.00
0.00
13
1,00
0.47
0.43
0.40
0.36
0.31
0.24
0.17
0.05
0.01
0.01
0.01
0.01
0.00
15
1.00
0.50
0.38
0.32
0.26
0.20
0.13
0.03
0.01
0.01
0.01
0.00
0.00
17
1.00
0.49
0.45
0.41
0.37
0.32
0,26
0.17
0.03
0.00
0.00
0.00
0.00
19
0.48
0.40
0.37
0.33
0.06
0.01
0.01
0.01
21
1,00
0.47
0.40
0.35
0.32
0.28
0.23
0.15
0.03
0.00
0.00
0.00
0.00
0.00
23
1.00
0.44
0.38
0.35
0.33
0.30
0.27
0.18
0.02
0.00
0.00
0.00
0.00
0.00
25
1.00
0.46
0.38
0.35
0.32
0.29
0.26
0.15
0.01
0.00
0.00
0.00
0.00
0.00
27
1.00
0.46
0.41
0.39
0.36
0.34
0.30
0.19
0.03
0.00
0.00
0.00
0.00
0.00
29
0.47
0.43
0.01
0.00
31
1.00
0.48
0.45
0.43
0.42
0.40
0.37
0.30
0.10
0.02
0.02
0.01
0.01
0.01
[8], and RoSA [7], approximate weight updates using low-rank matrices, significantly reducing the number of trainable parameters while maintaining performance. These techniques provide effective strategies for adapting large models new tasks with less computational overhead.
J.2 Now-Rank-Basee Adaptation Methods
Inspired by the success of LoRA [1], a large number of Low-rank-brst adaptation methods have emerged over the past three years [2, 3, 4, 5, 8, 9, 10, 11]. LoRA assumes fixed rank for all incremental matrices across different layers, thereby ignoring the diverse importance of different weight parameters. To address this limitation, AdaLoRA [3]
proposed to allocates parameter budgets by adaptively adjusting the rank during training. Following AdaLoRA, research attention has shifted toward improving the initialization strategies of A and B LoRA, opposed relying on random initialization. Representative works include PiSSA [2] and LoRA-GA(S), In addition, by incorporating both low-rankness and sparsity constraints, RoSA [7] enhances low-rank adaptation to enable more efficient parameter utilization. In another line of work. FouRA [8] extends LoRA into the frequency domain, yielding disentangled feature spaces that enable fine-grained control and editing. Motivated by the success of tensor decomposition in data compression [37], LoTR [11] and LoRETTA [10] leverage Tucker decomposition and Tensor-train decomposition, respectively, to achieve more compact representations of the low-rank update matrices.
28

---

Table 24: Singular value distribution of the Laplacian matrices constructed from the principal components of the query weight matrices at each layer of the pretrained Mistral-7B model.
Layer
doo1
1001
01501
02001
02501
60001
03501
40001
44002
04053
4p06
44996
1.00
0.29
0.15
0.09
0.05
0.02
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
3 1,00
0.49
0.46
0,43
0.41
0.38
0.28
0.16
0.03
0.00
0.00
0.00
0.00
0.00
5
1.00
0.48
0.44
0.40
0.36
0.32
0.25
0.17
0.06
0.01
0.01
0.01
0.01
0.00
7
1.00
0.47
0.43
0.39
0.35
0.31
0.23
0.14
0.04
0.01
0.01
0.00
0.00
0.00
9
0.48
0.41
0.38
0.34
0.19
0.06
0.01
0.01
0.00
0.00
11
0.47
0.42
0.37
0.32
0.27
0.11
0.00
0.00
0.00
13
0.99
0.42
0.33
0.27
0.21
0.16
0.11
0.06
0.01
0.00
0.00
0.00
0.00
0.00
15
1.00
0.45
0.38
0.33
0.27
0.22
0.16
0.11
0.03
0.01
0.01
0.01
0.00
0,00
17
1.00
0.48
0.41
0.35
0,30
0.24
0.19
0.12
0.03
0.01
0.01
0.01
0.01
0.01
19
1.00
0.40
0.32
0.27
0.23
0.18
0.14
0.02
0.00
0.00
0.00
0.00
0.47
0.41
0.36
0.32
0.27
0.04
0.01
23
0.46
0.39
0.36
0.33
0.00
0.00
0.00
25
1,00
0.44
0.40
0.37
0.34
0,31
0.25
0.18
0,06
0.01
0.01
0.00
0.00
0.00
27
1.00
0.47
0.43
0.41
0.38
0.34
0.30
0.22
0.07
0.01
0.01
0.01
0.01
0.01
29
1.0
0.45
0.41
0.38
0.35
0.32
0.26
0.17
0.03
0.00
0.00
0.00
0.00
0.00
31
0.46
0.42
0.39
0.36
0.32
0.26
0.16
0.02
0.00
0.00
0.00
0.00
0.00
Table 25: Singular value distribution of the Laplacian matrices constructed from the principal components of the query weight matrices at each layer of the pretrained Gemma 7B model.
Layer
1 1.00
0.50
0.46
0.44
0.41
0.37
0.29
0.15
0.00
0.00
0.00
0.00
0.00
0.00
3
1,00
0.49
0.46
0.45
0.43
0.41
0.39
0.32
0.12
0.01
0.01
0.01
0.01
0.00
5
1.00
0.48
0.45
0.43
0.41
0.39
0.36
0.26
0.03
0.00
0.00
0.00
0.00
0.00
0.49
0.45
0.44
0,42
0.08
0.00
0.00
0.00
0.50
0.47
0.44
0.41
0.37
0.33
0.07
0.00
0.00
0.00
II
1.00
0.48
0.45
0.43
0.42
0.39
0.35
0.25
0.05
0.00
0.00
0.00
0.00
0.00
13
1.00
0.50
0.47
0,44
0.42
0.37
0.29
0.20
0.07
0.00
0.00
0.00
0.00
0.00
15
1.00
0.49
0.44
0.38
0.29
0.18
0.10
0.04
0.00
0.00
0.00
0.00
0.00
0.00
17
1.00
0.50
0.45
0.41
0.36
0.30
0.23
0.04
0.01
0.00
0.00
0.00
0.00
0.52
0.38
0.30
0.01
0.00
21
0.99
0.48
0.43
0.39
0.34
0.27
0.19
0.11
0.01
0.00
0.00
0.00
0.00
0.00
23
1.00
0.51
0.42
0.36
0.31
0.25
0.19
0.11
0.02
0.00
0.00
0.00
0.00
0.00
25
1.00
0.49
0.41
0.33
0.27
0.20
0.11
0.04
0.00
0.00
0.00
0.00
0.00
0.00
27
1.00
0.46
0.43
0.40
0.37
0.33
0.28
0.19
0.05
0.01
0.01
0.00
0.00
0.00
29


Reviews:
[{"review_id": "LmVCyoPoa3", "reviewer": "Reviewer_J6fY", "summary": "The paper introduces AdaMSS, a new parameter-efficient fine-tuning (PEFT) method for large language models. Rather than using fixed low-rank adapters (like LoRA), AdaMSS selects a set of low-rank subspaces adaptively with a learned budget controller. The method is supported by solid theoretical analysis and evaluated across NLP and vision tasks. It shows promise in reducing trainable parameters while maintaining performance.", "questions": "- How well can this method scales with long-context or multi-step reasoning tasks?\n- Can it be extended with positional tuning approaches like RoCoFT?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a novel idea and provides a strong theoretical foundation.\n- It provides an extensive experimental study to demonstrate the viability of the approach across multiple domains, including LLMs and vision transformers.\n- The approach achieves significant parameter efficiency while maintaining high performance comparable to SOTA methods. \n\nWeaknesses:\n\n- The paper does not include performance metrics such as GPU memory use or training speed, which are key for PEFT.\n- Some of the newer models, such as  LLaMA3, DeepSeek, or Gemini, are not included in the experimental study.", "was_revised": true, "final_justification": "The paper presents a novel idea with a good theoretical foundation. The authors responded and addressed my concerns. They plan to provide a revised version with the additional results and discussion.", "rebuttal": "We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised. \n\n---\n\n**R1. Performance metrics such as GPU memory or training speed**\n\n\nThank you very much for pointing this out. In the revised version, we have added a comparison of different PEFT methods in terms of both memory usage and training time. Since our method use a smaller number of trainable parameters, it demonstrates superior memory efficiency, as shown in **Tables 1 and 2**. \n\nRegarding training speed, as discussed in the paper, the computational complexity of our gradient updates is of the same order as that of  LoRA and PiSSA when $r = \\sum_{k=1}^{K} r_k$, where $r$ denotes the hyperparameter used in **LoRA** and **PiSSA**. However, in practice, the multi-subspace-based adaptive budget allocation allows for faster training by selectively updating only the most important subspaces (see **Tables 3 and 4**). \n\n\nBesides, the proposed   multi-subspace-based adaptive budget allocation provides greater flexibility in balancing training time and performance. In our paper, the number of trainable subspaces is gradually reduced to a target value $K_{target}$ following a smooth  decay schedule  with decay exponent $\\rho=3$. A larger $\\rho$ leads to a faster reduction in the number of trainable subspaces. **Table 5** presents an ablation study demonstrating how different choices of $\\rho$ affect training speed and final performance.\n  \n\n **Table 1: Optimizer memory consumption (MB) of different PEFT methods on `ViT-Large` using fp32 precision.**\n | Method        | LoRA $(r=16)$  | PiSSA $(r=8)$ | LoRETTA $(r=5)$ |AdaMSS$_{base}$ $(r_k=1)$|\n |---------------|------- |------- |--------|------|\n |Memory (MB) |  18.84 |  9.6 |  1.584  |  2.136 |\n\n\n**Table 2: Optimizer memory consumption (MB) of different PEFT methods on `Roberta-Large` using fp32 precision.**\n| Method        | LoRA $(r=8)$  | PiSSA $(r=8)$ | LoRETTA $(r=5)$ |AdaMSS $(r_k=1)$|\n|---------------|------- |------- |--------|------|\n|Memory (MB) | 9.6 | 9.6 | 1.58  | 0.54 |\n\n \n**Table 3: Average training time (in seconds) of different PEFT methods on `ViT-Large`.**\n| Datasets\\Methods       | LoRA  | PiSSA | LoRETTA |AdaMSS $(r_k=3, \\sum_{k=1}^{K} r_k\\geq 30)$|\n|---------------|------- |------- |--------|------| \n| FGVC  | 1207.7669| 1206.7017  |  1057.7328  | 1074.9871 | \n\n\n\n**Table 4: Average training time (in seconds) of different PEFT methods on  `Roberta-Large` for  natural language understanding task (STS-B).**\n| Method        | LoRA $(r=8)$ | PiSSA $(r=8)$ | LoRETTA $(r=5)$ |AdaMSS $(r_k=1, \\sum_{k=1}^{K} r_k\\geq 10)$ |\n|---------------|------- |------- |--------|------|\n| Avg. Time (s) |5325.77 | 5174.04 | 5441.47  | 4972.03|\n\n**Table 5: Training time (in seconds) and performance (PCC) of AdaMSS with varying $\\rho$ on  `Roberta-Large` for natural language understanding task (STS-B).** \n|   $\\rho$    | 5  | 10 | 15 |  \n|---------------|------- |------- |--------|  \n| Avg. Time (s) |4913.63 | 4780.97 | 4667.35 |  \n|  PCC   | 91.66±0.03 | 91.52±0.02 | 91.35 ± 0.03 |  \n\n\n---\n**R2. Experimental study on newer models (LLaMA3, DeepSeek, Gemini)**\n\nThank you  so much for the valuable comments. Our current experiments cover a range of models, including **ViT (Base/Large)**, **RoBERTa (Base/Large)**, **LLaMA 2-7B**, **Mistral-7B**, and **Gemma-7B**. Among them, **LLaMA 2-7B** and **Gemma-7B** were released in *2023*, while **Mistral-7B** was released in *2024*. \n\nDue to the time constraints of the rebuttal, we were unable to include results on **LLaMA 3** and **DeepSeek** in this version. However, we are actively working on these experiments and will include a more comprehensive comparison with newer models in the revised version.\n\n---\n**R3. Scalability to long-context or multi-step reasoning tasks**\n\nThank you for the question. Our method is scalable to long-context and multi-step reasoning tasks. This is supported by our experimental results on **GSM8K** and **MATH** — two widely recognized benchmarks for multi-step symbolic and numerical reasoning. Compared to other methods, our approach achieves **higher accuracy** while requiring **fewer trainable parameters**, demonstrating both its **efficiency** and **effectiveness** in handling complex reasoning scenarios. The detailed results are provided below for reference.\n\n**Table 6: Perfromance of diferent PEFT methods on LLaMA 2-7B.**\n | Method        | Trainable Parameters | GSM8K   | MATH         |\n| -------------------- | -------------------- | ------------- | ------------ |\n | Full FT              | 6738M                | 49.05 | 7.22 |\n | LoRA                | 320M                 | 42.30         | 5.50         |\n| PiSSA $(r=8)$         | 19M                  | 44.11         | 5.84         |\n| LoRA-PRO $(r=8)$     | 19M                  | 46.61         | 6.40         |\n| AdaMSS$_{base}$ $(r_k=3)$ | **4M**               | **51.10**     | **7.57**     |\n| AdaMSS$(r_k=3)$    | **4M**               | 50.80     | 7.22     |\n\n\n**Table 7: Perfromance of diferent PEFT methods on Mistral-7B.**\n| Method               | Trainable Parameters | GSM8K     | MATH      |\n| -------------------- | -------------------- | --------- | --------- |\n| Full FT | 7242M | 67.02  | 18.60   |\n| LoRA   | 168M    | 67.70     |  19.68  |\n| PiSSA $(r=8)$  | 20M |  71.00  |  20.40 |\n| LoRA-PRO $(r=8)$ | 20M | 69.59| 19.17 |\n| AdaMSS$_{base}$$(r_k=3)$ | 4M |**70.71**|**20.44** |\n| AdaMSS$(r_k=3)$| **2M** |  70.74  | 19.47|\n\n\n**Table 8: Perfromance of diferent PEFT methods on Gemma-7B.**\n| Method               | Trainable Parameters | GSM8K     | MATH      |\n| -------------------- | -------------------- | --------- | --------- |\n| Full FT | 8538M | 71.34 | 22.74|\n| LoRA | 200M  |74.90 |31.28|\n| PiSSA $(r=8)$ | 25M | 75.48 | 29.59 |\n| LoRA-PRO  | 25M | 75.90 | 29.25 |\n| AdaMSS$_{base}$ $(r_k=3)$ | 6M     | 75.33     | **29.73** |\n| AdaMSS $(r_k=3)$ | **4M**  | **76.41** | 28.64     |\n\n \n---\n**R4. Compatibility with positional tuning (e.g., RoCoFT)**\n\n\nWe thank the reviewer for this insightful question. Indeed, our framework could potentially be integrated with positional tuning methods such as **RoCoFT**.\n\nOne preliminary yet intuitive idea is to apply positional tuning directly to the **low-rank representation** $\\mathbf{Z}$, especially given that $\\mathbf{Z}$ often exhibits an approximately **block-diagonal structure**. Another promising direction, inspired by the **AdaMSS**, is to leverage **subspace segmentation** to guide RoCoFT in deciding **which columns should be updated and which can be frozen**, potentially enabling **adaptive budget allocation** during training.\n\nWe will include this discussion in the *Future Work* section of the revised version.\n\n\n\n\n---\nWe thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify."}, {"review_id": "Ffh12hqrOD", "reviewer": "Reviewer_Bhh4", "summary": "This paper proposes AdaMSS, a parameter-efficient fine-tuning (PEFT) method that adaptively updates a small subset of subspaces within a neural network's weight matrix. Unlike existing methods like LoRA or PiSSA that rely on a single low-rank subspace, AdaMSS identifies multiple subspaces via low-rank representation and performs fine-tuning in a block-diagonal, multi-subspace manner. It also introduces an adaptive budget allocation mechanism that selectively updates only the most relevant subspaces during training. The authors provide theoretical guarantees showing improved generalization bounds over prior PEFT methods. Extensive experiments across vision and language tasks demonstrate that AdaMSS achieves competitive or superior performance with significantly fewer trainable parameters.", "questions": "Please check the weakness part. will increase the score if the concerns are solved.", "limitations": "Please check the weakness part", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. Insightful empirical finding on multi-subspace structure\n\nThe paper presents an interesting and novel empirical observation: pretrained weight matrices across various models (e.g., ViT, RoBERTa, LLaMA) exhibit a multi-subspace structure. This is revealed through Low-Rank Representation (LRR) and spectral clustering, where the coefficient matrix shows an approximate block-diagonal form. This observation serves as a strong motivation for moving beyond single-subspace LoRA-style methods and is well supported by visual and quantitative evidence.\n\n2. Well-motivated Multi-Subspace-Based Incremental Update\n\nThe proposed AdaMSS framework is a conceptually clean and practically relevant solution. By projecting pretrained weights into multiple orthogonal subspaces and assigning fine-tuning capacity selectively based on subspace importance, the method provides a structured, task-aware approach to parameter-efficient fine-tuning. The use of residual connections further ensures expressiveness while maintaining modularity.\n\n3. Strong theoretical analysis\n\nThe paper is theoretically grounded, providing clear generalization bounds via Gaussian complexity analysis. The bounds support the core claim that multi-subspace adaptation leads to improved generalization compared to global low-rank methods. This theoretical support enhances the credibility and rigor of the proposed method.\n\n4. Solid empirical performance across domains\n\nAdaMSS demonstrates strong and consistent performance improvements over existing PEFT methods such as LoRA, PiSSA, and RandLoRA across a wide range of benchmarks. Notably, it achieves these gains with significantly fewer trainable parameters, which makes it especially suitable for large-scale or resource-constrained applications.\n\n# Weaknesses\n1. Sufficiency of empirical evidence for subspace structure\n\nFigure 2 is a good illustrative starting point, but on its own, it's not sufficient evidence to claim that pretrained models broadly exhibit a multi-subspace structure. More layer-wise or task-relevant analysis would be needed to make the case compelling.\n\n2. Lack of comparative baselines for adaptive subspace selection\n\nThe proposed adaptive budget allocation strategy, which freezes or updates subspaces based on gradient magnitude and variance, is intuitive. However, it would be useful to see direct comparisons with simpler or alternative selection strategies, such as random subspace selection or allocating budget based on principal singular directions within each cluster. This would help isolate the specific benefit of the adaptive mechanism.\n\n3. Sensitivity to preprocessing and hyperparameters\n\nThe method introduces several preprocessing steps and hyperparameters, such as the number of subspaces \n𝐾 clustering thresholds, and gradient-based pruning schedules. It remains unclear how sensitive the method is to these choices or how automated the process is in practice. More guidance or ablation studies would be helpful, especially for practitioners looking to apply the method to new domains.\n\n## Writing Clarity\nThe paper can be somewhat difficult to follow in parts. The writing is dense, particularly in Sections 3.1 and 3.2, where the mathematical derivation of subspace segmentation and reparameterization could be broken down more clearly. The interplay between different components is not always intuitive on first read. More visual aids or examples could significantly improve clarity for readers unfamiliar with subspace learning techniques.", "was_revised": true, "final_justification": "I have read the thorough rebuttal by authors. The paper has strong theoretical analysis with solid performance across boards.", "rebuttal": "We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised. \n\n---\n**R1. Sufficiency of empirical evidence for subspace structure**\nThank you so much for your constructive suggestions. In addition to **Figure 2**, we will include **more visualizations of empirical evidence for multi-subspace structures across layers and tasks** in the revised manuscript. Owing to space constraints, we  provide **partial numerical evidence** for multi-subspace structures across different layers.\n\nIn addition to the **approximate block-diagonal patterns** shown in Figure 2, another way to observe the distribution of weight column vectors is through the **singular values of the Laplacian matrix** constructed from the **principal components**. In **Table 1**, we show the singular value distribution of the Laplacian matrix computed from the principal components of the **query weight matrices** at each layer of a pretrained **`ViT-Large`** model. From Table 1, we make the following observation: for most layers, the **first 900 singular values remain significantly large**, while the trailing values are **very close to zero** (e.g., below 0.01). \n\nThe **number of near-zero singular values** of the Laplacian matrix can be interpreted as an estimate of the **number of disjoint subspaces** within the weight space. This provides an additional numerical evidence of the **multi-subspace structures** within the principal components of pretrained weights.\n\n\n\n**Table 1: Singular value distribution of the Laplacian matrices constructed from the principal components of the query weight matrices at each layer of the pretrained `ViT-Large` model.**  \n|Layer index|$\\sigma_1$|$\\sigma_{101}$|$\\sigma_{201}$|$\\sigma_{301}$|$\\sigma_{401}$|$\\sigma_{501}$|$\\sigma_{601}$|$\\sigma_{701}$|$\\sigma_{801}$|$\\sigma_{901}$| $\\sigma_{1001}$ | $\\sigma_{1024}$|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|1|1.00|0.87|0.78|0.59|0.38|0.21|0.09|0.02|0.00|0.00|0.00|0.00|\n|3|1.00|0.83|0.78|0.72|0.65|0.57|0.47|0.37|0.24|0.10|0.00|0.00|\n|5|1.00|0.83|0.81|0.79|0.77|0.74|0.70|0.64|0.54|0.38|0.05|0.00|\n|7|1.00|0.82|0.81|0.79|0.77|0.74|0.71|0.67|0.60|0.47|0.01|0.00|\n|9|1.00|0.82|0.81|0.79|0.78|0.76|0.74|0.71|0.67|0.56|0.00|0.00|\n|11|1.00|0.83|0.81|0.80|0.79|0.77|0.75|0.73|0.68|0.55|0.00|0.00|\n|13|1.00|0.83|0.81|0.80|0.78|0.76|0.74|0.71|0.66|0.52|0.00|0.00|\n|15|0.99|0.81|0.79|0.77|0.76|0.73|0.71|0.67|0.63|0.53|0.00|0.00|\n|17|1.00|0.82|0.80|0.79|0.77|0.76|0.74|0.71|0.68|0.60|0.00|0.00|\n|19|1.00|0.82|0.81|0.80|0.78|0.77|0.76|0.74|0.71|0.66|0.00|0.00|\n|21|1.00|0.82|0.81|0.80|0.80|0.79|0.78|0.77|0.75|0.72|0.00|0.00|\n|23|1.00|0.83|0.82|0.81|0.80|0.80|0.79|0.78|0.77|0.75|0.12|0.00|\n\n---\n\n**R2. Lack of comparative baselines for adaptive subspace selection**\n\n**Table 2:   Comparing the performance of different adaptive allocation scheme on `ViT-Large` model.**\n| Methods \\ Dataset | StanfordCars | CIFAR100 | FGVC |\n|-|-|-|-|\n|  Importance-score-based  adaptive budget allocation |**85.24±0.3**|**93.22±0.01**|**64.31±0.4**|\n| Random adaptive selection   | 83.63±0.30 | 93.11±0.10 | 59.36±0.72 |\n| Adaptive selection based on $\\ell_1$ norm of each subspace parameters  | 79.89±0.41 | 92.00±0.21 | 40.77±1.13 | \n\n\nThank you so much for your constructive suggestions. We agree that including comparisons against alternative strategies will help clarify the advantages of our adaptive allocation scheme.\n\nIn the revised manuscript, we include comparisons with the following two adaptive allocation strategies:\n- (1) **Random adaptive selection**, as suggested by the reviewer.\n- (2) **$\\ell_1$-norm-based adaptive budget allocation**, where the budget is assigned based on subspaces with the highest $\\ell_1$-norm, although we currently do not directly implement allocation based on **principal singular directions within each cluster** as suggested.\n\nAs shown in **Table 2**, $\\ell_1$-norm-based adaptive budget allocation and random adaptive selection  lead to significant performance degradation, and importance-score-based  adaptive budget allocation has achieved best performance among all compared methods.\n\n---\n**R3. Sensitivity to hyperparameters and preprocessing**\n \nThank you so much for your constructive suggestions. In the current version of the manuscript, we have already compared strategies **with and without subspace number estimation**, as well as the effect of using different fixed values of $K$.\n\nFollowing your suggestion, we have reorganized our existing ablation studies and added **new ablations** from the following two aspects:\n\n1. **Hyperparameters for subspace segmentation**, including the thresholds $\\tau$ and $K_0$ used for estimating the number of subspaces $K$;\n2. **Hyperparameters for adaptive budget allocation**, including the target number of subspaces $K_{\\text{target}}$ and the decay exponent $\\rho$ used in the proposed multi-subspace adaptive budget allocation scheme.\n\nBelow we summarize the findings of these new ablation studies:\n\n- **(1) Thresholds $\\tau$ and $K_0$ for subspace estimation**:\n  The threshold $\\tau$ is used to detect near-zero singular values in the normalized Laplacian matrix and is set as a small value. In our experiments, we set $\\tau = 0.01$. The parameter $K_0$ serves as a lower bound on the estimated number of subspaces.\n  In **Tables 3–5**, we evaluate various values $\\tau$ ∈ {0.001, 0.01, 0.05, 0.10, 0.15, 0.20} and $K_0$ ∈ {1, 5, 10, 15, 20} using **ViT-Large** with  AdaMSS$_{base}$ (without adaptive budget allocation). \n\n   The results show that when $K_0 \\geq 10$,  AdaMSS$_{base}$ exhibits robust  performance  across all three datasets for  different  $\\tau$ and $K_0$.\n\n- **(2) Target number of subspaces $K_{\\text{target}}$ and decay exponent $\\rho$**:\n  In the adaptive budget allocation scheme, the number of trainable subspaces is gradually reduced to $K_{\\text{target}}$ according to a **smooth cubic decay schedule** (default $\\rho = 3$). A larger $\\rho$ indicates a faster decay in the number of active subspaces.\n  In **Tables 6–8**, we evaluate the sensitivity to $\\rho$ ∈ {1, 2, 3, 4, 5} and $K_{target}$ ∈ {100, 200, 300, 400} using **ViT-Large** with  **AdaMSS**.\n  The results indicate that **AdaMSS is robust** to across a range of $\\rho$ and $K_{\\text{target}}$.\n\n\n**Table 3: Results of AdaMSS**$_{base}$ **under different hyperparameter settings for `StanfordCars`.**\n| $\\tau$ \\ $K_0$|1|5|10|15|20| \n|-|-|-|-|-|-|\n|0.001|84.25±0.23|85.40±0.19|85.40±0.22|85.38±0.25|85.67±0.42|\n|0.01|84.12±0.24|85.20±0.50|85.31±0.27|85.41±0.13|85.50±0.21| \n|0.05|84.77±0.16|85.02±0.26|85.23±0.28|85.59±0.27|85.70±0.24| \n|0.10|84.58±0.29|84.93±0.25|85.60±0.26|85.36±0.29|85.42±0.23|\n|0.15|84.72±0.39|85.12±0.23|85.38±0.33|85.61±0.20|85.63±0.45|\n|0.20|84.57±0.09|84.86±0.17|85.53±0.21|85.39±0.34|85.65±0.19|\n\n**Table 4: Results of AdaMSS$_{base}$ under different hyperparameter settings for `CIFAR100`.**\n| $\\tau$ \\ $K_0$|1|5|10|15|20|\n|-|-|-|-|-|-|\n|0.001|93.27±0.09|93.44±0.16|93.47±0.06|93.57±0.12|93.48±0.15|\n|0.01| 93.38±0.08|93.33±0.10|93.50±0.10|93.53±0.15|93.57±0.05| \n|0.05|93.42±0.12|93.43±0.12|93.51±0.09|93.45±0.10|93.56±0.09|\n|0.10|93.40±0.12|93.47±0.09|93.58±0.13|93.45±0.12|93.64±0.10| \n|0.15|93.43±0.12|93.43±0.14|93.52±0.06|93.55±0.06|93.60±0.12| \n|0.20|93.43±0.10|93.54±0.05|93.48±0.08|93.53±0.04|93.51±0.11|\n\n**Table 5: Results of AdaMSS$_{base}$ under different hyperparameter settings for `FGVC`.**\n|$\\tau$ \\ $K_0$|1|5|10|15|20|\n|-|-|-|-|-|-|\n|0.001|61.36±1.12|65.33±0.55|66.11±0.53|66.62±0.78|66.98±0.73|\n|0.01|61.90±0.55|64.58±0.69|65.27±0.64|66.10±0.69|66.60±0.68| \n|0.05|61.64±0.22|64.63±0.98|65.51±1.05|66.63±0.48|67.44±0.68|\n|0.10|62.42±0.33|64.49±0.87|65.88±0.68|66.29±0.21|67.16±0.82|\n|0.15|62.56±0.73|64.90±1.02|65.72±1.04|66.86±0.30|66.44±0.92|\n|0.20|62.00±0.51|64.76±0.78|65.68±0.93|66.65±0.41|66.86±0.60|\n\n**Table 6: Results of AdaMSS under different hyperparameter settings for `StanfordCars`.** \n\n|$\\rho$ \\ $K_{target}$|100|200|300|400|500|\n|-|-|-|-|-|-|\n|1|84.78±0.32|85.06±0.16|85.34±0.13|85.35±0.11|85.16±0.12|\n|2|84.88±0.36|85.11±0.19|85.21±0.33|85.26±0.20|85.21±0.41|\n|3|85.03±0.29|84.91±0.24|85.21±0.39|85.06±0.20|85.02±0.18|\n|4|84.69±0.20|84.95±0.13|85.03±0.29|85.02±0.21|85.19±0.13|\n|5|84.89±0.27|84.67±0.25|85.04±0.25|85.16±0.26|85.37±0.36|\n\n**Table 7: Results of AdaMSS under different hyperparameter settings for `CIFAR100`.**  \n|$\\rho$ \\ $K_{target}$|100|200|300|400|500|\n|-|-|-|-|-|-|\n|1|93.47±0.10|93.49±0.11|93.41±0.14|93.56±0.10|93.34±0.12|\n|2|93.41±0.14|93.44±0.07|93.54±0.08|93.42±0.13|93.47±0.07|\n|3|93.42±0.14|93.38±0.05|93.56±0.09|93.46±0.14|93.55±0.08|\n|4|93.25±0.11|93.53±0.09|93.40±0.10|93.48±0.15|93.50±0.07|\n|5|93.35±0.15|93.45±0.11|93.37±0.09|93.41±0.12|93.41±0.12|\n\n**Table 8: Results of AdaMSS under different hyperparameter settings for `FGVC`.**\n|$\\rho$ \\ $K_{target}$|100|200|300|400|500|\n|-|-|-|-|-|-|\n|1|64.76±0.55|65.56±0.60|64.98±0.81|64.90±0.98|65.73±0.48|\n|2|64.55±0.73|64.38±0.83|65.14±0.89|65.11±0.57|64.99±0.70|\n|3|63.73±0.31|64.16±1.22|64.77±0.34|65.33±0.64|65.77±0.21|\n|4|63.02±0.89|64.35±0.62|64.64±0.81|64.54±0.47|65.13±0.61|\n|5|62.74±1.53|63.43±0.63|65.12±0.83|64.28±0.99|65.32±0.73|\n\n---\n**R4. Writing clarity and visual explanation**\n\nThank you for highlighting this issue. We will revise **Sections 3.1 and 3.2** by simplifying the notation and derivations, and by adding **visual illustrations** for the proposed methods.  \nWe hope these revisions will enhance the **clarity** and **readability** of our work.\n\n---\nWe thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify."}, {"review_id": "jjdVKjEHkn", "reviewer": "Reviewer_icwe", "summary": "This paper introduces AdaMSS, a parameter-efficient fine-tuning method that divides neural network weights into multiple subspaces and adapts them with low-rank updates. By dynamically allocating parameters to the most important subspaces, AdaMSS achieves strong performance on vision and language tasks with far fewer trainable parameters than previous methods.", "questions": "- The paper claims that lower model complexity leads to better generalization, but this statement is not rigorous enough. Please provide a more detailed explanation.\n- Please discuss in more detail how the proposed method compares to other LoRA variants that utilize subspaces.\n- How does the proposed method perform in LLM scenarios? Please cover as many base models and parameter scales as possible.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The approach of further reducing LoRA’s trainable parameter count through multiple subspaces while largely maintaining performance is interesting. The introduction of Low-Rank Representation is relatively novel, and the final weight block update strategy is quite clever.\n- The paper analyzes the method’s complexity.\n\n**Weaknesses**\n- The writing of the paper could be improved, for example by including a method diagram to help readers quickly grasp the main idea. The presentation of experimental results is not intuitive enough, making it hard for readers to immediately see the advantages of the method. The explanation of the importance score in the paper also lacks intuitiveness.\n- The paper could include more discussion comparing with other LoRA-based methods that leverage subspaces.", "was_revised": true, "final_justification": "After reading the author's response, some of my concerns have been solved. I keep my previous justification of borderline acceptance.", "rebuttal": "We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised. \n\n---\n**R1. Clarity of presentation**\n\nWe appreciate the suggestions. Following the recommendations, in the revised version, we will include a **method diagram** to summarize the proposed adaptation process and improve **readability** by simplifying the notation and derivations.\n\nFor the experimental section, we will **reorganize the existing ablation studies** and introduce a new **discussion section** to better highlight and explain the advantages of our method.\n\nWe believe that these revisions will significantly enhance the **clarity** of our work.\n\n---\n\n**R2. Intuitive explanation of importance scores**\n\nThe **importance score** used in this work is defined based on the product of $\\bar{I}^{(t)}$ and $\\bar{U}^{(t)}$, where:\n\n- $\\bar{I}^{(t)}$   measures the **sensitivity of the weights**, and it is defined on the **magnitude of the gradient-weight product**, which approximates the **change in the loss function** due to that weight.\n- $\\bar{U}^{(t)}$ captures the **uncertainty in the sensitivity estimation**.\n\n\nTherefore, this importance score helps identify weights that either (1) contribute significantly to **loss reduction**, or (2) exhibit **uncertainty**, making them important candidates for training.\n\n---\n\n**R3. Clarifying generalization**\n \nThe **superior generalization ability** of a method refers to the following:  \n**Given the same number of training samples and comparable training loss**, the method achieves a **lower expected test error** than competing approaches.\n\nTo support this, *Theorem 1* provides a generalization bound for **AdaMSS**, showing that the **expected test error decreases with the Gaussian complexity** of the learned function class. Since AdaMSS induces a function class with **lower Gaussian complexity**, it is theoretically expected to generalize better than methods with higher complexity.\n\nThis theoretical advantage is further **validated by our empirical results**, which consistently demonstrate that AdaMSS achieves **lower average test errors** across a range of tasks.\n\n---\n**R4. Discussion in detail how comparing with other LoRA-based subspace methods**\n\nWe thank the reviewer for the suggestion to clarify how **AdaMSS** compares to other **LoRA variants that utilize subspaces**. Following your suggestion, we will add a detailed discussion comparing our method with other LoRA-based approaches in the revised version. Besides, we will include a new **Discussion Section** and introduce additional **experiments** to better illustrate the advantages of AdaMSS.\n\nBelow is a summary of the discussion for your reference:\n\n\n**1. Key Differences Between AdaMSS and Other Subspace-Based LoRA Variants**\n\nWhile many existing PEFT methods—such as **PiSSA**, **AdaLoRA** and **LoRA-GA**—rely on the assumption of a **single low-rank subspace**, **AdaMSS** departs from this by modeling the pretrained weight space as composed of **multiple distinct subspaces**.\n\nAdaMSS clusters the column vectors of the weight matrix into separate subgroups, each forming an independent subspace with its own low-rank representation.  \nUnlike AdaLoRA, which dynamically adjusts the rank of a single subspace, **AdaMSS performs explicit subspace segmentation and dynamically estimates the importance of each subspace**, enabling more **compact expressiveness** and **finer adaptation**.\n\n \n\n**2. Advantages of AdaMSS over other Methods**\n\n- **(1) Reduced Training Time and Memory Usage:**  \n  Thanks to its lower number of trainable parameters and the  proposed adaptive budget allocation, AdaMSS achieves a better performance in both **memory efficiency** and   **training speed**.  \n  Moreover, our budget allocation scheme offers greater **flexibility in balancing accuracy and training efficiency**.\n\n- **(2) Better Accuracy Across Multiple Tasks:**  \n  AdaMSS consistently outperforms other methods—including **PiSSA**—on several benchmarks, only using **fewer or comparable trainable parameters**.  \n  In the following table, we summarize tasks where AdaMSS achieves **at least 3% accuracy improvement** over competing methods. **Bold entries** indicate cases where AdaMSS  **reduces the number of trainable parameters by more than 75%** yet still achieves superior results.\n\n| Tasks | PiSSA | LoRA | LoRETTA |\n|-------|-------|------|----------|\n| IC    | ViT-Base | **ViT-Base**, **ViT-Large** | ViT-Large |\n| NLU   | RoBERTa-Base, **RoBERTa-Large** | RoBERTa-Base, RoBERTa-Large | RoBERTa-Large |\n| NLG   | **LLaMA 2-7B** | **LLaMA 2-7B** | LLaMA 2-7B |\n\n *In the revised version,  we have extended our experiments on **language models** (RoBERTa) to include **LoRA $(r=1)$** and **PiSSA $(r=1)$** under **minimal parameter settings**, to better highlight and explain the advantages of our method. Please refer to **Tables 1 and 2**.*\n\n\n- **(3) Theoretical Guarantee:**  \n  Our method provides a stronger theoretical foundation. **Theorem 1** formally derives an **upper bound on the expected test loss** of AdaMSS, offering a **generalization guarantee** that is not available in most other LoRA-based approaches.\n\n\n**Table 1. Results of LoRA  $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Base**\n|Model|Trainable Param|CoLA|MRPC|QNLI|RTE  |SST-2|STS-B|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|LoRA|55K|62.3±3.57|89.22±0.31 |90.59±0.30|**79.49±0.42**|93.74±0.45|80.81±20.64|82.7|\n|PiSSA|55K|62.6±1.44|**89.26±0.61**|90.61±0.41|74.87±1.22|93.28±0.20|89.97±0.28|83.4|\n|AdaMSS|**32K**|**64.5±1.1**|88.8±1.4|**92.4±0.1**|77.3±0.7|**94.6±0.2**|**90.4±0.1**|**84.7**|\n\n**Table 2. Results of LoRA $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Large**\n|Model|Trainable Param|CoLA| MRPC|QNLI|RTE|SST-2|STS-B|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|LoRA|147K|62.16±2.38|88.33±0.65|93.86±0.16|82.24±2.47| 95.67±0.41|78.15±29.72|83.4|\n|PiSSA|147K|56.58±6.19|84.90±3.35 |93.40±0.31|65.92±11.26|95.18±0.23|91.26±0.24|81.2|\n|AdaMSS|**45K**|**67.2±1.2**|**90.3±0.5**|**94.5±0.1**|**87.1±2.1**|**96.1± 0.0**|**91.9±0.0**|**87.9**|\n \n\n---\n\n\n\n**R5. Cover as many base models and parameter scales as possible**\n\nThank you so much for the comment. Our current experiments cover a diverse set of models, including **ViT (Base/Large)**, **RoBERTa (Base/Large)**, **LLaMA 2-7B**, **Mistral-7B**, and **Gemma-7B**, with parameter sizes ranging from **85.8M to 8.5B**.  \nIn most cases, our method achieves the **best performance** across these models.  \n\nDue to the limited rebuttal timeline, we were unable to include results on larger-scale models in this version. However, we are actively conducting these experiments and will incorporate a more comprehensive comparison with larger-scale models in the revised submission.\n\n\n\n---\n\nWe thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify."}, {"review_id": "zgCRWF0crp", "reviewer": "Reviewer_i4NE", "summary": "- AdaMSS proposes a new parametrization for low-rank adaptation based on subspace ssegmentation.\n- The weights in transformer models can be written as $\\hat{W}_0 = \\hat{W}_0 Z_0$, with $\\hat{W}_0$ being the rank-truncated SVD of the pretrained weights $W_0$, and $Z_0$ has an approximate block diagonal structure alon with an analytical solution.\n- This formulation is an application of the Low-Rank Representation (LRR) model.\n- The row space is segmented into $K$ smaller subspaces (blocks in the block diagonal matrix $Z_0$), where each block contains columns of $W_0$ assined to the block.\n- Based on this formulation, the weights can be written as $W_0^{(k)} = \\hat{W}_0^{(k)}(Z_0^*)^{(k)} + \\bar{W}_{res}^{(k)}$\n- The paper proposes a residual update $\\Delta Z^{(k)}$, so that the final low-rank update can be written as a decomposition $W^{(k)} = W_0^{(k)} + A^{(k)}B^{(k)}C^{(k)}$, with A and B initialized using the SVD of $W_0$ and C initialized using 0.\n- The paper further proposes to freeze ranks progressively based on the gradients of $B$ and $C$, leading to adaptive budet allocation.\n- AdaMSS achieves a lower theoretical bound on generalization.", "questions": "For all the experiments, which modules are fine-tuned (for e.g, Query Value, etc.)?", "limitations": "Please refer to point (1) in the weaknesses for a limitation that I have observed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The AdaMSS formulation enables fine-tuning with very few additional parameters\n- The update is theoretically motivated from subspace segmentation, and the verification experiments confirm the theory\n- The bound on generalization is lower than prior methods, potentially offering better generalization capabilities\n\n## Weaknesses\n1. For the amount of complexity added, the performance improvement over prior methods like PiSSA are small (especially since PiSSA is much simpler)\n    - The initialization of the A and B requires computing the SVD of weight matrices, which can be quite costly for large models\n    - The authors mention that the matrices can be randomly initialized, no empirical experiments have been performed\n    - While this method does reduce the number of parameters, in practice, rank 16 is very economical, and anything below that does not really reduce the training time/memory requirements much especially for large models.\n    - The question then is, what other benefits/insights does the method offer? Can the authors perform some experiments that demonstrate generalization?\n\n2. For the experiments on large language models, the comparison with equivalent trainable parameter counts for LoRA and PiSSA is missing\n    - A comparison with equivalent parameter count would be useful to see how quickly any method can adapt with minimal parameters, showing more benefit of the structured adaptation\n\n3. Missing citation and comparison\n    - The residual update formulation in equation 8 ($W^{(k)} = W_0^{(k)} + \\hat{W}_0^{(k)}\\Delta Z^{(k)}$) is very similar to [1]. Can the authors explain the difference and add a comparison?\n\nOverall, the idea is interesting and the evaluation is good, and I would consider increasing my score if the authors can address my concerns.\n\n---\n\n## References\n\n[1] Chinmay Savadikar, Xi Song, and Tianfu Wu. \"Generative Parameter-Efficient Fine-Tuning.\" arXiv preprint arXiv:2312.00700 (2023).", "was_revised": true, "final_justification": "My main concern was regarding the overhead of the method compared to LoRA and PiSSA, and whether it is justified. Because of the following two points, I have increased my score and recommend acceptance: \n\n- The authors have demonstrated that with AdaMSS can adapt models to downstream tasks better than prior methods with comparable number of parameters. This shows the presence of meaningful subspaces in pretrained model weights, which can potentially open more avenues for future research.\n- AdaMSS shows theoretical guarantees for loer test error.", "rebuttal": "We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised. \n\n---\n**R1. Performance improvement over prior methods like PiSSA**\nThank you so much for your comments. Our method consistently outperforms other approaches—including **PiSSA**—in terms of accuracy, often with **comparable or significantly fewer trainable parameters**.\n- For **Image Classification (IC)**, our method achieves over **6% higher accuracy** than PiSSA on **ViT-Base** with a **similar parameter budget**.\n- For **Natural Language Understanding (NLU)**, our method achieves over **7% higher accuracy** than PiSSA on **RoBERTa-Large**, while using much fewer parameters. \n  In our newly added experiments (as suggested by the reviewer), we have included results for **PiSSA with $r=1$** (see Tables 1–2).  These results show that, under similar parameter conditions, our method performs  **consistently** across tasks on both **RoBERTa-Large** and **RoBERTa-Base**, and achieves **substantially higher average performance** compared to PiSSA.\n- For **Natural Language Generation (NLG)** (see additional results in the supplementary material), our method outperforms PiSSA on **LLaMA 2-7B** by: **7% higher accuracy** on **GSM8K**, and **1.7% higher accuracy** on **MATH**, while using **fewer trainable parameters**.\n\nBelow we summarize tasks where our method achieves **at least 3% higher accuracy** on average compared to other methods. **Bold entries** indicate cases where AdaMSS **reduces the number of trainable parameters by more than 75%** yet still achieves superior results.\n\n|Tasks|PiSSA|LoRA|LoRETTA |\n|-|-|-|-|\n|IC|ViT-Base|**ViT-Base**, **ViT-Large**| ViT-Large|\n|NLU|RoBERTa-Base, **RoBERTa-Large** | RoBERTa-Base, RoBERTa-Large|RoBERTa-Large |\n|NLG|**LLaMA 2-7B**|**LLaMA 2-7B**|LLaMA 2-7B |\n\n\n \n---\n**R2. The cost of initialization of AdaMSS**\nIn our initialization phase, the most time-consuming step is the **estimation of the number of subspaces**. \nAs discussed in our ablation study (see Appendix), when the number of subspaces is fixed to $K = 10$, the performance of our method with fixed $K$ is **comparable to AdaMSS with subspace number estimation strategy**.\n\nIn the current submitted supplementary material, we further compare AdaMSS **without subspace estimation** to other PEFT methods (including **PiSSA**, **LoRA**, **LoRA-PRO**, and **LoRETTA**) on **LLaMA 2-7B**, **Mistral-7B**, and **Gemma-7B**. In most cases, our method still achieves the **best performance**, even without estimating the number of subspaces.\nTo make AdaMSS even more practical, we also apply **low-rank SVD** techniques to accelerate the subspace initialization process.\n\nAs shown in the current submitted supplementary material, our method initializes faster than **LoRETTA** and completes in just a few seconds on LLaMA 2-7B, highlighting its computational efficiency.\n\n\n---\n**R3. Comparison with random initialization**\nWe have included a **comparison between random initialization and orthogonal initialization** in the current submitted supplementary material (**Figure 1**). As shown by our experimental results, **orthogonal initialization significantly outperforms random initialization**.\n\n---\n**R4. Training time/memory comparisons**\nOwing to space constraints, please see our response given in R1 for  Reviewer J6fY. \n\n---\n**R5. Clarifying generalization and  experiments that demonstrate generalization**\nThe generalization ability of a method is reflected in its expected test error. As established in Section **Analysis of AdaMSS**, AdaMSS exhibits a lower upper bound on **the expected test error** compared to other methods, given the same number of training samples. This theoretical advantage is further supported by our experimental results, which consistently show **lower average test errors** for AdaMSS across a range of tasks.\n\n---\n**R6. Advantages of AdaMSS**   As discussed in  **R1** and **R4–R5**, our method offers the following key advantages.\n- **(1) Superior Accuracy Across Tasks.** \n- **(2) Reduced Training Time and Memory Usage.** \n- **(3) Stronger Theoretical Guarantee:** **Theorem 1** establishes an **upper bound on the expected loss** of AdaMSS, providing a formal **generalization guarantee** that supports its theoretical soundness.\n\n---\n**R7. Comparison with equivalent trainable parameter counts for LoRA and PiSSA for   language models**\nThank you so much for the valuable suggestion. Our original comparisons on **ViT models** already included **PiSSA $(r=1)$** as a baseline.\n\nFollowing your recommendation, we have extended our experiments on **language models** (RoBERTa) to include **LoRA $(r=1)$** and **PiSSA $(r=1)$** under **minimal parameter settings**, in addition to our method **AdaMSS$_{base}$**. Please refer to **Tables 1 and 2** below.\n\nAs the experimental results demonstrate, our method consistently outperforms both **LoRA** and **PiSSA** under a   low-parameter budget. Furthermore, **AdaMSS exhibits more stable performance across tasks**, highlighting the advantage of **structured subspace updates** in resource-constrained settings.\n\n**Table 1. Results of LoRA  $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Base**\n|Model|Trainable Param|CoLA|MRPC|QNLI|RTE  |SST-2|STS-B|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|LoRA|55K|62.3±3.57|89.22±0.31 |90.59±0.30|**79.49±0.42**|93.74±0.45|80.81±20.64|82.7|\n|PiSSA|55K|62.6±1.44|**89.26±0.61**|90.61±0.41|74.87±1.22|93.28±0.20|89.97±0.28|83.4|\n|AdaMSS|**32K**|**64.5±1.1**|88.8±1.4|**92.4±0.1**|77.3±0.7|**94.6±0.2**|**90.4±0.1**|**84.7**|\n\n**Table 2. Results of LoRA $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Large**\n|Model|Trainable Param|CoLA| MRPC|QNLI|RTE|SST-2|STS-B|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|LoRA|147K|62.16±2.38|88.33±0.65|93.86±0.16|82.24±2.47| 95.67±0.41|78.15±29.72|83.4|\n|PiSSA|147K|56.58±6.19|84.90±3.35 |93.40±0.31|65.92±11.26|95.18±0.23|91.26±0.24|81.2|\n|AdaMSS|**45K**|**67.2±1.2**|**90.3±0.5**|**94.5±0.1**|**87.1±2.1**|**96.1± 0.0**|**91.9±0.0**|**87.9**|\n\n---\n**R8. Add citation [1] and comparison with [1]** \nWe sincerely thank the reviewer for bringing this excellent work to our attention. **WeGeFT** is recently accepted method at **ICML 2025**. In the revised version of our manuscript, we have included a discussion of WeGeFT [1] and added experimental comparisons against it.\n\nBelow, we provide a summary of  the  differences between **AdaMSS** and **WeGeFT**, as well as empirical comparisons on **Natural Language Understanding (NLU)** and **Image Classification (IC)** tasks.\n- **(1) Motivation:** AdaMSS is motivated by the **multi-subspace structure** of pretrained weight matrices. It seeks compact representations via **lowest-rank representation** and **subspace segmentation**, followed by a **multi-subspace-based adaptive budget allocation** strategy to reduce the number of trainable parameters while preserving model capacity. \n\nIn contrast, WeGeFT focuses on **cross-layer parameter sharing**, under the assumption that pretrained weights already contain useful \"transferable knowledge\" for downstream tasks. \n- **(2) Methodology:** AdaMSS leverages **low-rank representation** and **subspace segmentation** to obtain a compact parameterization of the weights: \n  $\\mathbf{W}^{(k)} = \\mathbf{W}_0^{(k)} + \\mathbf{\\hat{W}}_0^{(k)} \\Delta \\mathbf{Z}^{(k)}$, where $\\mathbf{W}^{(k)}$ consists of  the weights   belonging to the $k$-th subspace, $\\Delta \\mathbf{Z}^{(k)}$ is   **low-rank matrix**, and $\\mathbf{\\hat{W}}_0^{(k)}$ represents the **principal components** of the corresponding weight subspace. \n\n  WeGeFT adopts the formulation $\\mathbf{W}^l = \\mathbf{W}_0^{l} + \\mathbf{W}_0^{l} \\Delta \\mathbf{Z}$, where $\\Delta \\mathbf{Z}$ is a  low-rank matrix shared across layers, and $\\mathbf{W}^l$ denotes the weight matrix of the $l$-th layer.\n\n**Comparison with WeGeFT**\n\n**Table 3** compares the performance of AdaMSS and WeGeFT on **NLU** tasks.  \n**Table 4** shows their results on **IC** tasks.  \nOverall, **AdaMSS outperforms WeGeFT on average** for NLU (RoBERTa) and has a comparable performance for IC (ViT).\n\n**Table 3. Comparison with WeGeFT on NLU (RoBERTa)**\n|Model|Trainable Param |CoLA|MRPC|QNLI|RTE|SST-2|STS-B|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|RoBERTa-Base (WeGeFT)|49K|63.5±1.3| **89.46±0.49**|91.23±0.42|**78.6±1.60**|94.13±0.46|**90.5±0.06**|84.57|\n| RoBERTa-Base (AdaMSS)|**32K**|**64.5±1.1**|88.8 ± 1.4|**92.4±0.1**|77.3±0.7|**94.6±0.2**|90.4±0.1|**84.7**|\n| RoBERTa-Large (WeGeFT)|65K|64.04±1.99|75.74±7.67|93.7±0.3|53.55±1.19|94.95±0.34|91.37±0.26|78.89|\n|RoBERTa-Large(AdaMSS)|**45K**|**67.2 ± 1.2**|**90.3±0.5**|**94.5±0.1**| **87.1±2.1**|**96.1±0.0**|**91.9±0.0**|**87.9**|\n\n**Table 4. Comparison with WeGeFT on IC (ViT)**\n| Model| Trainable Param |CIFAR10|CIFAR100| StanfordCars| OxfordPets| FGVC| EuroSAT| RESISC45|Avg.|\n|-|-|-|-|-|-|-|-|-|-|\n|ViT-Base(WeGeFT)|49K|98.46±0.06|91.46±0.19|76.18±0.20 |92.71±0.15|51.82±0.86|94.92±6.89|93.03±0.19|85.51|\n|ViT-Base(AdaMSS$_{base}$)|**42K**| **98.71±0.07**|**91.90±0.1**|**78.98±0.2**|**93.91±0.2**|**53.2±0.4** |**98.64±0.07**|**93.62±0.08**|**86.99**|\n|ViT-Large (WeGeFT)|**204K**|99.06±0.07 | 92.69±0.19|83.96±0.14|94.49±0.17|60.82±0.46|98.34±0.10|94.49±0.26 |89.12|\n| ViT-Large (AdaMSS)|241K|**99.12±0.00**|**93.22±0.1**|**85.24±0.3**|**94.87±0.1**|**64.31±0.4**|**98.93±0.1**|**94.87±0.1**|**90.13**|\n\n---\n**R9. Which modules are fine-tuned?**\n\nFor both **IC** and **NLU**, we fine-tune **only the Query and Value** projection modules across all methods for a fair comparison. For **Natural Language Generation**, we evaluate two settings: fine-tuning **only the Query and Value** projections, and fine-tuning **all major projection modules**.\n\n---\n We thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify."}]

Author rebuttal:

Meta-review: AdaMSS is a novel PEFT method that segments pretrained weights into multiple low-rank subspaces, overcoming the limited expressiveness of single-subspace approaches. This multi-subspace design enables richer adaptation without significantly increasing trainable parameters, striking a better balance between expressiveness and efficiency. The method further introduces adaptive freezing to focus updates on the most important subspaces and provides a theoretical generalization bound with lower Gaussian complexity. Experiments across vision and NLP tasks confirm strong gains over existing PEFT methods while using fewer parameters.

The rebuttal addressed concerns with added efficiency metrics, expanded evaluation (Mistral-7B, Gemma-7B), stronger ablations, and clarified theory. While exposition could be clearer and results on the very latest models are pending, the contributions are timely, impactful, and well-supported, making the work valuable to the PEFT and efficient LLM communities and likely to inspire further research on structured subspace-based fine-tuning.
Decision: Accept (poster)