SumRA: Parameter Efficient Fine-tuning with Singular Value Decomposition and Summed Orthogonal Basis
Authors: Chin Yuen Kwok
Yongsen Zheng
Jia Qi Yip
Kwok-Yan Lam
Eng Siong Chng
Venue: ICLR 2026 Poster (2026)
Keywords: low rank adaptation
automatic speech recognition
model adaptation
parameter efficient fine tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The paper introduces SumRA, a new PEFT method that expands LoRA’s representational capacity by initializing low-rank matrices with sums of multiple singular vectors instead of only the leading ones.

Abstract: Parameter-efficient fine-tuning (PEFT) aims to adapt large pretrained speech models using fewer trainable parameters while maintaining performance. Low-Rank Adaptation (LoRA) achieves this by decomposing weight updates into two low-rank matrices, $A$ and $B$, such that $W'=W_0+BA$. Previous studies showed that freezing $A$ and only updating $B$ can reduce trainable parameters and achieve performance close to standard LoRA, where $A$ is initialized using the principal singular vectors of $W_0$ obtained via singular value decomposition (SVD). However, because $A$ is typically initialized with only the leading singular vectors, its representation capacity is confined to a narrow subspace of the model’s knowledge. To overcome this limitation, we propose SumRA, which initializes each row of $A$ as a sum of multiple singular vectors chosen from beyond the leading components, thereby enabling $A$ to influence a larger portion of the model’s knowledge space. Experiments on multilingual automatic speech recognition (ASR) tasks show that by adapting Whisper to five new languages from Common Voice with only 10 hours of data each, our method improves word error rate from 14.42\% to 12.41\% over LoRA baselines while using 50\% less trainable parameters.

Published as a conference paper at ICLR 2025
SUMRA: PARAMETER EFFICIENT FINE-TUNING WITH SINGULAR VALUE DECOMPOSITION AND SUMMED ORTHOGONAL BASIS
Chin Yuen Kwokl.2 Yongsen Zhengl.2,, Jia Qi Yip', Kwok-Yan Lam Eng Siong Chng² 'Digital Trust Centre, Nanyang Technological University. Singapore "College of Computing and Data Science, Nanyang Technological University, Singapore fkwok0062, (laqi006eee ntu. edu.sg fyongsen. zheng, kwokyan. lam, aseschng)@ntu.eeen edt..
ABSTRACT
Parameter-efficient fine-tuning (PEFT) aims to adapt large pretrained speech mod- els using fewer trainable parameters while maintaining performance. Low-Rank Adaptation (LoRA) achieves this by expressing the weight update to a pretrained matrix Wo as the product of two low-rank matrices, A and B, yielding the adapted weight W" = Wo BA. Previous studies showed that freezing A and only updat- ing B can reduce trainable parameters and achieve performance close to standard LoRA, where A is initialized using the principal singular vectors of Wi obtained via singular value docomposition (SVD). However, because A is typically initial- ized with only the leading singular vectors, its representation capacity is confined narrow subspace of the model's knowledge. To overcome this limitation, we propose SumRA, which initializes each row of as a sum of multiple singu- lar vectors chosen from beyond the leading components, thereby enabling A influence a larger portion of the model's knowledge space. Experiments on multi- lingual automatic speech recognition (ASR) tasks show that by adapting Whisper to five new languages from Common Voice with only 10 hours of data each, our method improves word error rate from 14.42% to 12.41 % over LoRA baselines while using 50% less trainable parameters.
INTRODUCTION
Cnrameter-Efficien Tunnng (PEFT) provides an effective strategy for adapting large pretrained models to diverse downstream tasks while minimizing additional parameters and computational overhead WWag et al. (2024); Chu et al. (2023); Hu et al.] (2024b). Among PEFT methods, Low- Rank Adaptation (LoRA) (Hu al.) 2022, 2024a) is the most classical approach, adapting large speech models by inserting and updating only small low-rank matrices (Fig. 33). This design sub- stantially reduces the storage cost of additional parameters (Lin et 2024 for example, a single LoRA for B--paammetrr model typically requires only tens of megabytes. However, when scaling to thousands (Brüel Gabrie-nn et al. 2024) or even millions (Parada et al., 2025) of LoRAs, total storage can still reach up to 10 TB, posing significant scalability challenges. This issue is espe- cially critical in multilingual (Della Libera et al. 2024) and personalized (Joseph & Baby 2024) Automatic Speech Recognition (ASR), where each user requires a dedicated LoRA tailored to their language proficiency, accent, and domain-spectio
vocabulary (Kwok et al., 22025] Shim et al. 2024
Rolland & Abad, 2024). At scale, large user base can therefore require thousands or millions of
LoRAs, greatly amplitying storage and deployment costs.
Multiple lines of work have explored improvements to LoRA. AdaLoRA Zhang et al. (203b))
dynamically adjusts (and progressively reduces) the rank of the LoRA matrices during training-
LoRA+ LoRA+[Hayou al. (2024) improves LoRA by applying differential learning rates the projection matrices A and B. Complementary to these methods, LoRA-FA (Zhang et al. 2023a) reduces the
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Full fine-tuning
B. LoRA
C. PISSA
D. SumRA (ours)
Y
Y
Y
SVDDW
UEV
Pretrained Matrix
Pretrained Matrix
B ==0
Pretrained Matrix
Pretrained Matrix
=
W, Bdxk
Wo Raxk
W R
A A~2(0,03)
A AmEpea
Sum(e
X
X
X
X
Comparison of full fine-tuning. LoRA, PiSSA, and our SumRA method. Only the pre- trained matrix Wo of a linear layer is shown for simplicity. Blue modules indicate frozen parameters, orange modules are trainable. A) Full fine-tuning updates all of Wo: B) LoRA freezes Wo and trains two low-rank matrices; C) PiSSA initializes A and B from singular vectors; D) SumRA enhances PiSSA with a novel initialization of (see Fig. and
Figure 1:
trainable parameters by freezing the projection-down matrix A and updating only the projection-up matrix B, thus improving efficiency and reducing memory usage. Nevertheless, LoRA-FA typically initializes A randomly, missing the opportunity to leverage the pretrained knowledge embedded in Wo. To address this, PiSSA (Meng et al. 2024) proposes more principled initialization approach: unlike LoRA which initializes from normal distributions (Fig. 2 PiSSA initializes A using the leading singular vectors from the singular value decomposition (SVD) of Wo (Fig. (2)). This allows the low rank matrices to be fine-tuned in the direction that Wo has the most significant stretching or impact.
Initialization stratagy for LoRA's metris
source
ta rege JAssume rank R
SVD and principal axes:
e gaxk
(VO,
Pretrained
SVD
(A)
A a0,
Matrix
UE Baxe
gaok
g1/2i' Bkok
JAssume rank A Rank
Each axis encodes the knowlecige of specific concept:
(B)
A-D
encodes
Concepts in medical domain
encodes
Concepts in
g:/2y1 E kkok
Vsssume rank
Global knowledga update across multiple axes
E Uk 2xk
New knowledge Prefer "ae" over "e"
(D)
Update this knowledge for muttiple concept groups
sum
Transcribe "haemoglobin" instead of "hemoglobin"
summed now multiple concept groups Sum in interleaving pattern is crucial for performance gain
Transcribe "aesthetic" inetead "esthetic"
Figure 2: Initialization strategies for LoRA's A matrix. Pre-trained weight matrix is Wo E m
and is LoRA's rank. SVD(W)) UEVT. A) The A matrix is initialized from normal distribution. B) The A matrix is initialized from the top r rows (singular vectors) ///2/T, C) Assume that each singular vector encodes knowledge relevant
specific subset of the speech model's 's vocabulary. Assigning a single singular vector to each row of A limits that row's influence one subset at a time. D) To address this, we propose to sum multiple rows of 1/2 into each row of A.
In PiSSA initialization, each singular vector extracted from Wo encodes only knowledge associ- ated with specific concept (Merullo et al.) 2024 Pan et al.] 2024). Since PiSSA initializes A using only the top few singular vectors-typicee typically less than 5% of all available ones (Yang et al.) 2024)-a0 the resulting initialization restricts A's influence sequently, the adaptation primarily targets a limited portion of the model's learned knowledge. To overcome this limitation, propose summing multiple singular vectors into each of A, as il- lustrated in Fig. 2D. By aggregating vectors, A gains the capacity to represent a broader range of concepts, thereby extending its influence larger portion of the model's knowledge space. This
narrow set of concept groups. Con-
2

---

Published as a conference paper at ICLR 2025
enhances the effectivvness of adaptation, particularly for tasks that require global rather than local- ized knowledge transfer. By summing singular vectors, computations that are originally performed on the individual vectors combined. This mechanism of sharing model computations actoss mul- tiple concepts or dimensions has demonstrated effectiveness in related work (Reneau et al. 2025). To improve the effectiveness of the vector aggregation, we further propose two summing strate- gics-intericave sum and greedy sum (Figs 3 and C)-that distribute singular vectors across rows so that the sum of singular values each row balanced. This ensures that highly important vectors are evenly assigned rather than concentrated in single row, reducing interference between summed vectors. comparison between LoRA and our SumRA provided in TablcT
(A) block
(B) Our interleave
(C) Our gueedy
g1/201 R
E3/yy1 E Rxc
g4/yy1 koo
IR
A Bank
Re
== Sum(X-Ay
skewed importance
balanced importance
balanced importance
Asgurn each of assigned with an importanca value
Total importance Vuluec
Total valuec 6
Inta.ingortinee valee:
Importance
tanoo
valas:0
Figure 3: Strategies to E1/2TT into matrix A. To improve global knowledge update dis- cussed in Fig. [ we propose to compress and sum all singular vectors into A. However, summing the most important singular vectors the row shown in A) block is unideal causes destructive interference between them. Therefore, we propose B) interleave sum and C) greedy sum that evenly assign the important singular vectors the rows of A.
Table I: Comparison between LoRA and our SumRA method to update a linear layer's pretrained weight Wo Rd h and x the input and outputs of the linear layer respectively. is the rank of the low rank matrices and AW is the weight update. In this table. bold highlights the model's trainable component during adaptation. SVD is performed on Wo to obtain U, Σ and V.
LoRA
our SumRA
Forward
(Wo AW)x
(Wo + BA)x
(Wo AW) x = (Wo + BA)x
B 0 Rdxr
B 0 Rdxr
Initialization
H reshape /2 R x
A A~
sum (H, dim = 0) R
Fine-tune B while freezing A
Fine-tune both B and A Store both B and A
Comparison
Store separate Bs
Share As for each task
Empirical results on low-resource multilingual ASR tasks validate the effectiveness of SumRA. By adapting Whisper five new languages from Common Voice-each with only 10 hours of data-SumRA achieves 50% reduction in trainable parameters by freezing A, yet consistently improvement improvement in word error rate despite using 50% less memory to store additional parameters. These findings highlight value structured, knowledge-awant initialization strategies for scalable and effective PEFT in multilingual and low-resource settings as shown in Fig.
outperforms LoRA, LoRA-FA and PiSSA, achieving up to an 16% relative
2 PRELIMINARY
This section reviews PEFT techniques relevant to our work. We begin with LoRA (Secion 2.1 and its memory-optimized variant L-RA-FA (Secion 2.2). both designed to reduce the number of train-

---

Published as a conference paper at ICLR 2025
LoRA/Pissa/CorDA
Additional Modules for N
B. SumRA(ours)
Additional Modules
Use case 1: Limited bandwidth
for
Server
End user device
Pretrained Matrix W E gdxk
Pretrained Matrix
U
E [Rd*
Use
(um((24/2p
Millions of LoRAs adapt
8y
1M uscrs
X
X
Bx
1K tasks
Figure 4: Memory cost comparison of LoRA/PiSSA/Corft with our SumRA method. Blue/orange modules are frozen/updated. Storage cost is reduced instead of A) having separate trainable A for each task, B) the same frozen A can be shared between different tasks in SumRA.
able parameters. We then introduce PiSSA (Secion 2.3 and CorDA (Secion 2.4 which improve LoRA's initialization using SVD enhance adaptation efficiency and task alignment.
2.1 LORA
Inspired by findings on the low intrinsic dimensionality of language models (Aghajanyan et al,, 2020). LoRA constrains weight updates low-rank subspace to reduce the number of trainable parameters during fine-tuning, Given pre-trained weight matrix Wo Baxk LoRA approximates the update as Wo AW W, + BA, where B E Rdxr and E T with r < min(d, k)) During training, Wo remains frozen and only A, B are updated. The forward computation becomes:
h
Wa BAr,
where h E Rd and R are the input and output activations, and is constant in Tuning is roughly the same as tuning the learning rate of LoRA (Hu et al. 2022).
2.2 LORA-FA
While LoRA reduces the number of trainable parameters, it still incurs substantial memory overhead in fineggraine,, personalized settings (Parada et al. 2025). LoRA-FA (Zhang et al. (2023a) further reduces memory by freezing both Wo and i LoRA, updating only B. In this setup. is randomly initialized from normal distribution and remains fixed, while B is initialized to zero, similar to standard LoRA.
This design is inspired by findings that LoRA can be trained effectively using fixed, random basis (oohpaaygaani et al. 2023). Given than frozen, assuming an RQ decomposition RQ and defining B = BR, the low-rank term BAx in Eq. becomes:
BAr BQx,
where is an orthonormal matrix. In this formulation, training B effectively tunes B operate on features projected by the fixed, random basis in Q, enabling efficient adaptation with minimal memory usage.
2.3 PISSA
Building LoRA, PiSSA improves the initialization of the adapter matrices A and B using the principal singular vectors of the pre-trained weight matrix Wi. Given the SVD of Wo E d as Wo USV PiSSA uses the top-r components to initialize:
B Upai))
1/2
[Branr] Rddx
(3)
1/2 A 2 [Iraa)] [arr]] E Brxk
(4)
4

---

Published as a conference paper at ICLR 2025
The remaining components form the residual matrix:
W = (ra..)) [.r:)
(5)
which is frozen during fine-tuning. This preserves the original model output at initialization:
h Wox = (W + B BA)x,
(6)
Compared to LORA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance.
2.4 CORDA
While PiSSA performs SVD on the pre-trained weight Wo. CorDA rescales Wo before applying SVD. The core idea is to compute the covariance matrix C = XXT from input activations X R collected from n target task data samples, and preform SVD on WyC, which is a rescaled version of Wo oriented by the target task data distribution:
SVD(W;C) UEVT
To preserve initial model hehavior, the weight Wo is reconstructed as W:
W SVD(WWC)C-+ UEV C- UEİT
with VT = VTC- is is not invertible, is added its diagonal. This decomposition allows the extracted principal singular vectors to be aligned with the task-relevant directions in the activation space and allows for more targeted adaptation.
METHOD
Building upon PiSSA and CorDA, we first present our SumRA method as an improved LoRA matrix initialization strategy. It initializes each row of sum of multiple singular vectors obtained from the SVD of Wo (Section 3.1). Following that, three strategies are presented to distribute singular vectors across rows of A improve performance in Section 3.2
3.1 SUMMING MULTIPLE SINGULAR VECTORS
The intuition behind our method is as follows. PiSSA demonstrates that initializing the low-rank matrices in LoRA using the top singular vectors of pretrained weight matrix Wo guides fine- tuning along directions where Wo has the most sigmificant impact-ic.. directions associated with the largest singular velulinresuliing in improved performance. However, this approach may still be suboptimal, as (Staats et al. (0244) argue that singular vectors corresponding to smaller singular values can also be important for adaptation. Ideally, the A matrix in LoRA should be initialized using all singular vectors to reduce information loss. However, it is infeasible to assign each singular vector to a unique row of A as the rank of A is usually smaller than the rank of Wy. To address this, propose strategy that sums multiple singular vectors into each row of A. This allows to compress the full set of singular vectors into a lower rank matrix without discarding potentially useful directions for adaptation.
Specifically, our method first performs SVD on the pre-trained weight Wo similar to PISSA:
SVD(W)) = UEVT Σ a.
where o, is the i-th diagonal element in E, u, is the i-th column U and v, is the i-th column of V. Then, given a; E R is the i-th row of A E Rrik and S; be the subset of column indexes of V
that is assigned to a, we initialize a, as:
a
VoNY,
(10)
JES.
To compress the full set of singular vectors into A, each column index of V must be assigned to exactly one of the subset St such that Ul S, [1,..., k). Furthermore, all the subsets should be disjoint such that S, Sy 0 for all < i.j be orthogonal to each other such that a, 1 a, for all 1 S i.j S r and * j. The orthogonality constraint improves the optimization efficiency during model fine-tuning Cao & Song (004)..
and i # j. This allows all rows of A to
5

---

Published as a conference paper at ICLR 2025
3.2 SUMMATION STRATEGY
However, is inevitable that there will be interference (information loss) between the singular vec- tors if they are summed together. To prevent the destructive interference between the more important vectors (estimated by the singular values), they should not be summed into the same row as shown in Fig. 3A.
To address this, we compare three summing strategies. The first strategy naive block sum as shown in Fig. 3A.. Specifically, assume k is divisible by r, this strategy assigns singular vectors by having:
(i - 1)k (i - 1)k 1,
ik }
S:
+
(1I)
However, block sum is unideal it assigns all the important singular vectors to the same row of and causes destructive interference between them. Instead, they should be evenly assigned to prevent interference. Specifically, define the load Ly of a; as the sum of the singular values of all singular vectors assigned to that row:
Lj Σ9, oi iESy
(12)
The distribution most even when the maximum row load Lmax = max, L, is minimized. To reduce Lmax. we propose interleave and greedy sum shown in Fig. BB and C that evenly assign the important singular vectors to the rows of A. For interleave sum, we have:
k S = (0 +
1 (r 1) k + +1}
(13)
Compared to interleave sum, we further derived greedy sum, optimal summation strategy that always results the most even distribution. Specifically, Let there be K singular vectors (scaled by Σ) in ΣV", each associated with singular value, sorted in non-increasing order. Assume we are assigning the singular vectors to rows of one by one. Then, our greedy sum strategy each step, assign the next singular vector (in decreasing order of singular value) the with the currently minimum load. We proof that greedy sum produces an assignment that minimizes Lmax in Appendix A.1
at
4 CONNECTION TO MODEL AVERAGING
Our approach, which sums multiple singular vectors together, closely related to model averaging. Instead of compressing all singular vectors into a single low-rank matrix A, naive alternative is to initialize multiple versions of A using different subsets of singular vectors from D1/2TT and independently adapt the pre-trained model into multiple variants. Since each model is fine- tuned in a different direction-owing to distinct singular vector initialization-tee may capture complementary aspects of the adapted task. Model averaging can then be applied to merge these models, leveraging their diverse adaptations for improved performance. As illustrated in Fig. [5] our SumRA interleave-sum method achieves a similar effect, but more efficient manner: we sum the LoRA matrices before adaptation, effectively integrating diverse adaptation directions within single model.
EXPERIMENT
In this section, we begin by introducing the datasets and training configuration used for multilingual ASR adaptation in Sections 5.1 5.2 respectively. We then perform a comprehensive evaluation of SumRA method, where Sections.3] presents the main results on multilingual ASR adaptation and analyze the performance of our method under different summation strategies as discussed in Section 3.2

---

Published as a conference paper at ICLR 2025
Our SumRRA interleave sum method is closely related model averaging:
g1/2VT E I kxk
(C)
Where
sum
A R
(A)
A
Sum (g.v2y))
be seen summing multiple vers ions of LoRA weights together
A2
(B)
(u
=
A1
A2
A3
A3
Figure 5: Comparison between our SumRA interleave sum method with model averaging. A) As an example, consider our SumRA method which interleave sum the matrix / lg**k to ini- tialize the LoRA A matrix. B) This can be seen as performing weight summation three different LoRA A matrices, where C) each of the matrix is initialized from different part of 11/VTT
5.1 DATASET
Building on previous studies (Yu et al. 2023 Della Libera et al. 2024), we evaluate our approach using subset of the widely-used Common Voice MASR dataset (Ardila al. 2020 This pub- licly available dataset, collected via crowd-sourcing, contains short audio recordings paired with transcriptions across 108 languages, amounting to total of 17,690 validated hours. The data is or- ganized into training, validation, and test splits. Following the methodology of (Della Libera et al.) 2024), we select five languages for our experiments, including Esperanto (eo). Meadow Mari (mhr), Central Kurmanji Kurdish (kmr), Frisian (fy-NL), and Interlingua (ia). These languages are not seen in the training set of our pre-trained model. For each language, we follow (Della Libera et al. 2024) to partition the training, validation, and test sets consisting of ten hours, one hour, and one hour of audio, respectively.
5.2 TRAINING CONFIGURATION
Whisper (Radford al., 2023) is chosen the MSAR model in our study. Whisper employs an encoder-decoder transformer architecture (Gulati et al. 2020), where the encoder processes input Mel spectrograms to extract audio features. The regr-ssgress decoder then generates transcrip- tions conditioned on the encoder's hidden representations, following multitask framework that leverages special tokens task specifiers.
We adapt both small and 1arge-v2 variants of the Whisper model to accommodate new languages. Model adaptation is performed over two epochs with a training batch size of Validation carried out every 1/8 epoch across all methods. We use the Adam optimizer (Loshchilov & Hutter] 2019) in conjunction with a variant of the ReduceLROnPlateee the linear layers of all the feed-forward and attention layers in the decoder. for LoRA is equal to the rank following Lee et al. (Lee et al. 2023) Only the parameters the LoRA-basees modules and the normalization layers (Ba et al. 2016) are updated during adaptation. All experiments employ greedy decoding strategy.
learning rate scheduler]] LoRA is added to
We compare method against several LoRA- A-baeed adaptation baselines. including vanilla LoRA (Hu et al., 2022). : VeRA (Kopiczko et al. 2023). LoRA-FA (Zhang et al.) 2023a), DoRA (Liu et al.) 2024), PiSSA (Meng al.) (2024), and CorDA Yang et al. 2024). To further validate the effective- ness of these adaptation strategies, we also report results from the unadapted Whisper model and fully fine-tuned (FT) Whisper model, where all parameters are updated during adaptation.
mww.:moooii.....m crg/en
https::/ hea.teecss........oo.o.oo..o.... 10/ www.atest/tttt//eet//e modul ledulee// speschbrain/ nnet nnet/scheduleen html hae.

---

Published as a conference paper at ICLR 2025
We evaluate our methods using word enror rate (WER), which is a common metric used to evaluate the accuracy of ASR systems. It measures the percentage of words that are incorrectly recognized in transcription compared to reference (ground-truth) transcription.
Table 2: Comparison of SumRA with existing LoRA-based methods on the multilingual ASR adaptation task using WER. Lower is better. The best results are highlighted in bold, while the second-best results underlined. The additional storage cost (param.) does not include weights that can be shared between tasks. The differences in WERs are shown in subscripts, relative to LoRA's WER.
LoRA Conf.
WER()
Method
Model
parum.
rank
ia
fy-NL
mhr
kmr
unadapted
94.37
74.53
1.05e+2
1.1e+22
1.14e++2
FT
18.89
13.79
28.69
32.37
39.63
LoRA
whisper-smi small
0.5M
2
28.76 -.om
19.99
50.81 -c.a
53.75 4n
60.10....m
VeRA
0.2M
2
38,99
27.42
67.81
68.08
76.42
VeRA
0.3M
768
30.54
21.30
55.63 -1.3
56.60 (s.00
63.71
LoRA-FA
0.4M
2
39.92
29.14 -907s
68.54-15m
69.85
poms
77.63 1.633-2m
DoRA
0.7M
2
20.15 +o.rs
52.30-2.am
53.88 20.295
61,30-s.v
PISSA
0.5M
2
29.20
19.23
49.72
51.29
58.68 2.45
CorDA
0.5M
27.05 -6.00s
18.96
47.66
c.am
51.08
58.85 23115
SumRA (ours)
0.4M
2
26.29 A.05
17.23 137
44.92 12m
48.49-...e
54.32 0.005
unadapted
94.37
74.53
1.05e++2
1.15e++2
1.144e+2
FT
18.89
13.79
28.69
32,37
39.63
LoRA
whisper-smal smaler--mall
7.7M
32
23.39.
15.31 mo.m
39.34
40.63
48.51 -..95
VeRA
0.2M
36,93
25.66 (aRR
65.52 -005
63.98 109
73.74 -305
VeRA
0.3M
768
30.54 -6.95
-6.30
55.63 -5.57
56.60 (5:255
63.71 -6.055
LoRA FA
3.9M
32
26.77 1495
17.04 117
13.98 1775
46.55 10%
55.45 1445
DoRA
7,8M
32
21.79-
12.53
37.07
38.95
43,18
PISSA
7.7M
32
21.79 9999
13.95
34.93
38.24
46.23
4.75
CorDA
7.7M
32
21.12 175
13.11 1198
34.35
36.53 1r
44.85 1:005
SumRA A (ours)
3.9M
32
20.77
13.38
33.37
36.30 115
44.47 6.355
unadapted
62,54
47.96
1.05e++2
81.28
1.0 L.02e++2
FT
15.59
13.20
26.05
30,60
36.86
LoRA
whisger- large-/2
2.4M
15.96.6005
9.85 (0.05)
29. 29.24 -1:475
39.02, (z)
44.55 (6.055
VeRA
0.8M
2
64.18 mm9
47.14 (.58
41.78 1375
48.90 (009
55.06 1445
VeRA
1.2M
1280
17.18 mass
11.15
32.88 1255
39.15
46.29
LORA-FA
1.6M
2
61.42
49.44
44.68
52.52
58.74
DoRA
2.9M
2
16.76
11.10 11ri
33.99 1108
39.53 e1 599
46.08 [s
PISSA
2.4M
2
15.12-a.s0
9.70 896
27. 84 4:806
36.40 1m
38.
12
CorDA
2.4M
2
15,55
8.98 s0
27. 27.94 1.3%
34.83 110k
39.37
SumRA (ours)
1.6M
14.55
9.30
25.83
34.72 11%
38.63
unadapted
62.54
47.96
1.05e++2
81.28
1.02e++2
FT
15,59
13.20
26.05
30.60
36.86
LoRA
whispor- Jargo-12
34.3M
32
14.42 mno6s
8.67.uum
24.75 -c.at
32,39
37.72....5)
VeRA
0.8M
32
20.68
15.83
43.60
49,29
57.73-ss5:
VeRA
1.2M
1280
17.18
11.15
32.88
39.15 400a
46.29. 4..9%
LoRA-FA
17.6M
32
16.42-135
10.31 +10%
30.69
37.55 115%
44.24-um
DoRA
34.9M
32
13.45 -..m
8.28 4..%
23.38-1.88
29.67
35,59-s.mM
PISSA
34.3M
32
13.00 -uol
8.82
22.43
29.97
34.26 9.2%
CorDA
34.3M
13.13-oum
9.18
22.96-1.ax
29.20
36.33-3.1%
SumRA (ours)
17.6M
32
12.41
8.17 5..0
22.27
1009
27.19
1005
34.21 9.3%
5.3
PERFORMANCE ON MULTILINGUAL ASR
Table shows the results of adapting whisper-small and whisper-large-v2 to five new languages using different LoRA- methods and using different ranks (2 and 32). The results show that before adaptation, Whisper generally achieves close to 100% WER as it has never been pre-trained to transcribe the new languages. After adaptation, the WER drops significantly. For whisper-small, filetffintunig (FT) generally achieves better WER than the LoRA methods as all the model weights can be updated accommodate new language knowledge. However, for larger models like whisper-
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of different summation strategies. WERs of adapting whisper-small using SuMRA method with rank 32 is shown. Lower is better. The additional storage cost (param.) does include weights that can be shared between tasks.
Method
param.
co
ia
fy-NL
mhr
kmr
LoRA
7.7M
23.39
15.31
39.34
40.63
48.51
SumRA (block sum)
3.9M
21.68
13.91
35.38
37.35
47.30
SumRA (interleave sum)
3.9M
20.77
13.38
33.37
36.30
44.47
SumRA (greedy sum)
3.9M
20.73
13.16
33.91
37.53
44.72
large-v2, adaptation more prone to overfitting, and some LoRA-based methods performs better than FT as they can reduce overfitting by constraining weight updates.
For LoRA-based methods using rank 2, Vera, LoRA-FA and DoRA generally perform worse than LoRA as they are more restrictive the weight update than LoRA. As the rank is increased to 32, the trainable parameters increases and the adaptation is more prone to overfitting than the rank 2 case. As result, DoRA generally performs better than LoRA for rank 32 it better reduce overfitting. Next, PiSSA, CorDA and SumRA generally performs better than LoRA in all cases we hypothesize that the methods can reduce overfitting by initializing the LoRA matrices with singular vectors, and do not reduce the learning capacity of the matrices as no additional weight update constraints are imposed during training.
Finally, the results show that our SumRA method generally outperforms all other baseline LoRA- based methods while have low memory cost. This is because our method can share the LoRA A matrices between tasks, thus adding more tasks does not increase the memory cost of the A matrices. Furthermore, our SumRA method can outperform LoRA by up 14%, and outperform CorDA by up to 11 1, showing the effectiveness of our initialization strategy for LoRA matrix A.
5.4 SUMMATION STRATEGY ANALYSIS
We further perform ablation study to study the effect of different summation strategies. The results Table 3 shows that block sum performs worse than all other summation strategies as sums all the important singular vectors into the same row of A. Our interleave sum and greedy sum methods show consistent improvements over block sum as they can distribute the important singular vectors more evenly across the rows of A, validating the effectiveness of our summation strategies.
5.5 SUMRA'S S EFFECT UNDER DIFFERENT DATA SCALES
To broaden the experimental scope, evaluate SumRA under different data regimes (10h, 50h, and 100h) when adapting whisper-small to Esperanto, a new language with a larger available train- ing corpus. Table reports the resulting WER. Compared LoRA, SumRA reduces WER from 23.39% to 20.77% in the 10h setting and from 15.20% to 14.49% in the 50h setting, In contrast, the improvement diminishes at 100h. These results indicate that SumRA provides more substantial gains in low-resource scenarios. Furthermore, the results in Table row four shows that additional WER improvement is achieved when the previously frozen down-projection matrix A (as shown in Fig. [P) is trained, indicating that further performance gains can be obtained by updating the parameters in A. However, this improvement comes at the cost increased trainable parameters and higher training overhead.
6 LIMITATION
Our approach is most effective for adapting global attributes, such as accents or speaking styles, that influence a large portion of the vocabulary. In these scenarios, summing multiple singular vectors enables coordinated parameter updates across many vocabulary subsets, providing clear advantages methods that adapt only a small fraction of vectors. However, for local adaptations such as

---

Published as a conference paper at ICLR 2025
Table 4: WER (%) Esperanto under different training data scales. Lower is better.
Method
10h
50h
100h
FT
18.89
15.31
13.62
LoRA
23.39
15.20
13.28
SumRA (freeze matrix A)
20.77
14.49
13.39
SumRA (train matrix A)
20.14
13.75
13.02
adding a small set of domain-specific terms terms-this broad update mechanism offers less benefit, as changes are required for only limited subset of the vocabulary.
Consistent with this intuition, our preliminary experiments show that SumRA does not yield substan- tial improvements when adapting LLaMA models Touvron al. (2023) to GLUE tasks Waang et al. (2018) We hypothesize that these tasks primarily require learning task-specific decision boundaries over existing representations, rather than adapting global model attributes. As they do not involve broad shifts in representation space (e.g., accent speaking style), they resemble localized adapta- tions, where SumRA's coordinated global update mechanism provides limited advantage.
7
CONCLUSION
We introduced principled initialization strategy for LoRA that leverages singular vectors from pretraining improve adaptation. By summing orthogonal components and freezing our method reduces trainable parameters while enhancing performance on low-resource multilingual ASR tasks.
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclu- sions recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority,
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 2020.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A mussively-multilingue speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218 4222, 2020.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ 2016. htLLattpss/pppp.smmatiiicccccc
semant wwschoaatccccccacccc.c.cma. org. ID:
Rickard Gabri-.aa,,,,sson,, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, and Justin Solomon. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.000666 2024.
Yang Cao and Zhao Song. Sorsa: Singular values and orthonormal regularized singular vectors adaptation of large language models. arXiv preprint w4Xiv:2409.00055, 2024.
Timothy Chu, Zhao Song, and Chiwun Yang. Fine-tune language models to approximate unbiased in-context learning. arXiv preprint arXiv:2310.0333 2023.
Luca Della Libera, Poonch Mousavi. Salah Zaiem, Cem Subakan, and Mirco Ravanelli. CL-MASR: A continual learning benchmark for multilingual ast. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
10

---

Published as a conference paper at ICLR 2025
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: anvolution-augmennt transformer for speech recognition. In Interspeech 2020, pp. 5036-5040, 2020. doi: 10.21437/ Satrspeech.2020--22
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint ar arXiv:2402. /2354, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2::, 2022.
Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo, Zhao Song, and Han Liu. Computational limits of low-rank adaptation (lora) fine-tuning for transformer models. arXiv preprinl arXiv:2406.08 03 036,, 2024a.
Jerry Yao-Chich Hu, Wei-Po Wang, Ammar Gilani, Chenyang Li, Zhao Song, and Han Liu. Funda- mental limits of prompt tuning transformers: Universality, capacity and efficiency. arXiv preprint arXiv:2411.16525, 2024b.
George Joseph and Arun Baby. Speaker personalization for automatic speech recognition us- ing weight-decomposed low-rank adaptation. Interspeech 2024, 2024. URL URL https://ap/:
semant htt.pphc.........o.ooorraaaaaaa.
Soroush Abbasi Koohpayegani, KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pir- siavash. Nola: Compressing lora using linear combination of random basis. arXiv preprint arXiv:2310.02555 2023.
Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11455: 2023.
Chin Yuen Kwok, Hexin Liu, Jia Qi Yip. Sheng Li, and Eng Siong Chng. A two-stage lora strategy for expanding language capabilities in multilingual asr models. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. ArXiv, Abs/2308.07331 2023.
Tzu-Han Lin, How-Shing Wang, Yung Weng, Kuang-Chen Peng. Zih-Ching Chen, and Hung- yi Lee. Peft for speech: Unveiling optimal placement, merging strategies, and ensemble tech- niques. 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 705-709. IEEE, 2024.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representationss 2019.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular
vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-12107, 2024.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer com- munication in transformer language models. ArXiv, hts/2406.09519, 2024. URL https:
//api. weemanticsccoleee wwtp/orrgsusa.ooeooooo ID 27022205255
Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissecting query-key interaction in vi- sion transformers. ArXiv, abs/2405.1 148800 2024. URI.https://app http::///pppp.aaaa.aaaa/a semant wewanttccccoolll....... casholar.
htte/CooppsID::2000500
Pablo Peso Parada, Spyros Fontalis, Md Asif Jalal, Karthikeyan Saravanan, Anastasios Drosou, Mete Ozay, Gil Ho Lee, Jungin Lee, and Seokyeong Jung. persoda: Personalized data augmentation for personalized ast. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEFE, 2025.

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Ma- chine Learning, pp. 28492-28518. PMLR, 2023.
Alex Reneau, Jerry Yao-Chieh Hu, Zhongfang Zhuang, Ting-Chun Liu, Xiang He, Judah Goldfeder, Nadav Timor, Allen G Roush, and Ravid Shwartz-Ziv. Ndlinear: Preserving multi-dimensional structure for parameter-efficient neural networks. arXiv preprint arXiv:2503.17353 2025.
Thomas Rolland and Alberto Abad. aaaate-ad novel transformer-basee parameter efficient transfer learning approach for children's 's automatic speech recognition. In Interspeech, volume 2024, pp. 2370-2374, 2024.
Kyuhong Shim, Jinkyu Lee, and Hyunjae Kim. Leveraging adapter for parameter-efficiet asr en- coder. In Proc. Interspeech 2024, pp. 2380-23844 2024.
Max Staats, Matthias Thamm, and Bernd Rosenow. Small singular values matter: A random ma- trix analysis of transformer models. 2024. URL httpss://ap.. stc.p//cco..mmmttccccccc..o... org/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arxiv 2023. arXiv preprint arXiv:2302.13977 10, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-tase benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pp. 353-355, 2018.
Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter-efficient fine-tuning in large models: A survey of methodologics. arXiv preprint arXiv:2410.19878 2024.
Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models. ArXiv, abs/2406.05223, 2024. URL https: /api. wwwanttcscchollr..rr......
Corpus 1D:
Zhongzhi Yu, Yang Zhang, Kaizhi Qian. Cheng Wan, Yonggan Fu, Yongan Zhang, and Yingyan Ce- line Lin. Master-asr: achieving multilingual scalability and low-resource adaptation in asr with modular learning. In International Conference Machine Learning, pp. 40475 40487. PMLR, 2023.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint urXiv:2308.03303 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303.10512 2023b.
A APPENDIX
A.1 PROOF FOR THE OPTIMALITY OF THE GREEDY ASSIGNMENT
Theorem Let there be K right singular vectors in VT each associated with a singular value
I, 12 Tk. sorted in non-increasing order:
1
Tx
12

---

Published as a conference paper at ICLR 2025
Let there be rows in the LoRA matrix A, where S K. Each singular vector must be assigned to exactly one row in A. For any row j E (l...., . r), define its load Ly as the sum of the singular values of all singular vectors assigned to that row:
L = Σ 4
EES
where Sj is the set of singular vectors assigned row j. Let the cost of an assignment be the maximum row load:
Lmmx = max Lgg
3
Then, the greedy algorithm that, at each step, assigns the next singular vector (in decreasing order of singular value) to the row with the currently minimum load, produces an assignment that minimizes Limax- That is, the greedy algorithm is optimal.
Proof. We first observe that any assignment must satisfy the following lower bound on the maximum row load. Given that m the maximum row load for the optimal assignment:
K
L Lmx max
4 1
This is because the total singular value Σ I; must be distributed across the r rows, and the largest singular value / must be assigned to some row, which then has load of at least /
Let LG denote the maximum row load produced by the greedy algorithm. We aim to show that: max
IGG -Lmax
max
We prove this by induction on K, the number of singular vectors.
1. Base Case (K r): Each singular vector is assigned to exactly one row of the LoRA matrix A. Since the singular values are sorted in descending order and the algorithm always assigns the next largest singular value the currently lightest row (which are all initially empty), each row receives exactly one singular vector. The load of the heaviest row in this case exactly 4, and no other assignment can result in smaller maximum load, since / must appear in some row. Thus, the greedy assignment is optimal.
2. Inductive Step (Case K m + 1): Assume that the greedy algorithm produces an optimal assignment for any instance with K m singular vectors and r rows, where m 2 r. The greedy algorithm assigns the first singular vectors optimally by the inductive hypothesis. It then assigns Lm+1. the next largest singular value, to the row with the smallest current load.
Suppose, for contradiction, that there exists another assignment that produces a smaller maximum row load than the greedy one. Denote this alternative assignment by Q. and its maximum row load by T.Q.x < G Tmmax- Let be the row to which mm41 is assigned under the greedy algorithm, and let T.G. be its load after this assignment. By construction, j has the minimum load among all rows prior to assigning 4m+1. and therefore assigning 1m+1 to any other row (as might happen Q) would result in post-assignment load that is at least as large, if not larger than L.C In the case that L becomes the maximum row load after greedy assignment, the alternative assignment must therefore result in maximum row load equal to or larger than L,C. So, the assumption that the assignment Q can improve upon the greedy assignment to yield strietly lower maximum row load leads to a contradiction. So by contradiction, the theorem holds for K = m +1,
By induction, the greedy algorithm always produces an assignment that minimizes the maximum row load for any K r.
13


Reviews:
[{"review_id": "ROakoBubjP", "reviewer": "Reviewer_ckmZ", "summary": "This paper proposes SumRA, a parameter-efficient fine-tuning method that improves upon PiSSA by initializing each row of the LoRA A matrix as a sum of multiple singular vectors from the SVD decomposition of pretrained weights, rather than using only the top-r singular vectors. The authors freeze matrix A during training and only update matrix B, reducing trainable parameters by 50% compared to standard LoRA. They introduce three summation strategies (block sum, interleave sum, and greedy sum) to distribute singular vectors across rows while minimizing interference between important components. Experiments on multilingual ASR adaptation using Whisper show that SumRA achieves 12-16% relative WER improvement over LoRA baselines across five low-resource languages with only 10 hours of training data each, while using half the trainable parameters.", "questions": "1. Have you conducted any preliminary experiments on text-based fine-tuning like instruction-tuning LLMs or vision tasks that you could share? \n\n2. Can you provide experimental evidence for this multi-task scenario? What is the actual performance when you train one shared A matrix with task-specific B matrices across all 5 languages compared to task-specific (A,B) pairs? \n\n3. Have you measured the actual information loss when summing singular vectors, for example by comparing reconstruction error?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Well-motivated approach with clear theoretical foundation. The paper provides strong intuition for why summing multiple singular vectors enables broader knowledge coverage compared to PiSSA's approach of using only top-r vectors (Figure 2D and Section 3.1). The connection to model averaging in Section 4 further strengthens the conceptual framework, and the formal proof of greedy sum optimality in Appendix A.1 adds mathematical rigor to the method.\n2. Comprehensive experimental validation with consistent improvements. The experiments systematically compare SumRA against multiple baselines (LoRA, LoRA-FA, VeRA, DoRA, PiSSA, CorDA) across two model sizes (Whisper-small and Whisper-large-v2), two ranks (2 and 32), and five languages. SumRA consistently outperforms all baselines, achieving improvements across nearly all experimental configurations while using 50% fewer trainable parameters, demonstrating both effectiveness and practical efficiency.", "weaknesses": "1. Limited evaluation domains. The paper exclusively evaluates on multilingual ASR adaptation with only 10 hours of training data per language from Common Voice. This narrow experimental setting raises concerns about generalizability. No experiments are provided to validate this claim or explore other domains such as natural language understanding, vision tasks, or even ASR with different data scales (e.g., 100 hours, 1000 hours). The evaluation should be expanded to include at least one additional modality (e.g., text-based tasks using LLaMA or similar models) and different data regimes to demonstrate broader applicability.\n\n2. Incomplete comparison with related work. The paper lacks comparisons with several directly relevant PEFT methods and omits discussion of recent theoretical advances that would contextualize the contributions. Experimentally, the paper does not compare against AdaLoRA [1], which adaptively allocates ranks and could demonstrate whether fixed summation outperforms adaptive strategies, or LoRA+ [2], which improves LoRA through differential learning rates for A and B. Theoretically and methodologically, the paper misses several pertinent works: Hu et al. [3] establish computational limits of LoRA that could inform when summation strategies are beneficial; SORSA [4] uses orthonormal regularization of singular vectors, directly related to this work's orthogonality claims (lines 264-265); NdLinear [5] addresses preserving multi-dimensional structure in parameter-efficient methods, relevant to the claimed broader knowledge coverage; and recent theoretical foundations on fine-tuning limits [6,7] could strengthen the motivation for summing singular vectors. These omissions make it difficult to assess whether SumRA advances beyond current state-of-the-art or whether existing methods achieve similar benefits.\n\n3. Insufficient analysis of the summation mechanism. While the paper provides intuition about reducing \"destructive interference\" between singular vectors (lines 270-271, 283), there is no empirical analysis of what actually happens when vectors are summed. \n\n# Reference \n\n[1]: Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao: AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023.\n\n[2] Hayou, Soufiane, Nikhil Ghosh, Bin Yu: Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354.\n\n[3] Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo, Zhao Song, Han Liu:\nComputational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models. ICLR 2025.\n\n[4] Yang Cao, Zhao Song: SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models. arXiv:2409.00055.\n\n[5] Alex Reneau, Jerry Yao-Chieh Hu, Zhongfang Zhuang, Ting-Chun Liu, Xiang He, Judah Goldfeder, Nadav Timor, Allen G Roush, Ravid Shwartz-Ziv: NdLinear: Preserving Multi-Dimensional Structure for Parameter-Efficient Neural Networks. arXiv:2503.17353.\n\n[6] Timothy Chu, Zhao Song, Chiwun Yang: Fine-tune Language Models to Approximate Unbiased In-context Learning. arXiv:2310.03331.\n\n[7] Jerry Yao-Chieh Hu, Wei-Po Wang, Ammar Gilani, Chenyang Li, Zhao Song, Han Liu: Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency. \tarXiv:2411.16525.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FBzFTghOX", "reviewer": "Reviewer_Q2k7", "summary": "This paper can be seen as an upgrade of PiSSA and LoRA.\nTo address the issue in PiSSA that “A is typically initialized with only the leading singular vectors, which limits its representational capacity to a narrow subspace of the model’s knowledge,”\nthe paper proposes SumRA, which initializes each row of A as a sum of multiple singular vectors selected beyond the leading components, while B is initialized to zero, as in LoRA.\nExperimental results on multilingual automatic speech recognition tasks show that SumRA outperforms many state-of-the-art fine-tuning methods.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of initializing each row of A as a sum of multiple singular vectors chosen from beyond the leading components—thereby enabling A to influence a larger portion of the model’s knowledge space—is quite intuitive.\n\n2. The writing of the paper is very fluent and easy to read.\n\n3. In terms of experimental results, SumRA achieves significant improvements compared to various other methods.", "weaknesses": "1. This paper conducts experiments only on ASR tasks using the Whisper series of models.\nHowever, the compared methods such as LoRA, DoRA, PiSSA, and CorDA mostly focus on fine-tuning large language models (LLMs).\nTherefore, it is necessary to validate the effectiveness of SumRA on LLMs to enhance the credibility of the experimental results.\n\n2. Although the motivation behind SumRA is quite intuitive, the paper does not sufficiently reveal the underlying mechanism that explains why it works.\nIt would be helpful to include some theoretical justification or experimental analysis to support the claim that\ninitializing each row of A as a sum of multiple singular vectors chosen from beyond the leading components\nindeed enables A to influence a larger portion of the model’s knowledge space.\n\n3. In Figure 1, the depiction of PiSSA’s initialization of W_0 is incorrect; this initialization would break the model’s capability at the start.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8lIprXNj8", "reviewer": "Reviewer_VaDY", "summary": "This paper proposes SumRA, a parameter-efficient fine-tuning (PEFT) method that improves upon PiSSA by initializing each row of the LoRA matrix A as a sum of multiple singular vectors from the SVD of pretrained weights, rather than using only the leading singular vectors. The key insight is that PiSSA's use of only top-r singular vectors restricts adaptation to a narrow knowledge subspace. By summing multiple singular vectors per row (with proposed interleave-sum and greedy-sum strategies), SumRA aims to influence a broader portion of the model's knowledge space. The method is evaluated on multilingual ASR tasks using Whisper, showing improvements over LoRA baselines while using 50% fewer trainable parameters by freezing matrix A.", "questions": "1. What happens if you train A instead of freezing it? Is the improvement from initialization or from freezing?\n\n2. Can you provide empirical evidence that different singular vectors encode different vocabulary subsets in speech models?\n\n3. How does performance scale with the amount of training data (10h vs. 100h vs. full data)?\n\n4. What is the computational overhead of SVD during initialization, especially for large models?\n\n5. How do you choose which singular vectors to sum? Is there an automatic way to determine optimal subsets?\n\n6. Can you show t-SNE or other visualizations demonstrating that SumRA representations cover broader knowledge space than PiSSA?\n\n7. Why does greedy sum sometimes perform worse than interleave sum (Table 3, languages mhr and kmr)?\n\n8. How sensitive is the method to the rank r? What happens with very small (r=1) or very large (r=64) ranks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Clear motivation**: The limitation of PiSSA using only top singular vectors is well-articulated, and the proposed solution is intuitive.\n\n2. **Comprehensive baselines**: The paper compares against multiple LoRA variants (LoRA, LoRA-FA, PiSSA, CorDA, DoRA, VeRA).\n\n3. **Consistent improvements**: Results show consistent WER improvements across multiple languages and model sizes (up to 16% relative improvement).\n\n4. **Mathematical rigor**: The proof of greedy sum optimality (Appendix A.1) is mathematically sound, even if its connection to performance is unclear.\n\n5. **Reproducibility**: Training details and hyperparameters are clearly specified.", "weaknesses": "1. **Weak theoretical foundation**: The paper lacks evidence that singular vectors encode vocabulary-specific knowledge. This is the central premise but is not validated. The authors should:\n   - Provide empirical analysis of what different singular vectors represent\n   - Show correlation between singular vector indices and vocabulary subsets\n   - Validate the \"concept encoding\" hypothesis with interpretability experiments\n\n2. **Incomplete experimental analysis**:\n   - No analysis of which singular vectors are being summed and whether they truly represent complementary knowledge\n   - Missing ablation: What if A is trained instead of frozen?\n   - No visualization of learned representations to validate the broader knowledge influence claim\n   - Limited to speech; NLP experiments needed to claim generality\n\n3. **Unfair comparisons**: \n   - SumRA uses SVD initialization while baseline LoRA uses random initialization\n   - Should compare against: (a) PiSSA with frozen A, (b) SumRA with trained A\n   - Storage cost comparison (Figure 4) is misleading—the savings come from freezing A, not from summing\n\n4. **Statistical significance**: No error bars, confidence intervals, or multiple runs reported. With improvements sometimes <1% WER, statistical testing is essential.\n\n5. **Limited scope**: \n   - Only evaluated on low-resource (10h) settings; unclear if benefits hold with more data\n   - Only one model architecture family\n   - No analysis of computational cost during training/inference\n\n6. **Interference analysis missing**: The paper mentions \"destructive interference\" between summed vectors but provides no quantitative analysis of this phenomenon or how much information is actually lost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)