GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
Authors: Yeonjoon Jung
Daehyun Ahn
Hyungjun Kim
Taesu Kim
Eunhyeok Park
Venue: NeurIPS 2025 spotlight ()
Keywords: LoRA
Parameter-efficient fine-tuning
LLM
Adapation
Primary area: deep_learning
TL;DR: This paper proposes an improved parameter-efficient fine-tuning method that utilizes granular low-rank adaptation.

Abstract: Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient fine-tuning (PEFT) of generative models, valued for its simplicity and effectiveness. Despite recent enhancements, LoRA still suffers from a fundamental limitation: overfitting when the bottleneck is widened. It performs best at ranks 32–64, yet its accuracy stagnates or declines at higher ranks, still falling short of full fine-tuning (FFT) performance. We identify the root cause as LoRA’s structural bottleneck, which introduces gradient entanglement to the unrelated input channels and distorts gradient propagation. To address this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA) that partitions weight matrices into sub-blocks, each with its own low-rank adapter. With negligible computational or storage cost, GraLoRA overcomes LoRA’s limitations, effectively increases the representational capacity, and more closely approximates FFT behavior. Experiments on code generation, commonsense reasoning, mathematical reasoning, general language understanding, and image generation benchmarks show that GraLoRA consistently outperforms LoRA and other baselines, achieving up to +8.5\% absolute gain in Pass@1 on HumanEval+. These improvements hold across model sizes and rank settings, making GraLoRA a scalable and robust solution for PEFT.

GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficien Fine-Tuning
Yeonjoon Jung 1.2 Daehyun Ahn'
Hyungjun Kim SqueezeBiits *POSTECH
Taesu Kim
Eunhyeok Park2t
[yeonjoon.juns jung, daehyun ahn, hyungjun.. kim, taesu. wiw.skkkkinnsuueeetttt.....
[yeonjoon, jung, eh. park)eeeheeecc ac.kr kr
Abstract
Low-Rank Adaptation (LoRA) is popular method for parameter-efficient fine- tuning (PEFT) of generative models, valued for its simplicity and effectiveness. Despite recent enhancements, LoRA still suffers from fundamental limitation: overfitting when the bottleneck widened. It performs best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks, still falling short of full fine-tuning (FFT) performance.
identify the root cause as LoRA's structural bottleneck, which introduces gradient entanglement to the unrelated input channels and distorts gradient propagation. To address this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA) that partitions weight matrices into sub-blocks, each with its own low-rank adapter. With negligible computational or storage cost, GraLoRA overcomes LoRA's limitations, effectively increases the representational capacity, and more closely approximates FFT behavior. Experiments on code generation, commonsense reasoning, mathematical reasoning, general language understanding, and image generation benchmarks show that GraLoRA consistently outperforms LoRA and other baselines, achieving up +8.5% absolute gain in Pass @ on HumanEval+, These improvements hold across model sizes and rank settings, making GraLoR scalable and robust solution for PEFT.
Introduction
Task-specific fine-tuning enables wide range of applications and significantly improves the quality and effectiveness of generative models. However, the massive scale of these models poses substantial challenges for practical deployment. To address these limitations, Parameter-Ffffiieet Fine-Tunine Among them, Low-Rank Adaptation (LoRA) [3 has gained particular attention for its simplicity and effectiveness, introducing trainable low-rank matrices while keeping the pre-trained model weights frozen. Although the imposed rank-r bottleneck may lead slight performance degradation compared to full fine-tuning (FFT), its efficiency has led to widespread adoption in practice.
(PEFT) methods have emerged as a cost-effective alternative 12 32).
To maximize the benefits of LoRA, various studies have proposed techniques such improved initialization 4 21 23 29 and structural refinements [10 [5] [6]] 17 to enhance fine-tuning quality, While these efforts have advanced performance, substantial quality gap remains compared FFT, largely due to the inherent upper bound on the rank. Although using a higher rank, within hardware limits, appears be natural solution, unfortunately, current implementations of LoRA and its variants do not support such flexibility. Simply increasing the rank often leads to degraded accuracy many scenarios.
In this paper, we present theoretical analysis identifying the root cause of the rank limitation in LoRA. Our analysis reveals fundamental issue in LoRA's structure, channel dominance in the gradient, where a small subset of outlier channels disproportionately influences the update direction.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fixed Original Weight
Trainable Parameters
BAT
1
KS
LoRA
B
Wo
W
B
B2A²
8
A
By4g"
A444"
X
X
X,
X
X
GraLoRA
X
(a) LoRA Workflow
(b) GraLoRA Workflow
(c) Weight Update
Figure I: Illustration of LoRA architecture and GraL.RAA architecture. Gral.oRA consists of 122 small adapter pairs, where each input and output dimension is k times smaller than the original LoRA.
This dominance suppresses contributions from other channels, leading to under-utilization of the available rank and degraded performance in tasks that require nuanced distributed representations.
To overcome these expressivity bottlenecks, we propose Granular Low-Rank Adaptation (GraLoRA), novel architectural extension of LoRA. As shown in Figure 1 GraLoR divides the weight matrix into multiple sub-blocks and applies independent LoRA modules to each, enabling fine-grained updates. This design enhances the model's capacity to capture complex, localized, or multi-facete patterns, effectively mitigating the channel dominance issue and improving performance-espeeeee especially at higher ranks.
Extensive experiments show that GraLoRA consistently outperforms vanilla LoRA across range of NLP benchmarks, particularly in scenarios with high input heterogeneity task complexity. These results position GraLoRA as principled and practical advancement the PEFT landscape.
Details and Limitations of LoRA
2.1 Introduction to LoRA
LoRA is one of the most widely adopted strategies for PEFT. Given a pre-trained weight matrix Wo RMXN, where M and N represent the input and output channel dimension, respectively, LoRA keeps Wo frozen and introduces a trainable low-rank update defined as:
R= sBAT, AERNX, BERMxr
8
Here, rank and are user-defined hyperparameters. Then, for given input X E NxT the WoX + RX E MxT, where denotes the batch or token dimension. This low-rank decomposition allows the model to adapt using significantly fewer trainable parameters and reduced memory overhead.
output of the LoRA-adapted layer
While FFT updates the entire weight matrix, LoRA only updates the decomposed low-rank matrices and B. Note that we assume for simplicity, the gradient the loss with respect to R
aL
OL T MNN OY
n I.
OR
From this, the gradients with respect the LoRA parameters and A are given by:
aL OL
ÔL
B OL XT
XT A,
(3)
aB
0Y
0AT
0Y

---

Parameters effected by Outlier Channel
X
8T
aL
aL ØY
0,47
aL
aw
av
FFT Gradient
(b) Adapter Gradient
X
X
X
A
äL
aL 0Y
A
dL
aA
dB
aw
8B
(c) Adapter B Gradient
(d) LoRA Gradient
Figure 2: Gradient dynamics of FFT and LoRA in the presence of an outlier input channel. The red channel input X denotes the outlier. While FFT localizes the gradient impact, LoRA's entire gradient update becomes disproportionately influenced by the single outlier.
These result in the following reconstructed update in the fused weight space:
0L 0B
A + B OL
OL DAT OY
XT AAT + BBT OL
V R
OY XT.
(4)
This expression reveals how the structure of LoRA introduces non-trivial interactions between the gradients and the input, particularly through the rank-7 matrices.
Why Does LoRA Suffer from Larger Rank?
When fine-tuning with a large LoRA rank (e.g., r > 64), it is often observed that accuracy degrades compared to using a moderate rank. This counterintuitive behavior arises from the distinct gradient dynamics of LoRA, which differ significantly from those of FFT.
LoRA's structural design makes its gradients inherently sensitive the entire input space, illustrated In particular, we observe that outlier channels. input channels with abnormally high activations, can disproportionately dominate the gradient signal.
in Figure 2
In FFT, the effect of such outliers is typically localized, affecting only a single column of the weight matrix W that directly interacts with the outlier channel. In contrast, LoRA's low-rank constraint causes the entire gradient of the adapter matrix B, denoted aL/0B, to be influenced by these outliers. This results in distorted weight updates in the fused weight space, where the gradient signal from outlier channels overwhelms the contributions from other inputs. Consequently, LoRA fails to accurately replicate the gradient dynamics of FFT, limiting its ability to match FFT-level performance.
We observe that in certain layers, most notably the down pown-projcction matrix of Layer in LLaMA3.1-SB input activations exhibit severe channel-wise imbalance (Figure 3 (a)). As shown in Figure 4 these outlier channels disproportionately impact the adapter's gradient updates. Figure further illustrates that the gap between LoRA and FFT gradient updates widens the LoRA rank increases.
These findings reveal fundamental misalignment between LoRA updates and the gradient landscape shaped by FFT. The entangled influence input channels caused by the low-rank projection limits LoRA's ability to selectively learn from salient features, particularly under skewed input statistics. While the negative impact of outliers has been well recognized the context quantization 31 [8 their influence LoRA's behavior has not been systematically studied until now.

---

Average Absolute Input Channel
Layer 0
Average Abvolute Gradient
FFT
FFT
Layer
LoRA (r-32)
LoRA (r-128)
Layer 2
LoRA (1=64)
GraLoRAA (k-2)
Layer
(r-18)
GnLaRA (k-4)
Gril oRA (A-8)
7K
14K
Input Channel
Layer Number
Layer Number
(a) Outlier Channel Analysis
(b) LoRA Gradicnts
(c) GraLoRA Gradicnts
Figure 3: (a) Mean input channel values for the down-projection matrices across layers in
LLaMA3.I-8B. pronounced outlier exists in Layer channel 198 and 2427. (b) Gradient deviation between LoRA and FFT increases with rank, showing LoRA's susceptibility to input outliers. (c) GraLoRA gradient results rank 128. GraLoRA noticeably reduces gradient deviation between FFT.
Full Fine-Tuning (FFT)
LoRA Rank 32
LoRA Rank 64
LoRA Rank 128
Ang Magatude fhailent Magsitule
55
5B
5D-06
5B-06
IB-06
16-06
16-02
65-04
55.03
3F:04
7
4K
75
14K
1
14K
*
14k
Channel
Input Channel
Input Channel
Input Channel
Figure 4: Gradient distribution in Layer down-projection matrix. LoRA gradients show poor alignment with FFT, outlier channel increases the overall gradient scale, while less emphasizing the corresponding outlier channel.
3 Method
3.1
GraLoRA: Granular Low-Rank Adaptation
Motivated by observation in previous section, we propose GraLoRA, fine-grained and modular extension of LoRA. As illustrated Figure GraLoRA addresses the limitations of standard LoRA by partitioning the weight matrix into grid of & x l independent blocks, each equipped with its local low-rank adapter. Here, is a hyperparameter that determines the number splits along the input and output dimensions. When k 1, GraLoRA reduces to the vanilla LoRA formulation.
Specifically, the weight update R E TR. M is expressed as the concatenation of block-wits wise updates:
AI
AL
REIaLORA =
Ar, E E
B E
(5)
A
A
This Mlcck-wise reparameterization provides localized control over each spatial subregion of the parameter space. As detailed in Section 3.4 GraLoRA incurs the same parameter count and com- putational overhead as standard LoRA when using same rank. However, it introduces two key advantages; (1) Enhanced Expressivity and (2) Robustness to Input Outliers. By enabling in- dependent adaptation across h² subspaces, GraLoRA supports more fine-grained and specialized feature learning. In addition, Localized gradient updates ensure that only the adapters associated with

---

kr
kr
A11
81
M
AIR
AgE
Bx.1
Br
(a) Structure of AgratoRA
(b) Structure of BeratoRA
Figure 5: Regularized form of GraLoRA as multiplication sparse two matrices, AGULORA and
ORALL
the affected input regions receive large gradients, thereby reducing global gradient distortion and preserving inter-channel signal balance.
3.2 Expression Power Analysis
While the weight update of GraLoRA was expressed as concatenation block-wise updates (5), can also be regularized as the form of multiplication of two matrices as in the vanilla LoRA. The sparse matrix AGraLORA E R kr can be constructed as Figure 5 (a), where Ar, for i,j E {n E N n S k}
is located in position (i + (j 1) j, of AGmLORA. Other elements are masked out, thus the total
number parameter becomes X r.
Then, BGraLORA E N is constructed as Figure (b), where matrix Brj for i,j {n E N I k} is located in position (i.j + (i 1) k) of BILLORA. Similarly, other composition of the matrix is masked, therefore the total number of parameter becomes M x Then the weight update of GraLoRA can expressed as W Wo + RGMLERA = Wo + BGaaL AGaLLOA
Assuming that all columns of [B,,,, ... B. are linearly independent, the rank of BGraLORA becomes R (BGILOORA) kr. Similarly, if all columns of [A A are linearly independent, the rank of AGraLoRA is R (AGraLoRa) = kr. Applying Sylvester's 's rank incquality to derive the lower bound and the matrix product theorem for the upper bound, we obtain:
R (ERALLRR) + R. (AT - kr R(BELLRRA ( S mminnn (eiuLORR), R(A (ATTOOL))))
(6)
Thus, the effective rank of RAALORA becomes kr, which is k times higher than that of the vanilla LoRA method-effie eethhd-efffctivvel enhancing the model's expressive capacity. The rank analysis of fine- tuned LoRA and LoRA, summarized Taile Appendix, demonstrates that GraLoRA linearly scales the representational power of the adaptation matrix practical settings.
3.3 Gradient Dynamics Under Outlier Activation
GraLoRA effectively localizes the influence of outlier channels to a limited subset of adapter blocks. Because each block processes only a specific slice of the input, only the adapter pairs intersecting with the outlier channel are exposed to amplified gradients. In contrast, the remaining 1:2 k adapters maintain gradient magnitudes close to baseline levels. This selective gradient propagation resembles the behavior of FFT, where only weights directly connected to active inputs are significantly updated.
GraLoRA's impact on gradient dynamics can be observed by comparing gradient distributions the down-projection matrix in Layer with standard LoRA. As illustrated in the Figure 3 (c) and Figure 6 GraLoRA reduces the gradient deviation and limits the influence of outlier channels, overcoming the limitations of standard LoRA with larger ranks.
3.4 Tradeoff Analysis
As discussed, GraLoRA provides several advantages over standard LoRA. However, these benefits do not come without cost. In this section, we provide deeper analysis the overhead introduced by GraLoRA.
Computation Overhead Analysis: First, analyze the expected computational cost of LoRA in terms of FLOPs. To take advantage of the low-rank structure, LoRA computes the projection two sequential steps. The first computes A X e g followed by the reconstruction B(A X)
5

---

LoRA
GraLoRA (k
GraLoRA (k 4)
GraLoRA (k =8)
Avg. Magitude Grodient Magnitule
5B-415
1R-45
SE-06
1B-06
3E-04
7K
14K
7K
14K
7K
14K
7k
14K
Input Channel
Input
Input Channel
Input Channel
Figure 6: Comparison of gradient distributions under outlier activation. GraLoRA, only the blocks interacting with the outlier exhibit elevated gradients, mitigating global distortion and aligning with FFT behavior.
RAXT, These steps require 2NrT and 2r FLOPs, respectively, resulting in total complexity of ( + N)T).
Similarly, GraLoRA divides the computation into two steps involving k² adapter blocks. In the first step. the projection computes A R xT for each of the 122 blocks, incurring a total cost of 2. 2 = 2NrT. In the second step, each intermediate output is processed by its corresponding Brj producing Bi (A, E xT This step adds another 2 E N T k22 Mr. FLOPs to the total cost. Hence, the overall computational cost of GraLoRA remains O (r(M + NN)), maintaining efficiency comparable to vanilla LoRA while significantly enhancing expressive power.
detailed analysis of computational overhead is provided in Appendix C
Table 1: Maximal allocated memory during training LLaMA3. 1-BB model with batch size 1. Input length was set to 1024 and memory allocated for weight was removed for direct comparison.
LoRA
GraLoRA (k=2)
GraLoRA (k=4)
GraLoRA (k=8)
Vanilla Backward (GB)
10.0
10.1
10.2
10.4
Gradient Checkpointing (GB)
2.6
2.6
2.6
2.6
with classical LoRA, GraLoRA can be merged into the original weight matrix at inference time. Therefore, our analysis focuses on the memory overhead incurred during training. Although the number parameters and FLOPs are identical to those LoRA, the intermediate latent representation A X becomes times larger than the corresponding A X in standard LoRA. This expanded latent space allows for greater information preservation, which can be beneficial. However, also leads to increased memory consumption during training time. Fortunately, the rank r is typically much smaller than the input and output dimensions, thus the additional memory required remains marginal even for large k. as demonstrated in Table T Moreover, by applying recent techniques such as gradient checkpointing, the memory overhead from the expanded latent space can be effectively hidden, making the impact negligible practice.
Memory Overhead Analysis:
Selection of k While GraLoRA increases the total rank from r to kr, each individual block, rep- resented B,, R N x is constrained
reduced rank of -. As result, increasing beyond a certain threshold can degrade performance due to limited expressiveness within each block. This effect is especially pronounced when the overall rank is small. Empirically, maintaining a minimum block expressiveness of approximately r r/k² xs 8 yields stable performance across various configurations. Based this observation, adopted k = 2 for ranks 16 and 32, and for ranks 64 and 128 in our experiments. Detailed k-sweep results can be found in Section 4.7
observed that
3.5 Hybrid GraLoRA
On the other hand, for smaller ranks ttricsllp rank 16 or below-using k 2 may still lead to performance degradation or yield only marginal gains. To address this limitation, we introduce hybrid approach that combines the strengths of LoRA and GraLoRA. This method retains the

---

Hybrid GraLoRA (k 2)
GraLoRA (k-2)
LoRA
A
B,
4g
9
B'
B'2
82
+
A
Ap
A
A,
8:
X
P. Ro
X
X
X
R²
A'3
B'
A'4
Bg
A3"
B,
A4'
Figure 7: Hybrid LrRL architecture when GraLoRA k = 2. LoRA parameter becomes shared across small GraLoRA adapters the same row or same column.
fine-graines input handling and increased total rank offered by GraLoRA, while preserving the expressive power of larger block units through LoRA. Since LoRA shares the same parameters across both rows and columns, it can be naturally integrated with GraLoRA in concatenated form, which we refer to as Hybrid GraLoRA (see Figure 7). Empirically, we found that allocating up to of the total rank the LoRA component mitigated the limitations of GraLoRA in low-rank scenarios (7 <= 16), while fully allocating the rank to GraL better performed in high-rank circumstances.
4 Experiments
In order to validate the superiority of the proposed idca, we conduct an extensive analysis on large-scale dataset with the state-of-the LLMs. We evaluate GraLoRA across five challenging domains: code generation, commonsense reasoning, mathematical reasoning, general language understanding. and personalized image generation. Our experiments are designed to assess whether the proposed granular adaptation mechanism improves performance across varying model sizes, LoRA ranks, and tasks that require nuanced reasoning and high representational fidelity.
4.1 Experimental Setup
Code Generation. We fine-tumed LLaMA3.1 (9) with 4 A100 80G GPU the Magicoder- Evol-Instruct-I 10k [30] train dataset, a curated and decontaminated subset of WizardCoder [20], comprising high-quality instruction-response pairs for programming tasks. Evaluation
conducted on the Humaneval+ test dataset following He et al. [10], which samples 50 completions per problem using a temperature of 0.2. We report Pass @ 1, Pass 55, and Pass 10 accuracy following standard
protocol via BigCode Evaluation Harness [
Commonsense Reasoning We fine-tuned LLaMA3.2 LLaMA3.1-70B Qwen-2.5-1.5B and Qwen-2.5-7B ( (33) across 8 commonsense tasks: BoolQ (). PIQA 33 SIQA (27]. HellaSwag (35) ARC-Easy 77]. and OpenBook [22] We followed the training pipeline proposed by LLM-Adapters [1]. Training was performed on 2 H100 80G GPUs for 1.5-8B models, and 8 A100 80G GPUs for the 70B model. LLaMA3.1
WinoGrande (26), ARC-Challenge
Owen-2.5-1.5B, and Qwen-2.5-7B were trained with rank 64, using the optimal configurations proposed by Biderman al. 2. LLaMA3.2-3B
trained with rank 32, following the settings of Ponkshe al. 25 to ensure fair comparison with results reported the original paper.
Mathematical Reasoning We fine-tuned LLaMA3.2-3B MetaMathQA (34 train dataset using 4 H100 80G GPUs Evaluation was done on MATH ||| dataset, following the evaluation procedure and settings from He et al. [TO].
General Language Understanding We trained and evaluated RoBERTa-bass 19, an encoder- only architecture model, on the GLUE 28 benchmark composed of eight sub-tasks. Following the protocol from prior works ( tasks-whiin also meant we did not apply the MNL.-bbsse tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results. All trainings were done single H100 80G GPU,
we excluded MNLI and QQP-two time-intensive

---

Table Pass P and @ 10 results on LLaMA3.1 using LoRA, MoRA, RaSA, and GraLoRA across different ranks, Best results per group are in bold. indicates Hybrid GraLoRA.
Rank
Method
Training Time
Relative Time
Pass
Pass@10 Pass@5
LoRA
6.2h
1.00x
56.1%
65.3%
68.1%
MoRA
8.8h
1.42x
53.6%
62.2%
64.5%
16
RaSA
6.7h
1.0xx
53.7%
64.4%
GraLoRA**
6.7h
1.08x
58.0%
67.1%
70.1% %
LoRA
6.5h
1.00x
58.4%
68.0%
69.9%
MoRA
9.1h
1.40x
58.3%
66.7%
69.0%
RaSA
6.8h
1.05x
57.2%
67.9%
70.5%
GraLoRA
6.9h
1.0xx
58.9%
67.0%
69.0%
LoRA
6.7h
1.00x
58.1%
66.4%
68.5%
MoRA
9.7h
1.45x
57.2%
66.4%
69.2%
RaSA
6.9h
1.03x
56.6%
65 4%%
67.9%
GraLoRA
7.2h
1.07x
60.5%
71,2%
72.6% %
LoRA
7.0h
1,00x
55.8%
64.8%
68.6%
MoRA
9.9h
1,41x
52.8%
62.3%
65.3%
128
RaSA
7.6h
1.09x
57.5%
65.5%
67.5%
GraLoRA
7.7h
1.10x
64.3%
71.7%
73.7 %
Personalized Image Generation We fine-tuned SDXL 24 following the official training setup from Huggingface diffusers repository, using the Naruto-Blip Bia--Blii-Captonnn [5] dataset single H100 80G GPU. The dataset was split 90% for training and 10% for evaluation. The quality was measured through CLIP similarity and DINOv2 similarity scores.
Training Details We conducted experiments on five open-sourced LLMs-LLaMas 1-BB, LLaMA3.1-70B, LLaMA3.2-3B Qwen-2.5-1.5B. and
-.vverinn diverse architecture and sclaes across code generation, commonsense reasoning, and mathematical reasoning tasks. Fol- lowing common practice [16] 17),, we used pre-trained models rather than instruction-tun -ttmedd models. All PEFT methods were applied to the linear modules in both the attention WW.. WR. W., W. )and the feed-forward networks (W up. Wdown. Wgate). We adopted alpaca- chat instruction template for training and evaluation. We compared GraLoRA three representative PEFT methods: LoRA. MoRA 16 and RaSA [TO. We have also handled RoBERTa-baa and SDXL,, show the robust- ness and scalability of our method across differnt models and tasks. Hyperparameters for ArR followed those introduced in Kopiczko et al. [17], except for learning rate, which was reduced by a factor 5-10, as VeRA uses a learning rate approximately times larger than LoRA. Detailed training parameters can be found in Appendix
4.2 Results on Code Generation
As shown in Table 2 GraLoRA outperformed LoRA, MoRA, and RaSA across all tested ranks for Pass @1 accuracy. At rank 64, GraLoRA achieved an absolute improvement of +2.4% in Pass I, +4.8% in P and +4.1% 1 in Pass@ 10 LoRA. At rank 128, the gains were even more pronounced, with increases of +8.5% in Pass@ 1, +6.9% in Pass@5, and +5.1% in @ass 10. Notably, while other methods struggled fully utilize the increasing rank capacity- often reaching perfor- mance plateaus at lower ranks-GraLoke GraLoRA maintained a consistent upward trajectory, effectively overcoming the limitations of LoRA.
Even in low-rank settings (e.g., rank 16), where expressive capacity is typically constrained, the hybrid variant of GraLoRA demonstrated superior performance. These improvements highlight GraLoRA's enhanced capability to preserve diverse gradient signals and resist suppression from dominant outliers. The strong results the HumanEval+ benchmark further underscore the benefits of fine-grained adaptation in tackling complex, high-precision code generation tasks.
4.3 Results Commonsense Reasoning
GraLoRA outperformed other methods across a wide range of models and tasks. Notably, GraLoRA demonstrated superior performance across models of varying scales, achieving 1.1% improvement in average accuracy both Qwen2.5-1.5B and LLaMA3.1-700 It also yielded
As shown in Table 3

---

Table 3: Commonsense reasoning accuracy across models and tasks. Bold indicates the best per- formance per column. HS means HellaSwag, and WG WinoGrande. indicates values reported by
Ponkshe et al. [25]
Model
Method
BoolQ
PIQA
SIQA
HS
WG
ARC-c
ARC-e
OBQA
Avg.
LoRA
66.5%
84.0%
74.9%
83.6%
73.7%
75.2%
88. 1%
83.4%
78.7% 7%
MoRA
65.9%
82.2%
74.7%
82.6%
73.4%
72.6%
86.5%
82.8%
77.6%
Qwen2.5-1.5B
RaSA
67.5%
83.7%
75.7%
85.3%
72.9%
76.4%
89.8%
83.8%
79.4%
GraLoRA
67.2%
84.2%
75.9%
85.7% %
73.8%
77.5%
89.9%
84.4%
79.8%
LoRA"
72.3%
88.2%
79.2% %
92.9%
84.7 %
84.0%
93.6%
89.6%
85.6%
MoRA
69.9%
85.3%
78.5%
83.7%
81.4%
77.5%
88.6%
85.0%
81.2%
Qwen2.5-7B
RaSA
72.0%
88.5%
78.
93.6%
81.8%
94,2%
90.2%
85.7%
GraLoRA
73.4%
89.7%
79.0%
93.0%
84.0%
86.9%
94.5%
90.6%
86.4%
LoRA
70.0%
85.2%
79.1%
90.7%
82.2%
74.3%
86.9%
81.9%
81.3%
MoRA
72.4%
86. 1%
80.
92.3%
84.8%
76.8%
88, 8%%
84.8%
83.3%
LLaMA3.2-3B
RaSA
73.1%
87.5%
81.1 %
93.7%
85.3%
78.9%
88.9%
83.6%
84.0%
GraLoRA
74.1%
86.5%
80.8 8%
93.8 %
87.5
79.9% %
89.5%
84.8 %
84.6 %
LoRA
81.7%
93.4%
82.2%
97.5%
93.1%
90.2%
96.5%
95.6%
91.3%
LLaMA3.1-70B
GraLoRA
83.1%
94.7%
83.6% %
97.9 %
93.8 %
92.3% %
97.8%
96.2%
92.4%. %
Table 4: MATH dataset accuracy results Qwen2.5-1.5B using LoRA and GraLoRA across different ranks. Best results per group are in bold.
Rank
Method
Training Time
Relative Time
Accuracy
LoRA
S.3h
1.00x
23.6% 25.7%
64
GraLoRA
6.2h
1.17x
LoRA
5.Sh
1.00x
24.7%
128
GraLoRA
1.20x
28.9%
0.9% gain on the widely used mid-sized model, Qwen2.5-7B. Moreover, GraLoRA achieved a 3.3% improvement 1LaMA3.2-3B. surpassing broad range of baselines presented in Table
Furthermore, GraLoRA achieved the best results on 26 out of 32 tasks, consistently outperforming alternatives across benchmarks. These results support our analysis in Section 3.3 showing that GraLoRA's localized updates enhance alignment with FFT and promote robust generalization in multi-aspect reasoning tasks.
4.4 Results on Mathematical Reasoning
In mathematical reasoning task, regarded as one of the most challenging benchmarks, GraLoRA consistently outperformed LoRA across all configurations. Notably, in the high rank setting of r = 128, GraLoRA achieved a 4.2% improvement in accuracy (Table 4 mirroring the performance trends observed in the code generation experiments. These results further highlight the robustness of GraLoRA, demonstrating its capability to fully exploit the advantages enabled by increased rank capacity, thereby overcoming the inherent expressiveness constraints of previous PEFT methods.
4.5 Results on General Language Understanding
GraLoRA demonstrates strong performance even in the low-rank regime, outperforming all baselines terms of average score. The Hybrid GraLoRA variant achieves the most robust results, attaining the best performance on four out of six tasks, while both the original and hybrid versions consistently surpass all other baselines, shown in Table 7 Compared with LoRA, the best GraLoRA config- uration yields a 1.8% improvement in average accuracy, with gains observed across all sub-tasks. These findings indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and generalizes well to non-LLM architectures.
4.6 Results on Personalized Image Generation
In the image generation task, GraLoRA consistently outperformed LoRA both CLIP and DINOv2 similarity metrics, achieving 0.5% and 2.1% improvements, respectively (Table 8). These results

---

(b=2)
Gml.RRA (Im4)
GraloRA (e-6)
Ranki6
Rank
Rank 64
61%
Paseol Accunzy
Rank 32
59%
57%
Rank 128
55%
53%
52%
56%
60%
64%
0
1/2
Pass Accuracy
Ratio of Allocated LoRA
(a) GraLoRA Sweep Result
(b)Hybrid GraLoRA Ratio Sweep Result
Figure 8: (a) GraLoRA sweep results and (b) Hybrid GraLoRA Ratio sweep results for LLaMA3.1-8B code generation task. Ratio 0 implies default GraLoRA and ratio vanilla LoRA in (b).
further demonstrate the generality and effectiveness of GraLoRA beyond language models, extending its applicability vision-language and generative architectures such as diffusion models.
4.7 Ablation Study
GraL.RRA J: Sweep We evaluated the impact of varying on code generation accuracy. As shown in Figure 8 (a). l yielded the best performance rank 32, while k = 4 was optimal at rank 128. These results are consistent with the theoretical prediction that a smaller is preferable for lower ranks, reduced sub-block rank can be particularly detrimental when the overall rank limited.
Hybrid GraLoRA Ratio Sweep We assessed performance across different AorAA----rrLLRAA rank allocation ratios for the Hybrid GraLoRA configuration (Figure 8 (b)). At rank 16, partially allocating the rank to LoRA led optimal accuracy. However, for larger ranks, allocating rank to LoRA resulted in degraded performance. This suggests that Hybrid GraLoRA is advantageous in low-rank regimes, where the sub-block rank GraLoRA alone may be insufficient. In contrast, under higher-rank settings where GraLoRA's sub-blocks expressive enough, introducing LoRA components may lead to gradient entanglement, therehy hindering effective learning.
Conclusion
In this work, we introduced GraLoRA, novel PEFT method that extends LoRA with granular, block- wise decomposition. Motivated by rigorous analysis of LoRA's gradient behavior, we identified that input outliers can dominate the low-rank update, suppressing meaningful contributions from other input channels and misaligning with the localized gradient propagation observed in FFT.
GraLoRA addresses this limitation by dividing the adaptation space into k2 independently trained low-rank adapters, enabling spatially localized and context-aware updates. Our theoretical analysis shows that this design increases expressivity by factor of k, without additional parameters or computational cost. Moreover, under outlier activations, A effectively mitigates the global gradient distortion seen vanilla LoRA and better preserves inter-channel balance. Empirically, GraLoRA consistently outperforms standard LoRA and strong baselines such as RaSA across diverse tasks and model scales. On the code generation benchmark HumanEval+, it achieves up to +8.5% absolute gain in Pass GraLoRA also delivers significant improvements across other 4 additional tasks, highlighting its robustness and scalability across heterogeneous architectures and model sizes.
Future Work. While GraLoRA improves gradient locality and expressive power, its current design assumes uniform partitioning. Future extensions may explore adaptive learned partitioning schemes, sparsity-aware block activation, or task-driven dynamic rank allocation. Additionally, applying GraLoRA to vision transformers, multimodal architectures, or continual learning setups may further highlight its potential for robust and efficient model adaptation.
Overall, GraLoRA represents principled and practical step forward in the design of PEFT methods, bridging the gap between global low-rank reparameterization and local, fine-grained adaptation.
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by the Technology development Program MSS [RS-2023- 00258531], the Starting growth Technological R&D Program of MSS [S3358777], and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2025-02304188 Development of Optimization Code Conversion Technology for Heterogeneous Al Semiconductor Based Large-Scale Models).
References
[1] Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. htttps:/giite 2022.
[2] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King. Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney. and John P. Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024.
[3] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[4] Kerim Büyükakyüü.. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.01177, 2024.
[5] Hole Cervenka. Naruto blip captions. https://huniinngaaccc///aa/ttaat
baruto-bbiipccapiions 2022.
[6] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2924 2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 2018
[8] Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform, 2024.
[9] Aaron Grattafiori et al. The Ilama herd of models, 2024.
[10] Zhiwei Hc, Zhaopeng Tu, Xing Wang. Xingyu Chen, Zhijie Wang. Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang. RaSA: Rank-sharing low-rank adaptation. In Proceedings of the 2025 International Conference on Learning Representations (ICLR), 2025.
[11] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[12] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nip. 2019.
[13] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint 2tt.:2106.09685. 2021.

---

[14] Zhiqiang Hu, Lei Wang. Yihuai Lan. Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. Lim-adapters: An adapter family for crrameter-efficice fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5276, 2023.
[15] Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-rank adaptation for large language models. In Proceedings of the 2025 Interna- fional Conference on Learning Representations (ICLR), 2025.
[16] Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. MoRA: High-rank updating for parameter-effiieet fine-tuning, arXiv preprint arXiv:2400 2401:20000.2110, 2024.
[17] Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. VeRA: Eector-based random matrix adaptation. In Proceedings of the 2024 International Conference Learning Represen- lations (ICLR), 2024.
[18] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13355-13364, 2024.
[19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach, 2019.
[20] Ziyang Luo, Can Xu, Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
[21] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: vectors adaptation of large language models. Advances in Neural Information Processing 2024.
Principal singular values and singular
Systems,
[22] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. 2018
[23] Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin, Mare Peter Deisenroth, and Sepp Hochreiter. One initialization to rule them all: Fine-tuning via explained variance adaptation. arXiv preprint arXiv:2410.07177 2024.
[24] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[25] Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank tinettuning 2025.
[26] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[27] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint XXX::1904009722 2019.
[28) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[29] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approxima- tion. Advances in Neural Information Processing Systems, 37:54905-54931. 2024.
12

---

[30] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empower- ing code generation with OSS-instruct. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller. Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52632-52657. PMLR, 21-27 Jul 2024.
[31] Guangxuan Xiao, Ji Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024.
[32] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment, 2023.
[33] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.55115, 2024.
[34] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint 230iv:2309.1200 2023.
[35] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 2019.
13

---

A Rank Analysis in Wolld Scenarios
Table Average rank size in each projection layer across LoRA and GraLoRA variants. Rank r was set to 128 in all methods.
prooj
k_proj
v v_proj
0 o_proj
up up_proj
down_proj proj
gate_proj
LoRA
128
128
128
128
128
128
128
GraLoRA (k=2)
256
256
256
256
256
256
256
GraLoRA (k=4)
512
512
512
512
512
512
512
GraLoRA (k=8)
1024
1016
1022
1024
1024
1024
1024
As shown in Table GraLoRA denoted linearly increasing ranks as the & increased. The observation aligns with our theoretical analysis that increasing GraLoRA leads to higher expression power by increasing the latent space from to kr.
B Gradient Distribution of LoRA and GraLoRA
LoRA
GraLoRA (k-2)
GraLoRA (k-4)
GraLoRA (k=8)
Gradient Magittuk
1E:05
Rank 32
1E-06
Magitude
3E-04
7K
Gradient Magiiade
1B-05
Rank 64
1E-06
Aug Magittud
6E-04
3E-04
14K
14K
SE-05
Gradent Magnitude
Rank 128
1E.06
Magntude
3E.04
7K
14K
7
7
14K
1
14X
Input
Input
Input
Input
Figure 9: Comparison of gradient distributions under outlier activation for rank 32, 64, and 128 in LLaMA3.1-80 Layer down-projection matrix.
Figure 9 displays gradient distributions of LoRA and GraLoRA for varying ranks. GraLoRA, only the blocks interacting with the outlier exhibit elevated gradients, structurally solving the gradient entanglement discovered vanilla LoRA. This enables to mitigate global distortion and align with FFT behavior in all ranks.
14

---

C Precise Analysis on Computation Overhead
Computation Workflow in GraLoRA
X1
B (An xp
Step
Step
X
Step
AR
B BRRARR
Xx
B (Ax.) xx
Step
Step
Stop
GraLoRA
Input
Output
Figure 10: Computation workflow in GraLoRA is composed of 3 steps: two sub-block matrix multiplications and a following matrix addition.
In the previous "Computation Overhead Analysis" " section 3.4 we compared the computation of LoRA and GraLoRA with the big O notation on the two major matrix multiplication steps. In this section we further examine the exact computation requirement and compare their efficiency.
LoRA FLOPs LoRA performs the projection in two sequential steps to leverage its low-rank structure. In the first step, the computation of A X R'xT requires (2.N 1)r7 FLOPs. In the second step, the reconstruction BAA X) E RMXT incurs (2r 1)MT FLOPs. Therefore, the total FLOPs for LoRA is:
LORAFLOPS = (2N - 1)r7 (2r (2r-1) 1)MT
= 2r + NYT - MIT.
GraLoRA FLOPs In practice, GraLoRA computations can be divided into three stages, involving 2 adapter blocks: matrix multiplications followed by matrix addition shown in Figure 10
In the first stage (projection), each adapter block computes AT E REXT, which requires (2 1) ET FLOPs. Since there are k²2 such blocks, the total FLOPs for this step is (2n k)rT.
In the second stage (reconstruction), each adapter block performs (A, X,) E E x which costs (2 1) Y FLOPs. With 1a2 blocks, the total becomes (2r k)mmT. The final stage involves aggregating the outputs across projections for each row:
Σ B.,
which requires ( T) (k- ()) = mk---1) FLOPs per row. Across k rows, the total cost becomes (k - 1)mT.
Combining all three stages, the total FLOPs for GraLoRA is:
GraLoRAmLoR (2n k)rT + (2r k)mT + (k - 1)ml 22(m + n)T k(r + m) m)T + (k: 1)ml 2r(m + n)T krT - m..
This can also be expressed as:
GraLoRAFLOR: LoRAFLOPS -Kk 1)rT,
demonstrating that GraLoRA introduces reduced computation compared to LoRA.
15

---

D Additional Experiment Results
D.1
Results Commonsense Reasoning with extensive baseline comparison
Table 6: Commonsense reasoning accuracy across models and tasks. All values are percentages; bold indicates the best performance per row. HS means HellaSwag, and WG WinoGrande. indicates
values reported by Ponkshe et al. [25]
Method
Rank
#Parmas
BoolQ
PIQA
SIQA
HS
WG
ARC-c
ARC-e
OBQA
Avg.
Full-FT!
3.21B
70.4%
85.6%
80.5%
91.9%
85.0%
75.3%
88.5%
81.9%
82.4%
LoRA-XS!
181M
67.3%
83.4%
78.7%
89.0%
82.1%
72.6%
85.2%
78.9%
79.6%
LoRA-SB
1.8M
70.3%
84.8%
80.2%
91.6%
84.6%
74.7%
87.9%
81.2%
81.9%
LoRA'
32
48.63M
70.0%
85.2%
79.1% 1%
90.7%
82, 82.2%
74.3%
96.9%
81.95
81.3%
MELoRA
48.63M
71.3%
85.0%
78.6%
93.0%
79.7%
73.7%
85.5%
79.0%
80.7%
LsLRRA'
48.63M
69.8%
85.1%
78.9%
90.5%
82.0%
74.2%
86.7%
81.7%
81.1%
PISSA!
48.63M
70.1%
85.4%
79.4%
90.9%
82.7%
74.6%
87.2%
81.8%
81.5%
DoRA
19.40M
70.4%
85.6%
79.7%
90.8%
82.9%
74.9%
87.6%
82.0%
81.7%
BOFT
48.48M
72.3%
84.6%
79.1%
84.5%
73.7%
87.8%
80.6%
81.7%
LoRA-Pro
48.63M
71.3%
85.8%
74.4%
90.9%
83.4%
75.3%
87.2%
81.7%
81.9%
MoRA
32
48.63M
72.4%
86.1%
80.1%
92.3%
84.8%
76.8%
88.8%
84.8 %
83.3%
RaSA
48.63M
73.1%
87.5%
$1.1% %
93.7%
85.3%
78.9%
88.9%
83.6%
84.0%
GraLoRA
48.63M
74.1%
86.5%
80.8%
93.8%
87.5%
79.9%
89.5%
84.8%
84.6%
D.2 Results on General I Language Understanding (GLUE)
Table 7: GLUE dataset accuracy results on RoBERTa-base across different tasks. Best results per
task are bold.
Method
#Params
SST-2 (%)
MRPC (%)
CoLA (%)
QNLI (%)
RTE (%)
STS-B (%)
Avg (%)
Full-FT
125M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
95.1
90.6
84.2
VeRA
0.043M
94.6
89.5
65.6
91.8
78.7
90.7
85.2
FourierFT
0.024M
94.2
90.0
63388
92.2
79.1
90.8
85.0
GraLoRA
0.3M
95.2
65.3
85.8
Hyhrid oRR
0.3M
95.2
90.2
64.1
93.4
79.8
91.2
85.6
Best GraLoRA
0.3M
95.2
90.2
65.3
93.4
80.9
91.2
86.0
D.3 Results on Personalized Image Generation
Table 8: SDXL fine-tuning results personalized image generation.
Method
CLIP Similarity
DINOv2 Similarity
LoRA
91.4%
79.2% 81.3 %
GraLoRA
91.9%
E
Experiment Details
Baseline Methods. We compared GraLoRA with three main baseline methods. Key idea for each method is follows:
LoRA freezes pretrained model weights and injects trainable low-rank matrices into selected layers, allowing efficient fine-tuning with significantly fewer parameters, approximating weight updates as product of two small matrices.
MoRA employs single square matrix instead of low-rank matrices to achieve high-rank updating while maintaining the same number of trainable parameters.
RaSA enhances LoRA by sharing partial low-rank components across layers while keeping layer-specific updates.
16

---

Table Hyperparameters for Code Generation, Commonsense Reasoning, Mathematical Reasoning. and Personalized Image Generation tasks.
Task
Model
Method
Rank
L.R
Batch size
Epochs
Optimizer
LoRA
Code Generation
MoRA
LLaMA3.1-8B
(16, 64, 128)
2e-4
192
2
LionW
RaSA
GraLoRA
LoRA
MoRA
Qwen-2.5-1.5B
2e-4
LionW
RaSA
GraLoRA
LoRA
MoRA
Qwcn-2.5-7B
4c-4
LionW
Commonsense
RaSA
Reasoning
GraLoRA
BOFT
4c-4
MeLORA
4e-4
LLaMA3.2-3B
MoRA
32
2c-4
192
2
AdamW
RaSA
4e-4
GraLoRA
4c-4
LoRA
LLaMA3.1-70B
3e-4
192
LionW
GraLoRA
LoRA
2c-4
AdamW
Mathematical
GraLoRA
Qwen-2.5-1.5B
Reasoning
LoRA
4e-4
AdamW
GraLoRA
Personalized
LoRA
SDXT.
128
le-4
Adam
Image Generation
GraLoRA
Table 10: Detailed hyperparameter settings for each sub-tasks in General Language Understanding
Model
Task
Method
Rank
LR
Head-LR
Batch size
Epochs
Optimizer
GraLoRA
4e-3
SST-2
4e-4
AdamW
Hybrid GraLoRA
4e-4
GraLoRA
4c-4
MRPC
8
4c-4
128
30
AdamW
Hybrid GraLoRA
2e-4
GraLoRA
5e-4
5e-3
RoBERTa-base
CoLA
Hybrid GraLoRA
8
Sc-4
8c-4
256
80
Adam
GraLoRA
QNLI
8
5e-4
2e-3
128
25
AdamW
Hybrid Gral
GraLoRA
RTE
8
2e-4
2e-4
128
160
Adam
Hybrid GraLoRA
GraLoRA
STS-B
8
1e-3
le-2
128
80
Adam
Hybrid GraLoRA
We fixed LoRA a 2r which is known to be generally applicable in different models with different ranks (2). Detailed hyperparameter settings for our experiments are denoted in Table 9
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked use checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] is perfectly acceptable to answer "[No] providee proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
.
Do not modify the questions and only use the provided macros for your answers.
1.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we strongly denote our main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification: Yes, we discuss the limitations of our method and evaluate it's impact in the "Tradeoff Analysis" section. We further examine how to overcome the limitation in "Hybrid Gral.RA" section and it's practical result in "Results on Code Generation" section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, we provide complete and correct proof for each theoretical result in the "Expression Power Analysis" section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
19

---

Justification:
Yes, we do disclose information needed to reproduce the results in "Experiment Setup" section and in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we do provide open access to code with sufficient instructions as supple- mental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides www.dee//ooeuuumisstteeeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.ccppbbll// http://///ggiieee///eeeubbbbbbis
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
20

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: Yes, we specify all the training and test details necessary to understand the results in "Experimental Setup" section and in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: expensive.
No, error bars are not reported because it would be too computationally
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
21

---

Justification:
Yes. we do provide sufficient information on the computer resources in "Tradeoff Analysis" section and in "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, our research conforms with the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We propose generic method for optimizing efficient training of neural networks. It is not directly related social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
22

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our method solely works for fine-tuning pretrained model, thus the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the assets used in the paper are properly cited in "Experiments" section and in overall writings.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sesrsswithhcee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
23

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our core method development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non sta-staddard components.
Please refer to LLM policy for what should or should not be described.
24


Reviews:
[{"review_id": "eshfQX0kUS", "reviewer": "Reviewer_EFNJ", "summary": "The paper identifies bottlenecks in scaling LoRA to higher ranks and introduces GraLoRA, a granular low-rank adaptation approach. The authors observe that LoRA’s structure makes it sensitive to the full input, which allows outliers to disproportionately influence the gradient signal. To address this, they propose partitioning the weight matrix into sub-blocks, with each sub-block has its own LoRA adapter. Experimental results indicate that this block-partitioned strategy, GraLoRA, consistently outperforms standard LoRA and its variants across multiple settings.", "questions": "The authors could help \n1) Why are all the columns of B_i and A_i sub-matrix independent?\n2) How are sub block adapters initialized? Is the initialization same as in the case of LoRA.\n3) Are LoRA alpha factors same across all blocks? Unfortunately, couldn't find this form the paper.\n4) Would alternate approaches such as VeRA /LoRA-SB alter the impact of outliers in the gradient signal? Can VeRA/LoRA-SB style adapters be leveraged for GraLoRA.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths \n1) The paper identifies bottlenecks in the performance of LoRA and highlights the sensitivity of gradient signals to outliers.\n2) A novel approach is proposed—partitioning the weight matrix into sub-blocks and training a separate LoRA adapter for each block.\n3) The analysis of memory and computational complexity indicates parity with conventional LoRA.\n4) Experimental validation demonstrates that GraLoRA consistently outperforms other well-known LoRA variants.\n5) The paper is well written, with experimental results presented in a clear and concise manner.\n\nWeaknesses:\n1) Theoretical justification of GraLoRA’s gradient dynamics and its sensitivity to outliers is minimal. While empirical results suggest changes in gradients localized to interacting blocks, a more rigorous formulation would be valuable.\n2) The paper lacks an analysis of the subspaces learned by the adapters using Grassmannian geometry, which could provide insight into observed performance gains.\n3)The discussion on effective rank in Section 3.2 is incomplete. The authors assume independence of all columns in the submatrices ( B_{(i,k)} ) and ( A_{(i,k)} ), but this assumption is not clearly justified. Empirical support is provided for the higher implicit rank hypothesis, but additional clarification is needed.\n4) The GraLoRA structure may not be amenable to efficient computation on resource-constrained or on-device deployments. Suggestions for possible optimizations would enhance the paper’s applicability.\n5) In scenarios requiring support for multiple adapters, GraLoRA could face scalability or architectural challenges. The paper does not address this issue.", "was_revised": true, "final_justification": "The authors have tried to address all the questions I previously raised. I carefully reviewed their responses and comments during the rebuttal period and acknowledge their efforts to clarify the points of concern. My remaining reservations pertain to the following:\n- The invocation of random matrix theory to justify higher rank in finite settings warrants more rigorous analysis. While the asymptotic behavior is well-understood, a non-asymptotic treatment would be more appropriate and informative in this context.\n- Although I agree that GLoRA maintains the same parameter count, practical deployment on edge devices often necessitates adapter switching or fusion with varying weights. Such scenarios demand careful memory management to ensure efficient adapter switching with acceptable latency.\n- The experimental results convincingly demonstrate the benefits of high-rank adapters. For example, MoRA performs comparably to GraLoRA across several metrics. It remains unclear whether the authors have investigated sparse high-rank adapters, which was another elegant effort leverage high rank adapter.\nI sincerely thank the authors for their diligent efforts in addressing the concerns raised. Based on their responses, I have increased my score.", "rebuttal": "Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of our contributions and the insightful points raised.\n## (W1) Theoretical justification of GraLoRA's gradient dynamics\n\nWhile the gradient dynamics of LoRA are theoretically formulated in Equation (4) and Figure 2, and empirically validated in Figures 3 and 4, the analysis for GraLoRA in the main paper has thus far focused on empirical results (Figures 3 and 6). We now provide a more rigorous treatment of GraLoRA’s gradient dynamics.\n\nGraLoRA divides the adapter into $k^2$ sub-blocks. Each sub-block is defined as\n$$\nR_{i,j} = B_{i,j}A_{i,j}^\\top, \\ B_{i,j} \\in \\mathbb{R}^{\\frac{M}{k} \\times \\frac{r}{k}}, \\ A_{i,j} \\in \\mathbb{R}^{\\frac{N}{k} \\times \\frac{r}{k}} \n$$\nGiven an input submatrix $X_i \\in \\mathbb{R}^{\\frac{N}{k} \\times T}$, the gradient of the parameters is:\n$$\n\\frac{\\partial{L}}{\\partial{A_{i,j}^\\top}} = B_{i,j}^\\top \\frac{\\partial{L}}{\\partial{Y_j}}X_i^\\top, \\  \\frac{\\partial{L}}{\\partial{B_{i,j}}}=\\frac{\\partial{L}}{\\partial{Y_j}}X_i^\\top A_{i,j}\n$$\nThe corresponding gradient of the composed update matrix $R_{i,j}$ is:\n$$\n\\frac{\\partial{L}}{\\partial{R_{i,j}}}=\\frac{\\partial{L}}{\\partial{B_{i,j}}}A_{i,j}^\\top + B_{i,j}\\frac{\\partial{L}}{\\partial{A_{i,j}^\\top}}=\\frac{\\partial{L}}{\\partial{Y_j}}X_i^\\top A_{i,j}A_{i,j}^\\top + B_{i,j}B_{i,j}^\\top \\frac{\\partial{L}}{\\partial{Y_j}}X_i^\\top\n$$\nThis closely resembles the gradient dynamics of LoRA, maintaining the same underlying structure of interaction between the input and parameter gradients.\n\nNow if an outlier exists in input slice $X_{out}$, only $k$ sub-blocks $[R_{out, 1}, \\dots , R_{out, k}]$ directly process the outlier input and receive the amplified gradient signals. In contrast, the remaining $k^2-k$ blocks remain largely unaffected, as they operate on non-outlier slices aligning from the observations in Figure 6. This selective exposure sharply contrasts with standard LoRA, in which a single outlier channel can influence the entire low-rank adapter due to global entanglement. \n## (W2 & Q1) Independence of all columns in the submatrices\n\nThe independence of columns in the submatrices $B_{(i,k)}$ and $A_{(i,k)}$ is a critical condition to achieve the maximum rank $kr$ in GraLoRA, comparable to the condition in standard LoRA, where the rank is maximized when all columns of $B$ and $A$ are linearly independent.\n\nTo justify why this condition holds in practice, we refer to results from random matrix theory. Consider a random matrix $X \\in \\mathbb{R}^{r \\times r}$, where each entry is sampled independently from a continuous distribution (e.g., Gaussian). The event that such a matrix is rank-deficient is equivalent to it being $\\det(X) = 0$. Since determinant function is a multivariate polynomial over $r^2$ dimensional space of the matrix entries, the set of rank-deficient matrices forms a lower-dimensional algebraic hypersurface of dimension $r^2-1$, which has Lebesgue measure zero in $\\mathbb{R}^{r^2}$. Therefore:\n\n$\\mathbb{P}(\\mathrm{rank}(X) < r) = \\mathbb{P}(\\det(X) = 0) = 0$ which implies $\\mathbb{P}(\\mathrm{rank}(X) = r) = 1$\n\nThis logic extends naturally to non-square matrices $X \\in \\mathbb{R}^{m \\times n}$. Rank-deficiency in such cases would require all $k \\times k$ minors (where $k = \\min(m, n)$) to vanish simultaneously, which again corresponds to a measure zero. Therefore, the condition of independence in both LoRA and GraLoRA is well-supported by probabilistic arguments rooted in random matrix theory.\n## (W2) analysis with Grassmannian geometry\n\nTo compare the subspaces learned by the adapters, we examine the column spaces of the weight update matrices from LoRA and GraLoRA. Let $R_{LoRA} = B_{LoRA}A_{LoRA}^\\top$ and $R_{GraLoRA} = B_{GraLoRA}A_{GraLoRA}^\\top$, as illustrated in Figure 5 of the paper. In vanilla LoRA, assuming column-wise linear in both $B_{LoRA}$ and $A_{LoRA}$, the rank of $R_{LoRA}$ is $r$. Thus, its column space lies in the Grassmannian manifold $\\mathcal{C}(R) \\in \\mathcal{Gr}(r, M)$, which denotes the set of all $r$-dimensional linear subspaces in $\\mathbb{R}^M$.\n\nIn GraLoRA, each sub-block $R_{i,j}$ has rank $\\frac{r}{k}$, as shown in Section 3.2. The rank of each column block $R_i = [R_{i,1} R_{i,2} \\cdots R_{i,k}]$ becomes $r$ under the assumption that all sub-blocks are linearly independent, and $R_{GraLoRA}$ can reach the maximum rank $kr$. Consequently, the column space of the GraLoRA update list in $\\mathcal{C}(R_{gralora}) \\in \\mathcal{Gr}(kr, M)$.\n\nIn terms of dimension, the Grassmannian manifold $\\mathcal{Gr}(r, M)$ has dimension $r(M-r)$, while $\\mathcal{Gr}(kr, M)$ has dimension $kr(M-kr)$. Under the common setting where $r \\ll M$, this implies that:\n$$\\textbf{dim}({\\mathcal{Gr}(r, M)}) < \\textbf{dim}({\\mathcal{Gr}(kr, M)})$$\nTherefore, the subspace learned by GraLoRA spans a higher-dimensional manifold, providing greater representational flexibility and capacity than standard LoRA. This geometric perspective further supports our theoretical claim that GraLoRA enhances expressivity by expanding the effective rank of the adaptation subspace. \n## (W3) GraLoRA on resource-constrained scenario\n\nAs discussed in Section 3.4 (Tradeoff Analysis), GraLoRA introduces no additional computational or memory overhead during training. This is because GraLoRA maintains the same number of parameters and FLOPs as standard LoRA.\n\nDuring inference, GraLoRA adapters can be statically fused into the base model weights, just like in LoRA. Once merged, inference proceeds identically to that of the original model with updated weights, incurring no runtime penalty. Thus, for single-adapter deployments, such as in typical on-device or resource-constrained settings, GraLoRA remains as efficient as standard LoRA.\n## (W3) GraLoRA on multiple adapter scenario\n\nWe agree that multi-adapter settings introduce practical challenges due to the need to dynamically load and apply adapter weights without fusion. However, we emphasize that GraLoRA introduces no additional burden compared to LoRA in this scenario. Both methods involve the same number of parameters and identical FLOPs, resulting in equivalent memory and compute requirements per adapter.\n\nTo empirically validate this, we measured the inference latency of GraLoRA and LoRA under a multi-adapter setting using the LLaMA3.1–8B model with rank 32. We simulated a realistic scenario by generating 128 output tokens from 128 input tokens using a single NVIDIA H100 GPU, with adapters loaded dynamically and applied without fusion. The average end-to-end inference times were 5.1 seconds for LoRA and 5.3 seconds for GraLoRA.\n\nThis negligible difference demonstrates that GraLoRA scales effectively in multi-adapter deployments, and does not suffer from significant computational or architectural limitations compared to existing PEFT methods. We will include this measurement in the revised paper to clarify GraLoRA’s practicality in such scenarios.\n## (Q2 & Q3) Initialization and hyper-parameter settings of GraLoRA\n\nThank you for pointing this out. The initialization and hyperparameter settings for GraLoRA follow the same convention as in standard LoRA. Specifically, each sub-block adapter $A_{i,j}$ is initalized using Kaiming uniform initialization, while $B_{i,j}$ is initialized to zero. Regarding the scaling factor, we currently apply the same LoRA alpha value uniformly across all blocks. Consequently, each sub-block’s output is scaled identically, ensuring consistent training dynamics across the adapter grid.\n## (Q4) Applicability of VeRA and LoRA-SB to GraLoRA\n\nBoth VeRA and LoRA-SB primarily aim to reduce the number of trainable parameters, but they do not fundamentally address the gradient sensitivity to outlier channels. In VeRA, the shared matrices \\( A \\) and \\( B \\) still multiply the full input \\( X \\), so outliers can dominate the gradient signal. LoRA-SB adopts the structure $ R = B R A^\\top $, training only \\( R \\), but since \\( A \\) still interacts with the entire input, it also remains vulnerable to outliers.\n\nNevertheless, both styles can be incorporated into GraLoRA. A VeRA-style GraLoRA would share \\( A \\) and \\( B \\) across sub-blocks, increasing parameter sharing. While this raises the latent dimensionality from \\( r \\) to \\( kr \\), the blockwise structure of GraLoRA still localizes the influence of outliers, offering improved robustness. For LoRA-SB-style GraLoRA, each sub-block can be reparameterized as $ R_{i,j} = B_{i,j} R_{i,j} A_{i,j}^\\top $, preserving GraLoRA’s outlier isolation while benefiting from reduced trainable parameters.\n\nTo further validate GraLoRA’s generality, we have conducted additional experiments across diverse tasks, including commonsense reasoning, mathematical reasoning, GLUE, and diffusion model fine-tuning. These evaluations incorporate VeRA and LoRA-SB as baselines and consistently demonstrate GraLoRA’s robustness and effectiveness across domains. As this response focuses on mathematical and theoretical aspects, the corresponding experimental results are provided in our responses to other reviewers.\n\n**To conclude**, we sincerely thank the reviewer for the thoughtful and constructive feedback. The insightful comments guided us to develop additional theoretical analysis of GraLoRA’s gradient dynamics, independence assumptions, and subspace structure, which we have now included. Furthermore, we conducted new experiments, including inference latency and multi-task evaluation, that address practical concerns raised in the review. These theoretical clarifications and empirical results will be incorporated into the revised manuscript to improve clarity, completeness, and impact."}, {"review_id": "b8C2AY3sid", "reviewer": "Reviewer_f9ih", "summary": "This paper addresses a known limitation in LoRA: its performance degrades at higher ranks. Motivated by empirical observations and theoretical analysis, the authors propose GraLoRA (Granular Low-Rank Adaptation), which partitions the weight matrices into sub-blocks to mitigate the issue of “outlier channels” disproportionately influencing training.", "questions": "GraLoRA introduces regularization through structural partitioning, hence, it could be viewed as a regularization method. Thus could the authors also compare GraLoRA with more regularization techniques in LoRA?\n\nCan the authors include experiments on image-based tasks or vision transformers? These would serve as compelling evidence.\n\nA special requirement is the comparison with MELoRA (MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning). In MELoRA the update is restricted to diagonal-blocks, which can be regarded as a more specific GraLoRA. I think it is better to use the GraLoRA directly on the tasks presented in that paper, which will be more convenient for you and meanwhile could test the generalization capability across tasks.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThe paper is well-motivated. The logical flow from empirical observations (i.e., LoRA's degraded performance at higher ranks), to the hypothesis of outlier channel dominance, and finally to the proposed architectural change is clearly laid out and persuasive.\n\nThe concept of “outlier channel” is used to explain gradient entanglement in LoRA, which is an insightful analysis. Although similar patterns have been discussed in deep learning literature (see weakness below), framing this in the context of LoRA and fine-tuning is valuable.\n\nGraLoRA is directly motivated by this analysis, offering a structured method to partition updates and reduce interference.\n\nExperimental results show strong empirical gains, especially, the improvement for code generation across various model sizes and LoRA ranks is significant.\n\nWeakness\nThe authors use the name “outlier channel”. It seems to me that the authors are implying that this is not a good phoneme. But it is quite nature: many papers have shown that the learning dynamics of DNN (not only ViT) are in low dimensional spaces. Hence, the presence of dominant channels is expected. Particularly in fine-tuning scenarios, these dominant directions may actually assist in fast adaptation. A more nuanced discussion is warranted to distinguish harmful dominance from useful representation compression.\n\nAlthough GraLoRA mitigates gradient interference via sub-block isolation, from a flexibility and capacity standpoint, the hierarchy seems to be: Full Fine-Tuning > LoRA > GraLoRA. That is, while GraLoRA may improve stability, it also constrains learning to smaller regions of the parameter space. The paper does not fully address this potential trade-off.\n\nThe evaluation is limited to natural language tasks. Given the generality of the proposed method, it would be helpful to test GraLoRA on vision-related fine-tuning tasks. This would further establish the robustness and generalizability of the method across modalities.\n\nMy own experience tells me that fine-tuning performance on NLP tasks often exhibits significant variance depending on initialization, data quality, and learning rate schedules. Some evaluation of this variance (e.g., standard deviations or box plots) would make the results more convincing.", "was_revised": true, "final_justification": "This paper addresses a known limitation in LoRA: its performance degrades at higher ranks. Motivated by empirical observations and theoretical analysis, the authors propose GraLoRA (Granular Low-Rank Adaptation), which partitions the weight matrices into sub-blocks to mitigate the issue of “outlier channels” disproportionately influencing training.\n\nThe rebuttal, which contains additonal explanation and experiments, have addressed most of my concerns. Now I vote for acceptance. Although the \"outlier channel\", or similar names, is not a new finding. But the authors design is new and the improvement is well supported. Thus, I suggest acceptance.", "rebuttal": "We are grateful for your careful reading of our paper and your positive remarks. Your comments have helped us clarify several important aspects.\n\n## (W1) Influence of Outlier Channels\n\nAs mentioned, prior studies have shown that outlier channels and weights are common in large models [1][2], and the impact of such outliers has been actively addressed in areas like quantization [3][4]. We also agree that existence of outlier channels and weights are nature, which usually show high correlation with tokens like \".\", or special tokens. \n\nOn the other hand, while the existence of outlier is not a problem in general situations, it can give negative influences when training LoRA adapters. As illustrated in Figure 2, while full fine-tuning localizes the gradient impact, LoRA’s entire gradient update becomes disproportionately influenced by the single outlier.\n\nSuch structural limitation of LoRA leads to misalignment between LoRA updates and the gradient landscape shaped by full fine-tuning. In addition, since outlier only exists in certain layers, it results in different gradient scale across layers making the model harder to learn. \n\n## (W2) Flexibility and Capacity analysis of GraLoRA\n\nTo comapare the flexibility and capacity, we examine the column spaces of the weight update matrices from LoRA and GraLoRA within the framework of Grassmannian manifold. \n\nLet $R_{LoRA} = B_{LoRA}A_{LoRA}^\\top \\in \\mathbb{R}^{M \\times N}$ and $R_{GraLoRA} = B_{GraLoRA}A_{GraLoRA}^\\top \\in \\mathbb{R}^{M \\times N}$, as illustrated in Figure 5 of the paper. In vanilla LoRA, assuming column-wise linear in both $B_{LoRA}$ and $A_{LoRA}$, the rank of $R_{LoRA}$ is $r$. Thus, its column space lies in the Grassmannian manifold $\\mathcal{C}(R) \\in \\mathcal{Gr}(r, M)$, which denotes the set of all $r$-dimensional linear subpaces in $\\mathbb{R}^M$.\n\nIn GraLoRA, each sub-block $R_{i,j}$ has rank $\\frac{r}{k}$, as shown in Section 3.2. The rank of each column block $R_i = [R_{i,1} R_{i,2} \\cdots R_{i,k}]$ is there fore $r$ under the assumption that all sub-blocks are linearly independent, maximum rank of $R_{GraLoRA}$ can reach up to $kr$. Consequently, the column space of the GraLoRA update list in $\\mathcal{C}(R_{gralora}) \\in \\mathcal{Gr}(kr, M)$.\n\nIn terms of dimension, the Grassmannian manifold $\\mathcal{Gr}(r, M)$ has dimension $r(M-r)$, while $\\mathcal{Gr}(kr, M)$ has dimension $kr(M-kr)$. Under the common setting where $r \\ll M$, this implies that:\n$$\n\\textbf{dim}({\\mathcal{Gr}(r, M)}) < \\textbf{dim}({\\mathcal{Gr}(kr, M)})\n$$\nTherefore, the subspace learned by GraLoRA spans a higher-dimensional manifold, providing greater representational flexibility and capacity than standard LoRA. This geometric perspective further supports our theoretical claim that GraLoRA enhances expressivity by expanding the effective rank of the adaptation subspace.\n\n\n## (W4 & Q1 & Q3) Experiments with New Baselines and Variance Evaluation\n\n### 1. Commonsense Reasoning with Extensive PEFT Baselines\n\nWe conducted a new evaluation on the commonsense reasoning benchmark using the LLaMA3.2–3B model, comparing GraLoRA with a wide spectrum of PEFT methods, including full fine-tuning, LoRA, LoRA-XS, LoRA-SB, rsLoRA, PiSSA, BOFT, MELoRA, MoRA, and RaSA. All models were trained with rank 32. For consistency, we used the results reported in the LoRA-SB paper for LoRA-XS, LoRA-SB, rsLoRA, and PiSSA. Here rsLoRA, BOFT, and MELoRA are popular regularization techniques in LoRA. For BOFT, MELoRA, MoRA, RaSA, and GraLoRA, we conducted a learning rate sweep over {2e-4, 4e-4, 1e-3, 2e-3}, selecting the best-performing configuration per method. All other hyperparameters followed the settings in LoRA-SB.\n\n|Method|Rank|Params|BoolQ|PIQA|SIQA|HS|WG|ARC-c|ARC-e|OBQA|Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|FullFT|-|3.21B|70.4|85.6|80.5|91.9|85.0|75.3|88.5|81.9|82.4|\n|LoRA|32|48.63M|70.0|85.2|79.1|90.7|82.2|74.3|86.9|81.9|81.3|\n|LoRA-XS|96|1.81M|67.3|83.4|78.7|89.0|82.1|72.6|85.2|78.9|79.6|\n|LoRA-SB|96|1.81M|70.3|84.8|80.2|91.6|84.6|74.7|87.9|81.2|81.9|\n|rsLoRA|32|48.63M|69.8|85.1|78.9|90.5|82.0|74.2|86.7|81.7|81.1|\n|PiSSA|32|48.63M|70.1|85.4|79.4|90.9|82.7|74.6|87.2|81.8|81.5|\n|BOFT|32|48.48M|72.3|84.6|79.1|91.3|84.5|73.7|87.8|80.6|81.7|\n|MELoRA|32|48.63M|71.3|85.0|78.6|93.0|79.7|73.7|085.5|79.0|80.7|\n|MoRA|32|48.63M|72.4|86.1|80.1|92.3|84.8|76.8|88.8|**84.8**|83.3|\n|RaSA|32|48.63M|73.1|**87.5**|**81.1**|**93.7**|85.3|78.9|88.9|83.6|84.0|\n|GraLoRA|32|48.63M|**74.1**|86.5|80.8|93.8|**87.5**|**79.9**|**89.5**|**84.8**|**84.6**|\n\nGraLoRA achieves the highest average accuracy and ranks first on 5 out of 8 tasks. These results, combined with those in Table 3 of the main paper, confirm GraLoRA’s scalability and consistent improvements across model sizes, ranks, and PEFT baselines.\n\n### 2. Mathematical Reasoning\n\nWe further evaluated GraLoRA on a mathematical reasoning task using the MetaMathQA dataset for training and testing on the MATH benchmark. Two models, LLaMA3.2–1B and Qwen2.5–1.5B, were fine-tuned, mostly following the settings from Hu et al. [1], with ranks 64 and 128. Following our paper’s heuristic, we used $k=4$ for both.\n\n|Model|Rank|Method|Accuracy|\n|-|-|-|-|\n|LLaMA3.2–1B|64|LoRA|14.9%|\n|||GraLoRA|**15.2%**|\n|Qwen2.5–1.5B|64|LoRA|23.6%|\n|||GraLoRA|**25.7%**|\n||128|LoRA|24.7%|\n|||GraLoRA|**28.9%**|\n\nGraLoRA consistently outperforms LoRA, especially on the larger model, highlighting its robustness across architectures and tasks requiring symbolic reasoning.\n\n### 3. General Language Understanding (GLUE)\n\nWe evaluated GraLoRA on the GLUE benchmark using RoBERTa-base, an encoder-only architecture with eight subtasks. To ensure fair comparison with recent PEFT methods designed for parameter efficiency, we included two additional baselines: VeRA and FourierFT.\n\nFollowing the protocol of prior work, we excluded MNLI and QQP—two time-intensive tasks—which also meant we did not apply the MNLI-based tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results.\n\nWhile VeRA and FourierFT involve fewer trainable parameters, their training time is comparable to or even longer than LoRA with rank 8. Therefore, we set the LoRA and GraLoRA ranks to 8. Since this is a relatively low-rank setting, we also evaluated Hybrid GraLoRA by splitting the rank equally between LoRA and GraLoRA (i.e., 4+4), which is expected to be beneficial under such constraints.\n\nAll hyperparameters for GraLoRA followed those used in the VeRA implementation, except for learning rate, which was reduced by a factor of 5–10, as VeRA uses a learning rate approximately 10× larger than LoRA.\n\n|Method|Params|SST-2 (%)|MRPC (%)|CoLA (%)|QNLI (%)|RTE (%)|STS-B (%)|Avg (%)|\n|-|-|-|-|-|-|-|-|-|\n|FT|125M|94.8|**90.2**|63.6|92.8|78.7|**91.2**|85.2|\n|LoRA|0.3M|95.1|86.5|63.4|93.3|76.2|90.6|84.2|\n|VeRA|0.043M|94.6|89.5|**65.6**|91.8|78.7|90.7|85.2|\n|FourierFT|0.024M|94.2|90.0|63.8|92.2|79.1|90.8|85.0|\n|GraLoRA|0.3M|**95.2**|89.7|65.3|93.0|**80.9**|91.1|85.8|\n|Hybrid GraLoRA|0.3M|**95.2**|**90.2**|64.1|**93.4**|79.8|**91.2**|85.6|\n|Best GraLoRA|0.3M|**95.2**|**90.2**|65.3|**93.4**|**80.9**|**91.2**|**86.0**|\n\nGraLoRA demonstrates strong performance in this encoder-only setting and low-rank regime, outperforming all baselines in average score. Hybrid GraLoRA delivers the most robust results, achieving the best performance on 4 out of 6 tasks. These results indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and in non-LLM architectures.\n\n## (W3 & Q2) Image Task Experiments\n\n### 4. Diffusion Model Fine-Tuning\n\nWe applied GraLoRA to diffusion models by fine-tuning SDXL for personalization. We followed the official training setup from the HuggingFace diffusers repository, using the lambdalabs/naruto-blip-captions dataset. The dataset was split 90% for training and 10% for evaluation.\n\n|Method|CLIP Similarity (%)|DINOv2 Similarity (%)|\n|-|-|-|\n|LoRA|91.4|79.2|\n|GraLoRA|91.9|81.3|\n\nGraLoRA consistently outperformed LoRA in both CLIP and DINOv2 similarity scores, further demonstrating its generality and effectiveness beyond LLMs—including vision-text and generative architectures like diffusion models. \n\n**To conclude**, we sincerely thank the reviewer for their valuable suggestions and thoughtful critique. Your feedback prompted us to clarify the role of outlier channels in LoRA, deepen the theoretical analysis of GraLoRA’s expressivity via Grassmannian geometry, and conduct comprehensive new experiments—including comparisons against regularized LoRA variants such as MELoRA, as well as evaluations on mathematical reasoning, GLUE, and diffusion tasks. These additions, derived directly from your comments, will be incorporated into the revised manuscript to strengthen both its theoretical depth and empirical breadth."}, {"review_id": "W8lFJzusDk", "reviewer": "Reviewer_J2Cd", "summary": "The paper proposes GraLoRA, an improved version of LoRA that divides weight matrices into smaller blocks for independent adaptation. Experiments demonstrate that GraLoRA outperforms LoRA and other methods in tasks like code generation and reasoning, offering better accuracy without increasing computational cost.", "questions": "* Does the hyperparameter $k$ require tuning for different models? Is there a simple way to find an optimal $k$?\n* How does the method perform on math tasks?\n* What would the performance gap be between GraLoRA and other PEFT methods if the number of epochs were increased to 3?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* GraLoRA effectively addresses the limitations of LoRA by enhancing model capacity through granular, block-wise decomposition, improving performance without additional computational cost.\n* The method consistently outperforms existing PEFT techniques across code and reasoning tasks, demonstrating scalability and robustness across model sizes and rank settings.\n\nWeaknesses:\n* The paper focuses on visualizing the gradient dynamics in the down-projection matrices of FFT but does not investigate other matrices, such as those in self-attention layers.\n* In the code generation task, the paper applies the same learning rate across all methods with varying ranks, without conducting a grid search for optimal learning rates, which may not ensure a fair comparison.\n* While the paper suggests that LoRA suffers from outlier channels in larger-rank optimization, it uses the LionW optimizer in experiments, which applies sign updates to mitigate the influence of large gradients, and momentum, which helps reduce significant gradient fluctuations.", "was_revised": true, "final_justification": "The authors have addressed my concerns with extensive experimental results.", "rebuttal": "We sincerely thank you for your encouraging evaluation and the valuable suggestions for improvement. Below, we address each of your points and questions clearly. \n## (W1) GraLoRA's gradient dynamics on matrices\nIn the paper, we identify a fundamental misalignment between LoRA updates and the true gradient landscape, particularly in the presence of outlier channels in the input. Prior studies have shown that outlier channels and weights are common in large models [1][2], and the impact of such outliers has been actively addressed in areas like quantization [3][4].\n\nOur empirical analysis focuses on the down-projection layer of LLaMA3.1–8B, based on the findings from [2] where most outliers mostly existed in the down projection layers. However, any linear projection matrix with LoRA adapters—such as the Q, K, V projections in self-attention—can exhibit similar behavior. These matrices are functionally equivalent in structure and are therefore equally susceptible to the effects of outlier activations.\n\nTo further support this claim, we measured the kurtosis of input for the first 10 layers in LLaMA3.1-8B. High kurtosis is indicative of a heavy-tailed distribution, which implies the presence of strong outliers.\n\n|Projection|0|1|2|3|4|5|6|7|8|9|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Q/K/V|1117|1796|144|71|87|77|59|57|78|79|\n|O|570|239|57|109|31|42|37|24|17|14|\n|Gate|93|7|32|2|22|4|4|5|8|13|\n|Up|93|7|32|2|22|4|4|5|8|13|\n|Down|30567|1888483|13063|386|15855|183|156|4498|2598|99|\n\nAmong these, down-projection layers exhibited the highest kurtosis values, indicating an extremely heavy-tailed distribution and thus a high likelihood of dominant outlier channels. However, most other layers are also leptokurtic, meaning they too are likely to be influenced by outlier activations.\n\nThis supports our central claim: the phenomenon of outlier-induced gradient distortion is not limited to the down-projection layer, but is broadly relevant across various parts of the model that use LoRA adapters.\n## (W3 & Q2) Experiments on additional tasks with different Optimizer\n### 1. Mathematical Reasoning\nWe further evaluated GraLoRA on a mathematical reasoning task using the MetaMathQA for training and testing on the MATH data. Two models, LLaMA3.2–1B and Qwen2.5–1.5B, were fine-tuned, mostly following the settings from Hu et al. [5]. Following our paper’s heuristic, we used k = 4 for both rank 64 and 128. In addition, we replaced the LionW to AdamW, since LionW itself can mitigate the influence of large gradients.\n\n|Model|Rank|Method|Accuracy|\n|-|-|-|-|\n|LLaMA3.2–1B|64|LoRA|14.9%|\n|||GraLoRA|**15.2%**|\n|Qwen2.5–1.5B|64|LoRA|23.6%|\n|||GraLoRA|**25.7%**|\n||128|LoRA|24.7%|\n|||GraLoRA|**28.9%**|\n\nAs shown in the table, all GraLoRA showed superior performance for math task, for both models. This highlights the robustness of GraLoRA, for different architecture and hyper-parameters including the optimizer. \n\n### 2. General Language Understanding (GLUE)\nWe evaluated GraLoRA on the GLUE benchmark using RoBERTa-base and included two additional baselines: VeRA and FourierFT.\n\nFollowing the protocol of prior work, we excluded MNLI and QQP—two time-intensive tasks—which also meant we did not apply the MNLI-based tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results.\n\nWhile VeRA and FourierFT involve fewer trainable parameters, their training time is comparable to or even longer than LoRA with rank 8. Therefore, we set the LoRA and GraLoRA ranks to 8 for fair comparison. Since this is a relatively low-rank setting, we also evaluated Hybrid GraLoRA by splitting the rank equally between LoRA and GraLoRA (i.e., 4+4), which is expected to be beneficial under such constraints. All hyperparameters followed those used in the VeRA implementation, except for learning rate, which was reduced by a factor of 5 to 10.\n\n|Method|Params|SST-2|MRPC|CoLA|QNLI|RTE|STS-B|Avg|\n|-|-|-|-|-|-|-|-|-|\n|FT|125M|94.8|**90.2**|63.6|92.8|78.7|**91.2**|85.2|\n|LoRA|0.3M|95.1|86.5|63.4|93.3|76.2|90.6|84.2|\n|VeRA|0.043M|94.6|89.5|**65.6**|91.8|78.7|90.7|85.2|\n|FourierFT|0.024M|94.2|90.0|63.8|92.2|79.1|90.8|85.0|\n|GraLoRA|0.3M|**95.2**|89.7|65.3|93.0|**80.9**|91.1|85.8|\n|Hybrid GraLoRA|0.3M|**95.2**|**90.2**|64.1|**93.4**|79.8|**91.2**|85.6|\n|Best GraLoRA|0.3M|**95.2**|**90.2**|65.3|**93.4**|**80.9**|**91.2**|**86.0**|\n\nGraLoRA demonstrates strong performance in this low-rank regime, outperforming all baselines in average score. Hybrid GraLoRA delivers the most robust results, achieving the best performance on 4 out of 6 tasks. These results indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and in non-LLM architectures.\n## (W2 & Q3)  Fair Comparison with Hyper-parameter search\nWe conducted an additional experiment on the commonsense reasoning task using the Qwen2.5–1.5B model, extending the number of training epochs from 2 to 3. In response to the concern about fixed learning rates, we also performed a grid search over learning rates {1e-4, 3e-4, 5e-4} for each method and selected the best-performing configuration individually. \n\n|Model|Method|BoolQ|PIQA|SIQA|HS|WG|ARC-c|ARC-e|OBQA|Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen2.5–1.5B|LoRA|**68.5**|82.3|74.4|85.3|**75.2**|71.0|86.2|**84.2**|78.4|\n||MoRA|67.1|82.3|74.1|84.8|74.7|73.5|87.1|81.6|78.2|\n||RaSA|68.4|**83.4**|74.5|84.5|73.6|73.9|**87.5**|81.2|78.4|\n||GraLoRA|67.7|82.7|**75.3**|**86.1**|74.0|**74.2**|**87.5**|80.8|**78.5**|\n\nGraLoRA continues to achieve the highest average accuracy and ranks first on 4 out of 8 tasks, demonstrating robust performance even under altered training schedules and per-method parameter tuning. \n\nInterestingly, we observed that overall accuracy slightly decreased compared to the 2-epoch setting reported in the main paper. We believe this is due to overfitting, since we observed that training loss continued to decrease in the third epoch, while evaluation loss began to increase.\n\n### 3. Commonsense Reasoning with Extensive PEFT Baselines\nWe conducted a new evaluation on the commonsense reasoning benchmark using the LLaMA3.2–3B model, comparing GraLoRA with a wide spectrum of PEFT methods, including full fine-tuning, LoRA, LoRA-XS, LoRA-SB, rsLoRA, PiSSA, BOFT, MELoRA, MoRA, and RaSA. All models were trained with rank 32. For consistency, we used the results reported in the LoRA-SB paper for full fine-tuning, LoRA-XS, LoRA-SB, rsLoRA, and PiSSA. For other methods, we conducted a learning rate sweep over {2e-4, 4e-4, 1e-3, 2e-3}, selecting the best-performing configuration per method. All other hyperparameters followed the settings in LoRA-SB, including the usage of AdamW optimizer.\n\n|Method|Rank|Params|BoolQ|PIQA|SIQA|HS|WG|ARC-c|ARC-e|OBQA|Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|FullFT|-|3.21B|70.4|85.6|80.5|91.9|85.0|75.3|88.5|81.9|82.4|\n|LoRA|32|48.63M|70.0|85.2|79.1|90.7|82.2|74.3|86.9|81.9|81.3|\n|LoRA-XS|96|1.81M|67.3|83.4|78.7|89.0|82.1|72.6|85.2|78.9|79.6|\n|LoRA-SB|96|1.81M|70.3|84.8|80.2|91.6|84.6|74.7|87.9|81.2|81.9|\n|rsLoRA|32|48.63M|69.8|85.1|78.9|90.5|82.0|74.2|86.7|81.7|81.1|\n|PiSSA|32|48.63M|70.1|85.4|79.4|90.9|82.7|74.6|87.2|81.8|81.5|\n|BOFT|32|48.48M|72.3|84.6|79.1|91.3|84.5|73.7|87.8|80.6|81.7|\n|MELoRA|32|48.63M|71.3|85.0|78.6|93.0|79.7|73.7|085.5|79.0|80.7|\n|MoRA|32|48.63M|72.4|86.1|80.1|92.3|84.8|76.8|88.8|**84.8**|83.3|\n|RaSA|32|48.63M|73.1|**87.5**|**81.1**|**93.7**|85.3|78.9|88.9|83.6|84.0|\n|GraLoRA|32|48.63M|**74.1**|86.5|80.8|93.8|**87.5**|**79.9**|**89.5**|**84.8**|**84.6**|\n\nGraLoRA achieves the highest average accuracy and ranks first on 5 out of 8 tasks. These results, combined with those in Table 3 of the main paper, confirm GraLoRA’s scalability and consistent improvements across model sizes, ranks, and PEFT baselines.\n\n### 4. Diffusion Model Fine-Tuning\nFinally, we applied GraLoRA to diffusion models by fine-tuning SDXL for personalization. We followed the official training setup from the HuggingFace diffusers repository, using the lambdalabs/naruto-blip-captions dataset. The dataset was split 90% for training and 10% for evaluation.\n\n|Method|CLIP Similarity (%)|DINOv2 Similarity (%)|\n|-|-|-|\n|LoRA|91.4|79.2|\n|GraLoRA|91.9|81.3|\n\nGraLoRA consistently outperformed LoRA in both CLIP and DINOv2 similarity scores, further demonstrating its generality and effectiveness beyond LLMs—including vision-text and generative architectures like diffusion models. \n## (Q1) Heuristic for Hyperparameter $k$\nWe have further analyzed the hyper-parameter sweep result in the ablation study section. Based on the sweep result and experience, $r/k^2 \\approx 8$ yielded stable performance under varying configurations including model and task. Thus, we set GraLoRA $k=2$ for ranks under 32, and $k=4$ for higher ranks (64, 128) in the experiments. \n\n**To conclude**, we sincerely thank the reviewer for the thoughtful feedback, which has significantly contributed to improving our work. In response, we have added new theoretical analysis on the generality of gradient distortion beyond down-projection layers, extended evaluations on mathematical reasoning and GLUE, and included comprehensive comparisons with regularized LoRA variants such as MELoRA. We have also investigated the impact of training duration and hyperparameter tuning, and validated GraLoRA’s generality through experiments on diffusion models. These additions will be incorporated into the revised manuscript to reflect the valuable insights raised in your review.\n\n[1] Bondarenko, et al., \"Understanding and Overcoming the Challenges of Efficient Transformer Quantization\"\n[2] Yu, et al., \"The Super Weight in Large Language Models\"\n[3] Xiao et al., \"SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models\"\n[4] Liu et al., \"SpinQuant: LLM quantization with learned rotations\"\n[5] Hu et al., \"LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models\""}, {"review_id": "0IS9xC42Dw", "reviewer": "Reviewer_QiZN", "summary": "This paper proposes Granular Low-Rank Adaptation (GraLoRA), a parameter-efficient fine-tuning (PEFT) method. The authors identify that Low-Rank Adaptation (LoRA) can show stagnating or declining accuracy at higher ranks (e.g., 32-64). The paper attributes this to a \"structural bottleneck\" where outlier input channels with high activation values disproportionately influence and distort gradient updates. To address this, GraLoRA partitions weight matrices into a grid of sub-blocks, with each block having its own independent low-rank adapter. The paper claims this design localizes gradient updates, enhances expressive capacity, and mitigates the outlier issue, all while maintaining a comparable parameter count and computational cost to standard LoRA.", "questions": "1. Could you address the motivation problem in weakness 1?\n\n2. Could you add baselines such as FourierFT and BOFT?", "limitations": "Not so clear. The future work part should be revised to discuss more directly on limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. The method is designed to incur the same parameter count and comparable computational FLOPs as standard LoRA. The authors analyze the trade-offs, such as memory overhead, and propose a \"Hybrid GraLoRA\" approach to improve performance in low-rank scenarios where the base method may be too constrained.\n\n2. The paper proposes GraLoRA, a method that partitions the weight matrix into independent blocks, each with a local low-rank adapter, to isolate gradient updates. The authors provide a theoretical analysis arguing that this structure increases the model's expressive capacity by raising the effective rank of the update matrix.\n\nWeakness:\n\n1. The paper's core motivation is that LoRA's performance degrades at high ranks (e.g., > 64) because outlier channels distort the entire gradient update. The explanation for why this happens is not fully elaborated. Intuitively, increasing the rank should provide the update matrix with more capacity, allowing it to better approximate the localized gradients of FFT and thus representing a smooth transition toward FFT performance. The paper does not provide a clear theoretical justification for why increasing the rank would intensify the \"entangled influence\" of outliers rather than providing more degrees of freedom to isolate their impact, creating a conceptual question around the mechanism that causes a performance dip at high, but not full, ranks.\n\n2. The method introduces a new hyperparameter, 'k', which defines how many ways the matrix is partitioned. The paper shows that performance is sensitive to this value and that an improper choice can hurt performance, adding a layer of tuning complexity not present in vanilla LoRA.\n\n3. The paper compares GraLoRA against three other PEFT methods: standard LoRA, MORA, and RaSA. While these are relevant and contemporary baselines, they all belong to the family of low-rank decomposition techniques. The evaluation does not include comparisons against other established families of PEFT methods, such as FourierFT [1] or BOFT [2], which claims high rank adaptation. Both of them are readily available in standard libraries like Hugging Face PEFT.\n\n[1] Gao, Ziqi, et al. \"Parameter-efficient fine-tuning with discrete fourier transform.\" arXiv preprint arXiv:2405.03003 (2024).\n\n[2] Liu, Weiyang, et al. \"Parameter-efficient orthogonal finetuning via butterfly factorization.\" arXiv preprint arXiv:2311.06243 (2023).", "was_revised": true, "final_justification": "The authors have addressed all my concerns.", "rebuttal": "Thank you for your review and for highlighting both the strengths of our work and areas for improvement. We have addressed each of your comments in detail below.\n## (W1 & Q1) Why increasing the rank intensifies the entangled influence of outliers\n\nAs discussed in Section 2.1 and 2.2 of the paper, we analyze the gradient dynamics of LoRA to explain why it suffers at higher ranks, particularly in the presence of outlier channels. \n\nGiven the LoRA weight update $R = BA^\\top$ where $B \\in \\mathbb{R}^{M \\times r}$, $A \\in \\mathbb{R}^{N \\times r}$, and input $X \\in \\mathbb{R}^{N \\times T}$, the gradients respect to $A$ and $B$ are:\n$$\\frac{\\partial{L}}{\\partial{A^\\top}} = B^\\top \\frac{\\partial{L}}{\\partial{Y}}X^\\top, \\  \\frac{\\partial{L}}{\\partial{B}}=\\frac{\\partial{L}}{\\partial{Y}}X^\\top A$$\nThe corresponding gradient of $R$ becomes:\n$$\\frac{\\partial{L}}{\\partial{R}}=\\frac{\\partial{L}}{\\partial{B}}A^\\top + B\\frac{\\partial{L}}{\\partial{A^\\top}}=\\frac{\\partial{L}}{\\partial{Y}}X^\\top AA^\\top + BB^\\top \\frac{\\partial{L}}{\\partial{Y}}X^\\top$$\nNote that $X$ appears inside both terms. In particular, the first term $\\frac{\\partial{L}}{\\partial{Y}}X^\\top AA^\\top$, shows that $X$ is multiplied between two. If $X$ contains an outlier channel, a row with abnormally high magnitude, the channel will influence the entire update due to the matrix multiplications. \n\nMoreover, as the rank $r$ increases, the norm of the Gramm matrx $AA^\\top$ also increases. This amplifies the effect of the outlier channel, widening the scale  gab between outlier and non-outlier contributions and distorting the overall gradient landscape. In other words, the entanglement becomes worse as $r$ grows, further suppressing gradients from informative but less dominant channels.\n\nThis phenomenon is empirically validated in Figure 4, where increasing rank leads to disproportionately larger gradients, driven by the outlier channel. These observations highlight the structural limitation of LoRA at high ranks, motivating the design of GraLoRA to localize such influence.\n## (W3 & Q2) Additional experiments with new baselines on varying tasks and models\n\nThank you for the suggestion. While the main comparisons in our paper focused on low-rank decomposition methods, we have since conducted additional experiments to evaluate the generality and robustness of GraLoRA across a broader range of model architectures, tasks, and PEFT baselines—including BOFT and FourierFT.\n### 1. Commonsense Reasoning with Extensive PEFT Baselines\n\nWe conducted a new evaluation on the commonsense reasoning benchmark using the LLaMA3.2–3B model, comparing GraLoRA with a wide spectrum of 10 PEFT baselines, including full fine-tuning and all models were trained with rank 32. For consistency, we used the results reported in the LoRA-SB paper for LoRA-XS, LoRA-SB, rsLoRA, and PiSSA. For other methods, we conducted a learning rate sweep over {2e-4, 4e-4, 1e-3, 2e-3}, selecting the best-performing configuration per method. All other hyperparameters followed the settings in LoRA-SB.\n\n|Method|Rank|Params|BoolQ|PIQA|SIQA|HS|WG|ARC-c|ARC-e|OBQA|Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|FullFT|-|3.21B|70.4|85.6|80.5|91.9|85.0|75.3|88.5|81.9|82.4|\n|LoRA|32|48.63M|70.0|85.2|79.1|90.7|82.2|74.3|86.9|81.9|81.3|\n|LoRA-XS|96|1.81M|67.3|83.4|78.7|89.0|82.1|72.6|85.2|78.9|79.6|\n|LoRA-SB|96|1.81M|70.3|84.8|80.2|91.6|84.6|74.7|87.9|81.2|81.9|\n|rsLoRA|32|48.63M|69.8|85.1|78.9|90.5|82.0|74.2|86.7|81.7|81.1|\n|PiSSA|32|48.63M|70.1|85.4|79.4|90.9|82.7|74.6|87.2|81.8|81.5|\n|BOFT|32|48.48M|72.3|84.6|79.1|91.3|84.5|73.7|87.8|80.6|81.7|\n|MELoRA|32|48.63M|71.3|85.0|78.6|93.0|79.7|73.7|085.5|79.0|80.7|\n|MoRA|32|48.63M|72.4|86.1|80.1|92.3|84.8|76.8|88.8|**84.8**|83.3|\n|RaSA|32|48.63M|73.1|**87.5**|**81.1**|**93.7**|85.3|78.9|88.9|83.6|84.0|\n|GraLoRA|32|48.63M|**74.1**|86.5|80.8|93.8|**87.5**|**79.9**|**89.5**|**84.8**|**84.6**|\n\nGraLoRA achieves the highest average accuracy and ranks first on 5 out of 8 tasks. These results, combined with those in Table 3 of the main paper, confirm GraLoRA’s scalability and consistent improvements across model sizes, ranks, and PEFT baselines.\n### 2. Mathematical Reasoning\n\nWe further evaluated GraLoRA on a mathematical reasoning task using the MetaMathQA for training and testing on the MATH data. Two models, LLaMA3.2–1B and Qwen2.5–1.5B, were fine-tuned, mostly following the settings from Hu et al. [1], with ranks 64 and 128. Following our paper’s heuristic, we used $k=4$ for both.\n\n|Model|Rank|Method|Accuracy|\n|-|-|-|-|\n|LLaMA3.2–1B|64|LoRA|14.9%|\n|||GraLoRA|**15.2%**|\n|Qwen2.5–1.5B|64|LoRA|23.6%|\n|||GraLoRA|**25.7%**|\n||128|LoRA|24.7%|\n|||GraLoRA|**28.9%**|\n\nGraLoRA consistently outperforms LoRA, especially on the larger model, highlighting its robustness across architectures and tasks requiring symbolic reasoning.\n### 3. General Language Understanding (GLUE)\n\nWe evaluated GraLoRA on the GLUE benchmark using RoBERTa-base, an encoder-only architecture with eight subtasks. To ensure fair comparison with recent PEFT methods designed for parameter efficiency, we included two additional baselines: VeRA and FourierFT.\n\nFollowing the protocol of prior work, we excluded MNLI and QQP—two time-intensive tasks—which also meant we did not apply the MNLI-based tricks for MRPC, RTE, and STS-B (as used in the original LoRA paper). Accordingly, we retrained LoRA on these tasks without this optimization and report updated results.\n\nWhile VeRA and FourierFT involve fewer trainable parameters, their training time is comparable to or even longer than LoRA with rank 8. Therefore, we set the LoRA and GraLoRA ranks to 8. Since this is a relatively low-rank setting, we also evaluated Hybrid GraLoRA by splitting the rank equally between LoRA and GraLoRA (i.e., 4+4), which is expected to be beneficial under such constraints.\n\nAll hyperparameters for GraLoRA followed those used in the VeRA implementation, except for learning rate, which was reduced by a factor of 5 to 10.\n\n|Method|Params|SST-2 (%)|MRPC (%)|CoLA (%)|QNLI (%)|RTE (%)|STS-B (%)|Avg (%)|\n|-|-|-|-|-|-|-|-|-|\n|FT|125M|94.8|**90.2**|63.6|92.8|78.7|**91.2**|85.2|\n|LoRA|0.3M|95.1|86.5|63.4|93.3|76.2|90.6|84.2|\n|VeRA|0.043M|94.6|89.5|**65.6**|91.8|78.7|90.7|85.2|\n|FourierFT|0.024M|94.2|90.0|63.8|92.2|79.1|90.8|85.0|\n|GraLoRA|0.3M|**95.2**|89.7|65.3|93.0|**80.9**|91.1|85.8|\n|Hybrid GraLoRA|0.3M|**95.2**|**90.2**|64.1|**93.4**|79.8|**91.2**|85.6|\n|Best GraLoRA|0.3M|**95.2**|**90.2**|65.3|**93.4**|**80.9**|**91.2**|**86.0**|\n\nGraLoRA demonstrates strong performance in this low-rank regime, outperforming all baselines in average score. Hybrid GraLoRA delivers the most robust results, achieving the best performance on 4 out of 6 tasks. These results indicate that GraLoRA maintains high effectiveness even under constrained parameter budgets and in non-LLM architectures.\n### 4. Diffusion Model Fine-Tuning\n\nFinally, we applied GraLoRA to diffusion models by fine-tuning SDXL for personalization. We followed the official training setup from the HuggingFace diffusers repository, using the lambdalabs/naruto-blip-captions dataset. The dataset was split 90% for training and 10% for evaluation.\n\n|Method|CLIP Similarity (%)|DINOv2 Similarity (%)|\n|-|-|-|\n|LoRA|91.4|79.2|\n|GraLoRA|91.9|81.3|\n\nGraLoRA consistently outperformed LoRA in both CLIP and DINOv2 similarity scores, further demonstrating its generality and effectiveness beyond LLMs—including vision-text and generative architectures like diffusion models. \n## (W2) Analysis for Hyperparameter $k$\n\nWe acknowledge that introducing $k$—which determines the granularity of matrix partitioning—adds a degree of tuning not present in vanilla LoRA. However, we empirically found that setting $r/k^2 \\approx 8$ provides consistently strong results across different models and tasks. Based on this, we use $k=2$ for ranks up to 32, and $k=4$ for higher ranks (64 and 128). All experiments in the main paper and during the rebuttal strictly follow this heuristic.\n\nThus, while $k$ is a new hyperparameter, we provide a practical and robust guideline that eliminates the need for exhaustive tuning in most settings. \n## (Limitations)\nWe appreciate the suggestion to more explicitly frame future directions around current limitations. Two key limitations were initially considered: Effectiveness at low ranks (e.g., $r=8$) and Hyperparameter sensitivity of $k$.\n\nIn small ranks, the per-block expressiveness of GraLoRA may diminish. To address this, we proposed the Hybrid GraLoRA architecture, which retains the fine-grained structure of GraLoRA while incorporating standard LoRA to maintain sufficient expressivity. Empirical results confirm that this approach remains competitive even in low-rank regimes (see GLUE experiment).\n\nAs noted, we provide a simple and effective heuristic for setting $k$. However, to eliminate manual tuning altogether, we believe this limitation can be addressed more fundamentally. As suggested in the paper’s future work section, adaptive or learned partitioning could allow the model itself to determine the optimal granularity, replacing the need for a fixed $k$. This would significantly enhance the usability and flexibility of GraLoRA in practical deployments.\n\n**To conclude**, we sincerely thank the reviewer for the thoughtful and detailed feedback. Your comments have helped us clarify the theoretical foundations of GraLoRA and broaden our empirical validation across diverse model families, tasks, and PEFT baselines. In response, we have incorporated additional theoretical analysis on LoRA’s gradient behavior and GraLoRA’s mitigation strategy, and we have conducted extensive new experiments—including comparisons against BOFT, FourierFT, and applications to GLUE and diffusion models. These results, along with revised discussions on tuning complexity and limitations, will be included in the updated manuscript.\n\n[1] Hu et al., \"LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models\", EMNLP 2023"}]

Author rebuttal:

Meta-review: This paper proposes GraLoRA, a granular low-rank adaptation approach designed to improve LoRA’s performance at higher ranks by partitioning weight matrices into sub-blocks. The work is well-motivated, technically sound, and supported by extensive experiments demonstrating clear improvements over standard LoRA.

Reviewers appreciated the clarity of the presentation, the empirical strength of the results, and the authors’ detailed rebuttal, which effectively addressed initial concerns. Several reviewers raised their scores after rebuttal, citing strong empirical evidence and useful clarifications.

Remaining concerns are relatively minor: some reviewers noted that the theoretical analysis could be more rigorous in finite settings, and that deployment considerations (e.g., adapter switching on edge devices) may need further exploration. These issues, however, do not undermine the overall contribution.

Overall, this paper received 1 Accept and 3 Borderline Accept.  The consensus is that this paper makes a meaningful and timely contribution to parameter-efficient fine-tuning. This meta-reviews concurs, and recommend to **Accept** this paper. The authors are encouraged to revise their paper carefully by taking into account all the comments and discussions.
Decision: Accept (spotlight)