Gradient Intrinsic Dimensionality Alignment：Narrowing The Gap Between Low-Rank Adaptation and Full Fine-Tuning
Authors: Jingqi Ye
haonan he
Minglei Li
Fujun Han
Tao Chen
Peng Ye
Venue: ICLR 2026 Poster (2026)
Keywords: PEFT
LoRA
Gradient Intrinsic Dimension
Adaptive Alignment
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Parameter-Efficient Fine-Tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA) and its variants, have emerged as critical tools for adapting large pretrained models under limited computational resources. However, a notable performance gap persists between these LoRA methods and Full Fine-Tuning (FFT). 
In this paper, we investigate a key yet overlooked cause of this gap: the relationship between LoRA's low-rank adaptation subspace and true effective update directions of FFT gradients, which we define as the **gradient intrinsic dimensionality**.
To systematically quantify this dimension, we first propose a novel entropy-based estimator, uncovering substantial discrepancies (up to more than 100x) between the rank of LoRA and the gradient intrinsic dimensionality. Motivated by this finding, we introduce **RaLoRA**, which adaptively aligns the ranks of LoRA adapters with layer-specific gradient intrinsic dimensions, without increasing the number of overall parameters. We further extend this approach into **RaLoRA-Pro**, integrating intra-layer rank alignment and inter-layer parameter reallocation guided by loss sensitivity, enabling finer-grained capacity relocation under comparable parameters. Extensive experiments demonstrate the effectiveness of our methods. Specifically, compared to vanilla LoRA, our methods achieve more than +5\% improvement on GLUE, +0.57 on MT-Bench, +5.23\% on GSM8K, +5.69\% on HumanEval, and +1.58\% on image classification, confirming consistent and substantial performance gains across diverse tasks and modalities.

Published as a conference paper at ICLR 2026
GRADIENT INTRINSIC DIMENSIONALITY ALIGNMENT: BRIDGING THE GAP BETWEEN Low- BANK ADAPTA- TION AND FULL FINE-TUNING
Jingqi Ye Haonan Hel2* Minglei Li
Fujun Han
Tao Chen
Peng Yel.5t
Shanghai Artificial Intelligence Laboratory
University of Science and Technology of China
Fudan University 'The Chinese University of Hong Kong, Shenzhen
"The Chinese University of Hong Kong "Shanghai Innovation Institute amzingye@naa 1. www.sstcce.. edu.cn
ABSTRACT
Parameter-Efficient Fine-Tuning (PEFT) techniques, such as Low-Rank Adapta- tion (LoRA) and its variants, have emerged critical tools for adapting large pre- trained models under limited computational resources. However, notable perfor- mance gap persists between these LoRA methods and Full Fine-Tuning (FFT). this paper, we investigate key yet overlooked cause of this gap: the relationship between LoRA's low-rank adaptation subspace and true effective update directions of FFT gradients, which we define as the gradient intrinsic dimensionality. To systematically quantify this dimension, we first propose a novel entropy-based es- timator, uncovering substantial discrepancies (up to more than 100x) between the rank of LoRA and the gradient intrinsic dimensionality. Motivated by this find- ing. introduce RaLoRA, which adaptively aligns the ranks of LoRA adapters with layer-specific gradient intrinsic dimensions, without increasing the number of overall parameters. We further extend this approach into RaL-oRA-Pro, inte- grating intra-layer rank alignment and inter-layer parameter reallocation guided by loss sensitivity, enabling finer-grained capacity relocation under comparable parameters. Extensive experiments demonstrate the effectiveness of our meth- ods. Specifically, compared to vanilla LoRA, our methods achieve more than +5% improvement on GLUE, +0.57 on MT-Bench, +5.23% on GSM8K, +5.69% on HumanEval, and +1.58% on image classification, confirming consistent and substantial performance gains across diverse tasks and modalities.
INTRODUCTION
Large language models (LLMs) have shown remarkable generalization abilities across broad range of downstream tasks YYu et al., 2025: Wang et al.) 2019] Adiwardana et al., 2020). However, full fine-tuning (FFT) LLMS task-specific scenarios remains challenging due to the high GPU memory requirement. To address this challenge, various Parameter- Efficient Fine- Tine-Tuning (PEFT) methods (Houlsby et al.] 2019 Hu al. 2022 Li & Liang 2021 Lester et al.) 2021 Nguyen et al. 2023) have been proposed to adapt pretrained LLMs by updating a small number of pa- rameters, hence reducing the memory requirement overhead. Among them, Low-Rank Adaptation (LoRA) (Hu et al., 2022) has become widely adopted method due to its advantages, including strong empirical performance, zero inference latency, and easy implementation. LoRA approxi- mates the weight update AW E Rddoutxdin using low-rank matrices A E Brxd... and B E Rdoue where r < min (din. d thereby enabling efficient fine-tuning tening by updating low-rank matrices with constrained resources. Despite its efficiency and strong performance simple tasks, LoRA still underperforms FFT on complex tasks such as mathematical reasoning (Biderman et al. 2024
Zhao et al. 2024).
Numerous variants LoRA have been proposed aiming to bridge this performance gap, broadly categorized into three directions (Mao al. 2025): (1) Rank Augmentation, which dynamically
"Equal contribution.
"orrespondiig author; 2.01 2001007200 201100200390 fudan. edu.en

---

Published as a conference paper at ICLR 2026
(o) Intrinsic Dimension
(b) Accuracy
Venilla LoRA
75.60
RaLoRA
324
GID
RaL-ORA- Pro
LS-
72
200
71
150
09
600
70
0
50
67.78
Loyer.0.. .0.qq
Loyer.8.. 8,
Layer 10. .k
Loyer .oerrr.1.00 33.00
LoRA
RoLoRA
RaLoRA-Pro LS-LoRA
Figure 1: Intrinsic dimensionality and performance comparison across LoRA Variants and FFT on LLaMA3. -SB-Base fine-tuned with MetaMathQA. GID denotes the gradient intrinsic dimension- ality. LS-LoRA uses loss sensitivity-guiel rank reallocation; RaLoRA and RaL A--Prrro are our methods. Note that all methods except FFT have comparable numbers of trainable parameters.
reallocates the LoRA ranks across layers based on importance scores (He al. 2025 Zhang al,
2023a b). or enhances expressiveness by stacking multiple LoRA modules (Ren et al.
2024 Lialin
al. 2023 Jiang et al., 2024); (2) Optimization of Training Dynamics, such stabilizing the scaling factor (Kalajzzievski 2023), assigning asymmetrical learning rates to A and B (Hayou al., 2024). or decoupling the directional updates and magnitude updates (Liu et 2024a); and
(3) Improved Initialization, including principal singular component-base (Meng et al. 2024) and
orthonormal QR-based initialization strategies (Büyükakyüz 2024). accelerating convergence and
adaptation efficiency.
Nevertheless, existing approaches largely overlook a fundamental issue causing LoRA's perfor- mance shortfall: the misalignment between LoRA's low-rank update subspace and the true effective update directions of FFT gradients, which we define as the gradient intrinsic dimensionality. Our empirical investigation in Figure [ reveals that the gradient intrinsic dimensionality in FFT typ- ically occupies a significantly larger subspace (up to 300) than the fixed-rank structure permitted by LoRA (usually set to 8), potentially limiting its expressiveness, particularly complex down- stream tasks. Indeed, as illustrated Figure [b), aligning LoRA's rank with the gradient intrinsic dimensionality substantially improves fine-tuning performance. However, realizing such alignment presents two primary challenges: (1) how to accurately estimate per-layer gradient intrinsic dimen- sionality-a largely unexplored issue; and (2) how to develop intrinsic dimensionality alignment strategies within fixed parameter budgets.
To address these challenges, we first propose a novel entropy-based estimator quantify the gradi- ent intrinsic dimensionality across layers systematically. This estimator is conceptually orthogonal to most existing LoRA variants and can even benefit other PEFT methods. Leveraging this esti- mator, we introduce RaLoRA, which utilizes block-diagone decomposition (Ren et al. 2024) to dynamically align LoRA's ranks with the intrinsic dimensionality of the corresponding layer-wise gradients without increasing the number of trainable parameters. We further extend this approach to RaL -RA-Poo which jointly optimizes intra-layer GID alignment and inter-layer parameter al- location guided by layer-wise loss sensitivity scores (Zhang et al. 2023b), offering more refined capacity relocation within comparable parameter budgets. Empirical results (Figure) I confirm that methods significantly reduce the performance gap between LoRA and FFT.
Our main contributions are summarized as follows:
1. We propose novel mnarepy-based estimator that systematically quantifies the intrinsic di- mensionality of gradients across layers. This estimator is conceptually orthogonal to most existing LoRA variants and can even benefit other PEFT methods.
2. We introduce RaLoRA, which adaptively aligns the rank of the LoRA module with the corresponding gradient intrinsic dimensionality without increasing the overall parame- ter count. We further present RaLoRA-Pro, which integrates GID alignment with loss sensitivity-guided parameter reallocation for fine- fine-grained capacity control.
2

---

Published as a conference paper at ICLR 2026
3. Extensive experiments demonstrate the effectiveness of methods across diverse tasks and modalities. Specifically, compared to vanilla LoRA, our approach achieves over 5% improvement on GLUE, +0.57 on MT-Bench, +5.23% on GSM8K, +5.69% on Hu- manEval, and +1.58% .58% on image classification. These consistent results confirm that our methods can significantly narrow the performance gap to full fine-tuning.
2 RELATED WORKS
2.1 LORA AND ITS VARIANTS
To efficiently adapt pretrained models under limited resources, PEFT methods such as LoRA (Hu eet al. 2022) have gained prominence, though a noticeable performance gap remains compared to full fine-tuning. To bridge this gap, numerous variants of LoRA have emerged, which can be broadly categorized into three representative directions (Mao et al., (2025). (1) Rank Augmenta- tion: Several methods aim to improve parameter utilization by dynamically reallocating the rank across layers based on importance scores (He et al. 2025 Zhang et al. 2023a(b), Others, such as MELoRA (Ren al. (2044) and ReLoRA (Lialin et al. 2023), increase the equivalent rank by stacking multiple LoRA modules during fine-tuning- Furthermore, recent approaches approximate high-rank or full-rank updates through novel structural compositions, such as incorporating fixed random matrices (Albert et al., 2025) or leveraging the Hadamard product (Huang al., (025). (2) Optimization of Training Dynamics: This line of work focuses on improving training dynamics. For instance, RSLoRA (Kalajdzievski 2023) stabilizes training by optimizing the scaling factor, while LoRA+HHayou et al.) (0024) assigns separate learning rates to the LoRA matrices A and B to stabilize the training process. DoRÁ (Liu et al. (024a). in contrast, decouples the directional updates and magnitude updates to better align with task-specific gradients. (3) Improved Initial- ization: To accelerate convergence and improve adaptation, PiSSA (Meng al. 2024) initializes LoRA modules with the principal singular components of the pretrained weight matrix, whereas OLoRA (Büyükakyūz (024) adopts orthonormal bases from QR decomposition for initialization, enhancing both stability and learning efficiency. While these methods offer improvements from various perspectives, they largely neglect critical factor underlying the performance gap between LoRA and FFT: the misalignment between the low-rank update subspace of LoRA and the gradient intrinsic dimensionality.
2.2 EFFECTIVE RANK
The concept of effective rank was first introduced stable, continuous proxy for matrix rank based on the Shannon entropy of singular value distribution (Roy & Vetterli 2007). Unlike conventional defined rank, effective rank smoothly quantifies the intrinsic dimensionality and redundancy in data, making it particularly suitable for optimization tasks under uncertainty. Effective rank has found success in various domains. has been applied signal processing for dimensionality reduction and source separation (Deligiannidis & Doucet 2019 ElMossallamy et al.] 2020), and in machine learning for feature selection by capturing latent data complexity (Paul & Drineas 2016). More recently, Yang et al. (Yang & Wang 2023) used effective rank to assess the evolving quality of em- beddings in Diffusion Probabilistic Models, while Nikitin et al. (Nikitin et al.) 2024) employed it to distinguish between semantic diversity and true uncertainty in L.MMs. Garrido et al. (Garrido et al.) 2023) introduced RankMe, a self-supervised evaluation metric based on effective rank, effectively predicting downstream performance without labeled data. Inspired by its success across these do- mains, our work is the first attempt to introduce effective rank as a novel estimator for the intrinsic dimensionality of gradient matrices in the context of LoRA, previously unexplored direction.
3
METHOD
3.1
BRIEF OVERVIEW OF LORA
LoRA builds on the hypothesis that fine-tuning updates to pretrained weights lie in low-rank sub- spaces. It models the weight update AW E ggdautxdnn using low-rank matrices E gexdis and B E Rdooo as AW a BA, while keeping the pretrained weight Wi frozen. This design yields strong empirical performance with minimal memory requirement overhead, making LoRA one of the most widely adopted PEFT methods.
3

---

Published as a conference paper at ICLR 2026
Theoretically, LoRA adapter acts implicit gradient compressor: at step t, it projects the full gradient G2 onto a low rank subspace (see Appendix for derivation):
( = - (B,B, G. + G,A, At).
(1)
Consequently, the rank of LoRA fundamentally constrains both its gradient compression rate and its expressive power, which directly causes the performance gap observed relative full fine-tuning.
3.2 GRADIENT INTRINSIC DIMENSIONALITY ESTIMATION VIA ENTROPY-BASED ESTIMATOR
Building on the view that LoRA adapters act as gradient compressors, we introduce the notion of Gradient Intrinsic Dimensionality (GID) for the first time, which characterizes the true effective up- date directions of gradients of full fine-tuning. When GID substantially exceeds the preset rank r, such fixed-rank projections induce severe information loss and critically limit performance. Cur- rently, most existing LoRA variants largely overlook this structural mismatch. This motivates us to develop principled and adaptive framework that estimates GID and aligns LoRA ranks accordingly, thereby maximizing representational capacity under a fixed parameter budget. A straightforward approach estimating the intrinsic dimensionality of G is apply singular value decomposition (SVD) and count the number of singular values above a small threshold 8:
rank(G) max (ilo; z),
where 01 > 02 > ..... on are the singular values of the gradient matrix G. However, this approach sensitive the choice of E, potentially limiting its robustness and generalizability when estimating layer-wise intrinsic dimensionalities across tasks.
To address these limitations, we introduce entropy- based estimator inspired by the concept of effective rank (Roy & Vetterli| 2007: [Su (025), which quantifies GID by examining the entropy of singular value distributions. Specifically, given the singular values σi, On of the gradient matrix G₁, the intrinsic dimensionality is defined as:
erank(Gi) exp
Σ log Pt
1j=1
where pt denotes the normalized singular value distribution. See Appendix
B for details.
Unlike threshold-based SVD methods that sensitive to hyperparameters and lack adaptability across layers and tasks, our entropy-based estimator offers a principled and robust measure of the gradient matrices (see the empirical experiment in Appendix D). By quantifying the effective degrees of freedom based on the distribution of singular values, this method adaptively captures the underlying structure of gradients derived from full fine-tuning. This underexplored in the context of LoRA- enables us to move beyond heuristic or importance-bart based rank assignment and instead align LoRA ranks with the actual gradient subspace complexity. As general and lightweight module, our estimator not only underpins the design of RaLoRA and RaLoRA-Pro but can also be incorporated into broader SoRA-basce or PEFT frameworks to guide adaptive GID alignment grounded in tasc-ipc gradient behavior.
3.3 RALORA: RANK-ALIGNED LORA WITH GRADIENT INTRINSIC DIMENSIONALITY
We introduce RaLoRA, which generalizes LoRA by dynamically aligning its rank with the intrinsic dimensionality of each layer's gradient. As depicted in Figure D adopt structured parallel decomposition (Ren et al. (2024) that splits the LoRA matrices A and B into ny mini-blocks, forming block-diagonal update matrix:
B1 0
0
BzA2
0
BA
Bay Ani
where A, E (da(d//ma) and B; E R(daa/nixxr The number of blocks n, for layer is determined by the estimated intrinsic dimension of its gradient G
crank(G())
6
log2
= 2 S.t. 1 nq S n mmaax:
(4)
4

---

Published as a conference paper at ICLR 2026
Vanilla LoRA
(I)
RaLoRA
(I)) RaLoRA-pro
Backward
Aliggmet
Algument
Wy
PW
Tal PWy
(2)
[2]
Adaptive & Importance Alggmment
Adaptive
Dimension
Alggnment
: Ki.lls.
W
Wi
(1.2) Adaptive Dimension Alignment
(IL.) Adaptive Dimension & Importance Alignment SVD
entrupy
Gx
or e" PuinoPu 2 (
Pulespurt 1o0s 10an
4
((W) ang(W O GDD de 21 4
Precat d/s + dl
Normalized
Intrinsic Dirvension
((1 dbe
Gradients
Singalae Valuae
(
Gg
Protat
done Foer
Figure 2: Overview of the proposed methods compared to vanilla LoRA. (I) RaLoRA adaptively aligns the rank of each LoRA adapter with the intrinsic dimensionality of full fine-tumne tine-tuning gradients estimated via an entropy-based estimator, using block-diagonal decomposition while keeping the parameters fixed. (II) RaLoRA-Pro extends this by integrating loss sensitivity-guiee -uiide inter-layer reallocation strategy, achieving dual alignment under comparable parameters.
where r is the vanilla LoRA rank, and the operator |-] ensures that is rounded down to the nearest power of 2, allowing it to evenly divide the input and output dimensions. The maximum expansion factor Ama serves
upper bound to maintain stability. This design expands the equivalent rank to ni X r while keeping the parameter count unchanged at r + dout).
Conceptually, RaLoRA is general extension of LoRA. Guided by our entropy-based estimator, RaLoRA adaptively adapts the number of blocks n; per layer: when GID is low, it reduces to n; = 1, identical to vanilla LoRA and focusing dominant directions; when GID is high, increases nu, trading fine-grained precision for broader expressivity across multiple gradient subspaces.
This design is effective because expressive power depends not only parameter count but also on architectural structure. By structurally aligning adaptation capacity with the GID, RaLoRA achieves more efficient use of parameters under the same budget, thereby improving approximation and task adaptability compared to vanilla LoRA. See theoretical analysis in Appendix
3.4 RALORA-PRO ENHANCING RALORA WITH LOss SENSITIVITY-GUIDER PARAMETER REALLOCATION
RaLoRA aligns the rank of LoRA with the intrinsic dimensionality of gradients at the intra-layer level. However, prior work has shown that the relative importance of different layers can vary significantly (He et al., 2025 Zhang et al. 2023b a) given different models and tasks. Motivated by this observation, we propose RaLoRA by integrating loss censitivity-guide design enables dual alignment of LoRA adapters both intra layers (to match gradient intrinsic di- mensionality) and inter layers (to reflect their relative importance), achieving finer-graine capacity distribution under a constrained parameter budget. Specifically, for a weight matrix W; E [doux X of the i-th layer, the importance score is defined as:
shown in Figure III.. which extends RaLoRA inter-layer reallocation strategy (Zhang et al., 2022). This
I(W) = avg(( O Gil),
(5)
where @ denotes element-wise multiplication. The importance score I(W;) quantifies the average influence of layer parameters on the loss. To control the total number of trainable parameters,
5

---

Published as a conference paper at ICLR 2026
normalize the importance scores across N target modules:
(6)
Ek=1 Is
where or [0, 1] indicates the relative importance of the (-th layer. This score guides proportional parameter allocation across layers. Assuming reference rank Tiets the total number of trainable parameters in vanilla LoRA with dimensionality smoothing is given by:
N
P Piotl
(v
d' d'uu
Tief
where d' and d'mg are the input and output dimensionalities of the (-th layer. Using the of calculated by the Equation 6 we reallocate the parameters (i.c., LoRA rank rt) to each layer, ensuring that more influential layers receive higher adaptation capacity. The rank is computed as:
Pioca as
s.t.
Tmin S rI S Tmax
VV dbat
where Ymin and Tmax are predefined bounds that maintain stability. After parameter reallocation, we align the dimensionalities of each layer according section 3.3 achieving a dual alignment of FFT intra and inter layers. The full processes of RaLoRA-Pro are shown in Algorithm [] Distinct from prior rank allocation methods that rely solely on sensitivity, RaLoRA-Pro represents the first dual-alignment framework that unifies inter-layer parameter reallocation with intra-layer geometric adaptation, ensuring that the allocated capacity structurally matches the effective update directions of full fine-tuning.
4 EXPERIMENTS
In this section, we present comprehensive evaluation of RaLoRA and RaLoRA-Pru across natural language understanding (NLU), natural language generation (NLG), and image classification tasks. We compare against established baselines (see Appendix G.I. following experimental protocols from prior work (He et al.) 2025 Wang et al. 2024b); full implementation details are provided in
Appendix G Results are organized follows: Section 4.] (NLU), Section 4.2 (NLG), and Sec-
tion 4.3 (image classification). All experiments are repeated with three different random seeds.
4.1 EXPERIMENTS ON NATURAL LANGUAGE UNDERSTANDING
We fine-tune the T5-Base (Raffel et al., 2020) model on five subsets of the GLUE (Wang et al., 2018). Detailed training configurations are provided in Appendix G.3
Results. As shown in Table both RaLoRA and RaL-oR-PPrs achieve the best overall perfor- mance among all LoRA variants, which corresponds to a substantial improvement of over 5% com- pared to vanilla LoRA. Notably, on tasks such as QNLI and MRPC, our approach even surpasses FFT, despite using significantly fewer trainable parameters. These results suggest that aligning the rank of LoRA with the gradient intrinsic dimensionality enables more effective capacity utilization under small trainable parameter counts comparable to vanilla LoRA.
4.2 EXPERIMENTS ON NATURAL LANGUAGE GENERATION
We fine-tune LLaMA-3.1-888 1-8B-Base (Grattafiori et al.) 2024) on three representative NLG tasks: math, code, and chat. Detailed training configurations provided in Appendix G.4
As shown in Table 2 RaLoRA consistently outperforms all LoRA baselines across the three benchmarks with comparable trainable parameter counts. achieves the best result on Hu- manEval, improving upon vanilla LoRA by +5.69, reducing the performance gap to FFT by 66.6%.
Results.
'The feature dimensionalities of different modules vary, which introduce biases in importance estima- tion. Our wtweeesiiooaaityssmtoheeee allocation mitigates this issue, promoting fair adaptation layers.

---

Published as a conference paper at ICLR 2026
Table 1: Result of fine-tuning T5-Base using full fine-tuning and different LoRA variants on the five subsets of GLUE benchmark.
Method
MNLI
SST-2
CoLAA
QNLI
MRPC
Averaage
Full
86.33:20000 94.75=0.21 80.7020.24 93.1920.222 84.56:20.73
87.91 82.08
LoRA
[lu el al.
2022
85.3010.04 94.04.0.11 69.3510.05 92.96.10.09
68.38.10.01
LoRA Variants with Rank Augmentation
MELORA
2024
85.57-0.19 0.30 77.28+0.36 89.594-5.24
84.114 1.45
86.03
MoRA
Jiang
83.46=0.11 94.0010.06 77.5740.14 92.19100.14 84.5240.36
86.35
AdaLoRA
Zhing
2023b
85,45.10.11 93.6910.20 69.1610.24 91.66.00055
68.14.10.28
81.62
LoRA Variants with Optimized Training Dynamics
DoRA i Lu al.
2024a
85.67-40.09 94.04-0.53 72.04+0.94 93.04+0.06 68.08++0.51
82.57
RSLoRA
Kalaydzievkki
2025
85.73:1.110 94.19:0.23 72.3211.12 93.12:1.09 52.86:12.27
79.64
LoRA+
layou et 2024
85,81 L..24 77.5310.20 93.14110.33
7..43.1..39
84.95
LoRA Variants with Improved Initialization
PISSA
Meng et
85.75+0.07 94.07=0.06 74.2740.39 93.1540.14
76.31+0.511
84.71
Buyukakyuz
2024
92.8510.05 76.8910.54 92.73:10.23
83.75.10.92
86.32
Ours
RRl RaLoRA-Pro
85.76-0.03
94.22-0.29 78.11+0.45 21.3610.14 84.7440.27
87.24
85.82+00113 94.00+0.17 27.85+0.35 93,38+0.04 85.09-+0.39
87.23
On GSM8K, RaLoRA reaches 72.25, resulting in +4.47 improvement over vanilla LoRA and re- covering 75.6% of the FFT performance. The performance of RaLoRA on MT-Bench is also highly competitive, matching the strongest baseline (i.e., MoRA) while being significantly more stable. These results suggest that RaL aligns the rank of LoRA with the corresponding GID, enhanc- ing intra-layer expressiveness significantly without increasing the number of parameters.
Table 2:
Performance of LLaMA-3. 1--B-B-------- -8B--ase fine-tuned with full fine-tuning and various LoRA variants on MT-Bench, GSM8K, and HumanEval. All methods use comparable trainable parame- ters, with LoRA rank and reference rank Teer in equation 7 set to 8.
Method
MT-Bench
GSM8K
HumanEval
Full
5.88
73.69
51.63:1.227
LoRA (Hu al. 2022)
6.15
1..78
43.09:0.33
MELORA Ren et al. (2024)
6.11+0.12
71.27+1.1
45.73+-..80
MoRA (Jiang et al. 2024)
6.38-10.12 12
12.1991.
29.6815.73
AdaLoRA (Zhagg et al.
2023b)
6.19:10.16
70.63+0.77
41.46:13.66
DoRA (Liu et al. 2024a)
6.24
69.1741.00
43.70.11.54
RSLORA (Kalaydzievskk
2023
6.18:00.9
68.36:10.7
45.78
LoRA+ (Hayou et al.
(2024)
6.35+0.10
71.29
44.51+2.11
PiSSA (Meng al. 2024)
6.08:+0.09
68.56+1.03
44. 44.10
OLoRA (Buyukakyūz 2024)
6.13+00.04
6..44
43.29--2.44
((Our))
6.38+00.07
+2.25-00.59
48.78 1.61
CaLoRA-Pro(Ouryy
6.72+00.04
73.01
48.37 +2.54
Beyond the strong performance of RaLoRA, RaLoRA-Pro achieves the best performance on MT- Bench and GSMSK, scoring 6.72 and 73.01, respectively. Compared to vanilla LoRA, these repre- sent improvements of +0.57 and +5.23, effectively closing 88.5% of the performance gap with full fine-tuning on GSM8K. Importantly, RaLoRA-Pro even outperforms full fine-tuning on MT-Bench by +0.84. These findings suggest that the joint optimization of dual alignment in RaLoRA-Pro enables finer-grained control over adaptation capacity, providing a principled and effective PEFT strategy for complex tasks.
4.3 EXPERIMENTS ON IMAGE CLASSIFICATION TASKS
We fine-tune the VIP-VVITB/1G (Radford et al. 2021) model on seven widely used image classifi- cation benchmarks. Detailed training configurations are provided in Appendix G.5
Results. As demonstrated in Table both RaLoRA and RaLoRA-Pro significantly outperform existing LoRA variants. RaLoRA-Prr achieves the best result of 90.66%, a substantial gain of 1.58% over vanilla LoRA, while RaLoRA also shows strong improvement of 1.45%. Notably, on challenging benchmarks such as Cars (fine-grained) and GTSRB (digit recognition), our meth- ods surpass strong baselines such OLoRA and MoRA, demonstrating robust performance across
7

---

Published as a conference paper at ICLR 2026
Table 3: Results of CLIP-ViT-B/I6 on image classification tasks using different LoRA variants.
Note that zero-shot results are reported following (Wang et al.
2024c).
Method
DTD
EuroSAT
GTSRB
RESISC4S
SUN397
SVIN
Zoro-shol
63:75
44.39
42.22
35.22
56.46
62.56
15.33
45.73
LoRA Ha et 2022
12.31400.8
76.97.0.51 1
98.38.10.20
97. 97.10
90.99
17.19.0.19
96.62:10.06
89.08
LoRA
Vartants with Ronk Augmentation
82.65-10.38
15.16.0.399
95.78
74.69.0.22
96.95.1.009
$8.96_00.15
MoRA Dhanp el mi 21p4
84.61+0.21
77.34+0014
98.65+0.16
98.68-0.18
96.33=0.19
78.12-0.06
97.17+0.15
90.13=0.16
AdaLoRxx paaing er 2023h
13.58::0.09
73.79:0045
96.96.0.12
58.87=0..88
89.07=0.00
12.00=00110
94.26:0013
79.79
LoRA Variants Optimized Training Dynamics
DoRA Li 2024g
82.44:10.26
76.86:20.44
98.4310.17
97.25=0.12
95.10:0.116
17.30::011
96.63:10.04
89.14:0.07
RSLoRk [Kalii (reesk 2023
83.94:00.22
77.64:0033
98.51:0.17
98.69=0.17
95 95.90=0.20
77.96=20.21
96 96.4400.66
89. 59.94:00006
LoRA+ (layou
86.6110.33
73.33
98.5611..44
98.99-0.00
96.06.10.38
76.80.10.34
96.98.1.08
89.62.0.19
Improved
PISSA Meng al. 2024
83.16:10.38
77.38.10.57
98 98.54100099
98.32=0009
95.92-10.40
77.46:10113
97.00.1..09
89.71
ORA (ayyuakk 2024
83.85++0.13
78.00+0.255
98 98.62+0003
98.49=0.14
90.01
77.30+00.08
97.15+0:14
90.00=0.15
RaloRA RaLoRA-Pro
86.63++0.31
77.75+0.30
6670
98.98-0.11
96. 06.62-00.28
77. 77,86+00.05
97.24+00.1
90.53-0.0
86. 10110.1
78.42:40.67
98.88.10.30
99.23-0.117
96.66.20.09
77.87-10.27
97.15
90.66:00111
diverse visual domains. These results underscore that RaLoRA's alignment with the gradient in- trinsic dimensionality and RaLoRA-Pro's dual alignment enable more efficient capacity allocation, yielding consistent improvements not only in NLP but also in vision tasks.
DISCUSSION
5.1
CHARACTERIZING GRADIENT INTRINSIC DIMENSIONALITY
We first quantify the gradient intrinsic dimensionality (GID) for each layer on Code-Feedback, MetaMathQO, and WizardLM datasets using our framework. As shown in Figure 4[a).
Fig Figure and Figure B), the estimated GID spans 30-1000, consistent with prior empirical findings that full fine-tuning gradients exhibit ranks 30-100x larger than standard LoRA settings (Biderman jet al. 2024). Moreover, Mean GID correlates with task complexity: WizardLM (=s 404), reflect- ing broad instruction diversity, exhibits the highest GID, followed by FeedbFec (>= 269) and MetaMathQA (+= 178), in line with their respective representational demands.
Feedbeee,,,,,, shown in Figure ((c) (see Appendix E.3 for implementation details). Across most layers, GID rises rapidly then stabilizes, highlighting both the robustness of our estimator and the persistence of dimensionalities far exceed- ing typical LoRA rank settings-ai observation that may help explain the performance gap between LoRA and full fine-tuning.
We further track GID dynamics during fine-tuning
Finally, reveal pronounced layer-wise heterogeneity in GID. Therefore, we further validate that our estimator captures such structure reliably, enabling finer-grained alignment strategies for parameter-efffiicnt adaptation, see Appendix E.4 for experimental details.
1000
700
400
00
330
Minettnnin Step
200
400
6000
150
400
50
2000
2000
4000
2000
800
(a) MetaMathQA Dataset
(b) Code-Feedhack Dataset
(c) Training Dynamics GID
Figure 3: Layer-wise gradient intrinsic dimensionality (GID) across tasks and during training. (a.b): Heatmaps estimated GID on MetaMathQA and Feedbaee, obtained with our entropy-based estimator (WizardLM heatmap in Appendix 4a). (c): Dynamics of GID during Code-Feedback fine-
tuning, showing rapid early growth followed by stabilization.
5.2 THE EFFECTIVENESS OF DIFFERENT STRATEGIES UNDER DIFFERENT RANKS
8

---

Published as a conference paper at ICLR 2026
We conduct controlled comparison of vanilla LoRA, LoRA only with loss sensitivity- rank reallocation (LS-LoRA), and our proposed methods RaL.oRA and Ral A-Pro across dif- ferent ranks, in order to isolate the contributions of intra-layer alignment and inter-layer reallo- cation under a comparable parameter budget.
Table Performance comparison of LoRA, LS-LoRA, RaLoRA, and RaLoRA-Pro MT- Bench, GSM8K, and HumanEval under different rank configurations.
Rank
Method
MT-Bench
GSM8K
EvaanEval
Full
5 5.88+0..3
73.69=0.23
51.63+ 1.27
LoRA
615+0.02
67.78+11.22 1.25
43.09=0.35
RaLoRA
6.19:0020
2142+0082
4..76:+1.27
As shown in Table 4 RaLoRA consistently outperforms vanilla LoRA across tasks and rank settings, with notable improvements on
LS-LoRA
5,9010.21
70.15:11445
39.8343.522
RaLoRA-Pro
6.1210008
72.23:10.32
46.95:11066
LoRA
6.16+0.12
67.83=00.1
42.68:00.61
GSM8K and HumanFval, further confirming the benefits of aligning rank with the gradient intrinsic dimensionality. Similarly, RaLoRA- Pro consistently surpasses LS-LoRA, espe- cially at higher ranks, demonstrating the addi- tional value of integrating intra and inter-layer alignment. Notably, LS-LoRA performs com- petitively on GSM8K at higher ranks but lags on MT-Bench and Saaaaannaall-uggstiiiiii that inter-layer reallocation alone is insufficient for tasks requiring finer-grained adaptation. con-
RaLoRA
6.28
02.12:40.79
48.3744..57
LS-LoRA
6.03++0225
71.97++1.18
43,70+ 1.76
RaLoRA-Pro
6.06+:0225
73.29::0.65
48.58:
LoRA
6.14::0.14
68.01:41.14
42.68-1 1.22
RaLoRA
6.224-0.14
1..37:+1.35
48.98-+2.75
32
LS-LoRA
6.1411010
13.7
44.10.10.93
RaLoRA Pro
6.2010.11
74.38
2..00
LoRA
6,28+00009
67.17:1.611
43.29:2.20
RaLoRA
6.32:0.06
46.55:10.46
21.22-1.11
L.S.
6,16100.10
74.48.10.63
46.95.1
RaLoRA-Pro
6.28+0.11
75.23:10.39
12.24
trast, RaLoRA-Par - joint modeling of local gradient structure and global task importance provides consistently strong performance, highlighting the complementary value of combining intra- and inter-layer alignment. Interestingly, as the rank increases, our method achieves performance that surpasses full fne-tuning on these three tasks.
5.3 THE EFFECT OF MAX EXPANSION FACTOR
As shown in Table 5 we evaluate the impact of the maximum expansion factor Ramax in Equation 4 which determines the number of parallel mini-LoRA blocks and thus the granularity of intra-layer adaptation. larger Rmmx enables finer alignment with gradient intrinsic dimensionality but reduces parameter capacity per direction, creating a trade-off between alignment fidelity and expressiveness.
Our results show that the optimal Maxx is task dependent. On GSM8K, performance peaks at 7max = 16 and degrades with larger values, suggesting wre-frramentttion harms tasks with lower intrinsic dimension- ality. In contrast, on MT-Bench and HumanEval, RaLoRA's perfor- mance improves with larger 7mmax. in- dicating that tasks with higher intrin- sic dimensionality gradients benefit from finer alignment. These findings validate the necessity of introducing Mmax to control block granularity: when properly balanced with per-direction capacity, structural alignment with gradient dimensionality significantly improves adaptation performance.
Table 5: Effect of the maximum expansion factor R'max the performance of RaLoRA and RaLoRA-Pro across MT- Bench, GSM8K, and Hvaa mGID stands for mean gradient intrinsic dimensionality.
MTT-Bench
GSMSK
HuumanEval mGID rs 269
Method
mGID se 404
mGID Rs 178
RaLoRA
6.18::0.1 5.75++0.08
72.25=0.59
45.12::1.22
Ral-oRAPPo
73.01+0053
48.37 +2.54
RaL.ORA
6.19+0.2
0..42++0.77
47.76
32
RaL-oRA-Pro
6.12+:0.12
7..23++0.2
46.95-t 1.06
Ral.RRA
6.38+00.07
71.19+0.79
48.78-11.61
RaLoRA-Pro
6.05=0.11
71.24-1.55
45.73++2.66
5.4 THE EFFECT OF RANK REALLOCATION RANGE
Rank reallocation guided by loss sen- sitivity is a critical component of RaLoRA-Pro. To maintain parameter efficiency comparable to LoRA un- der a reference rank Tres, the realloca-
Table 6: Performance comparison of different rank reallo- cation ranges in RaLoRA-Pro under fixed reference rank Tief = 8 and maximum rank 7max = 32.
Rank range
MT-Beech
GSMBK
HvmanEval
#Params
tion range (Tmin Tmax) in Equation g is typically chosen to satisfy the to-
8-8
6.19=002
71.42+0..7
47.76=1.27
6.82M
4-16
6. 6.1200.08
72.23++0.22
46.95=1.00 1.06
6.32M
4-32
6.70++0.33
7..39+1.02
45.32++1.76
7.00M
tal parameter budget constraint. As shown in Table 6 we investigate the
6-32
6.65++0.30
71.37+0.88
48.17=0.61
7.84M
impact of different rank allocation ranges under a fixed setting of Tief = 8 and Mmnax = 32. For
GSM8K, relatively narrow range yields the best accuracy, suggesting that smaller deviations from

---

Published as a conference paper at ICLR 2026
the reference rank may better preserve generalization in mathematical reasoning. On the other hand, broader ranges yield stronger performance on HumanEval and MT-Bench, indicating that tasks in- volving code generation and open-ended instruction following may benefit from more expressive and flexible adaptation. Overall, these findings highlight that appropriately selecting the rank real- location range can yield substantial performance gains under comparable parameter budgets.
CONCLUSION
In this work, we address a fundamental and underexplored limitation of existing LoRA methods, the mismatch between LoRA and the intrinsic dimensionality of full fine-tuning gradients. To bridge this gap, propose entropy-basee estimator that quantifies per-layer gradient intrinsic dimensional- ity in robust and scalable manner. This estimator is orthogonal to most existing LoRA variants and can potentially serve plug-in tool for adaptive alignment with the gradient intrinsic dimension- ality across broad range of PEFT methods. Building on this insight, we introduce RaLoRA, which structurally aligns each LoRA adapter's 's rank with the estimated gradient intrinsic dimensionality. We further propose RaLoRA-Pro, which incorporates inter-layer importance to enable dual-stage alignment mechanism. Extensive experiments across diverse NLPP and image classification tasks show that our approaches significantly enhance LoRA performance under constrained parameter budgets and substantially narrow the performance gap with full fine-tuning. Considering the page limit, the discussion of this work's ' limitations and future works is deferred to the appendix
ACKNOWLEDGEMENT
This work was supported by a locally commissioned task from the Shanghai Municipal Government and the Shanghai Artificial Intelligence Laboratory (AlLab). We also express our gratitude to our collaborators (Jingqi Ye, Haonan He, Minglei Li, and Fujun Han) for their valuable contributions during their intersships AlLab, both in-person and remotely.
REFERENCES
Daniel Adiwardana, Minh- Thah-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open- domain chatbot. arXiv preprint arXiv:2001. 09977, 2020.
Paul Albert, Frederic Z Zhang. Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. Randlora: Full-rank parameter-efficient fine-tuning large mod- els. arXiv preprint arXiv:2502.00988 2025.
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09675 2024.
Kerim Büyiikakyüüz. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint 47Xiv:2406.01775, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image classification: Bench- mark and state of the art. Proceedings of the IEEE, 106(10):1855-1883, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. arXiv preprint 1311.3631 2013.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.141 2021.
George Deligiannidis and Arnaud Doucet. Optimally designing the statistics of training data for learning. In AISTATS, 2019.
10

---

Published as a conference paper at ICLR 2026
Mohamed A ElMossallmmy et al. Reconfigurable intelligentt surfaces for wireless communication: Principles, challenges, and opportunities. IEEE Transactions on Cognitive Communications and Networking, 2020.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efffcient fine-tuning with discrete fourier transform. arXiv preprint arXiv:2405.03000 2024.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. Rankme: Assessing the downstream performance of pretrained self- self-supe representations by their rank. In Interna- tional Conference on Machine Learning, pp. 10929-10974. PMLR, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Fujun Han, Jingqi Ye, Chenglong Zhang, and Peng Ye. Ovid: Open-vocabulary intrusion detection. In The Fourteenth International Conference on Learning Representations.
Fujun Han, Peng Ye, Shukai Duan, and Lidan Wang. Ada-id: Active domain adaptation for intrusion detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7404- 7413, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint 12345 2024.
Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, and Lei Chen. Gora: Gradient-driven adaptive low rank adaptation. arXiv preprint vrXiv:2502.12177 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In The 2013 international joint conference neural networks (IJCNN), pp. 1-8. Ieee, 2013.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-effice transfer learning for nlp. arXiv e-prints, pp. arXiv-1902. 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 12):3, 2022.
Qiushi Huang, Tom Ko. Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: Parameter-effccien hadamard high-rank adaptation for large language models. In The Thirteenth International Con- ference Learning Representations, 2025.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng. Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parameter-efficient fine- tuning. arXiv preprint arXiv:2405. 12130. 2024.
Damjan Kalajdzievski. rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03772. 2023.
Yejin Kim, Eunwon Kim, Buru Chang, and Junsuk Choe. Improving fisher information estimation and efficiency for lora-based 11m unlcarning. arXiv preprint arXiv:2508.21300 2025.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412 69800 2014.
11

---

Published as a conference paper at ICLR 2026
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-gu ane-graindd categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Shashi Kumar, Yacouba Kaloga, John Mitros, Petr Motlicek, and Ina Kodrasi. Latent space factor- ization in lora. arXiv preprint arXiv:2510. 1..0.996440 19640, 2025.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. arXiv preprint vrXiv:2104.0866 2021.
Minglei Li, Peng Ye, Jingqi Ye. Tao Chen, et al. Eolora: Efficient and effective low-rank adaptation with -uroopy-guiedd adaptive sharing. In The Fourteenth International Conference on Learning Representations, 2026.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 2021.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates. arXiv preprint NxKv::2307.05655, 2023.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 11134892-344111 2023.
Shih- Yang Liu, Chien-Yi Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposee low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer. 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17X1v:1711.05101,1 2017.
Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. Adalomo: Low-memory optimization with adaptive learning rate. arXiv preprint 1xX1:22310.10155 2023a.
Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. arXiv preprint wxXiv:230600972, 2023b.
Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605, 2025.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-121001 2024.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
John Nguyen, Sid Wang, Ke Li, and Carole-Jean Wu. Read: Recurrent adaptation of large trans- formers. arXiv preprint arXiv:2305.1 /5348, 2023.
12

---

Published as a conference paper at ICLR 2026
Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinn.. Kernel language entropy: Fine- grained uncertainty quantification for Ilms from semantic similarities. Advances in Neural Infor- mation Processing Systems, 37:8901 2024.
Saurabh Paul and Petros Drineas. Feature selection for ridge regression with provable guarantees. Neural Computation, 28(():7116-74 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Melora: mini-ensemble low-rank adapters for parameter-efficient fine-tuning, arXiv preprint arXiv:2402.1726: 2024.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606 610. IEEE, 2007.
Jianlin Su. Effective rank, April 2025, https: http//paaeessas https::/spaces.ac. en/ http//rrhiees//108/ 1
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 07461,, 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Junjie Wang, Guangjing Yang, Wentao Chen, Huahui Yi, Xiaohu Wu, Zhouchen Lin, and Qicheng Lao. Mlae: Masked lora experts for visual parameter-efficient fine-tuning. arXiv preprint arXiv:2405.18897 2024a.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. Advances in Neural Information Processing Systems, 37:54905 -54455 2024b.
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint 2rXiv:2407.1824: 2024c.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 3485-3492, IEEE, 2010.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models follow com- plex instructions. In The Twelfth International Conference on Learning Representations, 2024.
Xingyi Yang and Xinchao Wang. Diffusson model as representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18938-18944 2023.
Peng Ye, Tong He, Baopu Li, Tao Chen, Lei Bai, and Wanli Ouyang.
beta -dorts robustdi wwwreeeeiiblerrccteeeereeeeeeeee CEEE Transactions Pattern Analysis and Machine Intelligence, 2025a.
+
Bi leveiregularzzationffoo forprozy
Peng Ye, Tong He, Shengji Tang, Baopu Li, Tao Chen, Lei Bai, and Wanli Ouyang. Stimulative training++: Go beyond the performance limits of residual networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025b.
13

---

Published as a conference paper at ICLR 2026
Fangchen Yu, Haiyuan Wan, Qianjia Cheng, Yuchen Zhang, Jiacheng Chen, Fujun Han, Yulun Wu, Junchi Yao, Ruilizhen Hu, Ning Ding, et al. Hipho: How far are (m) llms from humans in the latest high school physics olympiad benchmark? arXiv preprint ar arXhv:2509.078% 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12256 2023.
Wei Yu, Kuiyuan Yang, Yalong Bai, Tianjun Xiao, Hongxun Yao, and Yong Rui. Visualizing and comparing alexnet and vgg using deconvolutional layers. In Proceedings of the 33 rd International Conference on Machine Learning, volume 2, 2016.
Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. In- crelora: Incremental parameter allocation method for parameter-effiiiet fine-tuning. arXiv preprint arXiv:2308. 12043, 2023a.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International conference on machine learning, pp.
PMLR, 2022.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficien fine- tuning. arXiv preprint arXiv:2303.105//1 2023b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient I1m training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, 7i Lin, Zhuohan Li, Dacheng Li, Eric Xing. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinnerrpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
14

---

Published as a conference paper at ICLR 2026
A DERIVATION OF LORA AS GRADIENT COMPRESSOR
Low-Rank Adaptation (LoRA) (Hu et al. 2022 posits that weight updates during fine-tuning reside low-rank subspace of the pretrained model's s parameter space. Given a pretrained weight matrix Wo E RRot xdin, LoRA decomposes the update AW E RRd into low-rank matrices A E Rrxdm and Rdout X as AW = B A, while freezing Wo. Without loss generality, we omit the scaling factor a/r (common in theoretical analyses), yielding the forward pass:
Y= WoX BAX,
The gradients of the loss L with respect to A and B are derived via the chain rule. Let 0L/OW represent the full gradient of the loss with respect to the weight matrix (i.e., G = (OC/OY)XT E gdou where 0C/OY is the upstream gradient). The gradients are:
Oc BTG.
(10)
0A
OL GAT.
(11)
Under stochastic gradient descent (SGD) with learning rate 7). the updates for A and B are:
A41 A. 1] OL
A. A Gt
(12)
0A
dc
B = Br aB,
B, - r
(13)
where G. = OC/OW, is the gradient at step t. The change in the low-rank update A(BA) =
B441 B,A, is then:
A(BA) B1-1A441 BAA,
-nBB mBB.TG.. - B,A =-n (B,B G, + G.A, At) + G,A B, Gt.
(B. - No-N)AAT, (Ar, -
(14)
Since 17 1 in practice (typical values: le 1--5-e 3) the - is negligible. Thus, the first-order approximation is:
A(BA) -n (B, G. G.A, Ar).
(15)
This reveals that LoRA implicitly projects the full gradient Ge onto a low-rank subspace spanned by B, B, and A, A, Crucially, when the true effective update directions of G2 (termed the gradient intrinsic dimensionality of this paper) far exceed the LoRA rank r, this projection incurs significant information loss. Consequently, the expressivity of LoRA is fundamentally constrained by its rank r, necessitating a principled and adaptive gradient intrinsic dimensionality alignment framework to balance efficiency and performance.
B DETAILED EXPLANATION OF THE EFFECTIVE RANK ESTIMATION AND ITS PROPERTIES
B. EFFECTIVE RANK DEFINITION
Consider a gradient matrix G1 E R at layer I, with its singular value decomposition (SVD)
given by:
min(m..))
G
Σ
(16)
where 01 2 02 2 ... o, > are the singular values, r = rank(Gi), and u, E R"", U; E R are the corresponding left and right singular vectors.
15

---

Published as a conference paper at ICLR 2026
Let (01, Fg,,o,,,, o denote the vector of non-zero singular values. Define the normalized singular value distribution as:
P: | i= 1=,,2,....?"
(17)
where 1 is the l norm. The effective rank of G2, denoted erank(Gr). is defined as: crank(Gi) exp(H(p1. (p.................
(18)
where
H (P1,, ........Pt) -
P: log
(19)
the Shannon entropy of the distribution {p:}. This entropy, also referred to as the spectral en- tropy, quantifies the spread of the normalized singular values and provides robust measure of the gradient's intrinsic dimensionality for LoRA rank selection.
B.2 PROPOSITION: BOUNDS ON THE EFFECTIVE RANK
The effective rank of gradient matrix G₁ satisfies the following inequality:
S erank(G;) S rrank(Gi) S min(m,, n).
(20)
The lower bound erank(Gt) = is achieved if and only if o = (1al1,, 0, .,.....)) i.e., only one singular value is non-zero.
The upper bound erank(G;) rank(Gi) is achieved if and only if the non-zero singular values are all equal, i.e.,
|alla
|
Proof.
The Shannon entropy of the normalized singular value distribution satisfies
0 01........... - 0) & H(1, P2,...,,
Pr) P....P......PP. H (+.....,
#) .......))).........)
log
(21)
Therefore,
s crank(G.) = c .p)) Sr.
(22)
Equality erank(Gi) G.) = 1 holds when P1 = 1 and all others are zero, corresponding to o = (1 0)* Equality erank(G;) holds when p: 1/r for all mcaning all non-zero singular values are equal.
C ON THE EXPRESSIVE POWER OF RALORA: STRUCTURAL ALIGNMENT WITH GRADIENT INTRINSIC DIMENSIONALITY
C.I CORE ARGUMENT AND THEORETICAL REFINEMENT
We believe that expressive power is not solely determined by parameter count but critically depends on architectural design (He et al.] 2016: Ye et al.) 2025a Yu et al. 2016 Ye et al. 2025b). Consequently, our central argument is that RaL.RA enhances approximation quality by adaptively aligning with the gradient's intrinsic dimensionality (GID), thereby optimizing pa- rameter utilization under a fixed budget. This adaptive alignment enables RaLoRA to overcome limitations inherent LoRA when gradients exhibit high-diig -igh-dimensional structure.
RaLoRA is a generalized extension of LoRA. Specifically, when the number of parallel blocks ni = 1, RaLoRA reduces to LoRA, which provides fine-grained approximation along dominant gradient directions. As increases, RaLoRA adaptively trades precision in dominant directions for broader expressivity, capturing a more diverse spectrum of gradient subspaces. The selection of is adaptively driven by our proposed entropy-based estimator.
16

---

Published as a conference paper at ICLR 2026
C.2 THEORETICAL ANALYSIS: APPROXIMATION ERROR CHARACTERIZATION
To characterize the approximation capabilities of LoRA and RaLoRA, we analyze their respective errors representing the true weight update AW Rm*x under the Frobenius norm. Let the singular value decomposition (SVD) of AW be:
min(m.n)
AW Σ
(23)
where 01 2 02 ... ... 2 0 are the singular values, and Us R", D2 IR" are the corresponding left and right singular vectors.
LoRA's Approximation Error: LoRA approximates AW with a rank-r matrix BA (ignoring scaling for clarity). By the Eckart- Younn--uunggiin Mirsky theorem, the optimal rank-r approximation mini- mizes the Frobenius error, and the minimal error is:
min(m.n)
FLORA
min BAnk(
|AW BAWB
Σ
(24)
This implies LoRA captures only the top-r singular directions. When the gradient exhibits slow spectral decay-ic -ie., high GID the tail singular values 0,+,, 0r42.... remain significant, leading to substantial approximation loss.
RaLoRA's block-diagonal structure Approximation Error. RaLoRA approximates AW using bock-ddagonal structure with ny parallel rank-r blocks. We partition A AW into an ny ny block matrix (AW,.) p.a 11
AWL.1
AW
AW
AW21
AW2.2
2
AW
(25)
AW m1
AW AWm.2
AW mmm
Correspondingly, partition each singular vector into ny segments:
11,1
U4,1
U1.2
V122
04
U;
with Uipp E R/// Ui E Rm/m
(26)
Us,mz
Substituting Equation
23 into Equation
25
each block is:
miu(m,,))
AW p.g
Σ 2ml
(27)
RaLoRA approximates only the diagonal blocks (p = ②) with low-rank matrices Bp while off- diagonal blocks (p + 9) are implicitly set zero. The total approximation emrrr is therefore:
ERALORA = JAW - -diag( Bi
And ni
11
AW BpA
(28)
Truncation Erroe (off diagonal blocks)
Approximation Error (diagonal blocks)
The truncation error arises from discarding off- off-diagonal interactions, while the approximation error reflects suboptimal low-rank fitting within each diagonal block. Crucially, RaLoRA sacrifices fi- delity to global top singular directions but gains by parallelizing learning across heterogeneous
17

---

Published as a conference paper at ICLR 2026
subspaces. This structural reorganization allows it to capture richer gradient information when GID is high, as each block specializes in distinct directional components.
To conclude, our analysis reveals that LoRA's performance is limited by its rigid rank, which strug- gles to approximate sige--imensioall gradient updates. In contrast, RaLoRA's block-diagonal archi- tecture and adaptive GID alignment, guided by an entropy- based estimator, enable it to effectively capture complex, high-GID. Consequently, RaLoRA provides principled framework that aligns the adaptation structure with the GID, significantly enhancing the expressive power of low-rank adaptation.
D COMPARISON WITH THRESHOLD-BASED RANK ESTIMATION METHODS
To validate the robustness and superiority of our entropy-based estimator, we conduct a comparative study against conventional absolute threshold SVD methods for determining the gradient intrinsic dimension. The fhreshold-basee approaches count the number of singular values ø, exceeding a specific tolerance {10-1 10 10 1,1055 $))- The estimated rank, Test, is then converted block count fy using Eq. Results are averaged over 3 random seeds.
Table Performance under different rank estimation strategies on GSM8K and HumanEval. "Ours" uses the entropy-based GID estimator. Threshold-based methods are highly sensitive to E, while our approach achieves consistently superior performance without manual tuning.
Absolute Threshold (s)
Task
Ours
10-1
10-3
10-5
GSM8K
6..15 45.53 + 0.93
0.97
69.30 +: 1.32
72.25
0.59
HumanEval
44.71 + 1.27
43.29 + 1.61
48.78 + 1.61
Analysis. The high sensitivity of threshold-basee methods arises from a critical flaw: the spec- tral energy of gradients in large model fine-tuning decays slowly and non-uniformly, precluding a universal e for distinguishing "signal" from "noise." Heuristically setting a high threshold leads to capacity underutilization by aggressively truncating meaningful directions. Conversely, low threshold retains optimization noise, diluting the signal-to-noise ratio and harming generalization. This instability starkly demonstrated HumanEval, where a performance drop of over 2.2 points is observed as s changes from le-1 to 1
In contrast, our entropy-based estimator offers principled and robust alternative. It quantifies the true effective update directions of FFT gradients by measuring the entropy of singular value distri- butions. This data-driven, layer-adaptive approach naturally aligns with the underlying geometry of task-specific gradients, eliminating the need for manual tuning. This capability, previously underex- plored in LoRA, allows us to move beyond heuristic rank assignment and directly align LoRA ranks with the actual gradient subspace complexity.
E DETAILS FOR CHARACTERIZING GRADIENT INTRINSIC DIMENSIONALITY
E. GID ACROSS NLG TASKS AND LAYERS
To gain deeper insights into how gradient intrinsic dimensionality varies across different tasks and network layers, we employ our entropy-based based estimator to analyze the full fine-tuning gradients of L.LMA-3.11-8B Base on three representative datasets: Code-Feedbacke MetaMathQA, and Wiz- ardLM. Figure/Areveal distinct, non-random functional patterns regarding intra-layer roles and inter- layer depth. These patterns offer insights into how L.LMs adapt to varying task demands.
Intra-layer Functional Variation (W, Dominance). We observe substantial heterogeneity across attention sub-modules. Specifically, the output projection weight W. consistently exhibits signifi- cantly higher GID compared to W Wk. and W... This high intrinsic dimensionality suggests that W, serves a critical role in aggregating and synthesizing information from diverse attention heads, requiring higher-rank update subspace to effectively transform context-dependent features.
18

---

Published as a conference paper at ICLR 2026
Task-Specific Depth Dependency. The distribution of GID across the network depth varies by task, reflecting distinct adaptation requirements:
Open-ended Generation (Code-Feedback WizardI.M); These tasks display high GID in the upper layers. This indicates that upper layers in Transformers are responsible for high- level semantic abstraction and output formatting, which are critical for code generation and complex instruction following.
Logical Reasoning (Mtha Conversely, mathematical reasoning tasks exhibit higher GID concentrations the middle layers. This suggests that the adaptation required for logical reasoning relies heavily transforming intermediate representations rather than surface-level realization.
400
1200
800
330
300
800
600
400
100
200
WizardLM Dataset
(b) Code-Feedbane Dataset
(c) MetaMathQA Dataset
Figure 4: Heatmap of layer-wise gradient intrinsic dimensionality of L.LMA-3311-81 1-8--Basss on Code- Feedback, MetaMathQA, and WizardL.M M Datasets, Illustrating task-specifie and intra-layer varia- tions in representational complexity.
E.2 CORRELATION WITH FISHER INFORMATION AND LOSS CURVATURE
To determine if GID is merely a proxy for loss sensitivity, analyzed its relationship with the Fisher Information (FI) matrix. As established in prior work (Kunstner et al. (019). the Fl matrix approximates the Hessian, serving effective measure of second-order local curvature.
The diagonal of the empirical Fisher Information for model parameters 0 on dataset is defined Kim al. 2025):
(
Fo(D) ~
Σ RED
(29)
D
a0
Intuitively, Fe captures the sensitivity importance of parameters regarding the loss landscape. We denote the average FI value for a layer as F,.
Correlation Analysis. We computed the Pearson and Spearman correlation coefficients between the mnyer-wise estimated GID and the layer-wise Fisher Information F, The results, averaged across tasks, are presented Table
Table 8: Correlation analysis between GID and FI (Fi) across different module types. The low correlation values indicate that GID captures properties distinct from parameter sensitivity.
Module
Pearson (log Fi)
Pearson (1/F)
Spearman Rank
Wa
-0,051
-0.261
0.125
We
-0.179
0.141
-0.113
W.
-0.172
-0.214
-0.053
W.
0.277
-0.251
0.325
19

---

Published as a conference paper at ICLR 2026
Our empirical results demonstrate significant correlation between GID and Fisher Informa- tion. While initially counterintuitive, this finding provides profound insight into the optimization geometry of PEFT and theoretically justifies the design of our methods.
Magnitude * Dimension. The lack of correlation indicates that sensitivity (how much the loss changes, measured by Fl) and structural complexity (how many directions needed for the update, measured by GID) are decoupled properties.
A layer can be highly sensitive (steep loss landscape, high FI) but structurally simple, requiring updates along only few dominant principal directions (low GID).
Conversely, a layer may require processing diverse, fine-grained features necessitating high-rank update (high GID), even if it is not the most dominant factor in the overall loss magnitude (moderate FI).
Justification for RaLoRA-Pro. This "orthogonality" explicitly validates the dual-alignment strat- egy employed in RaLoRA-Pro. Since GID and parameter importance (Loss Sensitiviy//DD capture distinet aspects of the optimization landscape, an effective PEFT method must address both simul- taneously:
1. RaLoRA aligns ranks with GID to match the intra-layer structural complexity.
2. RaL-oRA-Pro further integrates Loss Sensitivity to match the inter-layer parumeter impor- tance.
If GID and FI were highly correlated, the dual-alignment strategy would be redundant. The observed lack correlation explains why MaeooRR--Prr-eee considers both fimensions-ccosiistenlll outperforms methods that rely principally on importance-based allocation (e.g., AdaLoRA (Zhang et al. 2023bb) or fixed-rank structures.
E.3 GRADIENT INTRINSIC DIMENSIONALITY DYNAMICS DURING FINE-TUNING
We tracked the gradient intrinsic dimensionality (GID) in section 5. using the entropy-based esti- mator for each layer every 5 update steps under the following configuration:
Model: L.lama-3.1885 Base
Dataset: Code-Feedback
Optimizer: Ad al. 2023a b) (layerwise SGD with adaptive learning rate, memory efficient and can be easily used to conduct analysis)
Batch size: 8 (per GPU) x 8 GPUs = 64
Total update steps: 1024
Learning rate: le-4 (common choice for Adalomo optimizer)
The first and last layers throughout training, visualized in Figure c). Key observations include:
Dynamic Evolution of GID: The GID exhibits a rapid increase during the early stages of training (=20 steps) before stabilizing. The subsequent stabilization confirms that the gradient subspaces converge to high-dimensions manifolds that far exceed the ranks of standard LoRA.
GID Magnitude vs. LoRA Rank: We find that the estimated GIDs consistently exceed typical LoRA ranks (e.g., 8-64). This suggests that LoRA's fixed low-rank bottleneck discards critical gradient information, which is key reason for its performance gap with full fine-tuning.
Lise Heterogeneity: The GID varies significantly across different layers, with some layers consistently maintaining a higher intrinsic dimensionality than others. This het- erogeneity provides the motivation for RaLoRA's adaptive, layer-specific rank allocation, which aligns LoRA capacity with the complexity of per-layer GID.
20

---

Published as a conference paper at ICLR 2026
Table 9: Performance and parameter efficiency comparison on GSM8K and HvmanEval. LoRA+GID assigns per-layer ranks based on estimated GID, matching parameter counts of stan- dard LoRA for fair comparison.
GSM8K
HumanEval
Method
Accuracy (%)
Params
Method
Pass@1 (%)
Params
FFT
73.69 + 0.28
1.34B
FFT
51.63 + 1.27
1.34B
LoRA (r=8)
67.78 + 1.25
6.82M
LoRA (r=8)
43.09 + 0.35
6.82M
LoRA (r=200)
74.12 + 1.23
170M
LoRA (r=380)
49.39 + 2.66
323M
LoRA+GID
75.41 + 0.84
175M
LoRA+GID
52.64 + 1.27
323M
E.4 EVALUATING ORTHOGONALITY AND FINE-GRAINED ALIGNMENT OF THE ENTROPY-BASED ESTIMATOR
We designed set of comparative experiments to concurrently assess our estimator's 's orthogonality with LoRA like methods and its reliability capturing layer-wise heterogeneity. To this end, we set the LoRA rank for each layer to the GID estimated by our estimator (denoted as LoRA+GID), For a fair comparison, we increased the rank of standard LoRA so that its number of trainable parameters was close possible to that of LORA+GID. We conducted these experiments on both mathematics and code generation tasks, keeping the implementation details consistent with those in Section 4.2 We also included performance comparisons against full fine-tuning and standard LoRA with a fixed rank of 8 to provide a more comprehensive and intuitive set of results.
Results shown in Table When the rank LoRA is increased, the performance improves signif- icantly, which demonstrates that LoRA's low-rank bottleneck restricts its performance. In addition, LoRA+GID still achieves significant performance improvements when the number of trainable pa- rameters is comparable to LoRA. This is because the proposed entropy-based estimator has good orthogonality and can effectively estimate the GID of each layer identify the heterogeneity be- tween different layers, thereby having a finer-grained alignment capability.
F ALGORITHM
The full processes of RaLoRA-Pro are shown in Algorithm
G EXPERIMENT DETAILS
G.1 BRIEF OVERVIEW OF BASELINE METHONS
We compare RaLoRA and RaLoRA-Pro with several relevant or widely used methods:
Full Fine- :unn-Tuning: Updates parameters in designated layers. This method serves as the upper bound in terms of performance but incurs the highest memory and computation cost.
2. LoRA (Hu et al. 2022): Introduces pair of low-rank matrices A and B per fine-tuned layer. Only these matrices are updated during fine-tuning, significantly reducing the number of trainable parameters.
LoRA Variants with Rank Augmentation: These methods aim increase the rank of LoRA while maintaining fixed parameter budget:
2023b): Prunes less important updates based on SVD and impor- tance scores, reallocating rank to more influential layers within constrained budget.
AdaLoRA (Zhang et al.
MELoRA (Ren et al. 2024): Employs multiple parallel mini-LoRA modules to enhance equivalent rank during training.
MoRA (Jiang et al. 2024]: Utilizes non- non-paramttric square matrices to implicitly achieve
high-rank updates via structured transformations.
21

---

Published as a conference paper at ICLR 2026
Algorithm The RaLoRA-Pro algorithm
Input: Model f(-) with target FT weight Wi,1 1,..., L, sample size N, loss C, reference rank fref- max blocks Mmax
Output: Initialized LoRA matrices {At,i, Bi,e)
D 1.... L and i 1....?
1: for 1= to do
2: G7 +0
3: (dout: din) f size(Wi) Wi)
4: Piona = (Vdin + dout) Fref
D. Pre-define trainable parameters budget.
5:
for to do
6:
Sample mini-batch B; = far, y/. compute 1 t- f(x
7: { e+-Cly: 9)
8:
for 1 to do
9:
D6 G7 += N aw;
Accumulate gradients of pretrained weights without optimizer step.
10: for = to L do
11:
I(W) + avg @ G)
12: Normalize importance: aj = Σ In
f L do (dout, din) & Wi))
14:
15:
Ti < clip(round (PporoundPPooaatCk Protat / din + dont) ) Timis Tmax)
16:
Compute crank(G, (G+""), ), set ny < (clip(Llogg (rt)), (.Pmma))
17:
Partition A, B, into n, blocks: Ar, Br,i
18:
for 1 to do
Initialize weights of the adapter.
A1.: +- Kaiming init
20:
B1,5 < Zero init
21:
Return {Ar,s Bt,,}
..... L and i .,...,
4. LoRA Variants with Optimized Training Dynamics: These methods have been proposed to enhance the optimization process during fine-tuning:
DoRA (Liu et al. 2024a): Decomposes pretrained weights into magnitude and direction,
updating only the direction component using LoRA for efficient adaptation.
RSLoRA (Kalajdzievski
2023): Optimizing the scaling factor to stabilize training.
LoRA+ (Hayou et al. and B.
2024): Assigning different learning rates the LoRA matrices A
5. LoRA Variants with Improved Initialization: These approaches aim to enhance LoRA perfor- mance through improved initialization strategies.
OLoRA Büyükakyüz 2024); Initializing LoRA matrices with orthonormal bases from QR
decomposition to improve convergence and stability.
PiSSA (Meng et al.) 2024): Using principal components from SVD of the pretrained weights to initialize LoRA modules for more effective adaptation.
G.2 HYPERPARAMETER SETTINGS BASELINE METHOODS AND METHIODS
In the baselines of this article, there several adjustable hyperparameters introduced LoRA variants. We detail the hyperparameter configurations for all compared LoRA variants. Unless otherwise noted, follow the recommended settings reported in the original papers ensure fair comparison.
LoRA (Hu et al. 2022): Unless specified,
the LoRA rank 8 (for RaLoRA-
Pro, this refers to the reference rank Tief), and the scaling factor = 16. The matrix A is initialized using Kaiming initialization, while matrix B is initialized to zero.
LoRA+ (Hayou et al. 2024): The learning rate ratio between matrices A and B is set to 16.
22

---

Published as a conference paper at ICLR 2026
AdaLoRA Zhang et al.
2023b): The initial rank is set to 12 and the target rank to with
t, 150 and ł 900.
PISSA Meng al.
2024
The number of SVD iterations is set to 64.
MELoRA [Renetal 2024):
only the input dimensionality of matrix A and the output dimensionality of matrix B, keeping the rank dimensionality fixed.
set set n = 2 and only
MoRA (Jiang et al.
2024): We apply a rotation-based compression and reconstruction
scheme.
RaLoRA (Ours): Unless specified otherwise, the number of parallel blocks is set to 7max 32.
RaLoRA-Pro (Ours): Given reference rank Tid, we set Tmin Teet/2, 2,, Fmax 27s and Amax = 32 by default.
G.3 TRAINING CONFIGURATION FOR NATURAL LANGUAGE UNDERSTANDING
Models and Datasets. We fine-tune the T5-Base (Raffel al. 2020 model on five subsets of the GLUE (Wang et al., 2018) benchmark: MNLI, SST-2, CoLA, QNLI, and MRPC. Performance is reported validation sets using accuracy the metric.
Implementation Details. We use the Adam Kingma & Ba 2014) optimizer with 31 0.9, 32 = 0.999, weight .eegay.. The batch size is set to 32, and a cosine learning rate schedule with a warmup ratio of 0.03 is applied. All linear layers except the language modeling head are fine-tuned using FP32 precision. The peak learning rate set to 1e 4, and the maximum input sequence length is 128. Please refer to Appendix G.2 for the detailed hyperparameter settings of different LoRA variants.
G.4 TRAINING CONFIGURATIONS FOR NATURAL LANGUAGE GENERATION
Models and Datasets. We fine-tune L.I. -.MMA-3.1-B--BB- 1-8B-Base Grrattatioii et al. 2024 on three rep- resentative NLG tasks: math, code, and chat. Evaluation is conducted as follows:
1. Mathematical reasoning: Trained on a 100K subset MetaMathQA YYu et al. ated on GSM8K (Cobbe et al. 2021). Accuracy is computed using regex-based extraction.
2023) and evalu-
2. Code generation: Trained on a 100K subset of
(Zheng et al. 2024) with code-
only labels and evaluated on HumanEval (Chen et al. (021)) using PASS@ @ the metric.
3. Conversational dialogue: Trained on a 52K subset of WizardLM (Xu et al.) 2024). Evalua-
tion follows the MTT-Bench (Zheng al. 2023) single-turn dialogues evaluation protocol, reporting
average scores (0-10) from GPT-4o, 1ammini-155-Paa and I.LMMA33.1 1-0.-...1-77--sttreee following the standardized prompts of (Zheng al. 2023) for comparability.
Implementation Details. We use the AdamW (Loshchilov & Hutter 2017) optimizer with B1 0.9, 82 0.999, weight_decce
4. The batch size is 64. cosine decay learning rate schedule with decay rate of 0.1 and warmup ratio of 0.03 is used. The peak learning rate is set to 5e 4 for AdaLoRA, and 5e for full fine-tuning and all other baselines. Fine-tuning is restricted to the attention projection matrices: Wg. Wa. W.. and W.. For RaLoRA, we search over the set {16, 32, 64) for the expansion factor Rmax and report the best-performing configuration, which is further discussed in Section 5.3 For RaLoRA-Pro
grid search strategy is used, and the best result reported. Specifically, we sweep the maximum expansion factor Amax E (16, 32, 64) and evaluate three different rank ranges: (Tmin, Tmax) {(4, 16), (4, 32), (6, 32))-
G.5 TRAINING CONFIGURATION FOR IMAGE CLASSIFICATION TASKS
Models and Datasets. We fine-tune the CLIP-ViT-B/16 (Radford et al., 202 model seven
widely used benchmarks: StanfordCars (rause et al. 2013). DTD (Cimpoi et al.] 2013). Eu-
roSAT (Helber et al.) 2019). GTSRB (Houben al. 2013). RESISC45 (Cheng et al.) (017). SUN397 (Xiao et al. 2010), and SVHN (Netzer et al. 2011). Model performance is reported on the validation sets using accuracy as the evaluation metric.
23

---

Published as a conference paper at ICLR 2026
Implementation Details. During fine-tuning, only the linear layers of the vision backbone updated, while all other parameters remain frozen. Training is performed in FP32 precision. For classification, adopt prompt- based templates such as "a photo of a {class)." The Adam optimizer employed with hyperparameters 31 = 0.9, f = 0.999, weight decay = 0, and a batch size of 64. We use cosine learning rate schedule with 3% warmup and peak learning rate of le-4. Zero- shot results are taken from (Wang et al. 2024c), and all fine-tuned results are averaged over three different random seeds.
H EXTENDED COMPARATIVE ANALYSIS AND DISCUSSION
To provide a comprchensive assessment of our contributions within the broader PEFT landscape, this section extends our evaluation beyond standard LoRA variants. We present comparative anal- yses against non-LoRA approaches (Section H.I) and alternative high-rank adaptation strategies (Section H.2) followed by discussion recent
methods (Section H.3)
H.1 COMPARISON WITH NON-LORA PEFT ARCHITECTURES
We compare RaL.RA and RoL-RRrr against representative non-LoRA methods that employ distinct adaptation mechanisms:
(IA) (Liu et al.,
2022): Adapts the model by learning vectors to rescale inner activations (Key, Value, and FFN), prioritizing extreme parameter efficiency.
FourierFT (Gao et al. 2024): Learns sparse updates in the spectral domain via the Discrete
Fourier Transform, theoretically enabling efficient fine-tuning of large models.
Experimental Results. We evaluated these methods on the GSM8K and HumanEval benchmarks following the protocols in Section 4.2 As detailed in Table 10, both RaLoRA and RaLoRA con- sistently outperform (IA)² and FourierFT. Specifically on GSM8K, RaLoRA-Pro achieves substan- tial gain of + 19.54% over FourierFT and +6.39% over (IA)². These results suggest that structurally aligning adaptation rank with gradient intrinsic dimensionality offers significantly higher expressiv- ity than vector rescaling spectral updates.
Table 10: Performance comparison with non-LoRA PEFT methods on GSM8K and HumanEval " indicates failure converge.
Method
GSM8K
HumanEval
(IA)² Liu al.
2022
66.62 +: 1.01
FourierFT Gao et al.
2024)
52.71 + 4.69
LoRA (Hu et al.
2022
67.78 +: 1.25
43.09 + 0.35
RaLoRA (Ours)
72.25 +: 0.59
48.78 + 1.61
RaLoRA-Pro (Ours)
73.01 + 0.53
48.37 + 2.54
Analysis of Convergence on Complex Tasks. A critical observation is the failure of both (IA)² and FourierFT to converge when fine-tuning on the Code-Feedback dataset. We attribute this op- timization failure to the task's high complexity. As illustrated Figure of the main paper, the Code-Feedback dataset exhibits a significantly higher mGID (sz 269) compared to standard instruc- tion tuning. We hypothesize that the restricted adaptation subspaces of (IA)3 and FourierFT lack the sufficient capacity to capture these nigh-dimensional gradient updates. In contrast, RaLoRA adap- tively expands rank capacity to match this complexity, ensuring robust convergence and superior performance.
Implementation Details. For faimnsss, we utilized the official implementations from the PEFT library with hyperparameters aligned with their respective original publications: (IA)² used learning rate of 3e adapting (W,, Wa, W., W.. Wop); FourierFT used a learning rate of 3e - frequency=1000, and scaling factor 300.
24

---

Published as a conference paper at ICLR 2026
H.2 COMPARISON WITH HIGH-RANE LORA VARIANTS
We further compare approach against HiRA (Huang et al. 2025), representative method that utilizes the Hadamard product to achieve high-rank updates without increasing the parameter budget.
Theoretical Distinction. While HiRA effectively increases rank, it relies on heuristic Hadamard product mechanism that does not guarantce alignment with the underlying optimization landscape. In contrast, RaLoRA is grounded in the theoretical view of the adapter as gradient compressor (Appendix [ By employing our entropy-bast based estimator, RaLoRA explicitly identifies the GID of FFT gradients and constructs block-diagonal structure to match the GID. Thus, RaLoRA represents a dala-driven adaptive alignment strategy, whereas HiRA remains structurally heuristic.
Empirical Results. We compared the methods GSM8K and HumanEyal using standard settings for HiRA (Ir = 1)-8). As shown in Table 11 both RaLoRA and RaLooR-Prr outperform HiRA across both benchmarks, validating the efficacy of GID-guided alignment.
Performance comparison between HiRA and our methods. All methods utilize a compa- rable parameter budget (cquivalent to rank
Table 11:
8).
Method
GSM8K
HumanEval
LoRA (r = 8)
67.78 + 1.25
43.09 + 0.35
HiRA (r pr=0 8)
69.93 + 0.39
45.73 + 1.22
RaLoRA (Ours) RaL.oRAPPro (Ours)
72.25 + 0.59
48.78 + 1.61
73.01 + 0.53
48.37 + 2.54
H.3 DISCUSSION ON RECENT PEFT ADVANCEMENTS
Finally, we clarify the distinction between our contributions and other recent works.
MLAE (Wang et al., 2024a) and FVAE-LoRA (Kumar et al. 2025) primarily address redundancy within LoRA adapters. MLAE utilizes masking strategies enforce independence among experts, while FVAE-LoRA employs s variational autoencoder framework to disentangle task-relevant infor- mation. While effective at reducing parameter redundancy, these methods do not explicitly measure or align with the GID.
Unlike the aforementioned approaches, our work the first to explicitly quantify the dimensional gap between LoRA and FFT using a novel entropy-based estimator. By aligning the adapter's 's rank with the estimated GID, we ensure that the adaptation capacity structurally matches the task-specific optimization landscape.
I LIMITATIONS AND FUTURE WORKS
This paper aims to address a central yet previously unexplored issue in LoRA: tween LoRA's low-rank adaptation subspace and the gradient intrinsic dimensionality. To quantify this mismatch, we introduce an entropy-based estimator systematically estimate the intrinsic di- mensionality of gradient matrices arising from full fine-tuning. Although our empirical analysis demonstrates that the entropy-basee estimator successfully captures meaningful differences in in- trinsic dimensionality across layers and tasks, the effectiveness of this estimator currently remains empirical and requires extensive theoretical analysis or broader empirical validation.
the mismatch be-
Moreover, the entropy-based estimator proposed in this work is conceptually orthogonal to most existing LoRA variants (Kalajdzievskij 2023 Hayou et al. 2024 Wang et al. 2024b Meng et al.) 2024 Büyükakkyiiz 2024| Li et al. 2026) and even to other PEFT approaches (Zhao et al. 2024). It general-purpose
thus holds promise as general purpose tool to facilitate adaptive rank selection based on the intrin-
sic gradient dimensionality of task sask-ipe full fine-tuning, potentially narrowing the performance gap between parameter-efficient methods and full fine- tuning. However, in this study, we have only explored three specific applications (RaLoRA, RaLoRA-Pro and LoRA+GID in Appendix [.4
25

---

Published as a conference paper at ICLR 2026
Investigating how to effectively integrate intropy-based estimator with other existing LoRA variants and PEFT approaches remains an important avenue for future research.
Finally, given the rapid advancement of multimodal large language models (MLLMs), which have become a pivotal area in artificial intelligence, it is crucial to evaluate the proposed method on various multimodal benchmarks to assess its generalizability and effectiveness. In future work, plan to extend our framework to multimodal settings and evaluate its performance on comprehensive benchmarks such (Liu al. 2024b 2023
Han al. 2024).
J LLM USAGE DISCLOSURE
During the preparation of this manuscript, we employed large language model solely for linguistic refinement, including addressing potential grammatical issues and enhancing clarity and readability. The model was not used to generate scientific content, ideas, experiments, or analyses. The authors take full responsibility for the accuracy and integrity of the manuscript.
26


Reviews:
[{"review_id": "QRn528yH9F", "reviewer": "Reviewer_E7b3", "summary": "The paper use the effective rank of the gradient matrix in full-parameters fine-tuning as the Gradient Intrinsic Dimensionality (GID) and based on it adaptively allocates the number of mini-LoRA to increase the overall rank without increasing the total parameter count.  Additionally, the paper integrates a loss-sensitivity-guided inter-layer reallocation strategy to dynamically redistribute trainable parameters across layers.", "questions": "In Figure 2 (Part II, Step 3: *Post-Double Alignment*), why the output dimension of ( A_1 ) in layer N is defined as ( r_1 / n_1 )?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The framework is intuitive and effective, combining several existing techniques into a cohesive and well-motivated design. Extensive experiments across NLP and CV tasks validate its robustness and superior performance. \n\n2. The visualization of gradient intrinsic dimensionality is particularly insightful, revealing similar GID distributions across different tasks.", "weaknesses": "1. The paper lacks efficiency comparisons, such as peak GPU memory usage and FLOPs, which is essential for a new PEFT method.\n\n2. Since the proposed methods is similar to the MELoRA with adaptive rank/number of mimi-LoRA for each layer, it would be helpful to include more MELoRA results with different configurations (e.g., *n* = 8, 16, 32) for a fairer comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QjkxGzeUK2", "reviewer": "Reviewer_keQx", "summary": "This paper introduces **Gradient Intrinsic Dimensionality (GID)** to explain the performance gap between low-rank adaptation (LoRA) and full fine-tuning (FFT). The authors argue that LoRA’s fixed low-rank subspace fails to align with the true gradient subspace of FFT. To address this, they propose **RaLoRA**, which estimates each layer’s effective gradient rank via an entropy-based metric and dynamically allocates LoRA capacity accordingly. A further variant, **RaLoRA-Pro**, performs inter-layer reallocation guided by loss sensitivity. Experiments on diverse NLP, code, and vision tasks show consistent and sometimes significant gains over existing PEFT methods without increasing parameters.", "questions": "1. How stable is the entropy-based GID estimator across mini-batch sampling, different random seeds, or noisy gradients?\n\n\n2. What is the computational overhead (in wall-clock time, GPU memory, or FLOPs) relative to standard LoRA?\n\n\n3. Does GID correlate with curvature or layer sensitivity measured via Fisher Information or Hessian spectra?\n\n\n4. Could the observed improvements of RaLoRA be attributed to implicit regularization effects rather than true gradient alignment?\n\n\n5. Have you tested dynamic rank scheduling during training instead of static pre-assignment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clear and meaningful motivation.**\nThe paper addresses a known limitation of LoRA—its rigid and sometimes misaligned update subspace—through a theoretically motivated concept (GID). This adds a fresh geometric interpretation to the efficiency–performance trade-off in PEFT.\n\n2. **Methodologically elegant design.**\nThe entropy-based estimator and adaptive rank allocation are conceptually simple yet principled. The proposed dual-level alignment (intra- and inter-layer) is coherent and implementable without modifying the base model.\n\n3. **Comprehensive experimental evaluation.**\nResults span multiple domains and include comparisons to strong baselines such as DoRA, LoRA+, PiSSA, and MELoRA. The reported gains are consistent and statistically significant, and the ablation analyses are generally convincing.  The ablation studies and visualizations (e.g., GID heatmaps) support the empirical narrative.\n\n4. **Readable and reproducible.**\nThe algorithm is clearly described, and the proposed method could be implemented with minimal modification to existing LoRA frameworks. The method can potentially generalize to other PEFT frameworks and adaptive subspace modeling approaches, making it of interest to the broader community.", "weaknesses": "**1. Incremental novelty.**  \nAlthough the paper introduces Gradient Intrinsic Dimensionality (GID) and rank alignment, the overall idea aligns closely with recent adaptive-PEFT variants such as GoRA, LoRA+, and DoRA. The contribution mainly reframes known intuitions in a new terminology without offering a fundamentally new optimization mechanism or theoretical insight.\n\n**2. Lack of theoretical rigor.**  \nWhile GID is intuitively appealing, the paper lacks formal analysis linking GID alignment to optimization dynamics or generalization improvement. No convergence guarantees, projection error bounds, or theoretical explanations are provided for why matching ranks to GID improves performance.\n\n\n**3. Computational overhead and scalability.**  \nThe GID estimator requires repeated singular-value decompositions (SVDs) on layerwise gradients, which can become expensive at scale. The paper omits concrete measurements of wall-clock overhead, GPU memory usage, or FLOP ratios compared with vanilla LoRA. It remains unclear whether RaLoRA scales to 70B-parameter LLMs or multi-GPU fine-tuning setups.\n\n**4. Interpretability and analytical depth.**  \nThe visualization of GID heatmaps provides little insight into model behavior. The work does not analyze whether GID correlates with layer function, curvature, or Fisher information. A deeper discussion of why certain layers have high or low GID would strengthen the scientific contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D8ffZgpasE", "reviewer": "Reviewer_rn9v", "summary": "This paper proposes RaLoRA and RaLoRA-Pro, methods for adapting the rank of LoRA adapters by estimating a new quantity called Gradient Intrinsic Dimensionality (GID) using an entropy-based “effective rank” estimator. The idea is that LoRA’s fixed low-rank subspace can misalign with the true gradient subspace of full fine-tuning; thus, estimating GID per layer and adjusting ranks could better use the parameter budget. The paper reports strong improvements on GLUE, MT-Bench, GSM8K, HumanEval, and image classification.", "questions": "Q1: The paper estimates Gradient Intrinsic Dimensionality (GID) using spectral entropy, which requires SVD decomposition.\nIs SVD computed for every layer at each update step, or is it precomputed and then frozen?\nWhat is the time complexity compared with other baseline methods?\nQ2: The current block-diagonal design may restrict gradient flow and information sharing. Could the authors discuss or compare this design with HiRA, or provide further justification for choosing this particular formulation?\nQ3: The visualization of importance estimation in RaLoRA-Pro is unclear. How does it differ from prior works that allocate LoRA ranks based on importance or sensitivity?\nQ4: The paper proposes a dynamic rank allocation method. Has it been fairly compared with other rank adaptation approaches? Could the authors analyze the comparison in terms of parameter count or efficiency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem motivation, mismatch between LoRA rank and true gradient structure, is interesting. \nThe core insight of this block-diagonal design hinges on the property that $\\text{Rank}(BA)= \\sum_{i=1}^{n_l} rank(B_iA_i)$. By employing block diagonalization, the authors aim to strategically augment the overall $\\text{Rank}(BA)$ to align with the estimated Gradient Intrinsic Dimension (GID).\n\n2. The proposed entropy-based GID estimator is conceptually clean and grounded in established ideas.\n\n3. Writing is mostly clear and well-organized.", "weaknesses": "1. The paper estimates Gradient Intrinsic Dimensionality using spectral entropy, which typically requires SVD and significant computation. Is SVD performed for every layer at each step, or precomputed and frozen? What is the time complexity compared to other estimation methods? It would be helpful to show robustness under different batches to assess the stability of the gradient SVD.\n\n2. The design of the block diagonal matrix may restrict gradient flow and information sharing.  If the goal is merely to increase $\\text{Rank}(BA)$, other approaches such as introducing a Hadamard product (as in HiRA [1], Eq. (4)) could also achieve a higher rank. A comparative discussion would further elucidate the distinctions between these approaches.\n\n3. The visualization of importance estimation in RaLoRA-Pro is unclear. How does it differ from prior works that allocate LoRA ranks based on importance or sensitivity?\n\n4. The paper adopts a dynamic rank allocation strategy. Has it been fairly compared with other methods, for example, in terms of parameter count or efficiency?\n\n5. Typos: \n     1) Chapter “Disscusion” → “Discussion” \n     2) Inconsistent capitalization of “GSM8K” (e.g., “GSM8k” in Table 2). Please standardize. \n     3) All equation references should use “Equation XX” instead of “equation XX”.\n\n[1] HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gF3cSwxZUp", "reviewer": "Reviewer_4tTg", "summary": "This paper introduces a novel and compelling concept: the mismatch between the fixed, low-rank subspace of LoRA and the high intrinsic dimensionality (GID) of full fine-tuning gradients is a fundamental limitation of existing LoRA variants. To address this, the authors propose an entropy-based estimator for Gradient Intrinsic Dimensionality (GID) and build two new methods upon it: RaLoRA, which performs intra-layer rank alignment, and RaLoRA-Pro, which adds inter-layer parameter reallocation. The paper is well-structured, the motivation is clear and well-argued, and the experimental evaluation across NLP and vision tasks is extensive and generally demonstrates the superiority of the proposed methods.", "questions": "1. What is the computational overhead (in time and memory) of the GID estimation phase compared to the standard LoRA initialization? Is this cost negligible for very large models (e.g., >100B parameters)?\n\n2. The GID is estimated once at initialization. Given that Figure 3(c) shows GID evolves, have you experimented with dynamic rank adaptation during training? What were the results and challenges?\n\n3. It is better if the authors can discuss some recent works:\n\n   - RandLoRA: Full-rank parameter-efficient fine-tuning of large models\n   - Mlae: Masked lora experts for parameter-efficient fine-tuning\n   - Latent Space Factorization in LoRA\n\n4. How does the performance and efficiency of your method compare to simply using a standard LoRA with a much higher rank (e.g., r=64 or 128) to cover the high GID, even if it means a higher parameter count? The results in Appendix E.3 (Table 8) are a start, but a more systematic comparison on a equal-parameter or equal-performance basis would be insightful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel and Well-Motivated Core Idea: The paper's central thesis—that aligning LoRA's structure with the intrinsic dimensionality of the full fine-tuning gradient is key to closing the performance gap—is both novel and persuasive. The analysis in Section 5.1 and Appendix E effectively characterizes this GID, providing strong empirical evidence for the problem statement.\n\nStrong Empirical Results: The experimental section is a major strength. The authors comprehensively evaluate their methods on a diverse set of tasks (NLU, NLG, Vision) and against a wide array of strong, recent baselines. The results consistently show that RaLoRA and RaLoRA-Pro outperform other LoRA variants and significantly narrow the gap with full fine-tuning, sometimes even surpassing it. The ablation studies (Section 5.2, 5.3, 5.4) are particularly valuable for disentangling the contributions of the different components.\n\nTheoretical Grounding: The paper provides a solid theoretical foundation. The derivation of LoRA as an implicit gradient compressor (Appendix A) is elegant and directly motivates the GID concept. The analysis of the approximation error for RaLoRA's block-diagonal structure (Appendix C) offers a principled explanation for why its architectural change can be more expressive than standard LoRA under a fixed parameter budget.", "weaknesses": "**1. Computational Overhead of GID Estimation:** A significant practical concern is the computational cost of estimating GID. The method requires computing the SVD of the full gradient matrix for each target layer over several mini-batches (Algorithm 1). For very large models (e.g., LLaMA 70B), the cost of these SVD operations, even if done only once during a warm-up phase, could be non-trivial. The paper would be strengthened by a discussion or analysis of this overhead, perhaps quantifying the extra time/memory cost compared to standard LoRA initialization.\n\n**2. Clarity on the \"One-Shot\" vs. \"Dynamic\" Nature of GID:** The methodology section and Algorithm 1 suggest that the GID estimation and parameter reallocation are performed as a one-time initialization step using gradients accumulated from a few mini-batches. However, the discussion of GID dynamics in Section 5.1 shows that GID evolves during training. This raises questions:\n\n   - Why is a one-time estimation sufficient? Could performance be further improved by periodically re-estimating GID and adapting the structure during training?\n\n   - How sensitive are the results to the specific mini-batches used for the initial GID estimation?\nA discussion of these points and perhaps an experiment on the stability of the initial GID estimate would alleviate these concerns.\n\n**3. Hyperparameter Sensitivity and Tuning Cost:** While the proposed methods are shown to be effective, they introduce new hyperparameters: the maximum expansion factor n_max and the rank reallocation range (r_min, r_max). The results in Tables 5 and 6 show that performance is sensitive to these choices, and the authors mention using a grid search. This somewhat offsets the \"parameter-efficient\" and \"easy-to-use\" appeal of standard LoRA. A more in-depth discussion on guidelines for setting these hyperparameters or strategies to reduce the tuning cost would be very helpful for practitioners.\n\n**4. Comparison to Other High-Rank PEFT Methods:** The baselines are comprehensive within the LoRA family. However, it would be valuable to see a comparison with other PEFT methods that also aim for higher rank/more expressive updates but through different mechanisms, such as (IA)³ or perhaps a memory-efficient implementation of a method like DiffPruning. This would better situate the contribution within the broader PEFT landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)