IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring
Authors: Xuan Cui
HuiYue Li
Run Zeng
Yunfei Zhao
Jinrui Qian
Wei Duan
Bo Liu
Zhanpeng Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Low-Rank Adaptation
Integrated Gradients
Parameter-Efficient Fine-Tuning
Uncertainty-Aware Scoring
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper proposes IGU-LoRA, an adaptive-rank LoRA method that leverages integrated gradients and uncertainty-aware scoring to improve parameter-efficient fine-tuning of large language models.

Abstract: As large language models (LLMs) scale to billions of parameters, full-parameter fine-tuning becomes compute- and memory-prohibitive. Parameter-efficient fine-tuning (PEFT) mitigates this issue by updating only a small set of task-specific parameters while keeping the base model frozen. Among PEFT approaches, low-rank adaptation (LoRA) is widely adopted; however, it enforces a uniform rank across layers despite substantial variation in layer importance, motivating layerwise rank allocation. Recent adaptive-rank variants (e.g., AdaLoRA) allocate ranks based on importance scores, yet typically rely on instantaneous gradients that capture only local sensitivity, overlooking non-local, pathwise effects within the same layer, which yields unstable and biased scores. To address this limitation, we introduce IGU-LoRA, an adaptive-rank LoRA that (i) computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation, and (ii) applies an uncertainty-aware scheme using exponential moving averages with deviation tracking to suppress noisy updates and calibrate rank selection. Theoretically, we prove an upper bound on the composite trapezoidal rule approximation error for parameter-space IG under a pathwise Hessian-Lipschitz condition, which informs the quadrature budget. Across diverse tasks and architectures, IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness. Ablations confirm the contributions of pathwise within-layer sensitivity estimates and uncertainty-aware selection to effective rank allocation.

Published as a conference paper at ICLR 2026
IGU-LORA: ADAPTIVE RANK ALLOCATION VIA IN- TEGRATED GRADIENTS AND UNCERTAINTY-AWAR SCORING
Xuan Cui', Huiyue Li', Run Zeng', Yunfei Zhao', Jinrui Qian', Wei Duan""*, Bo Liu".,, Zhanpeng Zhou?.* *Chongging Technology and Business University Shanghai Jiao Tong University "Corresoonding Author
ABSTRACT
As large language models (LLMs) scale to billions of parameters, full- callmparameter fine-tuning becomes compute- and memory- prooiiy-ro Parameter-efficient fine- tuning (PEFT) mitigates this issue by updating only small set of task-specific parameters while keeping the base model frozen. Among PEFT approaches, low- rank adaptation (LoRA) is widely adopted; however, it enforces a uniform rank across layers despite substantial variation in layer importance, motivating layerwise rank allocation. Recent adaptive-rank variants (e.g.. Adal allocate ranks based on importance scores, yet typically rely on instantaneous gradients that capture only local sensitivity, overlooking non-local, pathwise effects within the same layer, which yields unstable and biased scores. To address this limitation, we introduce IGU-LoRA, an adaptive-rank LoRA that (i) computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into layer-level score for rank allocation, and (ii) applies an uncertainty- moving averages with deviation tracking to suppress noisy updates and calibrate rank selection. Theoretically, zoidal rule approximation error for parameter-space IG under a pathwise Hessian- Lipschitz condition, which informs the quadrature budget. Across diverse tasks and architectures, IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness. Ablations confirm the contributions of pathwise within-layer sensitivity estimates and uncertainty- selection effective rank allocation. Our code is publicly available at https::/github. com/ http://wttowwwttttttee /tt.l////..ii.iii.i.
scheme using exponential
prove upper bound on the composite trape-
INTRODUCTION
Large language models (LLMs) have achieved remarkable success across a wide range of NLP tasks (Devlin et al. 2019 Brown et al.) 2020a; Han al. 2025). However, specialising these models for new downstream tasks remains challenging due to their large parameter counts and substantial computational and memory costs. Consequently, fine-tuning has emerged the standard way to adapt pre-trained LLMs to particular downstream tasks.
Early efforts in fine-tuning primarily relied on full-parameter fine-tuning (FPFT) (Lv et al. 2024 Qiu al.] 2020 Raffel et al. 2020), where all model parameters are updated during training. While effective for small to medium-scale models, such as BERT (Devlin et al., 2019) and RoBERTa- large [Liu et al. 2019). FPFT becomes increasingly impractical as model size scales exponentially. For example, GPT-3 (Brown et al. 20206) contains 175 billion parameters, making full fine-tuning prohibitively expensive terms of computation and memory.
To alleviate these challenges, parameter-efficient fine-tuning (PEFT) methods have been proposed, which adapt pre-trained models by updating only a small subset of parameters while keeping most of the model frozen. Notable PEFT methods include Adapter Tuning (Houlsby et al. 2019 Rückle
et al. 2021: Pfeiffer et al.) 1 He et al. 2022 Wang et al. 2022). Prefix Tuning (Li & Liang 2021; Wu et al. 2024), and Prompt Tuning (Liu et al. 2022b] Zhang et al. 2024 Yu et al. 2023
Cui et al. (2025). These methods significantly reduce the number of trainable parameters. However, they primarily affect shallow or intermediate layers, limiting their ability to capture deeper semantic representations.

---

Published as a conference paper at ICLR 2026
Complementary to the above, weight-delts methods (e.g.. Diff Pruning (Guo et al. 2020 Fang et al. 2023)) selectively update a sparse subset of important weights. While effective in reducing the scale of trainable parameters, these methods often rely unstructured sparsity, which poses challenges for optimisation and is less compatible with modern hardware acceleration. A more structured alternative Low-Rank Adaptation (LoRA) (Hu et al.] 2022a), which models the weight update A WW as the product of two low-rank matrices. By preserving the pretrained model architecture and introducing only small number of trainable parameters, LoRA achieves high efficiency without sacrificing model capacity. However, LoRA typically uses fixed rank across all layers, ignoring the heterogeneous contribution of different weight matrices. This static configuration may limit the adaptability and expressiveness of the model.
Building on this observation, several adaptive-rank PEFT methods have been proposed (Zhang et al. 2023, Xu et al. 2023, Ding et al. 2023; Valipour et al. 2023). For example, AdaL.RAA (Zhang et al. 2023) applies singular value decomposition (SVD) to the low-rank update matrices and dynamically adjusts rank sizes based on layer-wise wise importance scores. However, the scoring mechanism in AdaLoRA is primarily based on instantancous gradient signals, which fail to capture long-term parameter contributions and inter-layer interactions. As a result, the rank allocation may be suboptimal in complex optimisation scenarios.
To overcome these limitations, we propose IGU-LoRA(Fig. [c)), IG-driven PEFT framework that extends Integrated Gradients to the parameter space for scoring parameter importance. The IG path integral is efficiently approximated via a mini-batch stochastic quadrature that uniformly samples one node a E [o, 1) per mini-batch, thereby avoiding the O(N) forward-backwer trapezoidal integration where N denotes the number of discretization steps along the IG path and adding only batch-linear overhead. Compared with stable and globally informed importance estimates. Robustness further enhanced by modeling the predictive effect of parameter perturbations and by an uncertainty -ware score that couples an EMA mean with a dispersion term. On the theory side, we establish (i) a discretization-sapplhi
passes of
cotntttannnou-grrdiiien heuristics, this yields
error bound for the IG estimator of order O(N-3) + (M 1/23), where M the number of sampled mini-batches, and (ii) a high- gegh--robabblity stability guarantee for the EMA ratio score SNRt, the signal- to-noise ratio at iteration f. Empirically, across datasets (BoolQ, GSM8K, GI.UE.... and backbones (RoBERTaa-aarge. Qwen-2.5-0.5B. Llama-2-7B, Llama-3-8B, DeepSeek- 1-Distill- Srsstill-Qee-2.5-777 ( IGU-LoRA consistently improves accuracy over strong PEFT baselines (LoRA, AdaLoRA, DoRA) while matching their memory footprint and decoding latency.
My F4
oulput
prAce wauliity
Decalie intepeee gudiers
syn Redio
Never Wiigh Mark c
A gAre
Premza Weak:
Preses Wiigh
A Eeww
W. e
A c
We cA06
gra
Q R
Q C
ingut
ingut
impui
(4) LoRA
(b) AddL.RA
(6) IGU-LLRA (Curr)
Figure 1: Comparison of frameworks: left (a) LoRA, (b) Adal.oRA, (c) the proposed IGU-LoRA, IGU- LoRA builds LoRA and AdaLoRA, introducing integrated gradients (IG) compute parameter importance scores. Please zoom 300% for better clarity.
RELATED WORKS
2.1 PARAMETER EFFICIENT FINE-TUNING
Parameter-Efie Efficient Fine-Tuning (PEFT) received widespread attention for its effectiveness in effi
ciently adapting LLMs. Representative approaches included Adapter Tuning (Houlsby et al. 2019
Rücklé et al. 2021 Pieiffer et al.) 2 He et al. 2022 Wang et al. (2022). Prefix Tuning (Li &
Liang 2021 Wu et al. 2024), Prompt Tuning (Liu et al. 2022b Zhang et al. 2024 Yu et al. 2023
Cui et al. 2025), and P- Tuning v2 (Liu et al. 2021), which inserted lightweight trainable modules into
different layers of the model to enable efficient task adaptation. In parallel, reparamcterization based
PEFT approaches (Li et al. 2018
Aghajanyan al. 2021 Liu al. 2024a Hu et al. 2022a Zhang
2

---

Published as a conference paper at ICLR 2026
al. 2023) received increasing attention. Without modifying the model architecture, these methods modeled and optimized parameter updates in low-dimensional and efficient manner. Among them, Low-Rank Adaptation (LoRA) (Hu et al.) (022a) has become prominent method by expressing weight updates as the product of two low-rant matrices, which allows for tight control over the train- able parameter count while maintaining model performance. With the rapid release of open-source LLMs (Shao et al., [0244] Liu et al., 2019;; Dubey et al. (2024) and their increasing use in instruction tuning and other real-world applications, PEFT has emerged the mainstream paradigm for scalable fine-tuning and has been widely adopted in practical systems.
2.2 LOW-RANK ADAPTATION FINE-TUNING
LoRA (Hu et al. 2022a) is representative PEFT method that freezes pretrained weights and injects low-rank matrices, reducing parameter overhead with minimal performance loss. Several LoRA-base methods have been proposed to enhance efficiency and scalability. For example, Delta-LoRA (Zi
ett al. 2023) improves LoRA's expressiveness by updating weights with the temporal difference of
AB, addressing the limitations of small low-rank matrices. DoRA (Liu et al. 2024b) decouples optimization by factorizing W into a magnitude vector m and a direction matrix V. McLoRA (Ren
et al. 2024) aggregates outputs from parallel low-rank adapters in block-diagonal structure to improve model capacity. AutoLoRA (Xu et al., 2023) uses meta-learning to automatically assign optimal per-layer ranks, while AdaLoRA (Zhang et al., 2023) dynamically adjusts ranks during training using SVD and parameter importance scores. SalientLoRA (Ke et al. 2024) allocates ranks based on parameter saliency, optimizing the low-rank layers for improved performance. GoRA (He) et al. 2025) adapts low-rank adjustments dynamically using gradient-driven methods to meet task requirements while maintaining efficiency. These techniques enable efficient fine-tuning with fewer trainable parameters and strong performance.
2.3 INTEGRATED GRADIENTS
In interpretability research for deep learning, Integrated Gradients (IG (Sundararajan et al. 2017) a widely adopted attribution method that mitigates gradient saturation by computing the integral of gradients along the path from baseline input to the actual input. IG satisfies two key axioms, completeness and sensitivity, which ensure that it quantitatively reflects the contribution of each input feature to the model's s output. Subsequent studies extend IG in various directions. Theoretically, Lund berg & Lee (017) show that IG is equivalent to Shapley values under certain conditions. From computational perspective, Kapishnikov et al. [2021) propose adaptive sampling strategy that improves runtime efficiency by 3x. IG also demonstrates practical utility in high-stakes domains such medical imaging (Sayres et al. 2019), where it improves the localization of diabetic retinopathy markers. In this work, we extend IG to parameter importance estimation in large model fine-tuning. Our method addresses the limitations of instantancous gradient signals. which are prone to vanishing in deep networks. It introduces a redefined sensitivity scoring mechanism that more accurately captures long-term parameter contributions during optimization.
3 METHOD
3.1
PRELIMINARIES
Low-Rank Adaptation. Low-Rank Adaptation (LoRA et matrices into frozen pre-trained weights, substantially reducing the number of trainable parameters while preserving downstream task performance. Given a pre-trained parameter matrix Wo E Rd Xd2 for a specific layer of an LLM, LoRA updated the parameter matrix as:
2022al) injected trainable low-rank
W W, + AB,
where A E Ridix and B E Rexd2 were low-rank trainable matrices with < min(d1. dz).
Adaptive LoRA. A key limitation of LoRA is that it requires manually selecting the rank r, which is challenging due to the heterogeneity of intrinsic dimensionalities across layers and the lack of princi- pled guidance for determining appropriate values. To enable adaptive rank selection, singular value decomposition (SVD) is typically applied to the trainable low-rank product AB in Eq. (1) Zhang
et al. 2023):
W Wo + SVD(AB) Wo + PAQ.
where P E Rdi Q Rxxda are two orthogonal matrices, and the diagonal matrix A diag(> X2.....A,) E Rxr containing the singular values. We initialize r as an over- parameterized upper bound < min(d1, d2 then prune redundant dimensions via spectral analysis.
3

---

Published as a conference paper at ICLR 2026
To determine the final rank, define an importance score S; for each singular value A, which guides the pruming process. Unlike conventional methods that rely solely on magnitude, our proposed scoring method incorporates both the singular value and the sensitivity of its associated parameters, namely the elements in the i-th column of P and the i-th row of Q. Specifically, for each i E (1.....r) r). estimate S; by aggregating two components. First, sx(+) measures the intrinsic strength of the singular value: Second, s-).) quantifies the importance of the parameters with the i-th column of matrix P and the i-th row of matrix Q. The final score S; computed as Zhang al. (0023):
d,
de
S. = + d1 SANT (Psi) + d2 Σ
1
Σ k=1
8 Sanr (Qik);
where sx(A,) = |A,] denotes the magnitude of the singular value, and Senr (-) is a specific importance score function that measures the importance of individual weight the training loss function. Existing methods (Zhang et al. 2023) for measuring parameter importance are primarily based on simple gradient sensitivity Rey w.y C) where Wij is single parameter in model. However, this simple gradient sensitivity-bant based method suffers from the following limitations:
Lack of Structural Interpretability: Simple gradient sensitivity-based method evaluate weights independently, ignoring the structured interactions among parameter groups. In settings like LoRA, where parameters operate collectively within subspaces, such element-wist -wise assessments fail to capture their joint contribution, thereby limiting interpretability at the structural level.
Instantaneous Parameter Sensitivity: Simple gradient sensitivity- based method capture only the instantaneous impact of parameter on the loss function, overlooking its accumulated or long-term contribution throughout training. This limitation can result in unstable or misleading estimates.
Gradient Saturation: In transformer-based LLMs, activation functions such as ReLU may lead to gradient saturation in inactive regions, where the gradient signal vanishes entirely. As a result, the estimated importance of the affected parameters becomes unreliable.
Figure 2 illustrates why (a) the simple gradient method fails in gradient-satue saturated regions, while (b) the integrated gradient method provides more reliable parameter importance estimation through comparative demonstration. To address these limitations, we estimate parameter importance using Integrated Gradients (IG) in the parameter space. IG integrates the gradients along the path from 0 to 1, thereby capturing the non-local sensitivity and overall impact of the gradients. This method not only accounts for the cumulative effect of the parameter gradients along the integration path but also effectively bypasses saturation regions, where gradient signals typically vanish. By considering the entire path, this method ensures a more accurate estimation of parameter importance, particularly in regions where simple gradient-basee methods may fail due to vanishing gradients or saturation.
1.0
Satundies Regies
Bronpinned
0.6-
Earportance
Grodiret Cimppectane
Importance
3.6
0.4
0.2
0.0
0.23
0.50
3.75
1.50
3.25
3.30
1.75
2.50
6.25
1.50
2.75
1.50
1.25
1.53
2.75
2.00
CAL)
(a) Simple Gradient (Instantaneous)
(b) Integrated Gradients (IG)
Figure 2: Comparison of parameter importance scoring methods. (a) The simple gradient method fails in saturated regions, assigning near zero importance. (b) Integrated gradients compute importance by integrating along the path from initial to final parameter values, capturing the actual total contribution.
3.2 IMPORTANCE SCORING VIA INTEGRATED GRADIENTS
Integrated Gradients (IG (Sundararajan al. 2017 is attribution method originally developed to improve the interpretability of deep neural networks by attributing a model's output to its input features. It quantifies the contribution of each input feature by integrating the gradients of the output with respect to the input, along a path from a baseline to the actual input.
Inspired by this idea, we propose IGU-LoRA, which extends IG to the parameter space for importance estimation in LLMss. Specifically, integrate the gradients of the loss function with respect to
4

---

Published as a conference paper at ICLR 2026
model parameters along continuous path from baseline (e.g.. zero) to the actual trained weights, thereby capturing the cumulative influence of each parameter on the training loss function. This parameter-space IG formulation addresses key limitations of conventional gradient-based importance scores, such as limited structural interpretability, over-reliance on local (instantaneous) sensitivity, and susceptibility gradient saturation. Consequently, it provides more stable and comprehensive estimates of parameter importance for transformer-basee LLMs.
Formally, given weight matrix A AW, we denote by waj its (i, entry, representing a specific weight. Let L denote the loss function of the LLMs. Since Integrated Gradients (IG) requires baseline representing state of no information, and compute the importance score of Wii under IG as:
choose
the value for AW(") the baseline,
Aw!yy (0) [ AL(a(AW - AW ([loo)W-W(o))
al(oAW) do
Se(was) (wis
do
Wij
(4)
Ow'y
Owiy
where Aut (0)
E AW Due to the massive number of parameters in LLMs, the loss function L exhibits strong non-convexity and highly nonlinear dependencies in the parameter space. As result, Eq. 44 involves a high-dimensional integral that lacks closed-form solution. To approximate we discretize the path [0, 1J into N equal intervals with nodes ag k/N (k I,..., 1) and apply the trapezoidal rule, yielding:
OC(0) Twif
N-1 OC (axAAW OC(AW) Owi
(w(r))))
2
(5)
2N
Owij
Note that Eq. (5) requires gradient evaluations at N + points, which leads to O(N) forward- backward passes for each weight Waj. making it computationally expensive in large models. To mitigate this computational burden, we propose a stochastic approximation strategy: during fine- tuning, We randomly sample single integration point ar k/N for each mini-batch from set (1/N, (N -01)) 1)/N} that follows a uniform distribution. Consequently, for the p-th mini-batch, the importance score of Wij is approximated as:
aC(0) OL AW) + aC(AW)
L (wir)
2
(6)
2N Dwi,
Owij
Owi
At the end of the 1-th training epoch (which consists of M mini-batches), compute the aggregated importance score of wiy as follows:
Sagg (wi))
M Σ (way).
Theorem I bounds the error of our estimator, quantifying the gap between the exact IG score in Eq. and the epoch-level estimator in Eq. 7 the total error is O(N-2) (discretization) + O(M (sampling).
Theorem 1. Let se(wis) be the importance score based Integrated Gradients (IG) defined be the epoch-level estimator as defined in Eq. 77. Define 9ız(a) =
in Eq. (4) and let
ac(ad
aa 10,
owee
We assume the following:
1. Jij is twice continuously differentiable on [0, 1}. and there exists a constant C2 < 00 such that
sup oc(o.
(a)) s C2.
(8)
2. Let 01, 0220- O.M
O.M be i.i.d. samples drawn from the discrete uniform distribution over N=1 }, and let Sogg(Wi) be defined as in Eq. (7). N
{ W N....;
Then, for any N, M 2 1 and 6 E (0, with probability at least 1 - 8. the following bound holds:
|Wss|C2
log(1/6)
se (<(()))) - Sagg (wiz))
+ clweyl B
12N2
M
where 0 is an absolute constant, and B is constant such that [a)] S B for all E
N: N. ..... N-1 The proof is provided in Appendix A.1 A.I
5

---

Published as a conference paper at ICLR 2026
3.3 UNCERTAINTY-AWARE SCORING
Recent studies (hang et al. 2022 demonstrate that stochastic sampling and complex training dynamics result in high variance in importance score estimates via Eq. 7 thereby undermining their reliability. To alleviate this issue, we incorporate two complementary mechanisms: sensitivity smoothing and uncertainty quantification, defined respectively as:
(wis) 3, 318% (w:s) + (1 B1 B1)s (wiz);
(10)
(wij) (2U((1)) +1 B2) (/ (wiy) (
(11)
We define the final importance score as:
(1) s (wis) = SNR, (1(5)) (mns)) +
(wis)
(12)
where the numerator Se (wsy) captures the persistent influence of the parameter way via exponen- tial moving averaging of gradient-parameter correlations. The denominator (77) quantifies epistemic uncertainty by measuring deviations from the smoothed sensitivity across mini-batches. e is very small number to prevent the denominator in Eq. (12) from being 0. This ratio can be interpreted as a signal-to-noise ratio (SNR), providing criterion for assessing the importance of parameters. Specifically, a larger smoothed sensitivity s!¹) (w:t)) indicates that Wiy consistently exerts strong influence on the loss function. In contrast, a smaller uncertainty (w1r) suggests lower variability, reinforcing the reliability of the signal. A high-probability stability guarantee for the EMA ratio score SNR; is presented Appendix A..2 A.2 We summarize the complete workflow of IGU-LoRA
in Algorithm !
Algorithm IGU-LORA
Input: Dataset D; the number of total iterations T;; a pre-trained parameter matrix Wo E Bd xd2 of a large language model, number of mini-batches M; budget of remaining singular values b; randomly initialize trainable low-rank matrices A E R&4 x" and B jrxda; hyperparameters 31, Bz.
for t 1 do
2: for p to M do
3: Sample mini-batch from D and train A and B.
4:
Perform SVD on the matrix product AB to obtain PAQ SVD(AB), where A diag A2 -..... Ar]-
5:
Compute & Eq. (6) for every parameter in P.Q.
6:
end for
7:
Compute the aggregated importance score Sagg in Eq. ( for every parameter in P, Q.
8:
Compute the (c) in Eq. 10) and (1(+) in Eq. (11 for every parameter in P, Q.
9:
Update the final importance score sanr in Eq. 12)
10:
Compute the importance score of each singular value S; in Eq. for PAQ.
11: Find the top eigen value: d₁ .2....,,, Ås by importance score S,
Set A <- diag(A (),, λz,... ...... An, ......))
0)
13:
Update A P. a. A 1/2 B A
The subscript denotes the index set obtained by and Q descending order; æı:k represents indices the first selected columns; PimTL represents selecting the first columns according to the order defined by ा.
Q. QT
sorting the columns
14: end for
Output: W W=W-+ + AB
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Computational Resources. All experiments are implemented in PyTorch and conducted on an NVIDIA L40 GPU (48GB) running Ubuntu 18.04.1.
Pretrained Backbone Models. We use RoBERTa-large model (Liu et al. 2019) as the backbone for the GLUE tasks. For the remaining tasks, we adopt Qwen-2.5-0.31 model We further validate the robustness and generalization of IGU-LORA via backbone ablation, fine-tuning

---

Published as a conference paper at ICLR 2026
backbones (Llama-2-7B (Touvron al. 2023). Llama-3-8B (ubeyy et al. 2024 DeepSeek-RI. R1- Dastil-Qwee-225-77BBB on multiple datasets.
IGU-LoRA Configuration. For the BoolQ, ARC, GSM8K, and AQuA tasks, we perform instruction tuning. The initial LoRA rank set r(0) 32, and pruned to an average rank of r(1) 16, achieving pruning 50% rank reduction. For the GLUE tasks, we follow AdaLoRA's setup, using
classification or regression head, with r(0) = 2 pruned to average r(1) = 1. During the fine-tuning, IGU-LoRA selects the scaling factor a from .V 20 uniformly spaced values in the interval (0, 1). Rank pruning begins at epoch 2 and ends at epoch 5, performed at every one-fifth of an epoch. After pruning, we fine-tune the modules with early stopping (patience = 10 steps) to restore performance. Inference is performed using beam search with width of 3.
Reproducibility. Each task is run with 5 different random seeds, and report the median test performance. All predictions are generated using the model's language modeling head, which is conditioned given prompt instruction. Additional training configurations are available in Appendix
4.2 DATASETS AND EVALUATION METRICS
We group the tasks into 2 categories and compare the proposed IGU-LoRA against several baselines: (i) GLUE Benchmark Datasets (Wang et al. 2018) include a diverse set of language understanding tasks, such as paraphrase detection (MRPC, QQP), sentiment classification (SST-2), natural lan- guage inference (MNLI, RTE, QNLII,, and linguistic acceptability (CoLA). (ii) Mathematical and Common-Sense Reasoning Datasets include two mathematical reasoning tasks; AQuA (Li et al. 2024) and GSM8K (Cobbe al. 2021), ) and four common-sense question answering tasks: ARC-e, ARC-c (Clark et al., 2018). BoolQ (Clark et al.) 2019) and COPA (Roemmele et al. 2011. Detailed
dataset descriptions, statistical, and evaluation metrics are in Appendix
4.3 BASELINE METHODS
To evaluate the performance of the proposed IGU-LoRA method in fine-tuning LLMs, we compare against the following representative baseline: (i) LoRA and Its Variants. We evaluate four LoRA- based approaches: LoRA (Hu et al. 2022a), Adal (Zhang et al. (023). DoRA (Liu et al., 2024b), AutoLoRA (Xu et al. 2023) and GoRA (He et al., 2025). (ii) Other PEFT Method. We also evaluate the following LoRA parameter-efficient fine-tuning methods: Housbly-Adapp
(Houlsby et al. 2019), P-Tuning v2 (Liu et al. 2021), (IA)* (Liu et al. 2022a). and SSP (Hu et al. 2022b). (iii) Full Fine-tuning Method. For reference, we also include results from full-parameter fine-tuning (denoted as Full FT). All baseline methods are implemented using publicly available codebases. Hyperparameter settings are listed in Appendix C and additional descriptions of baselines are provided in Appendix
4.4
MAIN RESULTS
Table 1: Performance comparison of fine-tuning methods the GLUE task using RoBERTaalarge. All results are reported as the median over with different random sceds. Bold and Underline indicate the hest and the second-best results. The metric for each task is explained Appendix [.5
CoLAA
SST-2
MRPC
QQP
STS-B
MNLI
QNLI
RTE (acc)
Method
Params
(mcc)
(acc)
(ace-f1)
(ace-fl)
(corr)
(ace)
(acc)
Avg.
Full
355M
69.19
95.63
89.46
91.10
91.60
90.01
94.03
86.94
88.50
Housbly-Adapter
0.35M
67,80
94.38
89.75
89.
91.08
90.28
93.52
84.36
87,57
P-tuning v2
0.31M
67.35
93.13
88.49
88.63
90.41
89.19
91.94
82.42
86.45
(AA)
0.33M
68.62
93.82
59
89.78
90.84
89.87
92.60
82.75
87.23
SSP
0.36M
69.89
94.96
90.08
90.14
91.37
90.42
94.16
84.88
88.24
LoRA
0.33M
68.71
94.84
89.71
90.26
91.63
90.34
93.87
85.56
88.12
AdaLoRA
0.35M
70.04
95.62
90.34
90.37
91.57
90.18
94.29
87.06
88.68
DoRA
0.33M
70.26
95.80
90.12
90.16
91.68
90:43
94.17
87. 38
48.75
AuDLLORA
90.26
90.31
91.52
90.26
94.08
88.76
IGU-LORA
0.33M
71.93
96.17
90.69
90.68
91.95
90.76
94.72
88.46
89.42
GLUE Benchmark Results. We evaluate the performance of IGU-LoRA against baseline methods
on the GLUE development set using the RoBERTa-large model. The results are presented in Table
7

---

Published as a conference paper at ICLR 2026
Under the constraint of fine-tuning only 1% of model parameters, IGU-LoRA achieves performance that is comparable to or surpasses existing approaches across all tasks. Notably, on the CoLA task, IGU-LoRA achieves a Matthews correlation coefficient (MCC) of 71.93%, outperforming the best baseline by 1.5%. On the RTE task, it exceeds the second-best method, AutoLoRA, by 0.8% in accuracy (acc). Similar improvements are also observed on the remaining tasks, demonstrating the robustness of IGU-LoRA. Averaged across all tasks, IGU-LoRA achieves the highest overall performance. Importantly, it maintains strong parameter efficiency, requiring only 0.33 million trainable parameters, comparable to leading PEFT methods, while significantly outperforming full- parameter fine-tuning in both accuracy and efficiency.
Table 2: Performance comparison of fine-tuning methods on the Mathematical and rommon-sens, reasoning task using the 2w5n-22550... All results reported as the median over 5 runs with different random seeds. Bold and Underline indicate the best and the second-hest results.
BoolQ (acc)
ARC-e
ARC-e (acc)
GSM8K
AQuA (acc)
Method
Params
Avg.
(acc)
(acc)
Full
494.0M
81.74
74.82
54.98
34.64
48.72
58.98
Houusbly-ddapter
78.36
71.04
53,26
28.67
54.84
LoRA
8.8M
78.94
72.78
54.38
31.42
45.33
56.57
AdaLoRA
8.9M
80,32
73.90
54,23
33.27
46.58
57.67
GoRA
8.8M
79.24
71.20
51.91
32,07
45.81
56.04
IGU-LORA
8.8M
82.45
74.62
55.67
34.16
48.93
59.17
Mathematical and Common-Sense Reasoning Benchmark Results. We further systematically conduct mathematical and common-sense reasoning tasks using the Qwen-2.5-0.5B model, comparing four representative fine- -un-tt methods: Full Fine-tuning, Adapter, LoRA, AdaLoRA and GoRA. Table 2 summarizes the results, where IGU-LoRA consistently achieves performance advantages across most tasks. Specifically, IGU-LoRA achieves of results on BoolQ, ARC-c, and AQuA, outperforming the second-best method by 0.2% to 0.8% in accuracy. While it does not obtain the highest score on ARC-e and GSM8K, IGU-LoRA fine-tunes only 8.8M parameters, substantially fewer than full-parameter tuning (494.0M), yet delivering comparable performance. Across all evaluated datasets, IGU-LoRA consistently outperforms other parameter-efficient methods with similar parameter budgets, highlighting its strong generalization under tight resource constraints.
4.5 ABLATION STUDY AND ANALYSTS
Analysis of Training and Inference Efficiency. So far, we have shown that IGU-LoRA outperforms LoRA, AdaLoRA, and DoRA on BoolQ. A natural concern is whether these gains come at the expense of extra time or memory cost. We fine-tune the Qwen-2.5-0.55 model and report peak training GPU memory and wall-clock training time, as well as inference peak GPU memory and decoding latency, as shown in Table 3 All methods utilise similar memory due to the frozen backbone. LoRA trains the fastest but yields smaller gains; DoRA is slower because it maintains normalized weight directions while updating an additional magnitude vector p. which involves adding normalizatioon/recaalngg operations and optimizer states each step. AdaLoRA improves accuracy via sensitivity- based rank pruning in two-stage schedule; IGU-LoRA adopts similar two-stage design and thus achieves comparable training time while delivering higher accuracy. For inference, IGU-LoRA matches LORA, DoRA, and AdaLoRA in memory usage and decoding latency.
Table 3: The time cost, memory and speed for fine- tuning Qwen-2.5-0.5BB on the BoolQ task with different PEFT methods.
Table 4: Comparison of the performance of dif- ferent variants of IGU-LoRA on fine-tuning Qwen- 2.5-0.5B across BoolQ and GSM8K tasks.
Method
Training
Inference
Method
BoolQ
GSMSK
Avg.
Time cost (h)
GPU Mem (CB)
Speed (itss
GPU Mem (GB)
IGU-LoRA- (weo o)
81.87
33.76
$7.82
LORA
0.42
10:21
5.50
10.3
IGU-LORA-2 (N=10)
82.14
33,95
58.05
Adal
0.73
10,60
5.21
10.4
IGU-LORA-3 (N=4)
82.02
33.85
$7.95
DoRA
0.95
9.53
5,30
10.3
IGU-LORA (s. 8, 0)
82.28
33.69
$7.99
IGU-LoRA
10.32
5.23
10.3
IGU-LoRA
82.45
34.16
58.31
Ablation Study on Hyperparameterss and Importance Scoring. To assess the sensitivity of IGU-LORA to its key hyperparameters and scoring components, we perform the ablation study by incrementally disabling or simplifying individual modules. Specifically, we evaluate the following variants: (1) IGU-LoRA-I removes the gradient-integratee a coefficient used during both training and pruning; (2) IGU-LoRA-2 reduces candidate resolution of a from N = 20 to N = 10; (3)
IGU-LoRA A-3 further reduces the candidate set to N 4; and (4) IGU-LORA- A--44 replaces the final
8

---

Published as a conference paper at ICLR 2026
importance score in Eq. 122 with the alternative formulation in Eq. (11) from Zhang al. (2023). which combines sensitivity and uncertainty via AdaLoRA's multiplicative strategy As shown in Table 4 all variants exhibit performance degradation, particularly IGU-LoRA-3 and IGU-LoRA-4 which involve more aggressive simplifications. These results confirm that the default configuration of IGU-LORA, with high-resolution integrated gradient and uncertainty-aware scoring, is critical in achieving strong performance.
Hyperparameter Sensitivity Analysis. To investigate the sensitivity of IGU-LoRA to key hyperpa- rameters, we varied one hyperparameter at time while keeping others fixed. We analyzed the effects of mini-batch size M, the number of discrete points for a (denoted as N), and smoothing coefficients 31 and 32. Experiments were conducted by tine-tuning the Qwen2.5-0.55 model on the Boolq and GSM8K datasets. The results, shown in Figure and demonstrate that IGU-LoRA performs stably across a range of values. Performance improves with larger M and N, suggesting better adaptability with finer granularity in scaling factor selection. The coefficients, 31 and f3 show good robustness, with optimal performance a moderate range. These findings indicate that M, N. B1, and 32 are robust hyperparameters for IGU-LoRA.
(a) M parameter
(b) N parameter
(c) 31 coefficient
(d) f3 coefficient
Figure 3: The impact of different hyperparameters M, N, 91, B2 on performance when fine-tuning the Qwen2.5-
0.5B model on the Boolq dataset. Please zoom in 300% for better clarity.
15c
10
(a) M parameter
(b) N parameter
(c) [3 coefficient
(d) 92 coefficient
Figure The impact of different hyperparameters M, 32, Ba performance when fine-tuning the Qwen2.5 0.5B model on the GSM8K datasct. Please zoom in 300% for better clarity.
Cute
13
*p
Down
Dom
16
1b21
16.2
322
(3) BoolQ task
(b) GSM8K task
Figure Rank allocation by IGU-LoRA on the Qwen-2.5-0.5B backbone after fine-tuning for BoolQ and GSMBK tasks, Please zoom in 300% for better clarity.
Visualization of Rank Allocation in IGU-LORA. Figure 5 visualizes the pruned LoRA rank allocation produced in IGU-LoRA. The rank distributions vary significantly across tasks, underscoring the need for task- specific adaptation to achieve optimal performance. Even within a single task, different Transformer layers allocate ranks differently, reflecting the fine-grained sensitivity model
"AdaloRR LoRA Zhang
2023 for details.

---

Published as a conference paper at ICLR 2026
components low-rank updates. Despite this heterogeneity, consistent structural patterns emerge: in the self-attention mechanism, the Query and Key projections are most frequently prioritized for adaptation, while in the feed-forward network (FFN), the Up and Down projection layers receive the highest ranks. These observations reveal structural preferences in LoRA-based fine-tuning, offering valuable insights for designing generalized and efficient low-rank adaptation strategies.
Comparisons Rank Budgets. In the main experiments, fixed the initial rank budget at r 32 as standard configuration. To further evaluate the robustness and adaptability of IGU-LoRA, we vary the initial rank budget across {2, 4, 8, 16, 32, 64) and compare its performance with AdaLoRA, LoRA, and DoRA on the BoolQ and GSM8K tasks. The results, shown in Figure demonstrate that IGU-LoRA consistently outperforms AdaLoRA, LoRA and DoRA under all budget settings. This is attributed to its ability to allocate LoRA dynamically across Transformer layers, which enables more effective adaptation.
Accureyy
122
70-
OU-LARA
AlslintA
Lotk
30
DSRA
++ DeRA
10
30
Inttial (PP)
Initial Rank
(a) BoolQ task
(b) GSM8K task
Figure Performances across different initial rank budgets. The x-axis denotes initial rank r while the y-axis indicates the corresponding task performance. Please zoom in 300% for better clarity.
Comparisons on Different Backbone Models. To demonstrate the broad applicability of our method, we now conduct experiments on Llama-2-7B, Llama-3-8B and The results are reported in Table [S] We can see that on these three backbones, IGU-LoRA can also outperform the baseline methods.
Table 5: PEFT methods comparison on different backbones. Left: GLUE accuracy (%) with Llama-2-78 Right: BoolQ and GSM8K accuracy (%) with Llama-3-8B and DeepSeek-R -1eeck--R--iiiee Disttill-Qeen-2.55--- Results are reported as the median over 5 random seeds. Bold and underline indicate the hest and the second-best results.
Llama-2-7B
Llama-3-8B
Method
# Params
SST-2
RTB
QNLI BoolQ
COPA
Avg.
Method
BoolQ
GSMSK
Avg.
Pull FT
6738M
95.83
92.11
92.54
87.30
93,01
92.16
LoRA
AdaLoRA
91.65
88.48
75.82
73.54
83.74
81.01
Adapter
21.2M
94.15
82.12
93.10
87.03
91.10
89.50
DoRA
74.75
81.41
P-tuning
20.9M
93.42
79.62
92.64
84.73
90.30
88.14
IGU-LoRA
93.33
77.63
85,48
SSP
40.0M
94.14
83.11
93.10
87.11
91.65
89.82
DeepSeek- SeepSeek-RR-Distiine R1-Diitill-Qwen-22..
LoRA
20.0M
94.12
83.37
93.10
87.34
91.33
89.85
Method
BoolQ
GSM8K
Avg.
AdaLoRA
20.0M
94.12
83.51
93.20
87.11
91.62
89.91
LoRA
88.38
74.60
81 $1.49
DoRA
40.0M
94.24
82.12
91.23
85.51
90.01
89.02
AdaLoRA
90.54
73. 23.30
81. 92
DoRA
88.4%
69.52
79.00)
IGU-LORA
40.0M
84,33
88.11
92.10
IGU-LORA
92.82
74.28
83,55
CONCLUSION
In this work, we address the challenge of parameter importance estimation for efficient fine-tuning of LLMs. We propose IGU-LoRA, a robust scoring framework that integrates the concept of integrated gradients with an uncertainty-awar quantification mechanism. Unlike prior methods that rely solely on instantaneous gradient signals, IGU-LORA captures each parameter's global and long-term contribution to model performance. Experimental results across diverse tasks and model architectures demonstrate that IGU-LoRA consistently outperforms state-of-the-ar baselines, validating its effectiveness and generality. Nevertheless, the method incurs non-trivial computational overhead network models in networks with large parameter counts, and its performance can be influenced by the choice of integration paths and the precision of uncertainty estimation. In future work, we plan to extend IGU-LoRA to larger-scale models and cross-modee tasks to further explore its adaptability and generalization across architectures.
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
This paper proposes efficient fine-tuning framework, IGU-LORA, that adaptively allocates LoRA ranks to alleviate the inaccuracy of setiiiettseessttiit--bssse parameter importance estimation under gradient saturation, thereby enhancing the adaptability of large language models (LLMs) across diverse task domains. This study strictly adheres to ethical guidelines: no human subjects or sensitive data were involved. All experimental data are publicly available fine-tuning datasets, and no scenarios containing harmful content were used. While IGU-LoRA effectively improves the overall performance of LLMs, the models may still produce erroneous outputs or misjudgments: thus, we do not recommend deploying them in high-risk scenarios without thorough validation. We further declare that this work has no conflicts of interest, and all experiments and data processing comply with relevant ethical standards.
7 REPRODUCIBILITY STATEMENT
For clarity and reproducibility, summarize the critical details of our method in the main text and Appendix as follows.
Algorithmic Details: We provide a detailed description of the IGU-LoRA algorithm in
Section 3 including the integrated gradients computation (Section 3.2) and uncertainty-
aware scoring mechanism (Section 3.3 Pseudocode is provided in Algorithm
Theoretical Analysis: We present a theoretical analysis of the approximation error for
parameter-space integrated gradients Section 3.2 Appendix and Appendix A.2 includ-
ing all necessary assumptions and proofs.
Experimental Setup: We detail the experimental setup in Section and Appendix C
Code Availability: We adopt the code proposed by Zheng et al. (2024) for model training, which is publicly available at https:: //gi thul.. com/ hi miiyouga// LLaMA-F actory In addition, this work is accepted, we commit to releasing the source code of our method.
ACKNOWLEDGEMENT
This work was supported by the Natural Science Foundation of ChongqingJoint Fund for Innovation and Development (Grant No. CSTB2025NSCQ-L2200 34); the Science and Research Development Center of the Ministryof Education (Grant No. 2023ZY024); the Graduate Innovation Project ofChongging Technology and Business University (Grant No. yjscxxx2025-29911111 the Chongqing Technology and Business University ResearchFunds (413/1956019, 410000199911111 413/1952037 and 1752004).
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine- fine-tuning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 7319-7328, 2021.
Nadav Benedek and Lior Wolf. Prilora: Pruned and rank-increasing low-rank adaptation. In Findings, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020a.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel

---

Published as a conference paper at ICLR 2026
Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference Neural Information Processing Systems, 2020b.
Christopher Clark, Kenton Lee, Ming-Way Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 2018.
Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, 2021.
Chaoran Cui, Ziyi Liu, Shuai Gong, Lei Zhu, Chunyun Zhang, and Hui Liu. When adversarial training meets prompt tuning: Adversarial dual prompt tuning for unsupervised domain adaptation. IEEE Transactions on Image Processing, 34:1427-144 1440, 2025.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4133-41455 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle. The Ilama herd of models. ArXiv, abs/2407.21788? 2024.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16091-161100 2023.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint ar arXiv:2012.07460 2020.
Simeng Han, Frank Palma Gomez, Tu Vu, Zefei Li, Daniel Cer, Hansi Zeng, Chris Tar, Arman Cohan, and Gustavo Hernandez Abrego. Ateb: Evaluating and improving advanced nlp tasks for text embedding models. arXiv preprint arXiv:2502.16666 2025.
Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, and Lei Chen. Gora: Gradient-driven adaptive low rank adaptation. arXiv preprint arXiv:2502.1217) 2025.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kerg-Kirkpattick, and Graham Neubig. Towards unified view of parfmeter-efficient transfer learning. In International Conference on Learning Representations, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022a.
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang. Yasheng Wang, Zhiyuan Liu, and Maosong Sun. Sparse structure search for parameter-efficient tuning. ArXiv, abs/2206.07382, 2022b.
12

---

Published as a conference paper at ICLR 2026
Weizhong Huang, Yuxin Zhang. Xiawu Zheng, Yang Liu, Jing Lin, Yiwu Yao, and Rongrong Ji. Dynamic low-rank sparse adaptation for large language models. ArXiv, abs/2502.14816, 2025.
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: adaptive path method for removing noise. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5048-5056, 2021.
Wenjun Ke, Jiahao Wang, Peng Wang, Jiajun Liu, Dong Nie, Guozheng Li, and Yining Li. Unveiling lora intrinsic ranks via salience analysis. In Advances in Neural Information Processing Systems, volume 37, pp. 131575-131595, 131595, 2024.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. ArXiv, abs//804. 008.08838, 2018.
Ming Li. Yanhong Li, and Tianyi Zhou. What happened in Ilms layers when trained for fast vs. slow thinking: A gradient perspective. ArXiv, 2024.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 4582 4597, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin Raffel. Few-shot -effiiiee-effiiintt fine-tuning is better and cheaper than in-context learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022a.
Jiajun Liu, Wenjun Ke, Peng Wang, Jiahao Wang, Jinhua Gao, Ziyu Shang, Guozheng Li, Zijie Xu, Ke Ji, and Yining Li. Fast and continual knowledge graph embedding via incremental lora. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 2198 2206, 2024a.
Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 203.::2402.09333. 2024b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine- fine-tuning universally across scales and tasks. CoRR, abs/2110.07602.. 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61 68, 2022b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models. ArXiv, abs/2403.16187, 2024c.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4768-477 4777, 2017.
Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8187-8198. 2024.
13

---

Published as a conference paper at ICLR 2026
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, 2021.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63: 1872-1897, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 11:1-67,1 2020.
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. MELoRA: Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3052-3064, 2024.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical
formalizations of commonsense reasoning, pp. 90-95, 2011.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters transformers. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7930-7946, 2021.
Rory Sayres, Ankur Taly, Ehsan Rahimy, Katy Blumer, David Coz, Naama Hammel, Jonathan Krause, Arunachalam Narayanaswamy, Zahra Rastegar, Derek Wu, Shawn Xu, Scott Barb, Anthony Joseph, Michael Shumski, Jesse Smith, Arjun B. Sood, Greg S. Corrado, Lily Peng, and Dale R. Webster. Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy. Ophthalmology. 126:552-564 2019.
Zhihong Shao, Damai Dai, Daya Guo, Bo Liu (Benjamin Liu), Zihan Wang, and Huajian Xin. Deepseek-v2 -2: A strong, economical, and efficient mixture-of ex-eerssss language model. ArXiv, 2024.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: 008.00 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 331-3328, 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3274-3287, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-addapaation for parameter-efficient model tuning, arXiv preprint arXiv:2205. 12410, 2022.
Yue Wu, Yaoxiang Yu, Zhengming Yuan, Siwei Huang, and Bo Cai. Apt: Adaptive prefix-tuning on pretrained models for code intelligence. In 2024 International Joint Conference on Neural Networks (IJCNN), pp. 1-10, 2024.
14

---

Published as a conference paper at ICLR 2026
Xilie Xu, Jingfeng Zhang, and Mohan S. Kankanhalli. Autolora: A parameter-fre free automated robust linettuning framework. ArXiv, 2023.
Lang Yu, Qin Chen, Jiaju Lin, and Liang He. Black-box prompt tuning for vision-language model as service. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023.
Haozhen Zhang, Hualin Zhang. Bin Gu, and Yi Chang. Subspace selection based prompt tuning with nonconvex nonsmooth black-box optimization. Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4179-4190, 2024.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference Machine Learning, pp. 26809-280000 22822,, 2022.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficienl fine-tuning, In The Eleventh International Conference on Learning Representations, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13:: 13372, 2024.
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. ArXiv, abs/2309.0241 1,, 2023.
15

---

Published as a conference paper at ICLR 2026
A THEORETICAL PROOFS
A.1 PROOF OF THEOREM !
Proof. Fix wij and set f(()) fıs(a) giy(a) a[(aAW)/Ougg By Eq. se(wi))
|usl So f(a) da}. Define the composite trapezoidal approximation and its sampled variant:
N-1 E f( f(1)
Tu 1 2M f())] (13)
TN = 2N
f(0) +2
2N f(()
km
where JM 1 Ep f(ap) with ap ii.d. drawn from the discrete uniform distribution on (I/ (NV-)/N
Since Sagg (wir) |w |Tas and ||>|| |yll S j yl. the triangle incquality yields
se (wxy) - Saga (wx))
S |wy/| | TM - | So f TN + TN - -rrl). (14)
Step 1: discretization error. By assumption, f is twice continuously differentiable on [0, 1] and suPaejo.1] (a)) S C2. The standard error bound for the composite trapezoidal rule on (see. e.g., classical numerical analysis texts) yields
Jo f(a) da TN
(15)
12
Step 2: sampling error. Let μ N-T E ( denote the average of over the (N-1) interior
nodes. A simple algebraic manipulation gives
N-1
N N
IT -ml =
Σ - -
la - Full Σ - 1M- (16)
N
By assumption, faa is uniformly bounded the discretization nodes, which is discussed in detail in Appendix [B.]] there exists B < 0o such that [f(a) < B for all a E {I/N...., (N-1)/N }.
Therefore, each sample (app lies in [-B, B). and Hoeffding's inequality for bounded random
variables implies that, for any 8 E (0, 1).
2M
Mt
Pr(Mu Ml 2 1) S 2 exp
exp
(17)
(2B)2
2B
Setting the right-hand side equal to 5 and solving for yields that, with probability at least 1 - 6,
220((2
log(1/8)
| S < BV
cB
(18)
M
M
for an absolute constant с > 0. Combining with the previous display gives
log(1 15)
TT Tal I 1l - Farl s cB
(19)
M
with probability at least 8- 8.
Step 3: combining the bounds. Plugging Eq. 15 and Eq. yields that, with probability at least 8,
19 into the decomposition in Eq. 14
C2
log(1/8)
1e (e(ww)) Sagg (we)) <
|wgl
c
(20)
N2
M
which is exactly the claimed bound in Eq. 9
A.2 HIGH-PROBARILLTY STABILITY OF SNR,
The resulting SNR-based score favors parameters with consistent, high-impace contributions and suppresses those with volatile transient behavior. While the above formulation provides intuitive interpretation of SNR, it remains essential to ensure its statistical stability with high probability, which is formally addressed in Theorem 2
16

---

Published as a conference paper at ICLR 2026
Theorem 2. Let Y/t Sagg(Wa) be the per-epoch importance defined in Eq. 77. Since in Eq. 12) is very small constant, it can be ignored. Therefore, we have:
(1)
86
8E
SNR,
(21)
U
U
Assume that (ye) is an i.i.d. sequence of sub-Gaussian random variables with mean μ and variance o², and let d = EE [lve
p1]
> For 02 E (0,1), define the effective EMA window lengths
1+ 31
1+ B2 Nett (32) = 1 B2
Teft (51)
Reff = min (31), Dott (ot
(22)
1 B1
Then there exist universal constants C1, C22 Cn > 0 such that, for any ò E (0, 1) and all
C1
log C2
2 Iburn
(23)
1 min (,1, 12)
the following holds with probability least 8:
log(2/6) Netf
2v20 d
|NN; -m/dl VV
C
2c0 (o d).
(24)
d²
Proof. We analyze the EMA under the stylized assumption stated in Theorem 2 (21) is iid. Sub-Gaussian sequence with mean j4, variance proxy o², and = Ely - pl > O.
Recall that Eq. (10) and Eq. (TT) define the EMAS
s(t) B18 (1 B1)
T B2U,-1 + - 32) 1t 5
(25)
Unrolling the recursions (for &. large enough so that transients are negligible) shows that
Σ
uk (1)
Ut-ks
(1)
Bi)
T (1 (2)
8 yt- 51-x/-
(26)
420
420
Note that (w) )220 is geometric weight sequence with En W
(1)
(1)
and
1 B1
Σa 11-381)833 k20
81
(27)
+ B1
Noft (31)
Below write Nee min (.1), nett (B2)}.
Step 1: concentration of a(t) Since (ye) are i.i.d. sub-Gaussian with mean μ and variance proxy 0², any fixed weighted sum En w (1) y is also sub-Gaussiae with mean μ and variance proxy ) [2 o² /nott (31). Standard sub- GabbGanssian tail bounds then yield
o (2)
Cooff (B,) (
Pr (10 Pr(155) - ml 26) S 2exp
(28)
o²
for absolute constant
0. Setting the right-hand side to 5/2 and solving for gives
2o 4/5) nell (B1)
log 1og(4/6) (4/))
(3(0) 1l S vv
S v2ov
(29)
neft
with probability at least -8/2.
Step 2: concentration of (1)), We decompose (1) around d = =Eyy [4] as
-d d s (1-32) B2)
Σ
3 (1Vi -Ml- (1- (2)
W 5
Ui-k 11|/- (30)
k20
k00
17

---

Published as a conference paper at ICLR 2026
Define X, 1yve /4| 1ol-dd d, which is centered, aub-exponential random variable whose tail parameters depend only on (a, d) (because yt is sub-Gaussian). Let w = (1 - B2) denote the EMA weights for (T(0) Then Σ k20 "
(2) and
B2
(2)
11- 32
1+ A2
Nell (32)
M:0
Applying a Bernstein-typ concentration for weighted sums of i.id. sub-exponential variables (see, e.g., standard results Orlicz norms) yields the existence of absolute constant co >> 0 such that, for any 5 E (0, 1),
Pr (1- B2) Σ
36 x 2 colo 1)
log(4/8) Nott (52)
$ 2
8
(31)
A20
For the second term in Eq. 330 note that ||a - - la bi| S b cl for any a, b, E IR, so
|1 34-kl
-1b1o pl < _ s4/-
Thus
(1 - B₂)
11 - 3;-kl 1 - Hll v (1 32)
32 3 m/.
(32)
R20
k20
We now bound the right-hand side by splitting the sum into recent window and its tail. Let
C2 0
7. =
log
(33)
B2
for absolute constants C1, C2 > chosen large enough. For t. 2 L, we have
(1 - 32)
B - 1l v (1- 32)
B B - ml + (1 B2)
3 B - 14). (34) k>
Σ
A20
k=0
For the tail sum, (1 (1- 2 g5+11 < 8/(8c2). For the finite window - L}, we apply Eq. (29) and union bound
3
and, by choosing 61, 62 appropriately, we can ensure
over these (L + 1) indices obtain, with probability at least 6/2.
log(4L/6)
- - S V
for all 0 k S
(35)
neal (B1)
Combining these bounds and using Nel Nott (31) yields
1og(2/8) nell
(1 - B2) Σ 3 1 - Ml s 20p
(36)
1200
with probability at least - 5/22, for absolute constant ē 0.
Putting Eq. (31) and Eq. (36) back into Eq. 330 and recalling that neot S nettt (32), we obtain that, for 2 tour and with probability at least o.
1000 - d s Ca + di log(2/6)
(37)
Tofff
for absolute constant C%2- 0. By increasing Ci if necessary, may ensure that the right-hand side in Eq. (37) is at most d/2, that (T)) > d/2 holds on the same high-probability event.
18

---

Published as a conference paper at ICLR 2026
Step 3: bounding the ratio SN On the event {U() d/2)
control the ratio SNR;
2 (0() via the deterministic inequality
(t)
24 +
<
5(0)
1
1000 d). d
(38)
(7()
Combining Eq.
29 and Eq. (37) with Eq. (38), and noting that Neft I Datt (B1)
gives
2V20 H + 2e0
j lo + ta)) log(2/6)
SNR: - 1//d -
(39)
Nlfff
with probability least 1 - 6, for suitable absolute constant Co 0. This is exactly the claimed
bound in Theorem after setting C 2v22 + 2cn E (a + and burn (81 log % I-
B
THE DISCUSSION OF THE ASSUMPTIONS IN THEOREM
B.1 THE ANALYSIS OF THE ASSUMPTION IN THEOREM
In this section, we focus on how the assumption in Theorem that Pi is twice continuously differentiable on the interval [0, 1] with bounded second derivative, leads to the conclusion that giy (a) is bounded. First, consider the following form of (a)(a):
9iy gsy(a) al(oAW) Owi
a [0.1).
(40)
The analysis of Theorem [ relies solely the assumption that By is twice differentiable the interval [0, 1] and that its second derivative is bounded, which allows the application of the composite trapezoidal rule, leading to discretization error of O(N Specifically, numerical analysis typically assumes the existence of a constant C2 < Do such that:
sup aE/0.1]
K (a) S C2.
(41)
Under this assumption, we can derive the following error bound:
C2 J." (is(a) do Tn < 12N2
(42)
This equation provides the theoretical basis for the O(N-2) discretization error term in Theorem This requirement is essentially standard smoothness assumption in trapezoidal integration and does not involve any specific distributional assumptions. Furthermore, the condition of bounded second derivatives directly implies that Øsj itself bounded. By the fundamental theorem of calculus:
(')(a) (())=);((00) [ (1)(1) di, 91s(a) 01)(0) I g6 di,
(43)
We can obtain the bound for all e [0,
[,(ي(a)) s loly + [ (0) (0y( + Cz.
(44)
Thus,
(a)) S 19. + . (t)
(t)) dt < (0) + (g, C2 é B.
(45)
This implies that 9.j (a) is bounded on [0. 1]. When we sample a from the finite set {1/N.... 1)/N), the resulting random variable (a:(a:) bounded by constant B.
.......
19

---

Published as a conference paper at ICLR 2026
B.2 THE ANALYSIS OF THE L.I.D ASSUMPTION IN THEOREM 2
Theorem assumees that the per-epoch raw scores Yt Sagg (Way) form an i.id. sub-Gaussian sequence with a common mean and variance o². However, strictly speaking, y/ depends on the current model parameters W which are updated across epochs, so exact i.d. is an idealization.
Our goal is to model the regime in which the training dynamics have stabilized: after an initial transient phase (discarded via the burn-in time bbrra). the statisties of the gradient noise around the current solution change only slowly. Furthermore, within the effective EMA window netr (3), 32), the gradient sequence can be approximated as having nearly stationary mean and variance. In this regime, standard extensions of EMA concentration results to weakly dependent or mixing sequences apply. We chose the i.idd setting for clarity of presentation and keep the notation simple. It is important to note that Theorem 2 is derived under this stylized, locally stationary noise assumption, and is meant to provide intuition about how the EMA window size and variance control the stability of SNR,, rather than to capture every aspect of LLM training dynamics exactly.
To support this approximation empirically, we provide small diagnostic in Appendix G for representative layer on BoolQ. we plot the time series of y/ and its running mean/variance across epochs. We observe that, after the early epochs, both the mean and variance of y/: quickly settle into narrow band, and the lag-1 autocorrelation becomes small. Correspondingly, the SNR, curves are nearly flat after burn-in. These observations suggest that, in the regime where EMA-based importance is actually used for rank pruning, the i.i.d/local stationarity approximation is reasonably accurate.
Finally, we emphasize that these assumptions are used only in our theoretical analysis; the algorithm itself does not rely on them. Even when the exact assumptions relaxed, the qualitative conclusions remain the same: (i) our IG estimator trades off discretization error ( and sampling error
((M- and (ii) EMA-based SNR, scores become more stable as the effective sample size increases and the process enters locally stationary regime.
C HYPERPARAMETER SETTINGS
During the training process, we tune the learning rate from {5 x 10-4, x 10-4, 5 x 10-1,1 1,1 10-3 2 x 10-4) and pick the best learning rate for every method. For the MNLI, QNLI, and QQP, set the batch size to 128. For RTE, MRPC, CoLA, and STS-B, -B, the batch size is set to 32. For SST-2, we use batch size of 64. For all other tasks, the batch size is set 16. All baseline methods follow the same settings as IGU-LORA, as detailed in Table 6. In IGU-LORA, several key hyperparameters c, M,, N, 31 82 are set to 1 x 10-6, 6 16, 20, 0.85, and 0.85, respectively, as detailed in Table They remain constant throughout the experiment, and their sensitivity is discussed in the main text.
Table 6: Hyperparameter setup IGU-LoRA for training on different datasets.
Dataset
learning rate
batch size
Max. Sequence Length
# #epochs
Ay
MNLI
5x 10-1
128
512
25
0.1
500
20
10000
RTE
1x 10-3
32
512
0.1
300
2500
QNII
5x 10-4
128
512
25
0.1
400
20
10000
MRPC
1 10
32
512
25
0.1
300
2500
QQP
5x 10
128
512
0.1
500
10000
SST-2
10-3
61
400
5000
CoLA
1 10-3
32
512
25
0.1
300
5
2500
STS-B
2 10
32
512
25
0.1
300
2500
BoolQ
5 10
16
512
0.1
500
20
10000
ARC-e
ARC-e
5x 10
16
512
25
0.1
500
20
10000
COPA
1 10-3
16
512
25
0.1
500
20
10000
AQuA
1 10
16
512
25
0.1
500
20
10000
MMLU
1x 10
128
512
0.1
500
10000
VQA
2
32
512
0.1
300
10000
GAQ
5 10-4
32
512
25
0.1
300
20
10000
MVLR3
5 10
32
512
25
0.1
300
20
10000
COCO
2 10
32
512
0.1
300
20
10000
20

---

Published as a conference paper at ICLR 2026
Table 7: Setting of the 5 hyperparameters (c, N, 9r, .2) in IGU-LORA.
Hyperparameter
M
NN
B1
32
Value
1 10
16
20
0.85
0.85
D ABLATION STUDY ON HIGH-IMPACT PARAMETERS
To further validate the effectiveness of IGU-LoRA in identifying high-impact parameters, we conduct an ablation study high-impact parameters. Specifically, remove the high-rank and low-rank modules with the highest IGU-LoRA scores from different layers of the Qwen2.5-0.5D model and evaluate the performance drop on the Boolq and GSM8K datasets. As shown in Table removing the high-rank modules from the K module in Layer 3 (L3_K) and the V module in Layer 10 (L10_V) results performance drop of 1.30 and 1.33 points on Boolq, respectively. Similarly, removing the high-rank modules from the Q module in Layer 22 (L22 Q) and the K module in Layer 17 (L17 K) results in performance drops of 1.80 and 1.73 points on GSM8K, respectively. the low-rank modules from the K module in Layer (L.I_) and the V module in Layer (I.3_V results in only minor performance drops of 0.05 and 0.10 points on Boolq, respectively. The same trend is observed on GSM8K when removing the low-rank modules from the Q module in Layer 8 (L8-QL Q) and the module in Layer 6 (L6.M) L), resulting in performance drops of 0. and 0.15 points, respectively. These results demonstrate that IGU-LoRA effectively identifies high-impact parameters, as their removal leads to significant performance degradation compared to low-impact parameters.
contrast, removing
Table 8: Ablation study the impact of removing high-rank and low-rank modules from different layers Qwen2.5-0.55 0.B..B model performance. The numbers in parentheses indicate the performance drop compared to the model with no modules removed. The left table and the right table represent results on Boolq and GSM8K, respectively.
Module Removed
Rank
Booly
Module Removed
Rank
GSM8K
L33K
10
81.15
(-1.30)
L22_Q
12
32.35
(-1.0)
LIO.V
10
81.12 (-1.33)
L17_K
11
32.42 (-1.73)
3
L3
K/LIO V
10/ 10
80.44 (-2.01)
3
L22 Q/L17 K
12/ 11
31.15 (-3.00)
4
LLK
5
82.40 (-0.05)
4
L8.Q
6
34.05 (-0.11)
L3.V
5
82.35 (-0.10)
L6_K
6
34.01 (-0.15)
LL.KL3.V
5/5
82.30 (-0.15)
188 Q/16.K
6/6
33.84 (-0.32)
82.45
34.16
E GENERALIZATION SUPPLEMENTARY EXPERIMENTS
To further validate the generalization performance of IGU-LoRA, we conduct additional experiments on the MMLU benchmark using the Llama2-7B model. As shown in Table 9 IGU-LoRA achieves an average accuracy of 51.07%, which is very close to the full fine-tuning method (51.54%) and outperforms LoRA (49.94%). Notably, IGU-LoRA demonstrates superior performance in Science, Technology, Engineering, and Mathematics (STEM) and Social Science subjects, achieving accuracies of 41.71% and 12%. respectively. These results further confirm the effectiveness IGU-LoRA in enhancing the generalization capabilities of fine-tuned models across diverse subject areas.
Table 9: The generalization performance of fine-tuning the Llama2-7B model the MMLU benehmark using different methods, reporting the average results over random seeds.
Method
Humanities
STEM
Social.
Other
Avg.
Full FT LoRA
49.91
41.70
57.53
57.02
51.54
46.15
40.84
56.63
56.23
49.94
IGU-LoRA
47.33
41.71
58.12
57.10
51.07
F MULTIMODAL BENCHMARK SUPPLEMENTARY EXPERIMENTS
To further demonstrate the effectiveness of IGU-LoRA in multimodal tasks, conduct additional experiments on the VQAv2, GAQ, NVLR² and COCO Captioning datasets using the VL-BART (Su et al. 2019). As shown in Table TO IGU-LoRA achieves average score of 77.47, outperforming
21

---

Published as a conference paper at ICLR 2026
LoRA (74.31) and DoRA (77.40), and closely approaching the performance of full fine-tuning (77.35). These results further validate the capability of IGU-LoRA to effectively adapt multimodal models while maintaining high performance across different tasks.
Table 10: Performance comparison of different fine-tuning methods on the VQA, GAQ, NVLR* and COCO datascts using the VL-BART model. The results averaged over 5 random seeds.
Method
VQAv2
GAQ
NVLR
COCO Captioning
Avg.
Full FT
66.91
56.72
73.71
112.04
77.35
LoRA
64.32
54.10
71.25
109.56
74.31
DoRA
65.81
54.71
73.14
115.93
77.40
IGU-LoRA
65.78
55.32
73.42
115.36
77.47
G THE VERIFICATION OF THE I.I.D.LLOCAL STATIONARITY APPROXIMATION IN THEOREM 2
To validate the i.idd local stationarity approximation used in Theorem conduct empirical analysis of the importance score statistics during the fine-tuning process. Specifically, we monitor several representative modules (e-g.. the L16 Q module for the 16-th layer's component and the L5_K module for the 5-th layer's K component) across multiple training iterations on the BoolQ dataset. We observe that, after the initial epochs, the mean and variance of ye quickly stabilize within a narrow range, and the first-order lag autocorrelation becomes very small. Correspondingly, the SNR, curve becomes nearly flat after the burn-in period. These observations suggest that the i.i.d//ocal stationarity approximation is reasonable and accurate during the stage when EMA-based importance- ranking pruning applied in practice.
Variance
Vanance 139,00
Moan 1LL
12
Mean HN
2.0-
0.8
Nest ILL. V
1.2
Varann
$22.9
Dumn-i hatod
30
50
18
20
30
1.00
(11.9)
5M0 13
0.50
(111.4
546 (1113)
0.60
Natocr 1L16,
-0.52
0.4
1.80
Aatooo 1110
02
Figure Empirical analysis importance score statistics during tne-tuning, The plots show the changes Us. mean and variance of ye, the first-order lag autocorrelation, and SNR, across training iterations for representative module parameters.
H EFFECTS OF SAMPLE ORDER AND BATCH SIZE
To investigate the effects of sample order and batch size the performance of IGU-LoRA, we conduct experiments using the Qwen-2.550.555 model on the BoolQ dataset. The results are summarized as follows:
22

---

Published as a conference paper at ICLR 2026
Sample Order / Random Seed. trained with fixed batch size using five different random seeds. These seeds control the data shuffling and the sampled integration nodes ag.. The downstream accuracy varies slightly across seeds (within Aace absolute points, indicating small change). which demonstrates that the sample order has high stability on the results.
Batch Size. We further vary the batch size (e.g.- 2, 4, 8, 16, 32) while keeping all other hyperparame- ters fixed. The resulting test accuracy again shows only minor variation. This proves that batch size does not have significant impact on the results. The detailed results are presented in Table []
Table 11: Effect of Batch Size BoolQ Accuracy across Different Random Seeds
Batch Size
Seed 1
Seed
Seed
Seed 4
Seed 5
2
82.46
82.47
82.45
82.46
82,45
82.45
82.46
82.44
82.45
82.44
8
82.44
82.45
82.43
82.44
82.43
16
82.45
82.46
82.44
82.45
82.44
32
82.40
82.41
82.39
82.40
82.39
I DATASETS AND METRICS
I.1 GLUE BENCHMARK TASKS
Single-sentence Classification Tasks. (I) CoLA (Corpus of Linguistic Acceptability): Determine
whether sentence adheres grammatical rules (binary classification). (2) SST-2 (Stanford Sentiment
Treebank): Movie review sentiment analysis (positive/negative binary classification).
Sentence-pair Classification Tasks. (1) MRPC (Microsoft Research Paraphrase Corpus): Determine whether two sentences semantically equivalent (binary classification). (2) QQP (Quora Question
Pairs): Determine whether two Quora questions are semantically identical (binary classification).
(3) RTE (Recognizing Textual Entailment)) Determine whether sentence pair entails relationship (three-class classification:
Similarity and Regression Task. STS-B (Semantic Textual Similarity Benchmark): Calculate the semantic similarity between two sentences (continuous value from 5).
Question-answering Task. QNLI Question-answweiing NLI). Determine whether a sentence contains the answer to a given question (binary classification).
Natural Language Inference Task. MNLI (Multi-Genre Natural Language Inference). Large-scale cross-domain textual entailment classification (three-class classification).
1.2 MATHEMATICAL AND COMMON- REASONING TASKS
Mathematical Reasoning Tasks. (1) AQuA (Algebra question answering): Derive the correct answer from given algebraic problem (multiple-choice and generate the corresponding solution process (Rationales). (2) GSM8K (Grade school math 8K): Perform multi-step reasoning on mathematical problems described natural language.
Reasoning Tasks. (1) BoolQ (Boolean questions). Determine whether the answer to given question, based on the provided paragraph, is "Yes" (True) or "No" (False). (2) ARC-e (AI2 reasoning challenge- easy): Select the most reasonable answer from a given set of scientific questions (Multiple-choice question). (3) ARC-c (AI2 reasoning challenge challenge): Combine multi-step reasoning and cross-domain knowledge to provide answers. (4) COPA (Choice of plausible alternatives). Select the most plausible cause effect for given premise from two provided alternatives. The task requires understanding of causal relationships and commonsense reasoning in everyday scenarios.
Common-Sense
1.3 MULTIMODAL. BENCHMARK TASKS
Visual Question Answering Tasks, (1) VQAv2 ( (Visual Question Answering v2). Given an image and related question, select the most appropriate answer from multiple choices. (2) GAQ (Generalized
23

---

Published as a conference paper at ICLR 2026
Question Answering). This task extends VQA to generalized setting, where the model is asked to answer a wider range questions based on visual context.
Visual- Linguu--ingiistic Reasoning Task. (1) NLVR2 (Natural Language for Visual Reasoning 2). Given a pair of images and natural language statement, determine whether the statement accurately describes the relationship between the two images.
Image Captioning Task. (1) COCO Captioning. Generate descriptive captions for images in the COCO dataset, evaluating the model's ability understand and describe visual content accurately.
Table 12: Summary of the benchmark datasets.
Datasets
# train
# dev
# test
Type
Metrics
Common-Sense reasoning tasks
BoolQ
9427
3270
Common- reasoning
Acc
ARC-e
2251
570
2376
Common-Sense reasoning
Acc
ARC-c
1119
299
1172
Common-Sense reasoning
Acc
COPA
400
100
500
Common-Sema reasoning
Acc
Mathematical reasoning tasks
AQuA GSM8K
97467
254
254
Mathematical reasoning
Acc
7473
1319
Mathematical reasoning
Acc
GLUE benchmark tasks
SST-2
67k
872
1.8k
Sentiment
Acc
MNLI
393k
20k
20k
NLU
Acc
QQP
364k
40k
391k
Paraphrase
Acc-FI
MRPC
3.7k
408
107k
Paraphrase
Acc-FI
RTE
2.5k
176
3k
NLU
Acc
QNLI
108k
5.7k
5.7k
QA/NLI
Acc
CoLA
8.5k
1K
1K
Acceptability
Mee
STS-B
7k
1.5k
1.4k
Similarity
Corr
1.4 DATASET STATISTICS
In our experiments, we compare performance across multiple tasks, including the GLUE benchmark. which consists of eight datasets: CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE; three common-sense reasoning tasks (BoolQ, ARC-e, and ARC-c); and two mathematical reasoning tasks (AQuA and GSM8K). The dataset statistics are presented in Table
1.5 EVALUATION METRICS
As shown in Table 12 we strictly follow the official settings of GLUE and use the same metrics as Wang et al. (2018 8). For MNLI, we report the average of the accuracy scores on the matched and mismatched test sets. For MRPC and QQP, we report Acc-FI, the average accuracy, and F1 scores. For STS-B, we report Corr, which denotes the average of the Pearson and Spearman correlation coefficients. For CoLA, we report Mcc, which the Matthews correlation. For all other tasks, we report accuracy (Acc). Since the common sense and math reasoning tasks usually come with definite answer choice, we will directly consider the correctness the final answers. Thus, we report accuracy (denoted as Acc).
BASELINE DETAILS
Full fine-tuning the most common approach for adaptation. During fine-tuning, the model is initialized with pre-trained weights and biases, and all model parameters undergo gradient updates.
LoRA (Hu et al. 2022a) is representative efffcietr-efficient fine-tuning (PEFT) method. It introduces two low-rank matrices to parameterize the incremental weight updates, and only these lightweight components updated during fine-tuning. The number of trainable parameters is
24

---

Published as a conference paper at ICLR 2026
determined by the rank and the number of inserted adaptation matrices allowing for fine-grained control over the adaptation budget.
AdaLoRA (Zhang et al,, 2023) extends the conventional LoRA framework by introducing a dy- namic rank adaptation mechanism. It parameterizes the low-rank adapters using singular value decomposition (SVD), and evaluates the importance of each parameter based on the magnitude of its corresponding singular value. This importance score then guides progressive rank pruning process, allowing the model to dynamically reallocate its limited parameter budget to more critical layers or modules.
DoRA (Liu et al. 2024b) enhances the learning capacity and adaptability of pretrained models by decoupling weight matrices into two distinct components: magnitude and direction. The key idea is to keep the magnitude fixed and apply LoRA-style low-rank updates only to the directional component. This separation allows for more expressive and geometry-awant adaptation while preserving the norm of the original weights, which helps stabilize training and maintain alignment with the pretrained model. Since only the direction is modified, DoRA introduces no additional inference overhead, making it efficient and scalable for deployment.
AutoLoRA (Xu et al. 2023) is meta-learning- based fine-tuning approach designed to automatically determine the optimal rank for each layer in Low-Rank Adaptation (LoRA). It introduces learnable selection variable for each rank-l matrix and dynamically adjusts these variables using a meta- learning strategy. By jointly optimizing the rank configuration along with the LoRA parameters, AutoLoRA significantly improves fine-tune teeettnnng efficiency and overall performance.
Adapter (Houlsby et al.,, 2019) inserts lightweight bottleneck modules between each layer of the pretrained model, updating only these newly introduced modules during fine-tuning while keeping the original model parameters frozen.
P-tuming v2 (Lu et al. 2021) is an improved prompt tuning method that inserts trainable prompt tokens the input layer and across multiple model layers. This design increases the trainable parameters from approximately 0.01% % to 0.1% of the full model, while maintaining parameter efficiency. P-tuning v2 enhances optimization stability and improves performance across various tasks by integrating task-specific information deeper into the model.
(IA)² (Liu et al. 2022a) introduces learnable scaling vectors at key locations in the Transformer architecture, such as the keys and values in the self-attention mechanism and the intermediate activations in the feed-forward networks. These vectors are applied via dlement-wise multiplication to modulate the internal activations, enabling flexible control over the model's output without modifying the original model parameters.
SSP (Hu et al.) 2022b) leverages structural sparsity to guide the automatic search for parameter insertion locations, activating trainable parameters only in the most important substructures. This enables higher efficiency without sacrificing model performance.
GoRA (He et al.) 2025) leverages -adiven adaptive low-rank adjustment to dynamically adjust the rank of low-rank adaptation layers during training. By using gradient information, GoRA ensures that the model can allocate computational resources more efficiently, adjusting the rank based on the importance of each layer for different tasks and training stages. This method maintains computational efficiency while improving model performance, adapting the low-rank configuration to meet the specific needs of the training process.
K ADDITIONAL RELATED WORKS
K.1 DYNAMIC RANK ALLOCATION
Dynamic rank allocation gains increasing attention in deep learning model optimization, with various methods proposed to improve adaptability and efficiency. Several other notable approaches are intro- duced beyond AdaLoRA (Zhang et al., (202)) and AutoLoRA (Xu et al. (002).. LoSA (Huang et al., 2025) integrates sparsity and low-rank adaptation, dynamically adjusting both using representation mutual information and reconstruction error. PRILORA (Benedek & Wolf 2024) employs a heuristic strategy that linearly increases ranks from lower to higher layers, motivated by the observation that higher layers often require greater adaptability in transfer learning. ALoRA (Li et al.) (024c) further incorporates novel mechanism, AB-LoRA, which assesses the importance of individual LoRA
25

---

Published as a conference paper at ICLR 2026
ranks and incrementally prunes redundant components, reallocating the freed budget to critical Transformer modules. These methods provide diverse rank allocation strategies that contribute to more efficient fine-tuning of large models.
L
THE USE OF LARGE LANGUAGE MODELS
During the preparation of this manuseript, large language models (LLMs) were employed in several auxiliary capacities. First, at the writing stage, LLMs were utilized to refine and translate the text, thereby enhancing the overall fluency, readability, and precision of academic expression. Second, in relation to experiments and results presentation, LLMs assisted in generating parts of the code for data visualization and figure plotting, which facilitated more efficient presentation research findings. Third, in surveying the research landscape and related work, LLMs provided support for literature searches, helping us locate and summarize relevant studies in the field systematically. Finally, in the theoretical component of this work, LLMs offered auxiliary support in structuring complex proofs and verifying critical derivation steps, contributing to the clarity and rigor of our theoretical analysis. It should be emphasized that all uses of LLMs were strictly auxiliary in nature; the formulation of research questions, the design of methods, the core theoretical derivations, and the experimental analyses were all carried out independently by the authors.
26


Reviews:
[{"review_id": "Zo6kv5MZ65", "reviewer": "Reviewer_bPkZ", "summary": "This paper proposes IGU-LoRA, an adaptive rank allocation method for PEFT of LLMs. It integrates Integrated Gradients (IG) to compute within-layer parameter sensitivities and introduces an uncertainty-aware scoring mechanism based on exponential moving averages. The approach aims to overcome the instability and bias of gradient-based rank allocation methods such as AdaLoRA. Theoretical results include an error bound on the trapezoidal approximation of IG under a Hessian-Lipschitz condition. Empirically, IGU-LoRA shows consistent improvements over LoRA, AdaLoRA, and DoRA across GLUE, BoolQ, ARC, and GSM8K benchmarks.", "questions": "1. The reproduced results for some baseline methods differ substantially from those reported in the original papers, despite using the same model architectures and datasets. It would be helpful if the appendix could include detailed evaluation settings, such as whether multiple-choice questions are treated as open-ended generation or evaluated by choice-probability ranking. In addition, the statement “Each task is run with 5 different random seeds, and we report the median test performance.” is unconventional. Since five independent runs are already conducted, it would be more informative to report the mean ± standard deviation, which is the common practice for measuring performance stability and variance across seeds.\n\n2. How sensitive is the final performance to the number of mini-batches M used for importance estimation in Algorithm 1? Has a sensitivity analysis been conducted to quantify how varying M (e.g., smaller or larger batch groups) affects both the stability of the importance scores and downstream accuracy?\n\n3. The contribution of this paper is mainly a new importance-score method, and I hope it really is very important. Could the authors perform an ablation experiment to verify the interpretability of the proposed importance scores? Specifically, if a parameter (or singular direction) with a very high IG-based importance score is forcibly assigned a rank of zero (i.e., removed), does the model performance drop sharply? Such an experiment would provide more direct evidence that the IG-derived importance values truly capture critical structural contributions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a new importance-scoring mechanism that replaces instantaneous gradient magnitudes with parameter-space IG.\n2. The experimental evaluation is comprehensive, covering multiple model scales (RoBERTa-large, Qwen-2.5-0.5B, Llama-2/3, DeepSeek) and diverse benchmark types.", "weaknesses": "1. In the example of Figure 2(b), any two parameter curves with the same integrated area would yield identical importance scores under Eq. (4). Consequently, the IG formulation cannot distinguish between parameters that are early-important and those that are late-important along the training.\n\n2. Missing Recent Baselines. The paper omits several recent and closely related adaptive-rank methods, such as GoRA [1] (gradient-driven adaptive rank adjustment) and SalientLoRA [2] (saliency-based rank allocation). \n\n[1] Gora: Gradient-driven adaptive low rank adaptation. NeurIPS 2025.\n\n[2] Unveiling LoRA Intrinsic Ranks via Salience Analysis. NeurIPS 2024.\n\n3. Some hyperparameters, such as $\\beta_1$ and $\\beta_2$ in Eq. (10), are not explicitly reported. Moreover, although Theorem 2 provides a stability guarantee under the condition $t \\ge t_{\\mathrm{burn}}$, the paper does not explain how this condition is verified or ensured in practice during training.\n\n4. Eq. (4) assumes a zero baseline $\\mathbf{W}^{(0)} = 0$, but the physical meaning of a zero-weight baseline is unclear for pre-trained models whose parameters are already non-zero. Should the pre-trained weights instead serve as the baseline? Furthermore, a key assumption of IG is path independence; yet, in a non-convex loss landscape, the straight-line path from 0 to $\\mathbf{W}$ may not be meaningful or optimal. The paper does not analyze how different integration paths (e.g., stochastic or learned trajectories) might affect the resulting importance estimates.\n\n5. Insufficient justification for division-based SNR vs. AdaLoRA's multiplication: The paper uses $\\mathrm{SNR} = \\frac{\\bar{s}_e}{\\bar{U}} \\quad (\\text{Eq. 11})$, interpreting uncertainty as \"noise\" to be penalized. However, AdaLoRA uses multiplication $s = \\bar{I} \\times \\bar{U}$, interpreting uncertainty as \"task diversity\" to be rewarded. Table 4 shows only 0.32% improvement over multiplication (57.99→58.31), suggesting the choice may not matter much for these tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZhK2i20Gc1", "reviewer": "Reviewer_4NAn", "summary": "This paper introduces IGU-LoRA, an adaptive variant of LoRA that allocates ranks across layers using Integrated Gradients (IG) computed in parameter space, combined with an uncertainty-aware scoring mechanism. The method aims to overcome the limitations of existing adaptive LoRA approaches that rely on instantaneous gradients. The authors provide theoretical bounds on the approximation error of the IG estimator and the stability of their uncertainty-weighted score. Empirically, IGU-LoRA is evaluated on GLUE and several reasoning benchmarks, showing consistent improvements over strong PEFT baselines like AdaLoRA and DoRA.", "questions": "1. How would IGU-LoRA perform on models with 30B+ parameters? What is the expected training time and memory overhead compared to standard LoRA or AdaLoRA?\n2. How sensitive are the results to the number of IG samples (N and M)\n3. The author needs to thoroughly explain the impact of sample order and batch size on the results, supplemented with corresponding experiments and theoretical analysis. If these factors do not affect the results, why would different samples accumulate gradients with the same score? If they do have an impact, what is the training variance, how significant is the influence of batch size and shuffle order, and are there any visual comparisons of the score differences obtained for the same sample?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. good theoretical contribution. The error bound for the stochastic IG approximation and the stability guarantee for the SNR-based score are welcome additions.\n2. IGU-LoRA maintains comparable training/inference latency and memory usage to baselines while delivering better performance\n3. novelty in importance score, addresses critical limitations like gradient saturation and unstable rank allocation, with clear theoretical justification for IG approximation error.", "weaknesses": "1. While experiments include Llama-3-8B, results for models larger than 10B parameters (e.g., Llama-3-70B) are absent. Given that PEFT is most critical for very large LLMs, this limits confidence in IGU-LoRA’s scalability.\n2. The method requires O(N) gradient evaluations per parameter group during training, which is downplayed but still significant, especially for larger models. \n3. What worries me the most is that due to the use of gradient accumulation, the training process of the model can be heavily influenced by factors such as the order of sample shuffling and the size of the batch. This raises significant concerns for me, as slight changes or randomness in the training setup could drastically affect the training outcomes. For the training of large language models (LLMs), this is unacceptable—training the same model with the same data could result in completely different importance scores for the same samples simply because of differences in sample order.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5K5UT0ZOY", "reviewer": "Reviewer_oiw5", "summary": "This paper proposes IGU-LoRA, a novel adaptive low-rank PEFT framework for LLMs. The method addresses the limitations of existing PEFT approaches like LoRA and AdaLoRA, which either use fixed ranks across layers or rely on instantaneous gradients for rank allocation. IGU-LoRA introduces two key innovations:\n\n1. Integrated Gradients Sensitivity Scoring: A pathwise sensitivity measure that captures the global and long-term contributions of parameters within a layer, overcoming issues like gradient saturation and local bias.\n2. Uncertainty-Aware Mechanism: A scoring framework that uses exponential moving averages and deviation tracking to suppress noise and stabilize rank selection.\n\nThe paper provides theoretical guarantees for the IG approximation error under a Hessian-Lipschitz condition and demonstrates IGU-LoRA's superior performance across diverse tasks and architectures. Empirical results show consistent improvements in accuracy and robustness over strong PEFT baselines, with comparable memory and computational efficiency.", "questions": "What is EU-LoRA in table 5?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of integrated gradients in the parameter space for rank allocation is a novel and impactful idea, addressing key limitations of gradient-based approaches.\n\nThe paper provides a solid theoretical foundation for the proposed method, including error bounds for IG approximation and stability guarantees for the uncertainty-aware scoring.\n\nThe method is evaluated on diverse benchmarks (e.g., GLUE, mathematical reasoning, and common-sense reasoning tasks) with multiple backbone models, demonstrating its robustness and generalization.", "weaknesses": "While IGU-LoRA achieves strong performance, the use of integrated gradients introduces additional computational costs compared to simpler methods like LoRA. This could limit its scalability to extremely large models or real-time applications.\n\nWhile training efficiency is discussed, the impact of IGU-LoRA on inference latency is less emphasized, which could be relevant for deployment scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vl6DJEkmER", "reviewer": "Reviewer_FGvV", "summary": "This paper introduces IGU-LoRA, an adaptive low-rank fine-tuning method for large language models (LLMs). IGU-LoRA computes the parameter importance using Integrated Gradients (IG) and combines it with exponential moving average smoothing and deviation tracking to develop an uncertainty-aware scoring mechanism. It then allocates rank budgets per layer by keeping the top singular values of the update $AB$ according to these IG-based scores. Consequently, IGU-LoRA improves upon LoRA, which uses a fixed rank across all layers, and AdaLoRA, which uses instantaneous gradient magnitudes to estimate importance. Theorem 1 provides a theoretical bound that supports the proposed approximation of the IG-based integral, while Theorem 2 establishes the statistical stability of the uncertainty-aware scoring. Empirical results show that IGU-LoRA achieves comparable or better performance over LoRA, AdaLoRA, DoRA, and other PEFT baselines across different benchmarks such as GLUE, BoolQ, and GSM8K.", "questions": "- My main concern is the sensitivity of the method to the hyperparameters $N$ and $M$, both are used in approximating the IG-based importance scores. The authors present a preliminary study for small values of $N$ in Table 4, but it is somewhat surprising that there is little performance decrease when $N$ decreases from $20$ to $4$. Could the authors provide more insight into why the fine-tuning performance appears not sensitive to $N$? Additionally, how is $M$ selected here and throughout the experiments?\n\t\n- The justification for using Eq (6) to approximate Eq (5) is unclear. If the approximation needs to be unbiased, shouldn't it include a factor of $N-1$ so that  $(N-1)\\cdot \\mathbb{E}{\\frac{\\partial \\mathcal{L}(\\alpha_k W)}{\\partial w_{ij}}}=\\sum_{k=1}^{N-1}\\frac{\\partial \\mathcal{L}(\\alpha_k W)}{\\partial w_{ij}}$? \n\t\n- Regarding the assumptions in the theoretical results, Theorem 1 assumes that $g_{ij}(\\alpha)$ is sub-Gaussian, but additional justification or discussion is needed. Since the loss function for LLMs training is highly nonlinear and complex, this assumption may be difficult to hold in practice. Theorem 2 assumes that $y_t$ are i.i.d. with same mean and variance parameters across $t$, which seems unrealistic given that $W$ is updated after every epoch. Some empirical evidence or discussion would be helpful.\n\t\n- For computational efficiency with large $N$, $M$ must be much smaller than $N$. In that case, Theorem 1 shows that the second term dominates the first, and the total error should be understood as $O(1/\\sqrt{M})$. This suggests $M$ is the more critical factor than $N$. It would be nice to see additional experiments that vary $M$ to confirm this.\n\t\n- Please use different notation for Eq (7) and Eq (11).\n\t\n- In Algorithm 1, clarify what $k$ means and what $P_{:,\\pi_{1:k}}$ in line 13 refers to. \n\nMinor comments:\n- Eq (3): $P_{kj}, Q_{jk} \\Rightarrow P_{ki}, Q_{ik}$.\n- Eq (4): $\\alpha(W-W^{(0)})\\Rightarrow \\alpha(W-W^{(0)}) + W^{(0)}$?\n- In Eq (9), (10), clarify how $\\bar{s}^{(t)}_e(w_{ij})$ and $\\bar{U}^{(t)}(w_{ij})$ are defined for $t=0$.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well motivated and addresses the weakness of prior adaptive low-rank fine-tuning methods such as AdaLoRA. The authors provide a rich motivation for their new scoring mechanism based on IG and exponential moving average smoothing, and they conduct extensive experiments across a wide range of benchmarks and model scales.", "weaknesses": "The paper’s contribution appears somewhat incremental given that it mainly replaces the gradient-based importance scores in AdaLoRA with IG-based scores. The theoretical results require strong assumptions that are unlikely to hold in practice, though the theoretical results do not constitute a substantial part of the paper’s overall contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHqvXAO9ow", "reviewer": "Reviewer_8KC4", "summary": "This paper proposes IGU-LoRA, a parameter-efficient fine-tuning method that employs an adaptive rank mechanism. This paper has two main contributions but looks complex and hard to be followed by other researchers. It computes integrated gradients sensitivities within each layer and aggregates them into a layer-level score to enable adaptive rank allocation. Moreover, it introduces an uncertainty-aware mechanism based on exponential moving averages and deviation tracking to suppress noisy updates and ensure more stable rank selection. In addition, the method is supported by solid theoretical guarantees that justify the soundness of the proposed approach.", "questions": "[1]. How about the performance on MultiModal Benchmark?\n\n[2]. How about the performance on diffusion model?\n\n[3]. What is the EU-LoRA in Table 5?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1]. The paper introduces IGU-LoRA, a new PEFT framework based on adaptive ranks.\n\n[2]. This paper integrates Integrated Gradients into LoRA. Unlike prior adaptive-rank methods, which rely on instantaneous gradients. This paper has sufficient novelty.\n\n[3]. The authors provide solid theoretical guarantees for their approach. They derive an upper bound on the approximation error for the integrated-gradient estimator, quantifying discretization and sampling errors. They also present a stability guarantee for their uncertainty-aware signal-to-noise ratio based scoring. These analyses add strong mathematical rigor and credibility to the method’s design", "weaknesses": "[1] The paper contains formatting issues. It uses the preprint template instead of the anonymous submission template.\n\n[2] The abstract on OpenReview still includes unprocessed LaTeX commands.\n\n[3] There are several typos in the paper. For example, Martix -> Matrix, and revealstructural -> reveal structural.\n\n[4] Some data statistics are insufficiently explained. For instance, the meaning of EU-LoRA is unclear and causes confusion.\n\n[5] The proposed method is overly complex, making it difficult for other researchers to reproduce or follow the approach.\n\n[6] The generalization is not clear. This paper does not report the performance on MMMU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)