FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation
Authors: Muqing Liu
Chongjie Si
Yuheng Jia
Venue: ICLR 2026 Poster (2026)
Keywords: PEFT; Dynamic Rank; LoRA
Primary area: foundation or frontier models, including LLMs

Abstract: Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs. 
Parameter-efficient fine-tuning (PEFT) has thus become a mainstream paradigm. 
Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility. 
Dynamic rank allocation methods mitigate this issue by pruning redundant directions; however, they often rely on heuristic, element-level metrics that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation. 
To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability. 
By addressing granularity, flexibility, and stability limitations, FlexLoRA provides a more principled solution for PEFT. 
Extensive experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks.

Published as a conference paper at ICLR 2026
FLEXLORA: ENTROPY-GUDDED RANK ADAPTATION
FLEXIBLE Low-
Muqing Liu Chongjie Si Yuheng Jia
'Chien-Shiung Wu College, Southeast University "MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University "Shooo of Computer Science and Engineering, Southeast University "Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China 1i amm230sseu edu.cn on chengjies (us.ii..uuuuu.....u
yhjialseu. Uhjialseu.edu.co edu.cn
ABSTRACT
Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs. Parameter- efficient fine-tuning (PEFT) has thus become a mainstream paradigm. Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility. Dynamic rank allocation methods mitigate this issue by pruning redundant direc- tions; however, they often rely on heuristic, element-level metries that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation. To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adapta- tion framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability. By addressing granularity, flexibility, and stability limitations, FlexLoRA provides more principled solution for PEFT. Extensive experiments show that FlexLoRA consistently outperforms state-of of ofe-the-rt baselines across benchmarks.
INTRODUCTION
Since large pre-trained models have advanced the state of the art in numerous tasks (Kirillov al. 2023 Devlin et al., 2018,LLi et al. [2099] Peng et al. (002)), adapting them to downstream tasks has
become a prevailing way recently. However, their adaptation requires fully fine-tuning, i.c., updating
all parameters, which incurs substantial computational and memory costs (Ma et al. 2024 Raffel
al., 2020 Quu al., 2020). To address this challenge, parameter-efficien fine-tuning (PEFT)
methods have been proposed (Zhang et al. 2022a;; Si et al.) 2024 Pfeiffer et al.) 2020 Houlsby
cet al. 2019 Hu et al. 2021] He et al.) 2021a). which adapt pre-trained models by updating only small subset of parameters while preserving competitive performance.
Among PEFT approaches, Low-Rank Adaptation (LoRA) (Hu al., (202T) and its variants (Zhang et al., 2022a Wu et al. 2024) have emerged as representatives, which introduces trainable low- rank matrices to approximate task-specific updates of pre-trained weights. Despite its effectiveness,
LoRA employs fixed-rank across all layers, thereby limiting flexibility in allocating model capacity.
To address this issue, a series of dynamic rank allocation methods have been proposed, such as AdaLoRA (Zhang al. (023), SaLoRA (Hu al.] 2023a), and AutoLoRA (Zhang et al.) (2244). These methods usually compute a heuristic importance score such as parameter gradient for each individual rank direction. The scores from all ranks across all matrices then aggregated and globally sorted, after which the least important directions are pruned. These strategies partially alleviate the limitations of LoRA.
" "Corresponding author.
I Equal contribution.

---

Published as a conference paper at ICLR 2026
However, these dynamic rank allocation methods still suffers from three key limitations. First, the importance metrics are typically heuristic, relying on approximations such as parameter sensitiv- ity rather than principled criteria. Second, all rank directions from different matrices globally sorted and pruned together, ignoring matrix-level distinctions and thereby risking the removal of structurally important directions. Third, the allocation is unidirectional, it only prunes redundant ranks without mechanisms to expand capacity in layers that demand additional expressive power. Together, these limitations hinder the ability of existing methods allocate model capacity in principled and adaptive manner, motivating the need for a more flexible framework.
W
Q
Bottom 4
Prune
P
P2
Ax
Q₂
Matrix
Importance I
Qx
Pr
Q
Expand
Top &
W₂
Pk
Ag
Px
Ax
Pruned
Remained
Expanded
Figure 1: Framework of FlexLoRA. For each weight matrix WA. FlexLoRA represents the update an SVD-like form AW P&A Qs. where Aş is a diagonal matrix. It then computes a spectral entropy-baat based importance score for each AW. All scores globally ranked under given rank budget: matrices with lower scores prune the least significant direction in Ag, while those with higher receive additional ranks. The newly allocated ranks initialized with zero-impact scheme to preserve the original input while enabling subsequent learning.
To this end, we propose FlexLoRA, a novel dynamic low-rank adaptation framework. LoeX reallocates computational resources to the most critical layers, ensuring sufficient capacity for im- portant layers while removing redundancy from less important ones. Specifically, it evaluates the importance of each low-rank matrix the matrix level using spectral entropy and dynamically ad- justs the rank allocation accordingly: pruning the least significant directions in low-importance lay- ers while expanding the rank layers that demand additional capacity. For expansion, FleX adopts zero-impact initialization strategy, where newly added singular directions are initialized with zero singular values and Gampsed singular vectors, thereby preserving the original input while enabling stable training. Extensive experiments across diverse benchmarks demonstrate that FlexLoRA consistently outperforms strong PEFT baselines under the same parameter budget.
Our contributions are as follows:
We propose FlexLoRA, a novel framework that supports both pruning and expansion of ranks under global budget, enabling dynamic reallocation of model capacity across layers.
We introduces a spectral entropy-based criterion to assess the importance of low-rank ma- trices at the matrix level, overcoming the limitations of heuristic, element-wise metrics.
Extensive experiments demonstrate that FlexLoRA achieves superior performance com- pared with state-of-the-art PEFT baselines under identical parameter budgets.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 CLAAMETER-EFFICEER FINE-TUNING
Large pre-trained models exhibit strong generalization across diverse downstream tasks (Xu et al., 2023
Han al., 2024 Lin al. 2024 Wu et (025), yet full fine-tuning remains computa- tionally costly and infage-ineffcieet. Parameter-effiiint fine-tuning (PEFT) methods address this issue by reducing trainable parameters while retaining most of the performance of full fine-tuning. Early PEFT approaches fall into three main categories: (1) adapter-based methods (Houlsby et al.,
2019
Chen et al.
2022, Luo et al.) 2023; Pfeiffer et al., 2020 He et al. 202 Mahabadi et al.
202
which insert lightweight trainable modules; (2) prompt- based methods (Lester et al. 2021
Razdaibiedina et al. 2023] Wang et al. 2023 Fischer et al. 2024, Yang et al. (023). which learn
task-specific prompt vectors; and (3) low-rank methods (Hu et al.) [00]]022]] Liu et al. 2024 Zhang
et al.
2022a Kopiczko et al. 2023 Qiu et al. 2023 Renduchhntala et al. 2023 Wang et al. 2024
which approximate parameter updates via low-rank factorization.
Among these, LoRA (Hu et al. fectiveness. By injecting trainable low-rank matrices into linear transformations, LoRA achieves competitive performance with far fewer trainable parameters. However, its fixed-rank design across all layers limits flexibility and prevents adaptive capacity allocation task-specific requirements.
2021) is the most widely adopted due to its simplicity and ef-
2.2 LORA VARIANTS WITH DYNAMIC RANK ALLOCATION
To overcome the limitation of fixed-rank LoRA, recent studies have explored dynamic rank allo- cation, where the rank of each LoRA module is adaptively adjusted during training. Existing ap- proaches can be grouped into three families. (1) SVD-driven allocation: AdaLoRA (Zhang et al., 2023) and SaLoRA (Hu et al. (023a) periodically decompose low-rank matrices and prune less important singular directions while reallocating capacity to critical layers. (2) Singular rank decom- position (SRD): DoRA (Mao et al. 2024), AutoLoRA (Zhang et al. 2024). and SoRA (Ding et al.) 2023) dynamically split or merge singular components based on gradient statisties or structural pri- ors. (3) Sampling-biis based allocation: DyLoRA (Valipour et al.) (022) and QDyLoRA (Rajabzadch
ctt al. 2024) stochastically vary ranks across iterations to improve robustness through randomiza- tion. Besides rdnk-adjusting approaches, methods like MLAE (Wang et al. 2024) decompose low- rank matrices into rank-1 experts and apply expert-level stochastic masking, implicitly modulating effective capacity without altering the nominal rank, thus offering complementary perspective to dynamic-rank LoRA variants. The effectiveness of these methods largely depends on heuris- tic sensitivity-basseee metrics, which aggregate gradient-weight products across singular directions (Liang et al. [021]] Sanh et al., 2020 Zhang et al. 2022b) and smoothed with moving averages or uncertainty terms.
Despite their progress, these strategies remain component-level heuristics: they estimate the impor- tance of each singular direction or parameter independently, while neglecting the structural inter- actions at the matrix level. As a result, they are prone to gradient noise, lack stability, and may overlook the coordinated role of singular directions within the entire matrix, leading to suboptimal rank adjustments.
2.3 ENTROPY-GUIDED METRICS
Entropy, a fundamental concept in information theory for quantifying uncertainty (Shannon 1948 Jaynes| 1957). has recently been adopted to measure redundancy and information content in neural networks (Achille & Soatto 20.7). Entropy-based criteria have also been applied to pruning and compression, guiding the removal of uninformative components while preserving capacity (Liao) cet al. 2024 These works highlight that entropy is not only statistical measure of uncertainty but also principled indicator of representational richness, providing insight into the information distribution of entire matrices rather than individual parameters.
3

---

Published as a conference paper at ICLR 2026
3 METHOD
We propose FlexLoRA, flexible low-rank adaptation framework that dynamically allocates ranks across layers of large pre-trained models. FlexLoRA consists of three key components: (i) matrix- level entropy-guiide and (iii) zero-impact initialization. We first introduce the basic formulation of FlexLoRA, and then present method in detail in the subsequent subsections.
importance metric, (ii) rank pruning and expansion under global budget,
3.1 SVD-BASED LOW-RANK ADAPTATION
Low-Rank Adaptation (LoRA) (Hu et al. 2021) represents task-specific updates to pre-trained weight matrix W E IR x di using two trainable low-rank matrices:
AW BA, A A
gduxr
(1)
where r < min(dont dm)) is a small fixed rank to ensure parameter efficiency. Here, AW serves as low-rank update that is added to the frozen pre-trained weights, yielding the effective parameter matrix W
W + AW. Similar to LoRA's formulation and those in prior works (Zhang et al,, Meng et al. 2024), FlexLoRA adopts an SVD--based formulation, where AW PAQ. Here, P I gu X and Q E TR represent singular vectors, and A = diag(A1 A )r) E R is
2023
a diagonal matrix of singular values. To maintain stability and preserve the SVD property (P I, QQ = I), we introduce orthogonality regularization term:
R(P, Q) = |P - 1| + ||QQ - 1
(2)
where || || is the Frobenius norm. However, the rank of LoRA and its variants is fixed across layers, which may hinder adaptation flexibility and prevent efficient utilization of model capacity. To overcome this limitation, FlexLoRA introduces three synergistic mechanisms that enable dynamic adjustment of the effective rank and maximize its utilization.
3.2 MATRIX- ENTROPY-GUIDER IMPORTANCE METRIC
A key challenge in dynamic rank allocation lies in determining the importance of singular directions. As discussed Related Work (Sec. [.2), prior methods largely rely on sensitivity- heuristics that design importance metrics the level of individual parameters singular directions. Such approaches overlook the structure of the entire matrix, leading to noisy optimization.
To overcome these limitations, propose matrix-level entropy-guided importance metric. Unlike sensitivity- based measures, entropy-basee based evaluation captures the intrinsic geometry of the matrix throughout training. Given singular values A, the spectral entropy importance score is defined
X
((A) log
sy log(s; + +e) 8, Σ X
where e a small constant to avoid numerical issues. The entropy is normalized by log to ensure that the spectral entropy is bounded within (0, 1] and comparable across different ranks; the rationale for this normalization is provided Appendix B Intuitively, a low entropy indicates that energy is concentrated in few singular values, suggesting redundancy and suitability for pruning, while high entropy reflects a more balanced distribution, implying richer structural capacity.
3.3 RANK PRUNE AND EXPANSION WITH LERO-IMPACE INITIALIZATION
With the spectral-entropy -base based confidence score in place, we can dynamically adjust the rank of each matrix during training. Unlike prior approaches that primarily focus on rank reduction, our strategy supports both expansion and pruning, enabling more flexible capacity reallocation. Specif- ically, at each training step t, we define a rank budget b(t), which specifies the maximum number of singular directions that be either added removed in that step (the detailed design of b(t) is described in Sec. J.2p. This budget acts as a global constraint, ensuring that rank adjustments remain stable and computationally tractable while still allowing sufficient adaptivity.
4

---

Published as a conference paper at ICLR 2026
To remove redundancy, first rank matrices by their importance score and identify the b(t) least important ones with rank greater than one. Within each selected matrix, we reduce the rank by dis- carding the singular component associated with the smallest singular value. This choice is grounded in the SVD principle that directions with minimal singular values contribute least to the matrix's representational power. Moreover, the corresponding importance score T(Amin) also the smallest, indicating minimal spectral contribution; a formal proof of this monotonicity is provided in Ap- pendix C For expansion, we select the b(t) most important matrices. Within each selected matrix, we add new singular direction with its singular value initialized to zero, while the corresponding vectors are sampled from Gaussian distribution. This eero-impact initialization ensures that the sertion does not perturb the current output and allows the new direction to be gradually optimized during training, which guarantees stability while enabling effective capacity growth when needed.
By leveraging the matrix-level spectral entropy as confidence score, FlexLoRRA adaptively adjusts ranks through both pruning and expansion, while zero-impact initialization ensures stable incorpo- ration of new capacity. This joint design enables FlexLoRA to allocate model capacity more flexibly across layers, preserving redundancy-free directions while amplifying structurally informative ones. The full procedure is summarized in Algorithm
Algorithm 1 FlexLoRA
Require: S: total steps, T: bidirectional rank allocation period, r: initial rank, Require: 6: total ranks pruned/expanded per step
1: Initialize: (Po:N-1, Ao.v-1 Qo:v-1) for all weight matrices with rank r
2: for / = 1 to S do
UPDATEWEIGITTSCP A, Q) 4: Io:N-1 CALCULATEEIPPORTANEEEA
3:
5: if E then
6:
L <- LLAASTTAPPORRAATTAATTTCCEEEE b)
Identify top-b least important matrices Identify top-b most important matrices
7:
M + -MMOSTTAPOORTTANTTTTTIIIESSS b)
8:
PRUNERANKS(L, A, Q) EXPANDRANKS(M, P, A, A, Q)
9:
end if 11: end for
return Fine-tuned parameters (P,A,
EXPERIMENT
4.1 MODELS, DATASETS AND BASELINES
We evaluate Foe both language and vision models three tasks to demonstrate its gener- ality. For natural language processing, we adopt DeBERTa (He et al. 2021b). widely used encoder model for understanding tasks, and the LLaMA family large language models (AI@ Metal 2024), which represent the current state-of-the-an in generative modeling. For computer vision, we extend FleXLoRA transformer based backbones and assess its adaptability to recognition tasks.
We consider three representative categories of benchmarks:
Natural Language Understanding (NI.U): We use the GLUE benchmark (Wang et al. 2018), which includes diverse sentence- and pair-level classification tasks.
Commonsense Reasoning (CR): We evaluate on eight widely used benchmarks covering diverse reasoning forms, including yes/no question answering (BoolQ), physical and social reasoning (PIQA, SIQA), narrative completion (HellaSwag), pronoun resolution (Wino- Grande), and multiple-choice knowledge-intensin reasoning (ARC-e, ARC-c, OBQA), as well as additional tasks such as CommonsenseQAA and SocialIQA (Sap et al., 2020).
Visual Recognition (Vision):To test generality beyond NLP, we employ the Visual Task Adaptation Benchmark (VTAB) (Zhai et al. 2019), which spans 19 image classification datasets across natural, specialized, and structured domains. We use ViT-B/16 pretrained ImageNet-22K the backbone to ensure consistent evaluation setup.
5

---

Published as a conference paper at ICLR 2026
We compare FlexLoRA against strong baselines under unified parameter budget. Specifically, consider: (i) standard LoRA with a fixed-rank (Hu et al.) 2021), and (ii) AdaLoRA (Zhang et al. 2023), both initialized with the same rank budget for fairness.
4.2
IMPLEMENTATION DETAILS
All methods are implemented PyTorch (Paszke et al., 2019). based on Huggingface Transformers (Wolf et al. 2020). For FlexLoRA, introduce a dynamic rank scheduler to stabilize rank adjust- ment. At training step t, the number of ranks adjusted, denoted b(t), defined by a cubic decay
schedule:
b(t) = round bo (0-(1-1 t twarmup 3
(4)
Uiiall
where is the initial adjustment size, /warmup marks the start of rank adaptation, ffinal denotes the beginning of the final freeze phase, and T is the total number of training steps. We clamp b(t) to the range [0, by] and further restrict it so that adjustments never exceed the number of available modules or per-module rank limits. This schedule allows for aggressive rank reallocation in early training, when model capacity is being explored, and gradual stabilization towards convergence, ensuring consistent optimization in later stages. All methods are implemented in PyTTorch (Paszke et al. 2019) with the HuggingFace Transformers library (Wolf et al. 2020). Experiments are conducted on NVIDIA A100 GPUs, and unless otherwise specified, hyper-parameterss follow recommended settings in prior work to ensure fair comparison.
4.3 NATURAL LANGUAGE UNDERSTANDING
Table reports the GLUE benchmark results, comparing FlexLoRA with LoRA (r 8) and AdaLoRA under the same parameter budget. Details of the dataset and pyper-parameters settings can be found in Appendix (.I] FlexLoRA consistently delivers the best competitive performance across all tasks. The gains are particularly pronounced on CoLA and RTE, where FlexLoRA sig- nificantly surpasses both baselines, highlighting its advantage in handling linguistically challenging tasks. Overall, FleXLoRA achieves the highest average score of 89.1, outperforming o 188.1) and LoRA (81.8), thereby demonstrating the effectiveness of entropy-guided bidirectional rank allocation.
Table Results on GLUE development set with DeBERTaV3-base. We report mean of runs using different random seeds.
CoLA Mcc.
MNLI Acc.
MRPC Acc.
RTE
QNLI Acc.
SST-2 Acc.
STS-B Corr.
QQP Acc.
Method
Params.
Avg.
Acc.
Full FT
184.3M
69.2
89.9
90.2
83.8
91.0
95.6
91.6
92.1
88.3
BitFit
0.1M
67.0.2.5
89.4.0.2
87.8
78.720.9
92.2+a.2
94.8-o.3
91.440.2
88. 88-420.2
86.2
H-Adapter
1.2M
62.643.2
86.5g.0.4
89.9
80.4g2.0
92.8
98.720.4
90.2
900.g0.1
85.9
P-Adapter
1.2M
63.921.7
86.8.0.3
89.520.8
80.5g..9
92.6ga.2
93.8_o.2
90.7
90.52..
86.0
AdapterFusion
1.2M
68.8.0.2
90.340.3
89.5+a.1
85.2+0.5
94.33
95.6-0.6
91.5+a.1
92,0+0.4
88.4
LoRA,_
1.3M
68.5.2.6
89.8
90.720.7
84.8
94.1
94.0=0.2
91.249.a
87.920.3
81.7
AdaLoRA
70.0a1.8
80.0g1.6
90.9a1.5
88.1g0.9
94.1g1.1
94.620.
91.240.3
87.2
88.1
DoRA
1.3M
65. 65.4-0.4
87.8
90.1
81.721.
93.0
91.3-26
91.3+0.0
91.3+0.
86.5
DyLoRA,
0.9M
30.5 23.00
86.8
91.440.
77.620.6
93-04a.a
94.420.4
91.149.2
89.9
85.5
FlexLoRA
1.9M
71.820.
90.0+a.7
90.9go.
88.8.0.7
94.2+0.2
95.2=0.4
91.5
90.3go.7
89.1
4.4 COMMONSENSE REASONING
We next evaluate FlexLoRA commonsense reasoning benchmarks. Details of the dataset and eyper-parameter settings can be found in Appendix D.2 Results Table 2 show that Foex consistently delivers strong performance across all settings. On the LLaMA-3 models, Flex achieves clear improvements over standard LoRA, underscoring the necessity of adaptive rank al- location when operating under constrained parameter budgets. At rank 8, FORA achieves

---

Published as a conference paper at ICLR 2026
average score of 85.2 LLaMA-3, slightly surpassing AdaLoRA (85.1). When the rank is in- creased to 32, FlexLoRA further improves to 85.5, establishing the best overall results among all parameter-effiiiet baselines, including LoRA-Dash, NoRA+, and PrecLoRA. These findings sug- gest that Flexl not only adapts effectively to earlier model generations but also remains com- petitive the latest LLaMA-3, demonstrating robustness across model iterations while exploiting higher ranks without redundancy.
Table 2: Results on commonsense reasoning tasks.
Model
Method
Param.
BoalQ
PIQA
SIQA
HellaS.
WinoG.
ARC-
ARC-c
OBQA
Avg.
ChatGPT
68.5
78.5
79.9
74.8
77.0
Full FT
81.5
95.8
87.4
LoRA,
14.2M
62.2
86.5
80.3
94.5
84.4
77.7
88.0
85.8
82.4
AdaLoRA,
21.2M
74.1
88.4
80.3
95.7
84,4
80.1
91.0
87.0
85.1
21.2M
81.0
85.2
LLMMA3-8B
LoRA,
56.6M
75.6
89.5
81.2
95.1
85.1
80.1
90.3
86.2
85.4
AdaLoRA,
56.6M
71.3
88.7
80.1
94.5
78.8
90.2
85.8
84.5
LoRA-Dash
56.6M
80.2
95.7
86.8
90.7
80.2
85.6
85.4
NoRA+
56.6M
71.2
85.1
79.5
92.2
83.4
85.9
72.3
83.2
81.6
PrecLoRA
56.6M
70.7
85.8
78.9
91.9
83.7
85.1
71.1
82.4
81.2
HexLoRA
56.6M
72.8
89.1
80.7
96.0
86.4
81.3
90.8
87.2
83.5
4.5
VISUAL TASK
To further assess the generality of FleXLoRA beyond NLP, we evaluate it the Visual Task Adap- tation Benchmark (VTAB) and compare against LoRA (r = 14) and AdaLoRA under identical experimental conditions. Details of the dataset and eyperparraettess settings can be found in Ap- pendix D33 As shown in Table 3 Loe achieves the highest average accuracy of 67.8%, outperforming both LoRA (66.7%) and AdaLoRA (64.7%) under comparable parameter budgets. Notably, FoeX yields substantial improvements on natural image datasets (e.g., +8.9 CI- FARI00 over LoRA), highlighting its ability to capture domain diversity. also performs strongly specialized datasets such as Camelyon and Retinopathy, demonstrating robustness in medical and fine-grained visual recognition. Overall, these results confirm that FlexLoRA's cntropy-guidee rank allocation is not confined to language but generalizes effectively to vision.
Table 3: Results on VTAB benchmark. Accuracy (%) across Natural, Specialized, and Structured domains.
Natural
Specialized
Stencturel
Caleehiol
Camel VM
Red Zed.oopahhy
Clow Count
CloveDoat
KITTE-DGA
NORB-EE EI
Cifanti
SWHN
Restoks
DMLah
4Sp ON
DTD
Full FT
327M
97.2
58.6
65.5
46.7
1.20M
19.4
FLaRA
1.11M
72.0
LicRA
1.118
98.4
78.5
Mil
1.11M
PSSSA
1.11M
77.5
1.11M
87.1
69.8
70.1
57.4
74
43.2
FURTHER STUDY
To better understand the design choices of FlexLoRA, we conduct series of ablation studies and exploratory analyses. We compare alternative importance metrics, examine the necessity of bidirec- tional rank allocation, evaluate different initialization strategies, and analyze the final distribution of ranks after training. Together, these studies shed light on why FlexLoRA works and provide insights into future research directions.

---

Published as a conference paper at ICLR 2026
5.1 IMPORTANCE METRICS
We first investigate the role of importance metrics in guiding rank allocation. In addition to our proposed entropy-based criterion, compare two widely adopted alternatives: sensitivity-based and norm-based metrics.
kensitivity-basee metries. As in AdaLoRA (Sec. level grediett-weight products:
2.2 importance can be estimated by element-
((wsy) Way way C),
(5)
which approximates the sensitivity of each parameter to the training loss. To reduce the noise and instability such estimates, AdaLoRA further applies exponential moving average smoothing and an uncertainty term obtain refined importance score:
$00) (w:r) (w,y) (7( (wrr),
where 7(0) is the smoothed sensitivity and (? quantifies local variation. Although intuitive, such heuristics are highly sensitive gradient noise, unstable across iterations, and fail incorporate matrix-level structural information.
Norm-based metries. The nuclear and Frobenius norm summarize the overall strength of singular values by aggregating spectral magnitudes. While these metrics capture cumulative energy, they cannot reflect the distributional structure of singular components. For consistency, both norms are normalized by the number of singular values n.
[ nucleaa
TF
Results. Table 4 shows that entropy consistently outperforms both sensitivity- and norm-based al- ternatives on representative GLUE tasks. These results highlight entropy's superior discriminability and robustness, confirming it as a more principled criterion for dynamic rank allocation.
Table 4: Comparison of different importance metrics on GLUE with DeBERTaV3-basee Entropy consistently outperforms the other norm-based alternatives.
CoLA
MNLI
MRPC
RTE
QNLI
SST-2
STS-B
QQP
Method
Params.
Avg.
Mcc.
Acc.
Acc.
Acc.
Acc.
Acc.
Corr.
Acc.
Full FT
184.3M
69.2
89.9
90.2
83.8
94.0
95.6
91.6
92.4
88.3
AdaLoRA
1.9M
70.0
89.0
90.9
88.1
94.1
94.6
91.2
87.2
88.1
Nuclear
1.9M
69.8
89.1
90.2
84.8
94.4
94.8
91.1
87.5
87.7
Frobenius
1.9M
69.4
88.8
86.5
86.3
94.1
94.7
90.8
86.4
87.1
FlcxLoRA
1.9M
71.8
90.0
90.9
88.8
94.2
95.2
91.5
90.3
89.1
5.2
RANK PRUNING AND EXPANSION
We next assess the necessity of combining pruning and expansion. Specifically, we compare FlexLoRA with two variants: (i) Prune-only, which removes low-importance singular directions but never expands capacity; and (ii) Expand-only, which continually adds directions without prun- ing.
As shown in Table 5 prune-only leads over ever-pruning and lacks the flexibility recover capacity, while expand-only wastes parameters by retaining redundant directions. In contrast, FlexLoRA jointly prunes and expands ranks under a global budget, achieving consistently superior results. This confirms that bidirectional rank adjustment is critical for balancing efficiency and adaptability, and motivates further exploration adaptive scheduling strategies.
5.3 ZERO-IMPACT IMPACT INITIALIZATION
We further analyze initialization strategies for newly added singular directions.
8

---

Published as a conference paper at ICLR 2026
Table 5: Results of further study pruning and expansion. We report results four representative GLUE benchmarks (CoLA, MRPC, RTE, and STS-B) using DeBERTaV3-basse Results show that FlexLoRA's pruning and expansion strategy outperforms prune-only and expand-only strategies.
CoLA
MNLI
MRPC
RTE
QNLI
SST-2
STS-B
QQP
Method
Params.
Avg.
Mec.
Acc.
Acc.
Acc.
Acc.
Acc.
Cor..
Acc.
Full FT
184.3M
69.2
89.9
90.2
83.8
94.0
95.6
91.6
92.4
88.3
Prunc-only Expand-only
1.9M
66.8
89.4
90.4
85.9
94.2
94.5
91.4
87.6
87.5
1.9M
68.3
89.5
89.5
87.7
94.3
93.8
91.3
86.5
87.6
FlexLoRA 1.9M
71.8
90.0
90.9
88.8
94.2
95.2
91.5
90.3
89.1
Zero-impact initialization, adopted in FlexLoRA, sets the singular value while sampling vectors from a Gaussian distribution. As alternatives, we examine: (i) Small-init, which assigns small non-zero values to new singular values and samples orthogonal vectors via (ii) Zero-init, which sets both values and vectors to zero, freezing new directions until gradients accumulate: (iii) Orthogonal-init, which sets values serve independence.
but samples orthogonal vectors pre-
Table shows that FlexLoRA with zero-impact initialization achieves the highest average score (85.8), consistently outperforming all alternative initialization strategies across the four GLUE benchmarks. This demonstrates that pero-impact initialization provides the most favorable balance between stability and learnability, ensuring that the expanded capacity is fully utilized to enhance task-specific adaptation.
Table 6: Results of further study zero-impact initialization. We report results four repre- sentative GLUE benchmarks (CoLA, MRPC, RTE, and STS-B) using DeBERTaV3-base Results show that FlexLoRA's zero-impact initialization achieves the best balance, outperforming the other strategies.
CoLAA Mec.
MNLI Acc.
MRPC
RTE
QNLI
SST-2
STS-B
QQP
Method
Params.
Avg.
Acc.
Acc.
Acc.
Acc.
Corr.
Acc.
Full FT
1 184.3M
69.2
89.9
90.2
83.8
94.0
95.6
91.6
92.4
88.3
Small-init
1.9M
69.2
89.4
89.7
85.9
94.4
91.2
87.5
87.8
Zero-init
1.9M
66.4
89.5
88.2
84.8
94.4
94.6
91.1
87.4
87.1
Orthogonal-init
1.9M
70.4
89.8
90.0
87.0
94.2
94.7
90.7
87.5
88.0
FlexLoRA
1.9M
71.8
90.0
90.9
88.8
94.2
95.2
91.5
90.3
89.1
5.4 RANK DISTRIBUTION AFTER ALLOCATION
Finally, we analyze how FlexL allocates ranks across layers and modules during training. Fig- ure vieulvssullizes the evolution of effective ranks on the CoLA task, where darker colors denote higher capacity allocation and lighter colors indicate stronger pruning.
The results reveal several interesting patterns. Contrary to the common assumption that deeper layers should dominate task-specific adaptation, observe that shallower layers (LO-L3) undergo sub- stantial rank expansion. This indicates that early layers capture task-relevant syntactic and semantic cues that require increased modeling capacity, highlighting their underestimated role in linguistic adaptation. In contrast, deeper layers (I.-II11) are consistently pruned across training, suggesting that these layers encode more task-agnostic or redundant representations that can be preserved in compressed form without harming performance. Middle layers (L4-L8) exhibit mixed behavior. Within these layers, some modules maintain moderate rank allocations, while others are gradually pruned depending
their relative contribution to the task. Notably, attention output and intermedi- ate dense modules in these layers tend to preserve higher ranks, suggesting that they play a central role in shaping task task-specific decision boundaries and information propagation.
Overall, these findings suggest that CoLA-relevant signals are primarily concentrated in shallow and middle layers, while deeper layers encode more generic features requiring less adaptation. This

---

Published as a conference paper at ICLR 2026
WAK
1240
12
12]
LUV
10
[4]
15
U.A.
180
130
11040
LI1I
L1140
[1]
111.0
Tairing Step
Figure 2: Visualization of rank allocation during FlexLoRA training the CoLA task. We selected the modules with the most significant changes in rank and sorted them by layer depth. Modules are sorted by layer depth from top (shallow) to bottom (deep). K, V, denote key, value, and query projections: A is attention output; I is intermediate dense; O is output dense. Darker colors indicate more capacity, lighter colors indicate stronger pruning.
demonstrates that FlexLoRA adaptively reallocates capacity in manner that aligns with linguistic task requirements, improving efficiency while also yields interpretable rank allocation patterns.
CONCLUSION
We presented FlexLoRA, flexible low-rank adaptation framework that addresses the limitations of existing PEFT methods. FlexLoRA introduces three synergistic components: a matrix-level entropy- guided importance metric, bidirectional rank allocation mechanism under global budget, and zero-impact initialization strategy. These components together ensure both stable optimization and efficient utilization of model capacity during fine-tuning. Extensive experiments guage understanding, commonsense reasoning, and visual recognition benchmarks demonstrate that Loe consistently outperforms strong state-of-the-arr baselines in both accuracy and parameter efficiency. Further analyses confirm the effectiveness of its core components and reveal interpretable rank distribution patterns that aligned with task-specific requirements, offering insights into the func- tional roles of different layers and modules. In summary, Foe provides a robust and general strategy for parameter-efficien fine-tuning, laying principled foundation for future research on flexible low-rank adaptation of large pre-trained models across diverse modalities and tasks.
natural lan-
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant U24A20322 and Grant 62576094.
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations
through noisy computation. arXiv preprint 1rXivv1611.01133,, 2017.
AI@Meta Llama model card. 2024.
URL hattp:://gtthu. http://ttt.u..co//eee-tllllaaaaa
3 n MODEL. CARD. md
10

---

Published as a conference paper at ICLR 2026
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 1. Citeseer, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpin and Lucia Specia. Semeval-2017 task I: Semantic textual mumilarrty-muliilinng and cross- voos-linguull focused evaluation. arXiv preprint arXiv: 2017.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16644166678 2022.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint /9Xiv:/955.1000 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05477, 2018
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04800. 2018.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11690 2023.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005), 2005.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale.
arXiv preprint arXiv:2010.119900 2020.
Mare Fischer, Alexander Bartler, and Bin Yang. Prompt tuning for parameter-efficient medical image segmentation. Medical Image Analysis, 91:103024, 2024.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pp. 1-9, 2007.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Junxian He. Chunting Zhou, Xuezhe Ma. Taylor
and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110004366, 2021a.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disenangled embedding sharing. arXiv preprint 1.X552211..05443 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficice transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019.

---

Published as a conference paper at ICLR 2026
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.06683 2021.
Yahao Hu, Yifei Xie, Tianfeng Wang, Man Chen, and Zhisong Pan. Structure-aware low-rank adaptation for parameter-efficiet fine-tuning, Mathematics, 1 11(0))4317, 2023a.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy la-Wei Lee. -dmaatpters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint S..iv::2304011933 2023b.
E. T. Jaynes. Information theory and statistical mechanics. Physical Review, 106(4):620-630, May 1957. Reprinted from The Physical Review, Vol. 106, No. 4, May 15, 1957. Received September 4, 1956; revised March 4, 1957.
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation vision transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1060-1068, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 1454, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint (tt.::20040069/ 2021.
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compression to improving generalization. arXiv preprint arXiv:2105.12000 2021.
Zhu Liao, Victor Quétu, Van-Tam Nguyen, and Enzo Tartaglione. Nepenthe: Entropy-based pruning as a neural network depth's 's reducer. arXiv preprint arXiv:2404. 2024.
Shuxia Lin, Miaosen Zhang, Ruiming Chen, Xu Yang, Qiufeng Wang, and Xin Geng. Linearly decomposing and recomposing vision transformers for diverse-scale models. Advances Neural Information Processing Systems, 37:33188 011333188-331111 2221322, 2024.
Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint XXX::2402.00953, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. J 1692, 2019.
Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Ron- grong Ji. Towards efficient visual adaption via structural ee-paammetrrizatonn arXiv preprint arXiv:2302.08106, 2023.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1))654, 2024.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2/06.04489 2021.
Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution. arXiv preprint 13X. 2024.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.029) 02948, 2024.
12

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02785) 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536 2025.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning, arXiv preprint arXiv:200S.00247 2020.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10); 1872-1897, 2020.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu. Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, (6:79320-793666 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning. arXiv preprint arXiv:2402.10466, 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1rXiv:1606.05250 2016.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameter- ization. arXiv preprint arXiv:2305.03937, 2023.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter effi- ciency of lora with weight tying. arXiv preprint 1.XXi:23111095777 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine- tuning. Advances in Neural Information Processing Systems (NeurIPS), 53:20378-20332 2020.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint arXiv: 09728, 2019.
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. Commonsense reasoning for natural language processing. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 27-33, 2020.
Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27: 379-423, 623-656, 1948. Reprinted with corrections, July and October 1948.
Chongjie Si, Xuehui Wang. Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xi- aokang Yang. and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405. 14739, 2024.
13

---

Published as a conference paper at ICLR 2026
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631 (631-1642, 2013.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprini arXiv:2210.07558, 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint (Eti::8804.07460 2018.
Junjie Wang, Guangjing Yang, Wentao Chen, Huahui Yi, Xiaohu Wu, Zhouchen Lin, and Qicheng
Lao. Mlae: Masked Iora experts for visual parameter-efficient fine-tuning. 2024. URL https:
//arxivv.../////////// /aw.xiv.ogggaaaaeaa....aaaaaaaaa www.oorgggbbbaboeee http:/aas/224...2.2 18897
Yaqing Wang, Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Fred- crick Liu, Yi Liang, Bo Pang, Michael Bendersky, et al. Non-intrusive adaptation: Input- centric parameter-efficient fine-tuning for versatile multimodal modeling. arXiv preprint arXiv:2310.12100 2023.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Gfain-of-thhoulle prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2-24824-24887, 2022.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: hat:::770.05444 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface ' transformers: State-of- the-art natural language processing. 2020. URL
Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. Mixture-of-subspaces in low-rank adapta- tion. arXiv preprint 240iv:2406. 1909, 2024.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng. Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.00 056629, 2025.
Yi Xin, Siqi Luo, Xuyang Liu, Haodi Zhou, Xinyu Cheng, Christina E Lee, Junlong Du, Haozhe Wang, MingCai Chen, Ting Liu, al. V-petl bench: A unified visual parameter-efficient transfer learning benchmark. Advances in neural information processing systems, 37:80522 1..80522-80535.. 2024.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efffcient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint 2rXiv:2312.121121 2023.
Xu Yang, Yongliang Wu, Mingzhuo Yang. Haokun Chen, and Xin Geng. Exploring diverse in- context configurations for image captioning. Advances in Neural Information Processing Systems, 36:40924 40943, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1mXiv:1905.0784 2019.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, and et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arxiv: 048677,,.7 2019.
14

---

Published as a conference paper at ICLR 2026
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiee eeffiiierre fine-tuning. In The Eleventh Inter- national Conference Learning Representations, 2022a.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. International Conference on Machine Learning (ICML), PMLR, pp. 26809-26823, 2022b.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303. 105/2, 2023.
Ruiyi Zhang, Rushi Qiang. Sai Ashish Somayajula, and Pengtao Xie. Autolora: Automatically tun- ing matrix ranks in low-rank adaptation based on meta learning. arXiv preprint arXiv:2403.091 2024.
A THE USE OF LLM
We used large language model (ChatGPT) as a general-purpost writing assistant. role was limited the following:
Language polishing: Refining grammar, improving clarity, and ensuring consistent demic style in certain parts of the paper.
Formatting suggestions: Providing alternative sentence structures or paragraph organiza- tion for better readability.
No part of the research design, methodology, experiments, analysis was generated by the language model. All technical content, scientific claims, and conclusions are the sole work and responsibility of the authors.
The use of LLMs did not rise to the level of substantive contribution that would merit authorship.
B NORMALIZATION OF SPECTRAL ENTROPY
Recall the spectral entropy for normalized squared singular values
8;
X
0,
1,
==
is given by
H(s)
Σ
8 log 8):
defined on the probability simplex A r- {s E 00 Σ, = 1}.
Maximize (s) subject to E. = Form the Lagrangian
C(s, p)) = Σ
8; log 8; +
Σ 81 s
1=1
1=1
Taking partial derivatives and setting them to zero gives, for each i,
ac
(log + -
log 8; 1 1.
3s;
Hence e# is constant for all i. Using the constraint E; 8i = yields
en- 1
re
r
15

---

Published as a conference paper at ICLR 2026
and therefore
8; =
for all
Substituting the uniform distribution into gives
Hmax HHH (
4)
log 1
logr.
i=1
To show this stationary point is the global maximum, note that (s) is strictly concave A
Indeed, the Hessian matrix of H (with respect to the coordinates s,) is diagonal:
H 0s
1
HH
(i+
0s,0s;
For any nonzero vector x E RR,
x
S
because 8/ > O. Thus H is negative definite the domain, implying H is strictly concave. A strictly concave function on a convex set has at most one stationary point and that stationary point is the global maximum. Therefore the uniform distribution s; 1/r yields the unique global maximum and mmax logr.
Dividing (8) by logr yields normalized score
H(A)
I(A) logr
10,
with / = 1 at the uniform distribution and + when the distribution concentrates its mass on single component.
C
MONOTONICITY OF IMPORTANCE FOR A SINGLE SINGULAR VALUE
Given matrix with singular values λ A...., , Ay, recall that the spectral importance of the i-th singular value is
X I(A) Σ
8; Σ 2j=1
s, log(s; 8; + +c),
(8)
i=1
where is small constant to avoid numerical issues.
To see that 7(x) is monotonic in X; consider as function of A;:
Øs: 24, j
for A >
a^;
X3)2
Since -8; log 8; is an increasing function for small E (0, 1), we have that decreasing A decreases 8t which turn decreases its contribution to the total entropy 1 (AA).
Hence, the smallest singular value Amin always corresponds to the smallest importance I 1(2inn) justifying the pruning strategy in Sec
D DETAILS ON EXPERIMENTS
D. DETAILS ON NATURAL LANGUAGE UNDERSTANDING TASK
For the natural language understanding (NLU) experiments, adopt the General Language Un- derstanding Evaluation (GLUE) Wang et all (2018) benchmark. a suite of tasks designed to assess model's broad linguistic competence. GLUE comprises two single-sentence classification tasks,
16

---

Published as a conference paper at ICLR 2026
CoLA Warstadt et al. (2019) and SST-2 Socher et al. (2013). three tasks focused semantic sim-
ilarity and paraphrase detection, MRPC Dolan & Brockett (2005), QQP Wang et al. (2018). and
STS-B Cer et al. (2017), and three natural language inference tasks, MNLLWWlliae
et al. (2017).
QNLI Rajpurkar et al. 2016). and RTE Dagan et al. (2005); BBa--Haim et al. (2006); Giampiccolo
al.
2007): Bentivogli et al. (2009). The details of these datasets are shown Table
For this benchmark, we fine-tune DeBERTaV3-bass He et al. 2021b models. The settings for this task is shown in Table 8
Table 7: Details of GLUE dataset. Combined Score for MRPC and QQP is defined the average of Accuracy and Fl, while for STS-B is the average of Pearson and Spearman correlations
Dataset
Task
# Train
# Dev
# Test
Label
Metrics
Single-Sentence Classification
CoLA
Acceptability
8.5k
lk
lk
2
Matthews corr
SST-2
Sentiment
67k
872
1.8k
2
Accuracy
Similarity and Paraphrase
MRPC
Paraphrase
3.7k
408
1.7k
2
Combined Score
QQP
Paraphrase
364k
40k
391k
2
Combined Score
STS-B
Similarity
7k
1.5k
1.4k
Combined Score
Natural Language Inference
MNLI
NLI
393k
20k
20k
3
Accuracy
QNLI
QA/LLI
108k
5.7k
5.7k
2
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
Table 8: Hyper-parameter settings of FlexLoRA NLU task.
Hyper-parameter
CoLA
MNLI
MRPC
RTE
QNLI
SST-2 STS-B
QQP
Optimizer
AdamW
Warmup Ratio
0.1
LR schedule
Linear
Rank
8
16
b
4
Max Seq. Len.
64
256
256
512
256
256
256
320
Batch Size
32
32
32
32
32
32
32
32
Learning Rate
8e-4
5e-4
Ie-3
1.2e-3
5e-4
8e-4
2.2e-3
8e-4
Epochs
20
12
30
50
5
20
20
5
T Twormap
1000
5000
500
500
1000
5000
1000
5000
Trina
1000
5000
500
500
1000
5000
1000
5000
AT
200
1000
100
100
200
1000
200
1000
17

---

Published as a conference paper at ICLR 2026
D.2 DETAILS ON COMMONSENSE REASONING TASK
The commonsense reasoning benchmark suite comprises eight sub-tasks, each associated with specific dataset: BoolQ Clark et al. (2019) PIQA Bisk et al. (2020). SIQA et (2019). HellaS. 7ellers et al. (2019), WinoG. (Sakaguchi et al. (2021). ARC-e/ARC-e Clark et al. 2018), OBQA Mihaylov et al. (2018). Following the protocol described in Huu et al. (2023b), we aggregate the training portions of all tasks into unified corpus, referred to as the Commonsensel 170K dataset, and then evaluate performance separately on each task's ' test set. The hyper-parameter settings of
FlexLoRA are shown in Table. 9
We fine-tune LLaMA3-8B Meta (024)) on this task. For comparison, we also include re- sults from ChatGPT's implementation with the gpt-3.5-turbo API, particularly focusing on zero- shot Chain of Thought approaches Wei et al. (2022). The results of fully fine-tuning(Full FT) and
ChatGPT are cited from Liu et al. (2024).
Table 9: Hyper-parameter settings of FlexLoRA on commonsense reasoning task.
Hyper-parameters
LLaMA3-8B
Hyper-parameters
LLaMA3-8B
Rank r
& 32
Q
16
Learning Rate
3e-4
LR Scheduler
Linear
Dropout
0.05
Optimizer
A
Batch size
16
Warmup Steps
100
Epochs
b
T warmmup
5000
Timal
5000
AT
1000
Where
Q, K, V, Up, Down
D.3 DETAILS ON VISUAL. TASK
As shown in Table 10 the VTAB-IK dataset (Zhai et al. 2019) allocates 800 samples for training and 200 for validation during and evaluated the official test set.
tuning. The final model trained on all 1,000 samples
For this benchmark, we fine-tune ViT-B/16 Dosoviiskiy 2020) models. The results of fully fine- tuning(Full FT) are cited from Jie & Deng (2023). The typer-parameter settings for this task is shown in Table [I
Table 10: Details of VTAB-IK dataset.
Natural
Specialized
Struetured
Cide
Cálta
Flverinz
SVIN
Canelyon
Retaooottt
Clev-Onatt
Clevs Dit
d8pe Loc
uSpeOOi
ANORDS Ale
D7D
Dataaci
Classes
102
397
80070000
200
Texi
10000
6084
1890
6149
3669
26002
21750
52768
5400
6300
42670
15000
15000
22735
711
73728
73728
121.0
12150
E COMPARISON OF TRAINING COST
In this part, report the sestem-level metrics, including GPU memory and train runtime. As shown in Table 14 the results consistently show that FlexLoRA does not introduce noticeable overhead compared with AdaLoRA across all benchmarks. GPU memory usage remains nearly
18

---

Published as a conference paper at ICLR 2026
Table 11: Hyper-parameter settings of FlexLoRA visual task.
Optimizer
batch size
Learning Rate
Epochs
b
Twarmup
Tyinal
AT
A
32
le-3
100
4
500
500
100
identical across all settings, indicating that the rank reallocation strategy in FlexLoRA brings small computational cost.
As shown in Table [2 for NLU tasks (CoLA, RTE), FlexLoRA matches AdaLoRA in both runtime and memory while outperforming it in evaluation speed, and it incurs only small runtime differences compared with LoRA. As shown in Table 13 FlexLoRAA even reduces memory usage and shortens training time relative to Adal LoRA Table [4] for visual tasks (Cifar100, Resisc45, DMLab), FlexLoRA again shows almost the same training cost as AdaLoRA, with runtime differences typically under I 1%, while maintaining stable FLOP and memory footprints.
commonsense reasoning with LLaMA3-8B. As shown in
Table 12: Comparison of training cost on NLU tasks.
Dataset
Metric
FlcxLoRA
AdaLoRA
LoRA
Train Runtime (s)
2162.6 17446
2167.9 17487
2117.9
GPU Mem (MB)
17443
CoLA
Total FLOPs
4.88
10
4.90 10
4.88 10
Eval Runtime (s)
9.3231
9.2842
10.4523
Train Runtime (s)
1597.4
1574.9
1534.5
GPU Mem (MB)
17446
17487
17443
RTE
Total FLOPs
3.55 x 10
3.56 x 10
3.55 x 10
Eval Runtime (s)
2.4398
2.3954
2.6691
Table 13: Comparison of training cost on commonsense reasoning tasks.
Model
Metric
FlexLoRA
AdaLoRA
LoRA
LoRA-Dash
Train Runtime (h)
6.01 38.0
6.47
4.76
6.77
LLaMA3-8B
GPU Mem (GB)
43.3
37.6
51.8
F SINGULAR VALUE DISTRIBUTION
To further examine the rank allocation behavior, we randomly sampled ten LoRA modules trained on the CoLA task and visualized their singular value distributions. As shown the Figure 3 we can notice that no matrix with small magnitude and high entropy that has obtained large number of rank values, indicating that the magnitude of the singular values has only a minor effect on our rank allocation process, supporting the robustness of our adaptive strategy.
G MORE STUDY IMPORTANCE METRICS
To further validate the robustness of the spectral entropy metric and investigate the necessity of incorporating spectral energy into the importance metric, we designed and evaluated two hybrid variants that combine energy with entropy:
19

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of training cost on visual tasks.
Dataset
Metric
FleXLoRA
AdaLoRA
LoRA
Train Runtime (s)
650
654
581
GPU Mem (MB)
8414
8467
7787
C0far100
Total FLOPs
5.62 x 10
5.64 30.22
10
5.63 x 10 27.18
Eval Runtime (s)
29.94
Train Runtime (s)
646
647
566
GPU Mem (MB)
8439
8449
7785
Resisc45
Total FLOPs
5.62 x 10
5.64
10 19.78
5.63 x 10 17.82
Eval Runtime (s)
19.74
Train Runtime (s)
695
699
774
GPU Mem (MB)
8451
8465
7785
DMLab
Total FLOPs Eval Runtime (s)
5.62 x 10 67.06
5.64 10
5.63 x 10
66.98
59.91
Singulor Value Distribution of 10 Random LoRA Modules
5300
Ca
3175
117
313
110
5:6
5.4
Loarrd
175
322
315
013
4.00
Indes
Figure 3: Singular value distributions of ten randomly sampled LoRA modules trained on the CoLA task.
Elem-wise Energy Entropy (Jgtem) ) This variant calculates entropy at the element level within the singular value distribution, defined as:
X? + +), 8; = Σ, X3
Etem r rlogr
Σ A1s; log(s:
(10)
Matrix-wise Energy Entropy (TMar): This variant aggregates the singular values before computing the catropy-based importance:
1
st (Ixx st
log(s: +5) + 8;
X²
Ma
(1I)
rlogr
As shown in Table [5] neither variant outperforms the original FleA, reinforcing that our spec- tral energy entropy more reliable metric for measuring matrix importance.
20

---

Published as a conference paper at ICLR 2026
Table 15: Comparison of FlexLoRR and energy-considering variants NLU tasks using DeBERTaV3-base
CoLA Mcc.
MNLI
MRPC Acc.
RTE
QNLI
SST-2
STS-B Corr.
QQP
Method
Params.
Avg.
Acc.
Acc.
Acc.
Acc.
Acc.
Full FT
184.3M
69.2
89,9
90.2
83.8
94.0
95.6
91.6
92.4
88.3
Elem
1.9M
68.1
89.1
91.0
86.3
94.5
94.7
90.9
87.4
87.8
Mat
1.9M
69.0
89.5
89.3
83.8
94.6
95.5
91.8
87.2
87.6
FlexLoRA
1.9M
71.8
90.0
90.9
88.8
94.2
95.2
91.5
90.3
89.1
H
MORE EXPERIMENTS ON VISUAL TASK
To further verify the effectiveness of our method on visual tasks, we conducted experiments follow- ing the settings in Xin et al. demonstrate that FlexLoRA achieves superior average accuracy of 76.0%, outperforming LoRA (74.5%) and AdaLoRA (75.1% under comparable parameter budgets. These findings confirm that FlcxLoRA's entropy- entropy-guided rank allocation continually maintains consistent efficacy and robustness when extended to visual tasks.
rate = 2e-3, weight decay = le-3). Results in Table T6
Table 16: Results on VTAB benchmark with the experimental settings Xin et al. (2024). Accuracy (%) across Natural, Specialized, and Structured domains.
Natural
Specialized
Stractured
NNO00 Axim
Cilex
Caloebh0)
SWHNN
Cands
Rosto45
Rotnspaky
PME
KITT--W Die
degr
4NORB-ES
Method
Param./M)
Ang.
25.7
29.1
63.2
33.2
VPT-Deep
65.8
54.3
75.5
29.9
AdapiFormer
70.5
54.8
$4.4
76.3
49.3
LaRA
70.5
75.8
53.4
51.8
MLAE
0.34
65.7
90.7
70.4
99.1
91.1
85.4
53.9
82.4
04.
86
74.9
77.5
64.5
50.2
70.9
75.1
43.5
27.7
35.8
73.6
PISSA
0.31
69.4
92.3
72.4
99.0
91.3
89.6
54.6
86.7
953
56.4
75.7
61.8
63.9
54.1
90.4
61.0
45.0
29.8
41.4
75.8
AdiLAAA
0.37
70.5
90.3
71.9
99.1
91.4
57
56
83.9
95.3
55.6
75.2
$1.5
68.1
51.8
90.2
71.3
49.1
25.9
41.1
75.1
DERA
0.34
70.0
91.9
71.3
99
90.0
85.7
55.5
85.4
95.5
56.8
76.5
82.5
66.5
53.0
90.3
82.5
46.5
30.1
41.9
75.8
NOAH
0.36
69.6
92.7
70.2
99.1
90.4
86.1
53.7
84.4
95.4
835
75.
82.5
68.9
49.9
$1.7
81.8
43.5
32.8
44.2
75.5
PocT
0.07
70.6
90.6
70.8
99.1
90.7
88.6
54.1
88.8
96.2
84.5
75.7
82.5
68.2
29.8
90.7
90.8
67.4
53.2
43.0
75.6
SSD
0.24
89.0
92.6
75.1
99.6
91.8
90.2
52.9
87.
95.9
87.4
75.5
75.9
62.3
53.5
80.6
77.3
54.9
29.5
37.9
73.7
FEXLORA
0.34
71.2
92.5
72.1
99.1
90.7
90.6
54.7
364
96.0
97.5
76
51.9
55.5
53.9
90.2
90.6
45.6
29.9
43.1
76.0
21


Reviews:
[{"review_id": "cC993QprnB", "reviewer": "Reviewer_uSv1", "summary": "This paper introduces an entropy-based metric to estimate the importance score of each weight-update matrix and proposes FlexLoRA, a method that dynamically adjusts the rank of LoRA modules by reducing the rank of those with lower importance scores while increasing that of the more important ones.", "questions": "Since the framework adjusts the rank dynamically during training, it would be valuable to report how this process affects the overall training time and computational cost.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The framework is clear and intuitive. The authors conduct many experiments on GLUE, commonsense reasoning, and the Visual Task Adaptation Benchmarks, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The paper omits key system-level metrics such as peak GPU memory usage and FLOPs, which are essential for assessing efficiency.\n2. The proposed entropy-guided importance metric is closely related to the effective rank concept from [1], limiting the novelty of the contribution.\n3. The reported performance gains are marginal, particularly for large-scale models like LLaMA3-8B. Moreover, in Table 2, the number of trainable parameters in AdaLoRA is only about half that of the proposed model, making the comparison appear unfair.\n\n\n[1] Olivier Roy and Martin Vetterli, The effective rank: A measure of effective dimensionality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k3BhQPvhXx", "reviewer": "Reviewer_Z3z8", "summary": "The paper introduces FlexLoRA, a method for dynamically adjusting LoRA rank across layers using a matrix-level spectral entropy score. It periodically prunes low-entropy matrices and expands high-entropy ones while keeping a global parameter budget. New directions are added with zero-impact initialization, so the model output is not disturbed at the moment of expansion. Experiments on NLP and vision benchmarks show consistent, if sometimes modest, gains over standard LoRA and prior adaptive methods. Ablations suggest that the entropy score, bidirectional allocation, and the initialization choice each contribute to the improvements.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A simple, matrix-level entropy score guides where to prune and where to add rank, avoiding noisy element-wise heuristics.\n2. True bidirectional allocation with zero-impact initialization lets the model expand capacity safely while staying within a budget.\n3. Strong, cross-domain results (NLP and vision) with clear ablations show each component—entropy, bidirectionality, initialization—adds value.", "weaknesses": "1. Entropy ignores magnitude (energy).\nThe spectral‑entropy score is scale‑invariant: a matrix with very small Λ but uniform spread can have high entropy and thus be prioritized for expansion, even if its absolute contribution is negligible. The paper briefly compares against Frobenius/nuclear norms (Table 4), but does not explore combined criteria (e.g., entropy × energy) or energy‑gated expansion. This is a conceptual gap given the stated goal of measuring “importance.”\n2. Baseline anomalies & reporting details.\nIn Table 1, LoRA’s MRPC (68.4 Acc) and QQP (63.1 Acc) are unusually low for modern setups, reducing the average (81.7) and potentially inflating relative gains. GLUE typically reports MRPC/QQP F1 scores alongside, or instead of, accuracy. Please clarify the metrics, seeds, and hyperparameters per task, and include mean ± std across multiple seeds.\n3. Cost/overhead not quantified.\nMaintaining orthogonality (Eq. 2) and dynamically changing parameter shapes may incur compute/memory overhead vs. standard LoRA. No wall‑clock or throughput numbers are provided, especially on large LLM runs.\n4. Limited theoretical justification linking entropy to “importance.”\nAppendix B argues that the least singular value contributes least to entropy under certain conditions, which supports the pruning rule, but there’s no theoretical link to downstream loss reduction or generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRphQkXtZP", "reviewer": "Reviewer_SzUu", "summary": "In this work, the authors propose to evolve the rank of low-rank adapters using entropic regularization on the singular value set. The framework allows both to reduce and to inflate the rank of the adapters, enabling dynamic reallocation across layers.\nThe proposed method is backed up by a variety of different numerical results.", "questions": "1. From what I understood and from the numerical experiments, the proposed method should not be able to significantly reduce the number of trainable parameters. Is this correct?\n\nFor the rest, I would appreciate a comment from the authors about the \"weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The work is well written and clear. The proposed method, despite being very simple, seems to be effective against a variety of different baselines.", "weaknesses": "In the presentation of the algorithm, it is not clear which metric is precisely used to rank the importance of the single matrices. I would suggest that the authors include a more precise discussion about this in the revised version.\n\nThe proposed method is not able to significantly reduce the number of trainable parameters with respect to other methods like AdaLoRA or GeoLoRA [1]. It is also not clear how the global budget is allocated and kept under the maximal one, as the effect of inflating or not in terms of parameters depends on the layer one is considering. \n\n[1] S. Schotthoefer et al., GeoLoRA: Geometric integration for parameter-efficient fine-tuning, ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5n7L8SzAl4", "reviewer": "Reviewer_eq8b", "summary": "This paper proposes FlexLoRA, an entropy-guided dynamic low-rank adaptation method for parameter-efficient fine-tuning of large models. Unlike standard LoRA with fixed rank, FlexLoRA supports both rank pruning and expansion under a global parameter budget. It evaluates matrix importance using spectral entropy, enabling flexible capacity reallocation across layers. Additionally, a zero-impact initialization strategy ensures stable training when adding new rank directions. Experiments show that FlexLoRA consistently outperforms existing dynamic LoRA variants under the same parameter budget.", "questions": "Additional Questions and Suggestions:\n\n1. Prior study [1] have shown that randomly masking certain parameters within the LoRA matrices can effectively reduce redundancy and improve performance. This raises an important question: how does such a temporary stochastic masking approach compare to the permanent structural adjustments (i.e., pruning or expansion) proposed in this paper? An experimental comparison between random masking and FlexLoRA’s bidirectional rank adjustment would help clarify the relative advantages and trade-offs in terms of performance, stability, and generalization.\n2. The paper does not discuss the computational efficiency of FlexLoRA. Since the proposed method introduces additional operations for rank evaluation, pruning, and expansion, it likely incurs extra computational overhead. Given that the performance gains of FlexLoRA appear modest, it is crucial to report and compare the training and inference time costs across different methods to fairly assess their overall efficiency.\n\n------\nIf the authors can address the above concerns in response, I am willing to raise my score. \n\n[1] MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning. https://arxiv.org/abs/2405.18897", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses the key limitation of LoRA’s fixed rank by introducing a novel bidirectional rank adjustment mechanism that supports both pruning and expansion under a global budget. It also proposes a new matrix-level entropy-guided importance metric, offering a new perspective on evaluating low-rank matrices importance.\n2. The paper presents comprehensive experiments across multiple domains, including Natural Language Understanding, Visual Recognition, and Commonsense Reasoning, along with extensive ablation studies.", "weaknesses": "1. Although the paper introduces a bidirectional rank adjustment mechanism supporting both pruning and expansion, the idea — if we ignore the proposed importance metric — conceptually resembles a combination of AdaLoRA (for adaptive pruning) and IncreLoRA [1] (for rank expansion). Hence, its novelty may be somewhat limited at the framework level.\n2. Despite the conceptual soundness, the experimental credibility of the paper is questionable.\n   - The tables do not clearly indicate which results are reproduced and which are cited from previous works, making it difficult to assess fairness. Moreover, no detailed experimental settings are provided in either the main text or the appendix, hindering reproducibility and leaving uncertainty about how baseline methods were implemented.\n   - The improvements on commonsense reasoning tasks are marginal (e.g., LoRA 85.4% vs. FlexLoRA 85.5%), and given the high variance typically observed in such benchmarks, this raises doubts about the claimed effectiveness.\n   - For visual recognition, the reported performance appears lower than that of prior works such as MLAE [2] and GLoRA [3], yet the paper provides no explanation or comparison of configurations, leaving potential inconsistencies unresolved.\n   - The ablation study is also incomplete — only a subset of GLUE benchmarks (CoLA, MRPC, RTE, STS-B) is reported instead of the full suite, which may indicate selective reporting of favorable results.\n\n------\n\n[1] IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning. https://arxiv.org/abs/2308.12043\n\n[2] MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning. https://arxiv.org/abs/2405.18897\n\n[3] One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning. https://arxiv.org/abs/2306.07967", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)