Title: FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation
Primary area: foundation or frontier models, including LLMs
Keywords: PEFT; Dynamic Rank; LoRA

Abstract: Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs. 
Parameter-efficient fine-tuning (PEFT) has thus become a mainstream paradigm. 
Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility. 
Dynamic rank allocation methods mitigate this issue by pruning redundant directions; however, they often rely on heuristic, element-level metrics that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation. 
To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability. 
By addressing granularity, flexibility, and stability limitations, FlexLoRA provides a more principled solution for PEFT. 
Extensive experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks.