Title: DiHiRA: Diagonal High-Rank Adaption for Large Foundation Models
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: peft, lora

Abstract: Recent advances show that large pre-trained models excel on diverse downstream tasks, driving the popularity of parameter-efficient fine-tuning (PEFT) methods. Among them, Low-Rank Adaptation (LoRA) and its variants approximate weight updates with low-rank matrix products, achieving performance comparable to full fine-tuning while greatly reducing trainable parameters and memory cost. However, low-rank updates inherently constrain model capacity, whereas update weights derived from fully fine-tuning often produces nearly full-rank weight updates. To address this limitation, we propose DiHiRA, a simple yet highly efficient extension of LoRA. By augmenting low-rank updates with a learnable diagonal matrix, DiHiRA enables high-rank adaptation while retaining parameter efficiency. This design improves flexibility and strengthens transferability across diverse tasks. Extensive experiments demonstrate that DiHiRA achieves near full-rank adaptation with LoRA-level efficiency, consistently outperforming baselines on both computer vision and natural language processing benchmarks.