Title: SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection
Primary area: optimization
Keywords: parameter-efficient-training, fine-tuning, LoRA, optimization

Abstract: The success of language models in real-world applications depends on parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which inserts compact, trainable adapters into selected layers. We propose SimplexLoRA, a novel LoRA variant that adaptively adjusts adapter ranks during fine-tuning based on their learned importance. Unlike standard LoRA, which uses a fixed rank for all layers, SimplexLoRA optimizes adapter weights on a probabilistic simplex, dynamically allocating capacity where it is most needed without increasing the parameter budget. We also introduce efficient algorithms for rank expansion and compression using QR decomposition and truncated SVD. Experiments on the GLUE benchmark with DeBERTaV3 and Llama 3 models show that SimplexLoRA matches or surpasses standard LoRA and recent adaptive methods, while significantly reducing the number of trainable parameters. Our code is available at https://anonymous.4open.science/r/SimplexLoRA-C142.