Title: Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

LoRA

Fine-Tuning

Efficiency
Abstract: Large Language Models (LLMs) have yet to effectively leverage the vast amounts of edge-device data, and Federated Learning (FL) offers a promising paradigm to collaboratively fine-tune LLMs without transferring private edge data to the cloud. To operate within the computational and communication constraints of edge devices, recent literature on federated fine-tuning of LLMs proposes the use of low-rank adaptation (LoRA) and similar parameter-efficient methods. However, LoRA-based methods suffer from accuracy degradation in FL settings, primarily because of data and computational heterogeneity across clients. We propose Ravan, an adaptive multi-head LoRA method that balances parameter efficiency and model expressivity by reparameterizing the weight updates as the sum of multiple LoRA heads, $s_i\textbf{B}_i\textbf{H}_i\textbf{A}_i$, in which only the $\textbf{H}_i$ parameters and their lightweight scaling factors $s_i$ are trained. These trainable scaling factors let the optimization focus on the most useful heads, recovering a higher-rank approximation of the full update without increasing the number of communicated parameters since clients upload $s_i\textbf{H}_i$ directly. Experiments on vision and language benchmarks show that Ravan improves test accuracy by 2–8\% over prior parameter-efficient baselines, making it a robust and scalable solution for federated fine-tuning of LLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach to federated learning (FL) based on parameter efficient fine-tuning (PEFT) which works with computational and data heterogeneity while allowing for maximum effective rank with low communication overhead and exact aggregation across clients. Specifically, it proposes a multi-head variant of LoRA whereby the weight updates are factored as in LoRA-XS (i.e. \\Delta W = B H A) and each client updates a set of H matrices and their associated scalar scaling factors i.e. (\\Delta W = \\sum_{i=1}^{h}  s_i B_i H_i A_i), and the coefficients A_i and B_i are frozen. For a given parameter budget N, the method allows for a higher effective rank \\sqrt(N*h) compared to LoRA (N/2d). Also, it allows for exact aggregation across clients along with communication efficiency. Results on vision and language tasks show that the approach improves performance over other PEFT approaches (e.g. FedIT/Fed-SB). Ablations are shown to compare performance as a function of the mode of initialization of the B/A matrices, head selection strategies, computational heterogeneity, scaling factors, number of heads and scaling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Proposes a new PEFT method for FL which is able to tackle computational and data heterogeneity while allowing for a large effective rank.\n* Shows that the method outperforms various FL PEFT approaches on vision and language tasks.\n* Reports a number of ablations which show the impact of key factors such as initialization, scaling factors\n* Approach improves upon prior approaches which either have inexact aggregation (FedIT) or exact aggregation with high communication costs (FFA-LoRA).\n\n\nWeaknesses:\n* Results are reported on simulated data, where the heterogeneity has been introduced artificially (e.g. by drawing client specific class proportions based on a Dirichlet distribution with known parameters).  It is unclear if the method can capture heterogeneity in real datasets. It would be useful to report performance on alternative datasets used in FL research e.g. FedJAX (https://github.com/google/fedjax) contains prepackaged datasets such as EMNIST-62, Stack Overflow grouped by writers/username which would allow experimentation to study data heterogeneity.\n\nQUESTIONS:\n* Sec 4.3 / Figure 3: Computational Heterogeneity: Could you provide more details about how the clients choose  # of heads based on their computational capacity? Also, it would be useful to show the distribution of # of heads chosen across clients for each of the 3 distributions: Bell-shaped/Uniform/Skewed right\n* Effect of number of heads (Figure 4): It would be good to understand what happens when \\sqrt(N*h) exceeds d. Why does the effective rank of each head become smaller in this setup? Could you provide further insight?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a contribution to the field of Federated Learning with LLM models. A shortcoming of FL for LLMs is the high communication budget between clients and the server. Previous works in the literature, such as FedIT, proposed using LoRA to reduce this communication cost. This work refines that method, proposing a multi-head LoRA adaptation in the fashion of HydraLoRA. Here, however, the authors propose an augmented version si​Bi​Hi​Ai​ that allows for exact aggregation and increased expressivity. In this version, only si​ and Hi​ are trained and communicated, while Ai​ and Bi​ remain frozen. Experimental results advocate for good efficiency on a diverse set of architectures and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method is well-grounded and motivated, with a solid overview of related work. It is integrated into the broader context of federated learning and low-rank adaptation.\n- The method is evaluated on CIFAR-100, which is a challenging dataset in the context of federated learning.\n- The comparison to other methods in the same scenario is well-conducted, and the results are good for the proposed method.\n- RAVAN is easy to implement, which is a plus for practical applications.\n\nWeaknesses:\n\n- The claim \"To our knowledge, RAVAN is the first federated PEFT technique that simultaneously maintains parameter-efficient computation and communication and addresses both data and computational heterogeneity\" is too strong. The idea has been around for a long time, with works like FedPara and others. The authors should be more cautious with such claims.\n- Parts 3.1 and 3.2 of the paper raise questions about the underlying assumptions and motivations for the proposed strategies. For part 3.1: why does Gram-Schmidt perform better? Is there a theoretical justification? It would also be interesting to compare these results with a pre-trained initialization on the same domain using a random dataset.\n- The choice of 50 local epochs for client training is very problematic for comparing results with other works in the literature. In federated learning, it is common to see clients training for 1, 5, 10, or 20 epochs. Training for 50 epochs is almost equivalent to letting each client train their own model, which undermines the federated learning paradigm. It is absolutely necessary to perform new experiments with a more reasonable number of local epochs.\n- The paper mentions that the \"scaling factors\" are learned, but it is not clear how this learning process occurs. This aspect needs to be clarified for better understanding.\n\nQUESTIONS:\n- Could the authors provide more grounded explanations for the strategies proposed in parts 3.1 and 3.2? Specifically, why does Gram-Schmidt perform better, and is there a theoretical justification for this choice?\n\n- Can the authors perform new experiments with a more reasonable number of local epochs for client training? This would be necessary to position the results in the context of federated learning and make them comparable with other works in the literature.\n\n- How are the \"scaling factors\" learned in the proposed method? A clearer explanation of this process would be beneficial for understanding the overall approach.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RAVAN, a multi-head, low-rank adaptation method for the federated fine-tuning of large models. The proposed method addresses the challenges of data and computational heterogeneity in federated learning by reparameterizing weight updates as a sum of multiple augmented LoRA heads. By freezing the outer matrices and only training the small inner matrices and lightweight scaling factors, RAVAN aims to improve the expressive power of the model updates while maintaining parameter efficiency and ensuring mathematically exact aggregation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. **Exact Aggregation**: RAVAN's design provides an elegant solution to the inexact aggregation problem that affects many federated LoRA methods. By having clients only train and upload $s_iH_i$, the server can perform a direct, weighted average that is mathematically equivalent to averaging the full updates.\n2. **Communication and Computation Efficiency**: The method maintains the low communication cost of standard LoRA by only requiring clients to send the small, trained parameters. Furthermore, it directly addresses computational heterogeneity by allowing resource-constrained clients to train only a subset of the available heads, making the framework flexible and scalable in realistic FL environments. \n\nWeaknesses:\n\nOne of the paper's core argument is that RAVAN solves the non-I.I.D. issue, based on the observation that the spectrum of weight updates is broader in such settings.  While the empirical results are strong, this connection feels more like an intuition than a direct, proven solution. The observation in Figure 1 that $\\delta W$ has a more distributed rank does not necessarily mean that a low-rank approximation is the primary bottleneck for non-I.I.D. training. A more rigorous theoretical justification, perhaps related to convergence guarantees or generalization bounds, would be appreciated to formally connect the spectral properties of non-I.I.D. updates to the necessity of a higher-rank adaptation space.\n\nQUESTIONS:\nWhile RAVAN offers a high theoretical rank, does the optimization process for non-I.I.D. updates actually utilize this full capacity? I'm interested in an empirical analysis of the singular values of the learned update matrix which demonstrates how much of the available rank is used in practice. This would confirm if the increased rank is a fully leveraged resource in heterogeneous settings or primarily a theoretical upper bound.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}