Title: FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Venue: NeurIPS 2025
Accepted: yes
Keywords: low-rank adaptation

fly olfactory circuit

parameter-efficient fine-tuning

Mixture-of-Expert

model merging
Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in mitigating intra-task correlations in single-task instruction tuning, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where a frozen sparse random projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter-task interference due to the orthogonality property of random matrices. Extensive experiments across four domains---general knowledge understanding, scientific question answering, mathematical reasoning, and code generation---demonstrate consistent performance improvements over existing methods. Beyond empirical gains, FlyLoRA highlights how biological structures can inspire innovations in AI technologies. Code is available at https://github.com/gfyddha/FlyLoRA.


Peer reviews:
comments: SUMMARY:
This paper introduces FlyLoRA, a novel PEFT) method inspired by the fly olfactory circuit. FlyLoRA addresses the limitations of existing LoRA methods, specifically parameter interference and computational inefficiency, through implicit rank-wise expert activation and frozen sparse random down-projections. The method theoretically decouples intra- and inter-task interactions, leading to enhanced decorrelation in both single-task instruction tuning and multi-task model fusion. Experimental results demonstrate consistent performance improvements over existing PEFT methods and significantly reduced performance degradation in multi-task merging scenarios.

STRENGTHS AND WEAKNESSES:
Strengths:
1. FlyLoRA proposes a unique approach to PEFT by incorporating implicit rank-wise expert activation and frozen sparse random projections, inspired by biological neural structures, which is a compelling and innovative design.
2. By eliminating the need for explicit router parameters and leveraging inherent sparsity, FlyLoRA offers improved computational efficiency compared to other MoE-based LoRA variants.

Weaknesses:
1. The paper does not sufficiently explain why there is such a significant difference in results between merged and non-merged scenarios. A deeper analysis of this discrepancy would enhance the understanding of FlyLoRA's behavior in diverse contexts.
2. The evaluation results, while promising, are not comprehensive enough. The paper would benefit from testing on more challenging and common benchmarks such as GPQA, LiveCodeBench, and advanced Math reasoning datasets, to thoroughly assess FlyLoRA's robustness and scalability.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FlyLoRA, a novel LoRA variant inspired by the fly olfactory circuit, which improves parameter efficiency and task decoupling in both single-task and multi-task settings. The key innovations include: (1) using a frozen sparse random projection matrix $A$ to replace the conventional trainable down-projection, and (2) performing rank-wise top-k expert activation in the up-projection matrix $B$, eliminating the need for an explicit router. Theoretical analysis shows that the fixed sparse matrix preserves input distances and induces near-orthogonal subspaces, which mitigates parameter interference across tasks. Empirical results across several benchmarks using two backbones show FlyLoRA achieves strong performance with lower memory and compute overhead.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n* Quality: The paper presents a well-grounded method with both strong theoretical justifications (distance preservation, covariance reduction, orthogonality) and extensive empirical validation.\n* Clarity: The paper is well-written with clear motivation, architecture explanation, and figures. The supplementary materials provide thorough theoretical proofs and ablation studies.\n* Significance: FlyLoRA addresses a core limitation in existing LoRA and MoE-LoRA methods—parameter interference while improving computational efficiency. The method has broad applicability in fine-tuning large models under resource constraints.\n* Originality: The idea of combining sparse random projections with rank-wise MoE and top-k implicit routing, inspired by neuroscience, is novel and elegant. The routing-free design simplifies architecture while retaining strong performance.\n\n**Weaknesses**:\n\n* The initialization of the sparse random projection matrix $A$ uses a fixed Gaussian scheme. It remains unclear how sensitive the method is to this choice. Could other distributions or learned schemes improve or worsen performance?\n* Evaluation on larger backbone models is missing. It is unclear how FlyLoRA scales with increasing model sizes beyond 8B.\n* Only one MoE-LoRA baseline (Split-LoRA) is included. More recent or stronger adaptive/sparse fine-tuning baselines could strengthen the comparison.\n* In Table 2, performance of FlyLoRA + Llama-3.1-8B drops sharply on ScienceQA and GSM8K after merging. A deeper analysis of these drops would be useful.\n* The ablation in Table 3 shows marginal difference between ST+Trn and ST+Frz. It's unclear whether ST+Trn might outperform ST+Frz on other benchmarks.\n\nQUESTIONS:\n- Can the authors discuss the sensitivity of the frozen matrix $A$ to different initialization schemes? Could non-Gaussian or structured projections affect performance?\n- How does FlyLoRA perform on significantly larger models (> 8B)? Are there bottlenecks in memory or convergence?\n- Can more advanced MoE-based LoRA baselines be included for comparison?\n- What causes the large performance drop after merging FlyLoRA on ScienceQA & GSM8K tasks?\n- Could ST+Trn outperform ST+Frz on some tasks, and if so, what trade-offs would it bring in merging or stability?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FlyLoRA, a parameter-efficient fine-tuning method inspired by the fly olfactory circuit. It uses a frozen sparse projection for routing and rank-wise expert activation to reduce interference. This design improves both intra-task and inter-task decoupling, enabling efficient single-task learning and robust multi-task model merging. FlyLoRA outperforms standard and MoE-based LoRA methods across multiple domains with fewer trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n**Quality**: The paper is overall technically sound. Most empirical claims are well supported by experiments. The methods used are well-motivated by both biological inspirations, classic random matrix theories and very practical concerns for the trade-off between MoE-based LoRA performance and efficiency. The authors mentioned the weaknesses in the appendix and addressed it properly. \n\n**Clarity**: The submission is clearly written and well-organized and based on the writing it should be easy to reproduce based on the figure visualization and equation 11. The authors are able to guide readers through the motivation to propose FlyLoRA step by step from section 2 to 3. \n\n**Significance**: The results will be very impactful: first the FlyLoRA shows better generalization performance with lower parameters; second FlyLoRA shows better performance also on PEFT model merging, and both first and second topic are very widely-used techniques; third all theoretical analysis can be proved by empirical advantages.\n\n**Originality**: To my best knowledge the formulation of FlyLoRA is new, although some parts of the theoretical analysis framework are similar to Johnson–Lindenstrauss analysis. This is not a weakness though, as I appreciate authors who show how to apply these frameworks to the setup of FlyLoRA. Theorem 3.2 is also insightful to explain the success of TopK sparsity in other applications.\n\n## Weaknesses\n**Quality**: I have a few important clarification questions for the theoretical section. Please see Questions for details. Besides, the model merging part would be strengthened by more comprehensive LoRA merging baselines.\n\nQUESTIONS:\n1. Corollary 3.4: How can the additive norm preservation property benefit model merging? The orthogonality part is intuitive.\n2. Why can we apply Theorem A.1 to Theorem A.2 where in the statement of A.1, $\\mathbf{A}$ is a random matrix sampled iid from standard normal, but in A.2 the matrix $\\mathbf{A}$ has exactly $p$ non-zero entries as how you construct the matrix in FlyLoRA.  \n3. Can you explain why Theorem 3.1 extends to Top-k activation from Equation (2) in the Appendix? \n4. For A.3, it’s better to move the zero-mean gradient assumption (line 36) above and make it explicit in the theorem statement.\n5. In Table 2, the current baselines may be too weak because there are many papers focusing on reducing interference of LoRA for merging but split-LoRA is designed to improve single task PEFT performance. For example, you can try to compare with [1,2].\n\n[1] Stoica, George, et al. \"Model merging with SVD to tie the Knots.\" The Thirteenth International Conference on Learning Representations.\n\n[2] Tang, Anke, et al. \"Parameter-Efficient Multi-Task Model Fusion with Partial Linearization.\" The Twelfth International Conference on Learning Representations. 2023.\n\nLIMITATIONS:\nYes, in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper begins with the elegant observation that increasing number of experts while fixing total rank monotonically improves LoRA performance. However, existing routing mechanisms increase in parameter count linearly with number of experts, resulting in worse efficiency. To enable large numbers of low-rank experts as an efficient strategy, the authors introduce FlyLoRA, which removes the expensive routing mechanism in MoE-LoRA methods in favour of randomly initialised sparse (and provably near-orthogonal) random A projection, only learning the decoder B. This achieves empirically high performance on a variety of benchmarks against MoE- and standard LoRA, and has less degradation when merged.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper is elegantly characterised, explaining and empirically justifying each of the design decisions in FlyLoRA with experiments and theory. Overall I found the context well-presented and easy to understand.\n- The method consistently achieves large performance gains over baselines and also has less degradation when merged, while being more parameter-efficient than LoRA and MoE variants.\n- The ablations in table 3 and figure 4 show that the theoretical claims made by the authors seem to hold up empirically, particularly (which I found surprising) frozen near-orthogonal A seems actively helpful beyond the multi-task merging setting.\n- The biological justification is quite interesting!\n\nWeaknesses\n- None apparent.\n\nTypos\n- line 206: updateble\n\nQUESTIONS:\n- Previous PEFT methods have used orthogonality as a design decision with success even without MoE, e.g. [OFT](https://arxiv.org/abs/2311.06243), [ReFT](https://proceedings.neurips.cc/paper_files/paper/2024/hash/75008a0fba53bf13b0bb3b7bff986e0e-Abstract-Conference.html). Why do you think this is the case, and is it related to why FlyLoRA is successful (beyond just the merging setting)? If you enforced orthogonality on the A matrix, would that help performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}