NeurIPS 2025

FlyLoRA: Boosting Task Decoupling and Parameter
Efficiency via Implicit Rank-Wise Mixture-of-Experts
Heming Zou1∗
Yunliang Zang2∗
Wutong Xu1
Yao Zhu1
Xiangyang Ji1†
1Department of Automation, Tsinghua University
2Academy of Medical Engineering and Translational Medicine, Tianjin University
{zouhm24, xwt22}@mails.tsinghua.edu.cn
yunliangzang@tju.edu.cn, ee_zhuy@zju.edu.cn
xyji@tsinghua.edu.cn
Abstract
Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning
method for foundation models, but it suffers from parameter interference, resulting
in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA vari-
ants show promise in mitigating intra-task correlations in single-task instruction tun-
ing, they introduce additional router parameters and remain ineffective in multi-task
model merging where inter-task interference arises. Inspired by the fly olfactory
circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces:
(1) rank-wise expert activation in the up-projection matrix, and (2) an implicit
router that unifies expert routing and down-projection, where a frozen sparse ran-
dom projection matrix replaces the traditional dense trainable version. This design
resolves the trade-off between intra-task decorrelation and computational efficiency
by eliminating the need for an explicit router, while inherently mitigating inter-
task interference due to the orthogonality property of random matrices. Extensive
experiments across four domains—general knowledge understanding, scientific
question answering, mathematical reasoning, and code generation—demonstrate
consistent performance improvements over existing methods. Beyond empirical
gains, FlyLoRA highlights how biological structures can inspire innovations in AI
technologies. Code is available at https://github.com/gfyddha/FlyLoRA.
1
Introduction
Foundation models have demonstrated remarkable cross-domain capabilities with the scaling of
model parameters [1, 4, 15, 43, 57...