FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Authors: Heming Zou
Yunliang Zang
Wutong Xu
Yao Zhu
Xiangyang Ji
Venue: NeurIPS 2025 poster ()
Keywords: low-rank adaptation
fly olfactory circuit
parameter-efficient fine-tuning
Mixture-of-Expert
model merging
Primary area: deep_learning

Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in mitigating intra-task correlations in single-task instruction tuning, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where a frozen sparse random projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter-task interference due to the orthogonality property of random matrices. Extensive experiments across four domains---general knowledge understanding, scientific question answering, mathematical reasoning, and code generation---demonstrate consistent performance improvements over existing methods. Beyond empirical gains, FlyLoRA highlights how biological structures can inspire innovations in AI technologies. Code is available at https://github.com/gfyddha/FlyLoRA.

FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Heming Zou!#
Yunliang Zang²* Wutong Xu Yao Zhu!
Xiangyang Ji
'Department of Automation, Tsinghua University
2 AAcademy of Medical Engineering and Translational Medicine, Tianjin University {zouhm24, xwt22)@mailss tsinghua. edu. en yunliangzang@t edu. en, z_zhuuzzju. lhuy@zju.edu.cco edu.cn xyji@tsinghua. eeceh.aa.eeu....
Abstract
Low-Rank Adaptation (LoRA) is widely used parameter-efffcient fine-tuning method for foundation models, but suffers from parameter interference, resulting in suboptimal performance. Although ceature-of-Expereee (MoE)- -based LoRA vari- ants show promise in mitigating intra-task correlations in single-task instruction tun- ing, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, propose FlyLoRA, implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where frozen sparse ran- dom projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter- task interference due to the orthogonality property of random matrices. Extensive experiments across four domains general knowledge understanding, scientific question answering, mathematical reasoning, and code generation--dmmonsrraa demonstrate consistent performance improvements over existing methods. Beyond empirical gains, FlyLoRA highlights how biological structures can inspire innovations Al technologies. Code is available at attps://Et //ggttuu...... com/ com/gf gddhffyddh//////oooooo (aihaa/FFyLLR...
1 Introduction
Foundation models have demonstrated remarkable cross-domain capabilities with the scaling of model parameters [1, 4, 15, 43, 57, 63, 64]. To enhance their performance downstream tasks, Supervised Fine-Tuning (SFT) has become typical post-training approach. However, full-parameter fine-tuning (Full FT) incurs prohibitive computational overhead and storage costs, making customized deployment impractical for most individual users. To address this issue, Parameter- Errameter-Efficient Fine- Tuning (PEFT) [26, 27, 35, 40, 44, 47, 84, 90] has emerged significantly reduces resource consumption by keeping pre-trained weights frozen while fine-tuning only small set of additional injected parameters.
widely adopted technique that
Low-Rank Adaptation (LoRA) [27] is one of the most prominent PEFT methods. By leveraging the intrinsic low-dimensional properties of large language models [2, 36). LoRA approximates the parameter matrix update A AW e m as the product of two low-rank matrices, B e mx and A E xr where < min(m, nt). This method preserves much of the capability of Full FT across most tasks while substantially reducing both memory requirements and computational overhead.
"Equal contribution Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Winner-take-all (APL)
42.0
Active Trainable Params (96)
Accuracy (%)
0.85
Spit-LoRA LoRA (1=()
(f=32)
Split LoRA
LoRA
37.5
0:35
LORA
36.0
x
8 4
16 x
32 x1
8 x-4
16
x2
Expert Rank
Expert Rank
PNs
KCs
MBONs
(b)
(c)
Figure 1: (a) Accuracy comparison under fixed total rank = 32 and activation rank k = 8. Finer-grained rank allocation (from experts rank experts rank) yields consistent performance gains. (b) Activated trainable parameters (relative to Full FT) under the same budget. Increasing expert granularity leads to a monotonic rise in activated parameters due to router overhead. (c) Schematic of the fly olfactory circuit. Odor signals in projection neurons (PNs) are randomly projected to Kenyon cells (KCs), with each KC connecting to a fixed number of PNs (but not all), forming sparse connections. These signals are then selectively projected to mushroom body output neurons (MBONs), while lateral inhibition from an anterior paired lateral (APL) neuron suppresses weak KC-MBON connections, implementing winner-take- all strategy. Thus, the number of activated KCs is much smaller than the total dimension of the KC layer.
However, to achieve strong performance complex tasks, LoRA typically requires much higher ranks, which contradicts PEFT's core goal efficiency [31, 45]. Moreover, interference within LoRA's ranks can impair training [76], leading issues such as hallucination [22] and gradient explosion [58). thereby largely limiting its potential. We refer to this challenge as intra-task interference. Meanwhile, foundation models often need integrate multiple capabilities to handle complex downstream tasks, but retraining on multi-domain corpora is expensive [71], particularly when several specialized models are already available. Consequently, model merging [11, 28, 29, 54] is widely used to combine LoRA components trained on different domains in training- free manner. Arising from conflicts between different components, this process introduces another challenge: inter-task interference.
To address intra-task interference, several studies have incorporated the Mixture-of-Experr (MoE) architecture [19, 30, 34, 70, 78, 81] into LoRA [17, 20, 21, 37, 55, 76, 82, 89], where each expert learns specialized knowledge partially achieve task decoupling. We refer these approaches as MoE-based LoRA methods. They replace the original low-rank matrices with multiple experts and dynamic router to selectively activate them. By leveraging redundant parameters and sparse activations, these methods keep the computational budget comparable to LoRA with fewer total ranks. However, they may still suffer from interference within each expert. To investigate this issue, we conduct pilot studies on MMLU using Llama-3.1-8 adopting Split-LoRA as representative MoE-based method. As shown in Figure I(a), finer-grained rank allocation yields consistent performance improvements under a fixed budget. However, as illustrated in Figure (bb), pushing expert granularity to extremes also increases the number of activated trainable parameters due to additional router overhead. This trade-off makes it difficult to achieve both high performance and efficiency. Meanwhile, resolving inter-task interference has received limited emphasis existing MoF-bbased LoRA methods.
Therefore, seek to design improved MoB-based LoRA variant that simultaneously achieves:
Reduced parameter interference among different ranks within a single LoRA component;
Reduced parameter interference between different LoRA components;
Reduced activated trainable parameters in routers.
Inspired by the fly olfactory circuit [6, 13, 38, 42, 72, 100], which shows strong similarity to MoE- based LoRA, we introduce an implicit router to mitigate the trade-off between intra-task interference and efficiency. As shown in Figure 1(c), this leads to the design of FlyLoRA which (1) treats matrix A frozen sparse random projection that maps inputs into higher-rank space (c.g., r = 32 vs.

---

r 8); and (2) simulates the bio- pio-inspired wiineer-take-e all" mechanism by activating & rank-1 experts in B linked the top-k magnitudes after projection by A. This unifies the roles of and router G into a single frozen projection, jointly performing down-projection and expert selection. Without explicit router parameters, the resulting implicit rank-wise MoE structure maintains computational efficiency while reducing intra-task interference. Moreover, we theoretically show that distinct random projections A, and A, from different LoRA components naturally map task updates into approximately orthogonal subspaces, thereby alleviating inter-task interference.
In summary, the core contributions of our proposed FlyLoRA framework are:
Efficient Intra-task Decoupling: By using implicit wise MoE, we enable finer expert allocation with reduced parameter interference in single-task scenarios. Additionally. FlyLoRA surpasses MoF--hased LoRA in efficiency by eliminating the need for router parameters.
Efficient Inter-task Decoupling: In multi-task model merging scenarios, different random projections A, and A naturally form approximately orthogonal subspaces. This inherent property ensures different LoRA components operate in uncorrelated subspaces, thus achieves decoupling.
Neuroscience- Inspired Design: The efficacy of our algorithm, combined with its structural alignment with the fly olfactory circuit, establishes a promising bridge between neuroscience and artificial intelligence.
2 Revisiting MoE-based LoRA Methods
2.1 Preliminaries
LoRA (visualized in Figure 2(a)) simulates weight updates during fine-tuning by decomposing the update matrix into two learnable low-rank matrices. Given a pretrained weight matrix Wo E m the parameter update is computed
W' Wo + AW Wo + Q _A,
where E mx A E Rexn and the rank < min (m, n). The scaling factor a is typically set to 2r. For an input embedding x E R", the forward pass becomes:
(z) W' Wox + BAx.
(2)
Here, Wo remains frozen during training, while only (A, B) are updated. This approach reduces the number of trainable parameters from (mm) to O(r(m + n)), thereby achieving higher parameter efficiency. The low-rank structure allows LoRA maintain stable performance while significantly reducing both computational overhead and GPU memory requirements during fine-tuning.
2.2 MoE-based LoRA Framework
The MoE paradigm (visualized in Figure 2(b)) extends LoRA by decomposing the low-rank adaptation into N specialized experts. Each expert E, parameterized by pair of matrices (B, E mxx A, E grixn }. where denotes the expert-specific rank. The forward pass incorporates a gating mechanism G(=)): R" + R that dynamically routes inputs to activate the most relevant experts. Formally, the output combines the frozen pretrained weights Wo with sparse combination of expert contributions:
fMOE-LORA(E) (x) Wox Σ G(w); B.A.a,
(((e))
where the router G(x) typically follows a top-k selection policy via a trainable projection W, NNx, For simplicity, we omit the activation function, formulating the router as:
G(z) top-k(W> W,
By activating only k experts per input, this design maintains computational efficiency. The sparse routing strategy enables conditional computation, which expands the model's representational capacity without incurring a proportional increase in computational cost. In our work, we implement Split- LoRA under this framework as a minimal yet representative instantiation of MoE-based LoRA. Further implementation details are provided in Appendix C.3.
3

---

A
A
Wo
B
W
B B
B
W #
B
X
A
X
B
X
W
Router
(a) LoRA
(b) MoE-based LoRA
(c) FlyLoRA
Figure Schematic illustrations of different LoRA variants. (a) LoRA employs low-rank matrices A and B to simulate weight updates, where each row of A is fully connected to the corresponding column of B. (b) MoE-based LoRA decomposes the updates into multiple small experts (A,, ) and uses router to determine which experts should be activated. (c) FlyLoRA unifies the down- projection and router into a frozen matrix and selectively activates only the ranks in B linked to the top-k magnitude activations after projection through A.
2.3 Pushing MoE-based LoRA Architecture to the Extreme
Comparing Eq. and Eq. reveals that MoE-based LoRA can be viewed as finer-grained, sparsely activated variant of LoRA, where the separation of experts mitigates task conflict.. Taking this decomposition to the extreme motivates our rank-wise expert design, where each expert governs single rank, achieving the best decorrelating effect (see Figure (a)). Formally, for a rank-r LoRA, the matrices A and B can be decomposed into r rank-1 components:
Srank Trank-wise-Look
Wow Σ G(w); b,a.æ.
(5)
E.(m)
with a, Aii,:] Blxn and b; B E R
However, this approach introduces a scalability challenge: the router's linear layer W, E NxR grows linearly with the number of experts N (see Figure 1(b)). Under a fixed total rank, finer-grained experts with larger N make the explicit routing mechanism computationally prohibitive, undermining the efficiency gains of sparse activation.
To overcome this limitation, we seek an implicit routing mechanism that eliminates the need for the explicit router parameter W, entirely. This entails finding proxy that leverages intrinsic signals within the model to select the top-k experts, effectively approximating the function of the original router G. To address this, we draw inspiration from the perspective of Singular Value Decomposition (SVD), which can also be viewed as rank-wise decomposition. In SVD, the low-rank update matrix A AW can be decomposed as AW = Er= d,u.y, where denotes the i-th singular value (indicating the importance of the corresponding component), u, is the i-th left-singular vector, and v; is the i-th right-singular vector. Each component o,u.v is rank-1 update. The Eckart- Young- Mirsky theorem [18] guarantees that the top-k components, selected based on the magnitude of as, provide the best rank-k approximation to the original rank-r matrix in terms of Frobenius norm, thereby capturing the most salient features with minimal reconstruction error. While exact SVD is computationally prohibitive and thus impractical in our framework, this insight naturally suggests that the magnitude of each rank-1 term, [.,a,z] in Eq. 6, approximately reflects its importance:
fLORA Wox
Σ b,
(6)
Nevertheless, a naive approach of first computing all terms b,a, and then selecting the top-k would also forfeit the computational benefits of sparse activation, as the cost of computing all terms remains (rmm). This necessitates routing strategy that can identify the most important experts before fully computing their outputs. Furthermore, beyond efficient routing, another critical limitation of existing MoE-based LoRA methods is their lack of inherent support for multi-task deployment.

---

When merging models already fine-tume tine-tuned on different tasks, interference between LoRA adapters often leads to significant performance degradation, as the underlying architecture does not structurally encourage task-specific updates to reside in orthogonal or non-overlapping parameter subspaces.
These dual challenges motivate the following two key design requirements for improved MoE- based LoRA framework:
Implicit magnitude-bate based router for top-k activation, without explicit router parameters, abling expert selection prior to full computation;
Natvee support for training-free model merging through architectural properties that promote inter-task task interference mitigation.
3 FlyLoRA
Inspired by the fly olfactory circuit (Figure 1(c),, whose neural architecture inherently meets our requirements for MoE-based LoRA variants, propose FlyLoRA (visualized in Figure 2(c)). Section 3.1 presents its formal design, while subsequent sections analyze its key advantages: Section 3.2 shows how fixed A acts implicit router, Section 3.3 demonstrates intra-task decoupling, and Section 3.4 establishes inherent support for inter-task decoupling in model merging.
3.1 Formulation of FlyLoRA
In FlyLoRA, the matrix A E R"*" is sparse and frozen. It is randomly initialized at the beginning and remains frozen during training, implementing an intrinsic top-k operation in the projection space R" for implicit routing. Given input token E IR., this process is formulated as:
y = toy- = top-k (Ax),
where each row of A contains exactly p (p < n) non-zero entries independently sampled from N(0, _1 (a widely used standard initialization). We define the sparsity ratio as p P. After projection through A, only the columns b, E IR m (i E fl.....r)) in the up-projection matrix B E R linked to dimensions with top-k (k < r) magnitudes in Ax IR" are activated. Formally:
By'l = BBy) if the magnitude of [y]: is among the top-k values of y, otherwise.
10
(8)
To enhance training stability in this MoE structure, we incorporate a simple expert- expert-wise bias term IR" for loss-free load balancing, following [43]. This auxiliary term is updated manually via:
d d; sign(e, C;);
where u is a small learning rate, & represents the expected assignment frequency for expert i, c, tracks the actual assignment count, and sign(.) denotes the sign function. This bias term d is added Ax in expert selection to promote the activation of under-activated experts and suppress over-activated experts, thereby achieving load balancing. Thus, the activated experts are selected by:
Tupk {i1,...,44k .k where in arg max (Ax + d),-
(10)
if(ii.... 1)
The forward pass is then computed as:
(RyLORA(E)) Wo + AW Wox
"Ii(
T(i Impp)) b,a,
(11)
where T(.)) denotes the indicator function.
3.2
Fixed Sparse Random Projection as Implicit Router
The core objective of the MoE router is to select & out of experts that best approximate the effect of using all experts. However, established in Sec. 2.3, it is computationally impractical to determine the selection based on ||bl We therefore seek to perform the selection using only a;æ as surrogate. In FlyLoRA, since we fix A as sparse random projection, computing a,æ: is as efficient as in standard LoRA(r=))- We theoretically prove that this projection preserves pairwise distances (see Theorem 3. and thus can serve effective implicit router.
5

---

Activation Magnitude
0.030
0.015
0.010
0.005
10 20
30
50
70
60
90.00 10
Percentage
(a) Activation distribution
(b) LoRA- FA gradient correlation
(c) FlyLoRA gradient correlation
Figure 3: (a) Activation value magnitude distribution across dimensions, showing the mean actívation strength at different top-k selection percentages. (b-c) Gradient correlation matrices of (b) LoRA- FA (r=32)) versus (c) (ooLoRA(==ss 's B matrices (10 randomly sampled columns). For a simplified illustration, we use the LoRA module of proj in the middle layer of Llama-3. 1-BBB on MMLU.
Theorem 3.1. Given the matrix E Rexp with each row having exactly p non-zero entries randomly sampled from (0, for any >> 0,
P (1- y11
Ax
Ayll S (1+e)]læ w1²) 2 21-
20
ra
for any input embeddings æ, E R", where o² P nrz.
detailed proof is provided in Appendix A.I.
This theorem establishes a probabilistic guarantee for the approximate preservation of Euclidean distances under sparse random projection. Notably, the concentration bound tightens with reduced sparsity (p/n) and larger rank (r), which aligns with intuitive expectation. We further present its robustness empirically via hyperparameter sensitivity analysis in Section 4.4.
Building on this property, we posit that FlyLoRA can self-select experts based on the values of a,æ, which aligns with the finding in [49] that "an expert is aware of its own capacity to effectively process token, an awareness reflected in the scale of its internal activations." Unlike traditional trainable MoE routers that explicitly learn routing weights, FlyLoRA leverages the fixed geometry of A to perform implicit, activation-driver routing. This design not only eliminates the difficulty of learning routing parameters and avoids the separation between the router's 's decision- decision-making making and the experts' execution, but also reduces training instability by removing the stochasticity in router optimization. Because the projection A preserves pairwise distances, two semantically similar inputs and æ; mapped to nearby low-dimensional representations Aæ; and Az, and therefore routed to similar experts, while dissimilar inputs are routed to different experts. This geometry-induced consistency helps mitigate expert representation homogenization in MoE models, enabling each expert to focus on specialized knowledge. reduce internal conflicts, and improve sample efficiency (as supported by previous studies, c.g., [10]). In this sense, FlyLoRA resembles a variant of the hash router [66], which also achieves lightweight and stable expert assignment through fixed mapping. Consequently, the top-k operation naturally selects the most important experts according to the magnitudes of a, In Figure 3(a), empirically around t5p% of dimensions account for more than 80% "energy". Thus, it typically does not cause a large performance drop.
3.3 Gradient Decoupling via Top-k Sparsity
In the FlyLoRA framework, only matrix B requires updating. We theoretically demonstrate that our sank-wise expert allocation strategy, induced by top-k selection, inherently reduces gradient covariance between distinct experts, thus mitigating intra-task interference. Our analysis begins with Assumption 3.2 describing the sparsity pattern of activations. For analytical convenience, we consider a simplified condition where the top-k operation randomly selects k out of columns for activation.
Assumption 3.2 (Uniform Sparse Activation). During the top-k operation, each training sample activates exactly columns of the parameter matrix B E pmxrr with uniform selection probability p per column.
Based on Assumption 3.2, we derive Theorem 3.3 (proof in Appendix A.2).

---

Theorem 3.3 (Covariance Reduction Under top-k), Let E and E denote the gradient covariance matrices with and without top-k activation. When > off diagonal entries scale as:
k²
() Re E ((
Vi
,
This 0(12/r3) reduction factor quantifies how top-k sparsity promotes parameter decoupling by suppressing interference terms. When k (only rank is activated), the off-diagonal covariance almost vanishes, achieving full decoupling: when & = r (all ranks are activated, degenerating to LoRA-FA [94]), it recovers the dense training regime. Theorem 3.3 is proved in Appendix A.2. To empirically validate this theoretical result, we visualize the gradient correlation patterns of LoRA-FA and FIyLoRA in Figure 3(b) and (c), where correlations are computed using 10 randomly selected gradient columns (see heatmap visualization). The observed sparsity pattern strongly supports our theoretical prediction of reduced off-diagonal covariance under top-k selection.
3.4 Inter-Task Orthogonality Model Merging
Traditional LoRA model merging often suffers from parameter interference when combining task- specific components through weight averaging:
w' Wo
w. B,A,
(12)
We analyze how FlyLoRA's inherent subspace orthogonality enables effective multi-task model merging. We derive Theorem 3.4 (proof in Appendix A.3).
Theorem 3.4 (Approximate Subspace Orthogonality). For independent random matrices A,, Aj E R with sparse Gaussian entries (O, ( for < n randomly selected entries per row), the following holds,
1. Exact mean orthogonality: [[A O,xr
2. Polynomially decaying correlations: P(IIA,A] ll2 2 cr)
1 nr
This theorem establishes that sparse random projections naturally induce nearly orthogonal subspaces. The residual correlation bound of order O( indicates that interference becomes negligible under practical parameter scales. This property directly leads to Corollary 3.5 (proof in Appendix A.3).
Corollary 3.5. Let A,, A, E R be fixed sparse random projections after initialization. Then for any learned matrices A, and BjA they satisfy the pairwise orthogonality property:
B,A.,BAAjy
for i
This orthogonal decomposition provides key advantage for model merging: task-specific updates B,A, occupy nearly orthogonal subspaces, thereby preventing destructive interference. The "Pairwise Orthogonality" property captures FlyLoRA's behavior during multi-task aggregation. According to geometric intuition that orthogonality facilitates model merging [29, 56, 75], and consistent with theoretical analyses in [39, 92), FlyLoRA's fixed sparse projection design aligns with this principle. Empirically (see Section 4.3), the random projection A enables FlyLoRA preserve task-specific performance after merging, whereas the learnable in conventional LoRA exhibits significantly higher interference. A similar analysis of LoRA merging was conducted in concurrent study [93].
4 Experiments
4.1 Experimental Setup
Datasets and Backbones:
evaluate FlyLoRA's performance across four key domains: (1)
general knowledge understanding using the MMLU [25] benchmark with auxiliary training datasets for fine-tuning and test set for evaluation, (2) scientific question answering using the ScienceQA [48]
datasct for fine-tuning and evaluation, (3) mathematical reasoning GSM8K [12] problems for
7

---

Table 1: Performance Comparison of LoRA Variants in Single-task Evaluation. We evaluate
various methods across four benchmarks: MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass@k), with all metries reported in percentage (%). Param(%) indicates the percentage of activated trainable parameters relative to Full FT. The best results are highlighted in bold.
HumanEval Pass@ 5
Model
Method
Param( %)
MMLU
ScienceQA
GSM8K
Pass@1
Pass@ 10
LoRA (-8)
0.26
36.53=040
91.39=055
55.34=024
29.1
52,28=124
61.67=a61
LoRA (ERRAA-80 (- 32)
1.03
38.93=104
94.01=017
56,25=029
30.37
54.37=0.9
64.022004
Llama-3.1-8B
Split-LoR (4x8)
0.33
38.44=069
55.65=047 =2541=054
31.28
54.16-112
63.94
FlyLoAA (k=8)
0.13
40.88 161
94.15 0.36
58.76 0.74
36.88
191
62.40
1.82
73.34
LoRA, (-88
0,26
495
92.84=013
77.01
47.20=154
78.89=036
85,94=004
Qwen-2.5-7B
LoRA (2)))
1.05
=2307
-5.01=021
79.23=022
52.87=179
81.67=114
87,80=072
Split-LoR
0.33
10,68
93.08-041
77.12=076
48.65
79.30_091
86.05
FlyLoRA (k=8)
0.13
53.68 0.47
95.55
80.82
54.34 221 82.85
89.63.
fine-tuning and evaluation, and (4) code generation assessed via Code [7] for training and HumanEval [9] for evaluation. Except for HumanEval, which is evaluated via pass ( k metries, others are evaluated via accuracy. All benchmarks evaluated in zero-shot manner. We examine our framework in both single-task configurations, training with these four datasets individually, and multi-task settings, where the LoRA components trained in single-task setup for each dataset merged together in a training-fres manner. Most experiments are conducted using alama-3.1-81 [23] and Qwen-2.5-7B [86], respectively. See Appendix C for implementation details.
Baselines: For the single-task setup, we compare FlyLoRA against: (1) vanilla LoRA with identical activation ranks (LoRA(r=s)) and total ranks (r=)A)r=32)). and (2) representative MoE-based LoRA variants Split-LoRA(axe) (abbreviated for 4 expert rank). For the multi-task setup, we benchmark FlyLoRA against them using weight averaging fusion, and several advanced merging techniques. Across all datasets, FlyLoRA uses total rank 32 but activates only k = 8 ranks after the top-k operation between A and B, with the fixed sparse random A's sparsity ratio set to 8/32.
4.2 Single Task Performance
The single-task results are presented in Table 1. Despite operating under lower computational budget, FlyLoRA(k=s) outperforms LoRA variants with the same rank (ro)A(r=s)) across all datasets. This improvement can be attributed to its broader parameter space. Notably, FlyLoRA also achieves slightly better performance than LoRA variants with the same total rank (LoRA(r=32)) suggesting that significant portion of LoRA's parameters are redundant and may introduce interference. Additionally, (ly) demonstrates superior performance over Split-LoRA (1x8), highlighting the benefits of its finer expert allocation strategy within the MoE framework, which enables intra-task decoupling. The reduction in activated trainable parameters compared to these baselines shows FlyLoRA's efficiency. Extended results with larger models and further baselines are in Appendix B.
4.3 Multi-task Performance
For simplicity, we first employ the widely used weight averaging technique for model merging. Specifically, this corresponds to setting w; = in Eq. 12, yielding the merged weights W'
Wo+ + + Σt_ 1= B,A,, The multi-task results are presented in Table 2, where LoRA components from different domains are merged. Compared to both LoRA variants (r 8 and = 32) and Split- LoRA,4xs), FlyLoRA achieves higher accuracy both before and after merging, with significantly smaller performance degradation. This robustness stems from its inter-task decoupling enabled by approximate orthogonal random projection, as theoretically analyzed in Section 3.4. Additional results with advanced fusion techniques provided in Appendix B.
4.4 Ablation Study and Hyperparameter Sensitivity Analysis
We conduct an ablation study to analyze key properties of FlyLoRA by evaluating two critical modifications: (1) removing load- load-balancing strategies and (2) replacing the frozen matrix A with an
8

---

Table 2:
Multi-task Performance Comparison Before and After Parameter Merging, We evaluate LoRA variants across MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass@k) bench- marks. The table shows performance before merging, after merging, and the relative performance drop (A %.. The best results are highlighted in bold.
HvmanEval
Model
Method
Merge Status
MMLU
ScienceQA
GSM8K
Pass
P
Pass@ 10
Before
36.53+040
91.39=055
55.34202
19..5
52.28
61.672061
After
30.05
31.05=238
25.19
16,09
45,38
56.49+2.13
A (%)
-6.48
-60.34
-30.15
-13.04
-6.90
-5.18
Before
38,93+1.04
94.01=017
56.25
30.37
54.37
64.02
LoRA(r=59)
After
34.02:132
34.35:142
24.77.094
18,94
46.39
59.27.11.9
(%)
-4.91
-59.66
-31.48
-11.33
-7.98
-4.75
Llama-3.1-8B
Before
38 28.44:00.9
92.41=054
55.65
31.28
54.16a1.12
63.94
Spli-LoRA((xs))
After
33.58+1116
37.67=106
27.35=1.10
21.36=1.0
46.01
#0.7
59.52+005
(%)
-4.86
-54.74
-28.30
-9.92
-8.15
-4.42
Before
40.88:1161
15
58.762074
36.882191
62.40
73.34
FlyLoRA(=s)
After
38.8611.16
51.10.071
36.95
32.61
56.591286
69.7610.75
A (%)
-2.02
-43.05
-21.81
-4.77
-5.81
-3.58
Before
49.84=1156
92.84=015
=7.01=032
47.20=1.54
28.8
85 a5.94
LoRA(r=8))
After
44.62.1:25
60.07.2218
81.56
22.09
68.3810.3
80.4910.7
A (%)
-5.22
-32.77
+4.55
-25.21
-10.51
-5,45
Before
$2.07.1031
95.01:1011
79.231022
$2.87
81,67
87.801072
LoRA(r-32)
After
32.86+106
55.58=076
83.91=070
23.84+2113
66.23+145
79.27=100
A (%)
-19.21
-39.43
+4.68
-29.03
-15,44
-8.53
Qwen-2.5-7B
Before
50.681106
93.08:001
77.121076
48.65
79,30 19.30
86.05-104
Split-LoRA(xx8)
After
41.831192
59.37
81.701052
22.9811.35
67.02.0.59
81.011134
A (%)
-8.85
-33.71
+4.58
-25.67
-12.28
-5.04
Before
53.68+047
95.55=018
80.82=056
54.34-213
$2.85
89.63
FlyLoRA(4=))
After
60.23+a96
71.78=041
85.62=04x
33.11
25.28
11.15
A (%)
+6.55
-23.77
+4.80
-21.23
-7.57
-2.48
43
44
Table 3: Ablation study of FlyLoRA vari- ants analyzing: (a) Load balancing in single-task (ST) setting, (b) Matrix A freezing both ST and multi-task merging (MT) settings, where LB=Load Balancing, Fiz=A Frozen, Tm= Trainable.
accuracc (%)
26
38
Load Balancing
Frozen A
25
34
36
Variant
Acc (%)
Variant
Acc (%)
sparsity ratio
activated rank
total rank
(a)
(b)
(c)
w/ LB
10.88
ST+Frz
10.88
w/o LB
37.56:2287
ST+Trn
40.64+1.35
Figure 4: Accuracy comparison for: (a) Sparsity ratio
MT+Frz
38.86::146
in A, (b) Activated rank (with fixed total rank r 32),
MT+Tra
34.43++2.24
(c) Total rank (with fixed activated rank k = 8).
updatable version, as shown Table (MMLU and Llama-3. 18B). Our results demonstrate that load-balancing significantly improves MoE's training stability and boosts accuracy by 3.32%. For matrix A, we observe minimal performance differences between frozen and updatable versions in single-task settings. However, in multi-task merging scenarios, using updatable matrix Ieads to 4.43% performance degradation, as the updatable A does not satisfy the approximately orthogonal property. More ablation studies are provided in Appendix B.
Further sensitivity analysis on single-tase task performance (Figure 4) reveals three key insights. First, model accuracy increases monotonically with the sparsity ratio of A before saturating, exhibiting only marginal degradation unless the sparsity ratio becomes extremely small. Second, under fixed total rank budget, performance peaks at intermediate activation ranks: insufficient rank fails to capture taee-fpec features, while excessive rank induces parameter interference. Third, increasing the total rank while holding the activation rank constant consistently yields performance gains.

---

5 Discussion
5.1 Interference in Model Merging
In scenarios requiring training, gradient orthogonalization techniques are commonly employed to reduce task interference, as seen in multi-task learning [88] and continual learning [8. 91]. In our training-free model merging setting, all components are derived from the same base model through domain-specific SFT. Task interference can be quantified by measuring the orthogonality of parameter updates (relative to the base model) across different tasks [29]. For our LoRA components merging, these parameter updates correspond to B, We formally prove the near-orthogonality of FlyLoRA in Appendix A.3, which inherently reduces inter-task correlations.
5.2 FlyLoRA's Connection to Other Orthogonality- -Based Designs in PEFT
Representative orthogonality- PEFT methods like OFT [46, 59] and LoReFT [83] both operate on single tasks, and their orthogonal matrix R multiplies the pre-trained weight matrix Wo. differing from LoRA variants (including FlyLoRA) that add AW Wo. The multiplication scheme rotates the entire weight parameter space, and [46, 59] demonstrate this better adjusts semantic information compared to changing magnitude, explaining its success. In contrast, the additive scheme lacks this property since Wo cannot be rotated. In single-task settings, removing the MoE part with only random A reduces FlyLoRA to LoRA-FA [94] Asymmetry LoRA [97]. These variants can save resources but cannot improve performance. Thus, although all methods use orthogonality, FlyLoRA succeeds differently. We think the orthogonality design in LoRA excels in multi-task scenarios, such as model merging (this work and LoRi [93]) and continual learning (O-LoRA [80]), because it decouples parameter interference across multiple downstream tasks when fine- tnn-ttnnig from the base model.
6 Related Work
Low-Rank Adaptation LoRA [27] is a widely used PEFT strategy for fine-tuning LLMs. To enhance its expressive power, several improvements [45, 95] have been proposed. Recently, to address parameter interference in settings like multi-task and continual learning, several MoE-based LoRA variants [17, 20, 21, 37, 55, 76, 82, 89] have proven effective by forcing each expert to specialize in specific areas. Several works [73, 75, 93] also aim to reduce interference during LoRA merging. In this article, further develop the MoE-based LoRA structure. To improve LoRA's efficiency, LoRA-FA [94] and AsymmetryLoRk [97] show that fixing the down-projection matrix A saves memory for input activations without performance degradation. In our work, reconsider freezing A from a new perspective by showing that its orthogonality and distance- properties can be utilized to design an improved intra-/inter-task decoupling mechanism.
Fly Olfactory Circuit The fly olfactory circuit [6, 13, 38, 42, 72] serves as an exemplary model in bio-inspired Al due its structural simplicity and functional completeness. Its core mechanism- random projection followed by sparse selection-effectio separable representations. This biological principle has inspired algorithmic innovations across multi- ple AI domains, including Docality-sensitve hashing [14, 67, 69], word embedding [41], federated learning [65], and continual learning [100, 101]. The circuit's enduring relevance highlights the value of cross-disciplinary inspiration in advancing computational methods.
transforms high-dimensional inputs into
7 Conclusion
In summary, this work provides a comprehensive revisit of the MoE-based structure for LoRA and analyzes its drawbacks regarding parameter interference and efficiency. Inspired by the fly olfactory circuit, we introduce FlyLoRA, novel MoE-based LoRA variant that employs rank-wise expert activation in matrix B and fixed sparse random projection for matrix A as an implicit router. Through the theoretical properties of these components, FlyLoRA achieves both intra-task and inter- task decoupling, significantly improving decorrelation in single-domain instruction tuning and LoRA component fusion multi-task settings. Additionally, the implicit routing strategy and inherent sparsity ensure computational efficiency.
10

---

8 Acknowledgments
We thank Cheems Wang for his valuable suggestions on the manuscript. We also thank the anonymous reviewers for their positive feedback and constructive comments. This work was supported by the National Key R&D Program China under Grant 2018AAA0102801.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint S.ti::2333088744,, 2023.
[2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
[3] Nir Ailon and Bernard Chazelle. The fast Bachnson-lindenssten nearest neighbors. SIAM Journal on computing, 39(1):302 302-222,22222222 2009.
transform and approximate
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901. 2020.
[5] Nina Dekoninck Bruhin and Bryn Davies. Bioinspired random projections for robust, sparse classification. SIAM Journal on Imaging Sciences, 15(3) 2022.
[6] Sophie JC Caron, Vanessa Ruta, Larry F Abbott, and Richard Axel. Random convergence of olfactory inputs in drosophila mushroom body. Nature, 99777447)1133-1177 2013.
[7] Sahil Chaudhary. Code alpaca: An instruction-follooiing llama model for code generation. https://github. com/ wwwaa2a00111111 14/oo4/a1aaaa.a... 2023.
[8] Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33: 9900-9911 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov. Heidy Khlaaf, Girish Sastry. Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
[10] Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin- Yu Chen. Patch-level routing in mixtureoof-experts provably sample-efficient for convolutional neural networks. In International Conference Machine Learning, pages 6074-6114 14. PMLR, 2023.
[11] Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027, 2023.
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, arXiv preprint arXiv:21 10. 14168, 2021.
11

---

[13] Sanjoy Dasgupta, Charles F Stevens, and Saket Navlakha. A neural algorithm for a fundamental computing problem. Science, 358(6364) 2017.
[14] Sanjoy Dasgupta, Charles Stevens, and Saket Navlakha. A neural algorithm for a fundamen- tal computing problem. Science, 358(6364;;7937796 2017. doi: 10. htt.siiincce..mm9866.....
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pages 4171-44111 14166, 2019.
[16] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint 1. 2023.
[17] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao. Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang. Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint arXiv:2312.099: 09979, 2023,
[18] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-215 1936.
[19] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 13(12)) 2022.
[20] Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning for large language models. arXiv preprint 2rXhv:2403.03332, 2024.
[21] Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint 240.:2242.008662 2024.
[22] Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning Ilms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.0500 2024.
[23] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.2178, 2024.
[24] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint Xiv::25001 12948, 2025.
[25] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[26] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[27] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, I (2):3, 2022.
[28] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 13269, 2023.
[29] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
12

---

[30] Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991.
[31] Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parameter- efficient fine-tuning. arXiv preprint arXiv:2405. 12130, 2024.
[32] William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26(189-206);,, 1984.
[33] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning. pages 3519-3529. PMIR, 2019.
[34] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[35] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient 2021.
prompt tuning. arXiv preprint
[36] Chumyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint 18X4v:1804.000 08838, 2018.
[37] Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint arXiv:2404. 15159, 2024.
[38] Haiyang Li, Liao Yu, Qiang Yu, and Yunliang Zang. Seemingly redundant modules enhance robust odor learning in fruit flies. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[39] Hongkang Li, Yihua Zhang, Shuai Zhang, Meng Wang, Sijia Liu, and Pin- Yu Chen. When is task vector provably effective for model editing? generalization analysis of nonlinear transformers. arXiv preprint arXiv:2504. 10957, 2025.
[40] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint vxXv:2201.00019, 2021.
[41] Yuchen Liang, Chaitanya K Ryali, Benjamin Hoover, Leopold Grinberg, Saket Navlakha, Mohammed Zaki, and Dmitry Krotov. Can fruit fly learn word embeddings? arXiv preprint arXiv:2101.066 006887, 2021.
[42] Andrew C Lin, Alexei M Bygrave, Alix De Calignon, Tzumin Lee, and Gero Miesenböck. Sparse, decorrelated odor coding in the mushroom body enhances learned odor discrimination. Nature neuroscience, 17(4):559-568 2014.
[43] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1rXiv:2412. 19437, 2024.
[44] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot errameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 15:00-99 2022.
[45] Shih-Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: deiopo-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[46] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.0622 2023.
[47] Xiao Liu, Yanan Zheng Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 5:208-215, 2024.
13

---

[48] Pan Lu. Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chum Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurPPS), 2022.
[49] Ang Lx, Ruobing Xie, Yining Qian, Songhao Wu, Xingwu Sun, Zhanhui Kang, Di Wang, and Rui Yan. Autonomy-of-expers models. arXiv preprint M.Niv:2501.1007.. 2025.
[50] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine PMLR, 2023.
Learning, pages
[51] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:40587-4006 2023.
[52] Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learning with ood state correction and ood action suppression. Advances in Neural Information Processing Systems, 37:93568 93601, 2024.
[53] Yixiu Mao, Qi Wang, Yun Qu, Yuhang Jiang, and Xiangyang Ji. Doubly mild generalization for offline reinforcement learning. Advances in Neural Information Processing Systems, 37: 11433-51473, 2024.
[54] Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703- 2022.
[55] Mohammed Mugeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. arXiv preprint M.XXi:23306.37455. 2023.
[56] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36:66727-66777 2023.
[57] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
[58] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, Advances in neural information processing systems, 30, 2017.
[59] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320-79362 2023.
[60] Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Liu Line, Yang Feng, Lin Lai, Hongyang Qin, et al. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. Advances Neural Information Processing Systems, 066:2166-22115 2023.
[61] Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Björn Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models? arXiv preprint arXiv:2507.044 047777044322200 2025.
[62] Yun Qu, Qi Cheems Wang, Yixiu Mao, Yiqin Lv, and Xiangyang Ji. Fast and robust: Task sampling with posterior and diversity synergies for adaptive decision-maken in randomized environments. arXiv preprint arXiv:2504.19139 2025.
[63] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
14

---

[64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[65] Parikshit Ram and Kaushik Sinha. Federated nearest neighbor classification with a colony of fruit-flies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8036-8044, 2022.
[66] Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. advances in neural information processing systems, 34:17555-17566, 2021.
[67] Chaitanya Ryali, John Hopfield, Leopold Grinberg, and Dmitry Krotov. Bio-inspired hashing for unsupervised similarity search. In International conference on machine learning, pages 8295-8306. PMLR, 2020.
[68] Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36:77290-77312 2023.
[69] Jaiyam Sharma and Saket Navlakha. Improving similarity search with high-dimensional senssity- hashing, arXiv preprint arXhv:1812.01844 2018.
[70] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gatee mixture-of-cxperts layer. arXiv preprint arXiv: 006338, 2017.
[71] Jiayi Shen, Qi Wang, Zehao Xiao, Nanne Van Noord, and Marcel Worring. Go4align: Group optimization for multi-task alignment. Advances in Neural Information Processing Systems, 37:111382-11144 111405, 2024.
[72] Charles F Stevens. What the fly's nose tells the fly's brain. Proceedings of the National Academy of Sciences, 112(30):4400-944) 9465, 2015.
[73] George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd tie the knots. arXiv preprint arXiv:2410. 19735,
[74] Peng Sun, Yao Zhu, Yunjian Zhang, Xiu Yan, Zizhe Wang, and Xiangyang Ji. Unleashing the potential of large language models through spectral modulation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3892-3911, 2024.
[75] Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. arXiv preprint arXiv:2310.04742. 2023.
[76] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565 9584, 2024.
[77] Jiachen Tianhao Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for Ilm training in every iteration. Advances in Neural Information Processing Systems, 37:131 137-13 2024.
[78] Qi Wang and Herke Van Hoof. Learning expressive rete-repressnttttions with mixture of expert neural processes. Advances neural information processing systems, 35:26242-26255. 2022.
[79] Qi Cheems Wang, Zehao Xiao, Yixiu Mao, Yun Qu, Jiayi Shen, Yiqin Lv, and Xiangyang Ji. Model predictive task sampling for efficient and robust adaptation. arXiv preprint UrXiv:2501.10330 2025.
[80] Xiao Wang. Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. 2023.
arXiv preprint
15

---

[81] Ziqi Wang, Chang Che, Qi Wang, Yangyang Li, Zenglin Shi, and Meng Wang. Separa- ble mixture of low-rank adaptation for continual visual instruction tuning. arXiv preprint arXiv:2411.13943 2024.
[82] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404. Ttti::440..36266 13628, 2024.
[83] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. Ad- vances in Neural Information Processing Systems, 37:63908 63962, 2024.
[84] Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang. and Cees GM Snoek. Dynaprompt: Dynamic test-time prompt tuning. arXiv preprint arXiv:200.166444 2025.
[85] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. mies-megging: Resolving interference when merging models. Advances Neural Information Processing Systems, 36:7093-7115, 2023.
[86] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412. 2.X1::2412.11115, 2024.
[87] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In Forty-first International Conference on Machine Learning, 2024.
[88] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824-5836, 2020.
[89] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.0544 054444 2023.
[90] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: fine-tuning for transformer-bast masked language- modelss arXiv preprint arXiv:2106.10199 2021.
Simple parameter-efffcient
[91] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 18)):364-72, 2019.
[92] Siqi Zeng, Yifei He, Weiqiu You, Yifan Hao, Yao-Hung Hubert Tsai, Makoto Yamada, and Han Zhao. Efficient model editing with task vector bases: A theoretical framework and scalable approach. arXiv preprint 0010022502.01015 2025.
[93] Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation. arXiv preprint arXiv:2504.07448 2025.
[94] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308. 033033 2023.
[95] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprini arXiv:2303. 12512, 2023.
[96] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint urXiv:2507.18077 2025.
[97] Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Saez De Ocariz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry low-rank adapters of foundation models. arXiv preprint PrXiv:2402.168444 2024.
16

---

[98] Yao Zhu. Yunjian Zhang, Zizhe Wang, Xiu Yan, Peng Sun, and Xiangyang Ji. Patchwise cooperative game-based interpretability method for large vision- lision-language models. Transactions of the Association for Computational Linguistics, 13:744-759, 2025.
[99]
Heming Zou, Yixiu Mao, Yun Qu, Qi Wang, and Xiangyang Ji. Utility-diversity aware online batch selection for Ilm supervised fine-tuning. arXiv preprint vrXiv22510.66882 2025.
[100] Heming Zou, Yunliang Zang, and Xiangyang Ji. Structural features of the fly olfac- tory circuit mitigate the arXiv:2502.01427 2025.
dilemma in continual learning. arXiv preprint
[101] Heming Zou, Yunliang Zang. Wutong Xu, and Xiangyang Ji. Fly-cl: A fly-inspired framework for enhancing efficient decorrelation and reduced training time in pre-trained model-based continual representation learning. arXiv preprint 2500:
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have summarized the contribution and scope in the abstract, introduction,
and conclusion. Especially, we list our contribution in the last paragraph of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the possible limitations in Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

We have listed theorems in Section 3 of the main text, with complete proofs presented in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have listed the training details in Appendix C, from which people can repro- duce the main experimental results based on it. We also provide code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
19

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code in the supplementary materials. The open-source datasets and models are listed in Appendix C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
hetbbmmmssinoooicyyy...... for more details. httpc//ggiide//oeeeueeeeee
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.ea. cttpssscccppccccccccpccccccccc c/guidgusssseeeee /CodeS/mm ssi iasionPoiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss the experimental setup and training details in Section and Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the error bar over three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We summarize our computational resources in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: /ttp://eeeuusppeeeeeeeeeeer cc cc/public/Ethitiic (pppblic/Ethicccuidelllniee
didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We have read the Code of Ethics and make sure to preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discuss potential societal impacts in Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have used open- code and datasets, and have appropriately cited them.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: (https://nett http//rrpps.cccoooeeeeeeeeeeeeee ccc.ooooeeeeeccce/22///1/111 for what should or should be described.
24

---

A Theoretical Analysis
A.1 Distance Preserving Property for Sparse Random Projection
In this section, we prove that the fixed sparse random projection matrix A satisfies the distance- preserving property. This demonstrates that fixed projection A can function like hash router without an explicit router. Our result extends the well-known Johnson-Lindeentrrus Lemma [32].
First, [5, Corollary 3.5] provides the following bound:
Theorem A.1. Let A E Rexn be a random matrix whose entries Auj are sampled independently and randomly from a distribution that is symmetric around the origin with E(A?;) = 0² > 0.
1. Suppose C = =E(A))) < oc. Then, for any c <> 0.
1 P
(62 (-12-e)))
Aæ 12 S
S exp
for all x IR".
2( ([]
2. Suppose 3L > O such that for any integer k. > 0, E(A
1j
2k)
S 0² (2k)1 2
L
Then, for any
O,
P
1 Aal -
o² (1+
exp (-(62
for all R
According the definition of A mentioned in Section 3.1, the second moment of Air satisfies B(A?,)
n > So, the 2k-th moment of Aí is given by E[A?]] " X 2 + ?)
2
1f [[ where Xú NO ). From the property Gaussian distribution, we derive n
R
3 = (2k- 1)!! ( 26 (2k1
( 2k where denotes the double factorial. This leads to 2
(2R) 22 S 5 (2k) 1,2k B[A k) .p which is clearly finite. To satisfy the inequality E[4 2
2
p always satisfies the condition specified in Theorem A.1. Combining both bounds from Theorem A.1, we summarize the final result in Theorem A.2.
we require 2 loga* P. Due to monotonicity, simply choosing L loga
1 r
P (1
-)]lac y11²
<
Ax Ayll s (1 (1++)] y11
1 (
P
Ayl2 < - -))o²la:: -y113
(13)
-
P
Ayl1a > (1 e) æ y1
Theorem A.2. Given the matrix A E Rrxa with each entry i.i.d. from N(0, # and set to otherwise, for any >> 0,
P (11-c)lz--)) y11 y
1 r Ax Ayll (1+c)] v1²) 1 - 1
2(52+1)
for any input embeddings x, E IR", where o²
Since this construction does not differ significantly from our desired construction for A, which includes an additional constraint on non-zero entries per row and is more consistent with biological observations the fly olfactory circuit, we use this more analytically tractable form as surrogate to study the preserving property. In practice, these two construction methods show negligible performance differences.
A.2
Top-k Activation Promotes Rank- Wise Decoupling
Let A diag(A1..... Ar Rrxr denote a binary mask where A; if column is activated by top-k. The relation of the masked gradient between top-k version 0c and the dense version (0 Bn
0C
is:
Oc
Oc AB
A E mxr,
(14)
aB
25

---

Let 9: and E R" denote the gradient vectors for the i-th column of 0c and
0c respectively.
BB*
Their corresponding cross-column covariance are:
((,)) E [g. 9:
E gi
E [g)]
(15)
Σ ((,)) E [g. 93] E [
E Egg]
(16)
We can assume Elg: R = Elg:] = EEgg] 0m. these simplify to:
[g. 9:
(17)
Σ (a,)) = [9. 95]
(18)
Thus, the expected covariance depends only on the expected inner product of gradient vectors. From
Assumption 3.2, the difference between Σ (i.i) ((,))) is factored by the co-activation probability of columns i and
(--2) 2)
k(k 1) r(r - 1) y2
k2
P(A. = 1 =
(19)
This leads the following theorem:
Theorem A.3 (Covariance Reduction Under top-k). Let E and E denote the gradient covariance matrices with and without top-k activation. When r > k, the off-diagonal entries scale as:
k2 , Vi
B| R B[
(20)
A.3 Random Projection Induces Approximate Subspace Orthogonality
The orthogonality properties of random projections form the theoretical foundation for FlyLoRA's effectiveness in model merging. Consider two independent random projection matrices A,, A, E Rrx* with entries distributed as specified in Section 3.1. The expectation of their product reveals:
E B[A:A]] =E
Σ (Ai)mu mi (Az)ue Lk=1
Σ B[(A: mb mm [B(()))((AA)))))) = Orxr k=1
(21)
This zero-expectation result follows from the independence and zero-mean property the random matrices. To quantify how tightly the product concentrates around zero, analyze its variance:
Var ((A:4))ml mm)) =
Var Var((A:) ((A))mx) Var ((A))u)) + Var Var((Ai)mx) (EE (B(AA)us)
+ Var ((A))1a) (E( (B(A:)m)t)
Var Var((A,) ((A))m)) Var Var(( ((A))1)
p²
no
(22)
nr
Applying Chebyshev's inequality, we bound the probability of large deviations for each entry:
PP (((A,A])ml 2() Var ((A,A) mmm)
p²
(23)
2
2
The Frobenius norm characterization gives us:|| A
(A, ja1 To analyze its probabilistic behavior, employ the union bound principle: for any events Egy defined as
1 = Em
[(AAA] )mn) e, we have P (Um, Emn < Em. P(Emi). Using this union bound over all 2
entries yields:
P (I(A,A] )ml 2 12. D²
p²
P (IAAA) 2 ET)
(24)
422
2r222
1=1
Since the spectral norm is bounded by the Frobenius norm (I) A,l1 & ||A A, we conclude that when n > - the subspaces spanned by A; and A, are approximately orthogonal with high probability.
26

---

Theorem A.4 (Approximate Subspace Orthogonality). For independent random matrices A, E Brxa with sparse Gaussian entries (O(0, + for < randomly selected entries per row), the following holds:
1. Exact mean orthogonality: ] 0pxr
2. Polynomially decaying correlations: P(IIA,)) 1l2 er) S .र
We demonstrate the approximate orthogonality between distinct LoRA components B,A; and B through Frobenius inner product analysis following [75]:
B B = tr (BjA,) (B.A))
(A) BJ B,A.)
tr [B B.A,A))
2 tr [B B, 0pxr)
2
(25)
These analysis demonstrates that random projections naturally create nearly orthogonal subspaces, which helps prevent interference between different experts FlyLoRA. The small residual correlation becomes negligible when input dimension + co, leading to the effectiveness of our sparse random projection approach for model merging.
Formally, considering the conventional LoRA merging scheme:
W
W
+
Wi B,A;
(26)
2=1
When multiple FlyLoRA modules (B,A,) approximately orthogonal, the squared Frobenius norm of the merged weight matrix can be decomposed into the weighted sum of individual module norms.
Σ
W. B,A
B.A.
Σ i
(,W,(B,A,, B,A,)
(27)
F
BA
(28)
which aligns with the "Weight disentanglement" property for task arithmetic [56]. We summarize these findings in the following corollary:
Corollary A.5. Let A., A, grxn be fixed sparse random projections after initialization. Then for any learned matrices B,A; and B,A, the following properties hold:
1. Pairwise Orthogonality:
(B.A, B for i
2. Orthogonality's Outcome Merging:
B
Σ 3
B
Through sparse random projections, FlyLoRA inherently constructs nearly orthogonal subspaces for parameter merging from different tasks.
B
Additional Results
B.1
Evaluation Larger Models
We further conducted experiments using the Qwen-2.5-14B model, as shown in Tables and following the settings of Tables and 2. The results show that FlyLoRA remains superior in accuracy
27

---

(for both single-task and multi task settings) and is more parameter-effiiiet We encountered no memory or convergence bottlenecks when training FlyLoRA on the 14B model, which consistently outperforms LoRA and Split-LoRA. This scalability confirms that our method's s benefits extend effectively to larger model architectures without compromising training stability.
Table 4: Performance Comparison of LoRA Variants in Single-task Evaluation using Qwen-2.5- 14B. We evaluate various methods across four benchmarks: MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass @k) k), with all metrics reported in percentage (%). Param(% ()) indicates the percentage of activated trainable parameters relative Full FT. The best results highlighted in bold.
Method
Param %
MMLU
ScienceQA
GSMSK
HumanEval
LoRA(r=8)
0.23
56.7410.56
95.62.0.18
83.081074
51.6911.48
LoRAir=st (52
0.93
59.351079
97.0510.2
8523
54.8010.75
Split-LoRA(axes 0.29
58.261113
96.85
84.881051
54.65
FlyLoRA
0.12
60.17:1108
97.37 10.32
85,96
56.42::116
Table Multi-task Performance Comparison using Qwen-2.5-14B. We evaluate LoRA variants across MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass @ k) benchmarks. The table shows the relative performance drop (A %) before and after merging. The best results are highlighted in bold.
Method
MMLU
ScienceQA GSM8K
HumanEval
LoRA(raa)
-13.75
-25.20
-11.43
-18.60
LoRA (2))
-8.91
-20.45
-7.62
-16.34
Split-LoRA(axs)
-7.48
-21.97
-6.05
-1.87
FlyLoRA
-4.35
-17899
-2.18
-11.72
B.2 More Baseline into Comparison
We further compare several strong and widely used baselines AAaLLRA [95] (adaptive rank alloca- tion), SoRA [16] (sparse adaptation), and HydraLoRA [76] shown Tables and 7. The results demonstrate that FlyLoRA remains superior in terms of accuracy and efficiency in both single-task learning and multi-task merging scenarios. Notably, FlyLoR achieves this performance while maintaining a simpler training pipeline, as avoids the additional hyperparameter tuning required by adaptive and sparse methods.
using Qwen-2.5-7B, as
Table 6: Performance Comparison with More LoRA Variants in Single-task Evaluation using Qwen-2.5-7B. We evaluate various methods across four benchmarks: MMLU, ScienceQA, GSMSK (accuracy), and HumanEval (Pass @k), with all metrics reported in percentage (%). Param (%) indicates the percentage of activated trainable parameters relative to Full FT. The best results are highlighted in bold.
Method
Param(%)
MMLU
ScienceQA
GSM8K
HumanEval
AdaLoRA (r=))
0,26
51.22+021
93.48+02
77.65
47.96
SoRA
0.19
10.89
93,25
78. 78.461082
1098
Hydral -,
0.52
53.05
94.694034
79.31 #0.9
52.98
Olyl.RRA ((=8)
0.13
53.68
95.55
80.82
54.34
B.3 Training Time and Memory Consumption
We further conduct experiments comparing the training time and memory consumption of the LoRA variants discussed in Section 4. The results, presented in Table 8, demonstrate that LoRA, (rm32) requires more training time and memory than LoRA,,=s) due its higher rank. Split-LoRA shows intermediate values between these two approaches. Notably, FlyLoRA achieves both the fastest training times (resulting from having the fewest activated parameters) and the lowest memory consumption (mainly attributed to its frozen matrix A that significantly reduces memory for activation
28

---

Table 7: Multi-task Performance Comparison with More Baselines using Qwen-2.5-7B. We evaluate LoRA variants across MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass k) benchmarks. The table shows the relative performance drop (A%) before and after merging. The best results are highlighted in bold.
Method
MMLU
ScienceQA GSM8K
HumanEval
AdaLoRA
10.90
-33.15
+4.52
-25.46
(r-8)
-9.45
-34.81
+4.05
-26.50
Hydral.oRA
(18.8-118--))
-6.22
-34.67
+4.21
-24.94
FlyLoRA +6,55
-23.77
+4,80
-21.23
values (94]). We also illustrate the theoretical memory consumption for these LoRA variants in Table 9, which tightly aligns with the experimental results. It's clear to see that under a fixed total rank larger number of experts N causes Split-LoRA require significantly more activated trainable parameters and memory consumption, degrading the efficiency of MoE-based LoRA methods. These results and analysis demonstrate the robustness of FlyLoRA's parameter efficiency compared to MoE-based LoRA methods.
Table 8: Training Time (Hours) and Memory Usage (GB) of LoRA Variants on Different Datasets and Architectures. Comparison of LoRA(r=8), LoRA(r=2)) Split-LoRAyzxe) and FlyLoRA(k=s) fine-tuning Llama-3.1-80 and Qwen-2.5-7B on MMLU, ScienceQA, GSM8K, and CodeAlpaca-20ks The best results are highlighted in bold.
Llama-3.1-8B
Qwen-2.5-7B
Metric
MMLU
ScieneeQA
GSMSK
CodeAlpaca
MMLU
ScienceQA
GSM8K
CodeAlpaca
LoRA;
Training Time
4.79h
5.30h
0.52h
1.85h
4.45h
5.07h
0.54h
1.75h
Memory Usage
12.5GB
14.8GB
20.1GB
20.2GB
14.7GB
1688GB
22.9GB
22,9GB
LoRA
Training Time
5,09h
5.61h
0.58h
1.95h
4.76h
5.39h
0.60h
1.87h
Memory Usage
13.2GB
15.3GB
20.7GB
20.7GB
15.4GB
17.3GB
23,4GB
23.4GB
Split-LoRAaxss
Training Time
4.94h
5.46h
0.56h
1.92h
4.60h
5,23h
0.57h
1.79h
Memory Usage
12.8GB
15.0GB
20.4GB
20.4GB
15.0GB
17.1GB
23:2GB
23,2GB
FlyLoRA((=s)
Training Time
4.73h
5.23h
0.51h
1.82h
4.39h
4.99h
0.52h
1.70h
Memory Usage
10.6GB
10.7GB
10.9GB
10.9GB
12.1GB
12.2GB
12.4GB
12.4GB
Table 9: Theoretical Memory Consumption Comparison of Different LoRA Variants for Single Linear Layer. Param indicates the number of activated trainable parameters. Variables d, r, k. b, and N represent hidden dimension, total rank, activation rank, batch size, sequence length, and number of experts, respectively. We record memory usage for weights, gradients, optimizer states, and activations in bytes. These results are calculated under 16-bit mixed-precision training settings.
Method
Param
Weight
Gradient
Optimizer
Activation
2(d + 2dr) Split-LoRA 2dk AN 2( + 2dr + dN) 4dk + 2dN 24dk + 12dN 2bsd + 2bsk + 2bs
LoRA
2dr
4dr
24dr
2bsd 2bsr
FlyLoRA
dk
2(d² + 2dr)
2dk
12dk
2bsk
B.4 Multi-task Performance for Advanced Model Merging Techniques
Extending the results in Section 4.3, we also evaluate two advanced model merging techniques, TIES-MERGING [85] and DARE [87], for merging LoRA components from different domains. The results are listed in Tables 10 and 11, respectively. Overall, both TIES-MERGING and DARE outperform naive weight averaging by resolving parameter conflicts through intelligent selection (TIES' sign consensus and trimming) and selective rescaling (DARE's dropout- basod redundancy climination). Similar to weight averaging. these results demonstrate that FlyLoRA consistently surpasses LoRA(r=s). LoRA(r=32). and Split-LoRA across all comparisons, highlighting the
29

---

Table 10: Multi-task Performance Comparison Before and After Parameter Merging Using TIES-MERGING. We evaluate LoRA variants across MMLU, ScienceQA, GSM8K (accuracy), and HvaanEval (Pass@k) benchmarks. The table shows performance before merging, after merging, and the relative performance drop (A %). The best results are highlighted in bold.
HumanEval
Model
Method
Merge Status
MMLU
ScienceQA
GSM8K
Pass
P
Pass@ 10
Before
36.53+040
91.39=a35
55.340o4
29.13
52.28=1.44
61.67=061
LoRA(r=8)
After
31.83+004
35.9621 46
27.532108
1844
45.522054
56.63
A (%)
-4.70
-55.43
-27.81
-10.68
-6.76
-5.04
Before
38,9311.04
94.011017
56.251029
30.3711.0
54.3710.39
64.02.109
LoRA(r12))
After
34.45+173
36.63=079
26.59
20.98
47.15
59.97
A (%)
.4,48
-57.38
-2966
-9.39
-7.22
-4.05
Llama-3.1-8B
Before
38.444049
92.41
45.65=047
31.28
54. 16.1.21112
63.9420500
Split-LoRA(axe
After
33,92:057
40.81
29.022082
22.02:0.24
46.32:0.7
59.72:0025
A (%)
-4,52
-51.60
-26.63
-9.26
-7.84
-4.22
Before
40.881161
94.15
58.76
86.881191
62.40
73.34.124
FlyLoAA(--))
After
39.03:131
46.82::04
19.12.1.2
33.372062
17.362141
70.35
A (%)
-1.85
-39.33
-19.64
-3.11
-5.04
-2.99
Before
49.8410.16
12.84
77.01
47.20
18.894036
85.94.004
Lo
After
44.98+u.22
61.69=134
81.89
23.37=1.24
68,
68.96
81.25
A (%)
-4.86
-31.15
+4.88
-23.83
-9.93
-4.69
Before
52.07=a31
=5201
79.23
17.87
21.67
87.802072
LoRA
After
35.92.0.18
58
28.02.1277
13.98
24.79
67,50
80.35
A (%)
-16.15
-36.99
+4.75
-28.08
-14.17
-7.45
Qwen-2.5-7B
Before
50.68:1.06
93.08:0.41
77. 77.12=076
48.65
79.30
86.05
Split-LoRA(xs))
After
44,9610.6
60.591002
81.82.0.0
23,5310.74
67.59
82.34.0.00
(%)
-5.72
-32.49
+4.70
-25.12
-11.71
-3.71
Before
53.68:1047
15
80.822056
54.3412.15
82.85
89.63
FlyLoRA(=S)
After
60.51
73.46=030
86.24=031
35.37
76.05+1.25
87.97
A (%)
+6.83
-22.09
+5.42
-18.97
-6.80
-1.66
robustness of FlyLoRA's near-orthogonallity property in reducing inter-task decoupling and further enhancing model merging performance.
We also include comparison with more advanced model merging techniques in Table 12. KnOTS [73] and L-LoRA [75] are both built upon (r=2)) The results suggest that they achieve comparable performance to FlyLoRA, with each method excelling different datasets. It is noteworthy that FlyLoRA is not competitor to these methods; rather, they can be used in a plug-and-play manner with Fly to further improve performance after merging.
B.5 Additional Ablation Studies on Load-Balancien Strategies
We compare experimental results using different load- bad-bbalnccigg strategies. In Section 3.1, we employ an easy-to-implemer loss-free balancing strategy. This loss-agnostic approach effectively achieves load balancing with negligible computational overhead and memory footprint. Simultaneously, other loss-controlled load-balancing strategies like [19] are widely used in MoE-like structures. A comparison of them is shown in Table 13. Our results show that different routing strategies achieve similar effects performance. While FlyLoRA requires load-balancing strategies, it not sensitive to specific methods.
B.6 Additional Ablation Studies on K-Selection Strategies
To evaluate the impact of activation selection, we analyze different K-selection approaches in our experiments. In neuroscience, the fly olfactory circuit implements a "winner-take-all" strategy, simulated through top-k selection based on activation values across dimensions. To validate top- k's effectiveness, we test random-k selection and full activation (without selection) as baselines. Results in Table 14 show that both top-k and random-k outperform full activation, confirming sparse activation mitigates intra-task interference. Crucially, top-k surpasses random-l: because random selection cannot prioritize the most informative dimensions, leading suboptimal performance.
30

---

Table 11: Multi-task Performance Comparison Before and After Parameter Merging Using DARE. We evaluate LoRA variants across MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass@k) benchmarks. The table shows performance before merging, after merging, and the relative performance drop ()%). The best results are highlighted in bold.
HumanEval
Model
Method
Merge Status
MMLU
ScienceQA
GSM8K
Pass
P
Pass@ 10
Before
36.53+040
91.39=a35
55.34=02
29. 19.16
52.28+1.24
61.67=061
LoRA(r=8)
After
31.244040
34.37=125
26.7621110
17.35
45.492006
13.24
A (%)
-5.29
-57.02
-28.58
-11.78
-6.79
-4,43
Before
38.9311.04
94.0110.17
56.251029
30.371106
54.3710.39
64.02.109
LoRA(r-22)
After
34.75+0.3
37.56=039
26.89
19.36=1.38
46.67
59.85
A (%)
-4.18
-56.45
-2.36
-11.01
-7.70
-4.17
Llama-3.1-8B
Before
38.444040
92.41
45.65
31.28
54.
16
23.9420500
Split-LoRA(axe
After
34.02:024
48.58:008
28. 28.632072
23.52:143
46.84
26:84
60.30:041
A (%)
-4.42
-53.83
-27.02
-7.76
-7.32
-3.64
Before
40.881161
94.15
58.762074
86.881191
62,40
73.34.124
FlyLoAA(-S))
After
39.37:1100
52.34:005
38 10
33.34
23.34
17.14
70.24
A (%)
-1.51
.41.11
-20.56
-354
-5.66
+3.10
Before
49.8410.66
12.84
77.01
47.20
78.891036
85.94.004
LoRA(r-8)
After
45.20+
61.39=132
81.98=0m
23.49
69.04
81,22+027
A (%)
-4.64
-31.45
+4.97
-23.71
-9.85
-4.22
Before
52.07=aa11
=520
79.23gam2
12.87
81.67
87. 37.8020.2
LoRA
After
35.27:1168
56.791134
84.07
24.35
67,74
80.
80.12.1.32
(%)
-16.80
-38,22
+4.80
-28.52
-13.93
-7.68
Qwen-2.5-7B
Before
50.68
93.08:0.41
77.12=076
48.65
79.30s091
86.05
Spli-LLRAA(xs))
After
43.561004
62. 25
81.82.0.0
23,79
68,
18,96
81,53
(%)
-7.12
-30.93
+4.70
-24.86
-10.56
-4.52
Before
53.68:10.7
15.55
80.8220.6
14.34
82.85
89.63.20.5
FlyLoRA(=S)
After
61.35++047
72.94=021
86.34=036
34.47=0.44
70.97
87.64=1.4
A
+7.67
-22.61
+5.52
-19.87
-6.88
-1.99
Table 12: Multi-task Performance Comparison with More Advanced Merging Techniques using Qwen-2.5-7B. We evaluate different methods across MMLU, ScienceQA, GSM8K (accuracy), and HumanEval (Pass@k) benchmarks. The table shows the relative performance drop (A%) before and after merging. The best results highlighted in bold.
Method
MMLU ScienceQA GSM8K HumanEval
FlyLoRA
+6.55
-23.77
+4.80
-21.23
KnOTS
+10.76
-26.85
+4.68
-23.37
L-LoRA
+4.51
-22.48
+4.74
-20.85
KHOTS+FIYLORA
+11.47
-23.41
+5.25
-20.69
L-LORA+FIYLORA
+7.65
-21.42
+5.02
-19.85
These findings collectively demonstrate that biologically inspired top-k activation optimally balances efficiency and tasc-fpe feature selection.
B.7 Additional Ablation Studies on Matrix A initialization Schemes
We further compare three methods for generating the sparse projection with 7 sparsity:
1. Gaussian (our default): Each non-zero entry is drawn from NO, 3
2. Rademacher (non-Gaussian): Each non-zero entry is + with equal probability.
3. FJLT [3] (structured projection): A PHD, where D is random diagonal matrix with independent Rademacher variables on its diagonal, H a normalized Hadamard matrix, and P enforces the 2 sparsity.
4. Two-Phase (briefly-learned): The non-zero entries of are trainable for 5% of total steps as warm-up, then frozen for the remainder.
The results, shown in Table 15, indicate that almost all variants perform similarly. Non-Gaussian, structured, or leaanle initializations have little impact, except that the briefly-learedd scheme
31

---

Table 13: Performance Comparison of Different Load-Balancing Strategies. Evaluation on MMLU benchmark using 1aama-3.1-82
Goad-Balancing Strategy Accuracy (%)
Loss-Free
40.88
Loss-Controlled
40.59
No Load- Saad-aalncinng
37.56+2287
Table 14: Performance Comparison of Different K-Selection Strategies. Evaluation on MMLU benchmark using 1lama-3.1-8B.
K-Selection Strategy Accuracy (%
top-k
40.88=1.61
random-k
40.02=0026
full activation
39.40
shows a noticeable drop after merging. This demonstrates that FlyLoRA is robust to choice of initialization scheme for matrix A, and that learning may break the approximate orthogonality of the random matrix, making unsuitable.
B.8 Analysis of the Performance Gap Between Merged and Non-merged Scenarios
In Tables 2, 10, and 11, we can see that ScienceQA usually demonstrates large performance drop after merging for all LoRA variants. Intuitively, the four tasks tasks-general knowledge understanding (MMLU), scientific question answering (ScienceQA), mathematical reasoning (GSM8K), and code generation repressn significantly different distributions, so merging their adapters prone to substantial conflicts.
Empirically, following [73], we use centered kernel alignment (CKA) [33] quantify the alignment between the output representations of each single-task adapter and the merged adapter. A higher CKA indicates better output alignment, which is likely the inherent reason, and therefore, results in smaller accuracy drop after merging. Table 16 reports both CKA and accuracy drop (A) on Llama-3. 8B. Since there is no apparent difference between LoRA(r=S)) LoRA and Split-LoRA in model merging, that tasks with lower CKA (especially ScienceQA and GSM8K) suffer the largest accuracy drops. FlyLoRA consistently yields higher CKA than LoRA, which aligns with its consistently smaller A. This micro-level analysis corroborates why FlyLoRA outperforms LoRA in heterogeneous-tare merging.
use LoRA(r=8)) as representative to compare with (ly)) We observe
C Detailed Experimental Setting
C.1 Datasets
To comprehensively evaluate the effectiveness of proposed method across diverse domains and task types, we conduct extensive experiments five carefully selected benchmarks. These datasets span critical capabilities including general knowledge reasoning, scientific understanding, mathematical problem solving, and code generation. Table 17 summarizes the key characteristics of each dataset, while detailed descriptions are provided below:
MMLU [25] serves comprehensive benchmark for evaluating broad knowledge un- derstanding and reasoning capabilities. comprises multiple-choice questions spanning 57 distinct academic subjects, ranging from elementary mathematics and US history to computer science and professional law. The diversity domains makes particularly suitable for assessing model generalization across different knowledge types.
ScienceQA [48] provides multimodal framework for science question answering, with content derived from elementary and high school curricula aligned with California Common Core Content Standards. The questions originate from IXL Learning's expert-curated educational resources. Following the established practice [37], we utilize only the textual components to focus on linguistic understanding.
32

---

Table 15: Performance Comparison of Different Initialization Strategies. Single-task and
multi-task accuracy comparison on MMLU using Llama-3.1-88
A Initialization Strategy
Before
Aafter merging
Gaussian
40.88+161
-2.02
Rademacher
40.42.1023
-2.35
FJLT
40.57+1.34
-2.50
Two-Phuse
40.76*1104
-4.86
Table 16: CKA and Corresponding Accuracy Drop (A) Between Single-Task Adapter and Merged Model. Evaluating using Llama-3. 1-B.
Method
Task
MMLU ScienceQA GSM8K HumanEyal
CKA
0.78
0.39
0.58
0.75
(o
A
-6.48
-60.34
330.55
-13.04
CKA
0.85
0.53
0.71
0.84
A
-2.02
-43.05
-21.81
-4.27
GSM8K [12] offers 8,500 high-quality grade school mathematics word problems that demand multi-step arithmetic reasoning. Each problem is accompanied by detailed, step- by-step solution, making it ideal evaluating logical reasoning and procedural accuracy in mathematical contexts.
CodeAlpaca-20k [7] contains 20,022 synthetically generated instruction-response pairs specifically designed for code-related tasks. This dataset facilitates effective instruction tuning for programming applications by providing diverse coding prompts paired with corresponding solutions.
HumanEval [9] consists of 164 hand-crafted Python programming problems developed to assess functional correctness in code generation. Crucially, these problems were manually created to prevent data contamination, ensuring they do not appear in the training corpora of existing code generation models.
Table Details of MMLU, ScienceQA, GSM8K, CodeAlpaca and HumanEval Datasets. We list the number of training and testing samples and task types for the following datasets used in our experiments.
Dataset
Training Samples Testing Samples
Task Types
MMLU [25]
99,842
14,042
Multiple Choice
ScienceQA A [48]
12,726
4,241
Multiple Choice
GSM8K [12]
7,473
1,319
Math Problems
CodeAlpaca-20k [7]
20,022
Code Instruction
HumanEval [9]
164
Code Generation
C.2 Training Configuration
This section elaborates on the experimental setup and hyperparameter configurations employed throughout our study. Table 18 documents the shared training parameters applied consistently across all backbone models and datasets. To address the specific requirements of different tasks and model architectures, we additionally provide dataset-specific and model-specific configurations in Table 19, including learning rate schedules, batch size adjustments, and task-specific optimization strategies.
C.3 Split-LoRA
We implement Split-LoRA a representative MoE-based LoRA method, following the general framework described in Section 2.2. In our experiments, we incorporate a sigmoid activation function the router to normalize expert selection scores. Thus, the gating function operates as which ensures differentiable routing while maintaining the sparsity
G(x) = (tnpnidttopp(WW.... of expert activation. This configuration allows Split-LoRA to serve as a representative baseline for evaluating the effectiveness of MoE structures in LoRA.
33

---

Table 18: General Training Hyperparameters for FlyLoRA. Shared configuration across all
experiments, including rank settings, optimizer details, and architectural choices.
Parameter
Value
Total rank (r)
32
Scaling factor (a))
64
Activated rank
Target modules
{q, kq gate, down, upp_,,uppppo0 Adam
Optimizer
Warmup ratio
0.01
Gradient accumulated batch
128
Dropout rate
0.00
Table 19: Dataset-Specine and Model-Specific Training Configurations for FlyLoRA. Task- optimized settings for 1lmma33.1885 and Qwen-2.5-7B across four benchmarks, showing variations in epoch counts, learning rates, and sequence lengths based on dataset characteristics and model requirements.
Model
Parameter
MMLU
ScienceQA
GSM8K
CodeAlpaca
Epochs
20
2
Learning rate
3 X 10
x 10
3 x 10
3 x 10
Llama-3.1-8B
Max sequence length
128
256
512
512
micro batch size
8
Epochs
20
2 x 10
Qwen-2.5-7B
Learning rate
x
x
x 10
Max sequence length
128
256
512
512
micro batch size
8
8
8
8
C.4
Environments
Most experiments were conducted on a Linux server rumning Ubuntu 20.04.4 LTS, equipped with an Intel(R) Xeon(R) Platinum 8358P CPU 2.60GHz and 8 NVIDIA GeForce RTX 3090 GPUs, using CUDA version 11.7. Experiments with Qwen-2.5-14 2.B-14B were conducted on machine with NVIDIA A100 GPUs.
D Limitations and Future Work
In FlyLoRA, matrix A is randomly initialized and frozen during training, but there may still be room for improvement. Recent neuroscience studies [13] suggest that A need not be entirely frozen and random. indicating potential for more bio-inspired mechanisms to enhance task decoupling through an adaptable version of A. Moreover, recent works suggest that component-wiss interpretability [98] and spectral modulation [74] could inspire adaptive or frequency- FlyLoRA to improve efficiency, robustmess, and task decoupling.
modifications of A in
Recently, RL fine-tuning for LLMs has emerged as a promising approach that significantly enhances their reasoning ability [24]. However, stabilizing MoB RL training remains an open question [96], and further exploration will focus on the integration of FlyLoRA with RL training [61] and potentially extending it offline policy optimization [50-53, 60, 68). Additionally, integrating active data selection methods could be a promising direction to further improve data efficiency [62, 77, 79, 99].
E Broader Impact
Our proposed FoyLoRA resolves the trade-off between parameter interference and efficiency in MoF-hhased LoRA approaches. Additionally, this efficient decoupling mechanism, which is inspired by fly olfactory circuits, can be applied across various domains, helping researchers and developers leverage more powerful LoRA fine-tuning strategies. On the other hand, FlyLoRA could potentially be misused to fine-tune LLMs that exhibit biases or generate harmful content. We recommend implementing model access controls and minsmmoig frameworks when deploying this technique.
34


Reviews:
[{"review_id": "Rd92RKraHQ", "reviewer": "Reviewer_gKhQ", "summary": "This paper begins with the elegant observation that increasing number of experts while fixing total rank monotonically improves LoRA performance. However, existing routing mechanisms increase in parameter count linearly with number of experts, resulting in worse efficiency. To enable large numbers of low-rank experts as an efficient strategy, the authors introduce FlyLoRA, which removes the expensive routing mechanism in MoE-LoRA methods in favour of randomly initialised sparse (and provably near-orthogonal) random A projection, only learning the decoder B. This achieves empirically high performance on a variety of benchmarks against MoE- and standard LoRA, and has less degradation when merged.", "questions": "- Previous PEFT methods have used orthogonality as a design decision with success even without MoE, e.g. [OFT](https://arxiv.org/abs/2311.06243), [ReFT](https://proceedings.neurips.cc/paper_files/paper/2024/hash/75008a0fba53bf13b0bb3b7bff986e0e-Abstract-Conference.html). Why do you think this is the case, and is it related to why FlyLoRA is successful (beyond just the merging setting)? If you enforced orthogonality on the A matrix, would that help performance?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper is elegantly characterised, explaining and empirically justifying each of the design decisions in FlyLoRA with experiments and theory. Overall I found the context well-presented and easy to understand.\n- The method consistently achieves large performance gains over baselines and also has less degradation when merged, while being more parameter-efficient than LoRA and MoE variants.\n- The ablations in table 3 and figure 4 show that the theoretical claims made by the authors seem to hold up empirically, particularly (which I found surprising) frozen near-orthogonal A seems actively helpful beyond the multi-task merging setting.\n- The biological justification is quite interesting!\n\nWeaknesses\n- None apparent.\n\nTypos\n- line 206: updateble", "was_revised": true, "final_justification": "Satisfied by author response.", "rebuttal": "We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.\n\n**Q1: Connection between FlyLoRA and orthogonality design**\n\nBoth OFT [1,2] and LoReFT [3] operate on single tasks, but their orthogonal matrix $R$ is **multiplied** with the pre-trained weight matrix $W_0$, which differs from all LoRA variants (including FlyLoRA) that **add** $\\Delta W$ to $W_0$. The multiplication scheme **rotates the entire weight space**, and [1,2] demonstrate that this approach better adjusts semantic information compared to changing the magnitude. This is why the scheme succeeds. The additive scheme lacks this property, as $W_0$ cannot be rotated. Thus, although all use an orthogonality design, FlyLoRA succeeds in a different way.\n\nBy removing the MoE part and keeping only the random matrix $A$, FlyLoRA reduces to LoRA-FA [4] / Asymmetry LoRA [5], which saves resources but does not improve performance. We also tested enforcing the orthogonality constraint ($A^\\top A=I$) on an updatable $A$ (see Supplementary Table 1). The results show no significant accuracy difference compared to LoRA, confirming that **orthogonal design in LoRA does not boost single-task performance**.\n\nWe believe the **orthogonality design in LoRA excels in \"multi-task\" scenarios**—such as model merging (this work) and continual learning (O-LoRA [6])—because it **decouples parameter interference across multiple downstream tasks** when fine-tuning from the base model.\n\n**Supplementary Table 1: Single-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B.**\n\n|Method|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|\n|LoRA (r=8)|49.84±0.56|92.84±0.13|77.01±0.32|47.20±1.54|\n|Ortho-LoRA (r=8)|49.74±0.73|92.82±0.28|77.06±0.19|47.06±0.98|\n\n**Typos:**\n\nThanks for pointing it out. We will fix it in our the final version.\n\n**References**\n\n[1] Controlling Text-to-Image Diffusion by Orthogonal Finetuning. Neurips2023.\n\n[2] Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR2024.\n\n[3] ReFT: Representation Finetuning for Language Models. Neurips2024.\n\n[4] LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. Arxiv2023.\n\n[5] Asymmetry in Low-Rank Adapters of Foundation Models. ICML2024.\n\n[6] Orthogonal Subspace Learning for Language Model Continual Learning. EMNLP findings2023."}, {"review_id": "xdSBuGioM1", "reviewer": "Reviewer_vPRz", "summary": "The paper proposes FlyLoRA, a parameter-efficient fine-tuning method inspired by the fly olfactory circuit. It uses a frozen sparse projection for routing and rank-wise expert activation to reduce interference. This design improves both intra-task and inter-task decoupling, enabling efficient single-task learning and robust multi-task model merging. FlyLoRA outperforms standard and MoE-based LoRA methods across multiple domains with fewer trainable parameters.", "questions": "1. Corollary 3.4: How can the additive norm preservation property benefit model merging? The orthogonality part is intuitive.\n2. Why can we apply Theorem A.1 to Theorem A.2 where in the statement of A.1, $\\mathbf{A}$ is a random matrix sampled iid from standard normal, but in A.2 the matrix $\\mathbf{A}$ has exactly $p$ non-zero entries as how you construct the matrix in FlyLoRA.  \n3. Can you explain why Theorem 3.1 extends to Top-k activation from Equation (2) in the Appendix? \n4. For A.3, it’s better to move the zero-mean gradient assumption (line 36) above and make it explicit in the theorem statement.\n5. In Table 2, the current baselines may be too weak because there are many papers focusing on reducing interference of LoRA for merging but split-LoRA is designed to improve single task PEFT performance. For example, you can try to compare with [1,2].\n\n[1] Stoica, George, et al. \"Model merging with SVD to tie the Knots.\" The Thirteenth International Conference on Learning Representations.\n\n[2] Tang, Anke, et al. \"Parameter-Efficient Multi-Task Model Fusion with Partial Linearization.\" The Twelfth International Conference on Learning Representations. 2023.", "limitations": "Yes, in Appendix E.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strength\n**Quality**: The paper is overall technically sound. Most empirical claims are well supported by experiments. The methods used are well-motivated by both biological inspirations, classic random matrix theories and very practical concerns for the trade-off between MoE-based LoRA performance and efficiency. The authors mentioned the weaknesses in the appendix and addressed it properly. \n\n**Clarity**: The submission is clearly written and well-organized and based on the writing it should be easy to reproduce based on the figure visualization and equation 11. The authors are able to guide readers through the motivation to propose FlyLoRA step by step from section 2 to 3. \n\n**Significance**: The results will be very impactful: first the FlyLoRA shows better generalization performance with lower parameters; second FlyLoRA shows better performance also on PEFT model merging, and both first and second topic are very widely-used techniques; third all theoretical analysis can be proved by empirical advantages.\n\n**Originality**: To my best knowledge the formulation of FlyLoRA is new, although some parts of the theoretical analysis framework are similar to Johnson–Lindenstrauss analysis. This is not a weakness though, as I appreciate authors who show how to apply these frameworks to the setup of FlyLoRA. Theorem 3.2 is also insightful to explain the success of TopK sparsity in other applications.\n\n## Weaknesses\n**Quality**: I have a few important clarification questions for the theoretical section. Please see Questions for details. Besides, the model merging part would be strengthened by more comprehensive LoRA merging baselines.", "was_revised": true, "final_justification": "Although this paper still has some glitches for the mathematical rigor that cannot exactly explain the FlyLoRA properties (thus not a higher score), overall this is a high-quality paper, and the natural orthogonality with the random matrices is a very clever idea. Good job!", "rebuttal": "We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.\n\n**Q1: How can additive norm preservation property benefit for model merging**\n\n\"Additive Norm Preservation\" is a direct consequence of the \"Pairwise Orthogonality\" property described in Corollary 3.4. Both \"Pairwise Orthogonality\" and \"Additive Norm Preservation\" reflect the orthogonality property of FlyLoRA during model merging. According to the widely used geometric intuition (an assumption) that orthogonality benefits model merging [1–3], FlyLoRA's design aligns with this principle.\n\n**Q2: How can i.i.d sampling assumption applied to constraint on number of elements of each entry?**\n\nWe will modify the description in Theorem A.2 to: \"Given the matrix $A\\in\\mathbb{R}^{r\\times n}$ with each entry i.i.d. from $\\mathcal{N}(0, \\frac{1}{r^2})$ with probability $\\frac{r}{p}$, and set to $0$ otherwise,\" to make the proof more rigorous. Since this construction does not differ significantly from our original construction for $A$, we use this more analytically tractable form as a surrogate to study the distance-preserving property. We also compare the empirical performance of these two construction methods using Llama-3.1-8B in Supplementary Table 1 to support this approximation, with the difference in accuracy being negligible (all differences are within 0.2%). We will clarify this point in the final version.\n\n**Supplementary Table 1: Accuracy comparison of FlyLoRA using i.i.d versus Fixed-$p$ sparsity patterns.**\n\n|Method|MMLU|$\\Delta$ after merging|\n|:-:|:-:|:-:|\n|gaussian i.i.d. p|40.79±1.46|-2.13|\n|gaussian fixed p|40.88±1.61|-2.02|\n\n**Q3: How does Theorem 3.1 extend to Top-k activation?**\n\nWe apologize for the confusion. The goal of an MoE router is to select $k$ out of $r$ experts that best approximate the effect of all $r$ experts. In FlyLoRA, according to Theorem 3.1 and Appendix A.2, we assume that the value of $a_i x$ for $i = 1, 2, \\ldots, r$ is sufficient to serve as a surrogate for identifying the most informative experts. The top-$k$ operation, therefore, selects experts according to the magnitude of $a_i x$. We will clarify this assumption and its logical connection in the final version.\n\n**Q4: Placement of assumption**\n\nThanks for the suggestion. We will revise our text as recommended by the reviewer.\n\n**Q5: Comparison with other merging technique**\n\nWe include the comparison in Supplementary Table 2. KnOTS [4] and L-LoRA [3] are both built upon LoRA$_{(r=32)}$. The results suggest that they achieve comparable performance to FlyLoRA, with each method excelling on different datasets. It is noteworthy that FlyLoRA is not a competitor to these methods; rather, they can be used in a plug-and-play manner with FlyLoRA to further improve performance after merging.\n\n**Supplementary Table 2: Multi-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B. We report the accuracy drop relative to single-task performance.**\n\n|Method|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|\n|FlyLoRA|+6.55|-23.77|+4.80|-21.23|\n|KnOTS|+10.76|-26.85|+4.68|-23.37|\n|L-LoRA|+4.51|-22.48|+4.74|-20.85|\n|KnOTS+FlyLoRA|**+11.47**|-23.41|**+5.25**|-20.69|\n|L-LoRA+FlyLoRA|+7.65|**-21.42**|+5.02|**-19.85**|\n\n**References**\n\n[1] Editing Models with Task Arithmetic. ICLR2023.\n\n[2] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. Neurips2023.\n\n[3] Parameter-Efficient Multi-Task Model Fusion with Partial Linearization. ICLR2023.\n\n[4] Model merging with SVD to tie the Knots. ICLR2025."}, {"review_id": "E4zylOT7eG", "reviewer": "Reviewer_oMGp", "summary": "This paper proposes FlyLoRA, a novel LoRA variant inspired by the fly olfactory circuit, which improves parameter efficiency and task decoupling in both single-task and multi-task settings. The key innovations include: (1) using a frozen sparse random projection matrix $A$ to replace the conventional trainable down-projection, and (2) performing rank-wise top-k expert activation in the up-projection matrix $B$, eliminating the need for an explicit router. Theoretical analysis shows that the fixed sparse matrix preserves input distances and induces near-orthogonal subspaces, which mitigates parameter interference across tasks. Empirical results across several benchmarks using two backbones show FlyLoRA achieves strong performance with lower memory and compute overhead.", "questions": "- Can the authors discuss the sensitivity of the frozen matrix $A$ to different initialization schemes? Could non-Gaussian or structured projections affect performance?\n- How does FlyLoRA perform on significantly larger models (> 8B)? Are there bottlenecks in memory or convergence?\n- Can more advanced MoE-based LoRA baselines be included for comparison?\n- What causes the large performance drop after merging FlyLoRA on ScienceQA & GSM8K tasks?\n- Could ST+Trn outperform ST+Frz on some tasks, and if so, what trade-offs would it bring in merging or stability?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n* Quality: The paper presents a well-grounded method with both strong theoretical justifications (distance preservation, covariance reduction, orthogonality) and extensive empirical validation.\n* Clarity: The paper is well-written with clear motivation, architecture explanation, and figures. The supplementary materials provide thorough theoretical proofs and ablation studies.\n* Significance: FlyLoRA addresses a core limitation in existing LoRA and MoE-LoRA methods—parameter interference while improving computational efficiency. The method has broad applicability in fine-tuning large models under resource constraints.\n* Originality: The idea of combining sparse random projections with rank-wise MoE and top-k implicit routing, inspired by neuroscience, is novel and elegant. The routing-free design simplifies architecture while retaining strong performance.\n\n**Weaknesses**:\n\n* The initialization of the sparse random projection matrix $A$ uses a fixed Gaussian scheme. It remains unclear how sensitive the method is to this choice. Could other distributions or learned schemes improve or worsen performance?\n* Evaluation on larger backbone models is missing. It is unclear how FlyLoRA scales with increasing model sizes beyond 8B.\n* Only one MoE-LoRA baseline (Split-LoRA) is included. More recent or stronger adaptive/sparse fine-tuning baselines could strengthen the comparison.\n* In Table 2, performance of FlyLoRA + Llama-3.1-8B drops sharply on ScienceQA and GSM8K after merging. A deeper analysis of these drops would be useful.\n* The ablation in Table 3 shows marginal difference between ST+Trn and ST+Frz. It's unclear whether ST+Trn might outperform ST+Frz on other benchmarks.", "was_revised": true, "final_justification": "* The authors provided strong additional results in the rebuttal, including sensitivity analysis for the frozen projection matrix, evaluations on a 14B model, and comparisons with stronger MoE-based and adaptive baselines.\n* The explanation of merging-related performance drops using CKA and the detailed ablation studies between ST+Trn and ST+Frz effectively addressed my earlier concerns.\n* While some trade-offs remain inherent to the merging process, the core claims are now better supported both theoretically and empirically.\n* Overall, the rebuttal meaningfully strengthened the paper and justified an increase in my score.", "rebuttal": "We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.\n\n**W1&Q1: More sensitivity studies on initialization schemes of matrix $A$**\n\nAs the reviewer suggested, we further compare three methods for generating the sparse projection $A$ with $\\frac{p}{r}$ sparsity: \n\n- Gaussian (our default): Each non-zero entry is drawn from $\\mathcal{N}(0,\\frac{1}{r^2})$；\n- Rademacher (non-Gaussian): Each non-zero entry is $±\\frac{1}{r}$ with equal probability.\n- FJLT [1] (structured projection): $A=PHD$, where $D$ is a random diagonal matrix with independent Rademacher variables on its diagonal, $H$ is a normalized Hadamard matrix, and $P$ enforces the $\\frac{p}{r}$ sparsity.\n- Two-Phase (briefly-learned): The non-zero entries of $A$ are trainable for 5% of the total steps (warm-up), then frozen for the remainder.\n\nThe results, shown in Supplementary Table 1, indicate that almost all variants perform similarly. Non-Gaussian, structured, or briefly-learned initializations have little impact, except that the briefly-learned scheme shows a noticeable drop after merging. This demonstrates that FlyLoRA is robust to the choice of initialization scheme for matrix $A$, and that learning may break the approximate orthogonality of the random matrix, making it unsuitable.\n\n**Supplementary Table 1: Single-task and multi-task accuracy comparison on MMLU using Llama-3.1-8B.**\n\n|Method|before|$\\Delta$ after merging|\n|:-:|:-:|:-:|\n|Gaussian|40.88±1.61|-2.02|\n|Rademacher|40.42±0.23|-2.35|\n|FJLT|40.57±1.34|-2.50|\n|Two-Phase|40.76±1.04|-4.86|\n\n**W2&Q2: Evaluation of larger models**\n\nWe further conducted experiments using the Qwen-2.5-14B model, as shown in Supplementary Tables 2 and 3, following the setting of Tables 1 and 2. The results show that FlyLoRA remains superior in accuracy (for both single-task and multi-task settings) and is more parameter-efficient. We encountered no memory or convergence bottlenecks when training FlyLoRA on the 14B model, which consistently outperforms LoRA.\n\n**Supplementary Table 2: Single-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-14B.**\n\n|Method|Param(%)|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|LoRA$_{(r=8)}$|0.23|56.74±0.56|95.62±0.18|83.08±0.74|51.69±1.48|\n|LoRA$_{(r=32)}$|0.93|59.35±0.79|97.05±0.22|85.31±0.25|54.80±0.76|\n|LoRA-Split$_{(4\\times 8)}$|0.29|58.26±1.13|96.85±0.35|84.88±0.51|54.65±0.59|\n|**FlyLoRA$_{(k=8)}$**|**0.12**|**60.17±1.08**|**97.37±0.32**|**85.96±0.89**|**56.42±1.16**|\n\n**Supplementary Table 3: Multi-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-14B. We report the accuracy drop relative to single-task performance using weight averaging technique.**\n\n|Method|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|\n|LoRA$_{(r=8)}$|-13.75|-25.20|-11.43|-18.60|\n|LoRA$_{(r=32)}$|-8.91|-20.45|-7.62|-16.34|\n|LoRA-Split$_{(4\\times 8)}$|-7.48|-21.97|-6.05|-14.87|\n|**FlyLoRA$_{(k=8)}$**|**-4.35**|**-17.89**|**-2.18**|**-11.72**|\n\n**W3&Q3: More baseline into comparison**\n\nWe further compare several strong and widely used baselines—AdaLoRA [2] (adaptive), SoRA [3] (sparse), and HydraLoRA [4] (MoE)—using Qwen-2.5-7B, as shown in Supplementary Tables 4 and 5. The results demonstrate that FlyLoRA remains superior in terms of accuracy and efficiency in both single-task learning and multi-task merging scenarios.\n\n**Supplementary Table 4: Single-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B.**\n\n|Method|Param(%)|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|AdaLoRA$_{(r=8)}$|0.26|51.22±0.21|93.48±0.28|77.65±0.14|47.96±1.34|\n|SoRA$_{(r=8)}$|0.19|50.89±0.42|93.25±0.20|78.46±0.82|47.83±0.94|\n|HydraLoRA$_{(r=8, A=1, B=3)}$|0.52|53.05±0.16|94.69±0.34|79.31±0.49|52.98±1.57|\n|**FlyLoRA$_{(k=8)}$**|**0.13**|**53.68±0.47**|**95.55±0.18**|**80.82±0.56**|**54.34±2.13**|\n\n**Supplementary Table 5: Multi-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B. We report the accuracy drop relative to single-task performance using weight averaging technique.**\n\n|Method|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|\n|AdaLoRA$_{(r=8)}$|-10.90|-33.15|+4.52|-25.46|\n|SoRA$_{(r=8)}$|-9.45|-34.81|+4.05|-26.50|\n|HydraLoRA$_{(r=8, A=1, B=3)}$|-6.22|-34.67|+4.21|-24.94|\n|**FlyLoRA$_{(k=8)}$**|**+6.55**|**-23.77**|**+4.80**|**-21.23**|\n\n**W4&Q4: Analysis in performance drop between merge and non-merge settings**\n\nThe four tasks—general knowledge understanding (MMLU), scientific question answering (ScienceQA), mathematical reasoning (GSM8K), and code generation (HumanEval)—represent significantly different distributions. As a result, merging their adapters is prone to major conflicts that can cause performance drops.\n\nEmpirically, following [5], we use centered kernel alignment (CKA) [6] to quantify the alignment between the output representations of each single-task adapter and the merged adapter. A higher CKA indicates better output alignment, which is likely the inherent reason, and therefore, results in a smaller accuracy drop after merging. Supplementary Table 1 reports both CKA and accuracy drop ($\\Delta$) on Llama-3.1-8B. Since there's no apparent difference between LoRA$(r=8)$, LoRA$(r=32)$ and Split-LoRA$(4 \\times 8)$ in model merging, we use LoRA$(r=8)$ as a representative to compare with FlyLoRA$(k=8)$. We observe that tasks with lower CKA (especially ScienceQA and GSM8K) suffer the largest accuracy drops. This micro-level analysis helps explain the substantial performance decreases observed on ScienceQA and GSM8K. It is important to note that such degradation is an inherent phenomenon of model merging, not a weakness of FlyLoRA. In fact, FlyLoRA still outperforms its LoRA variants (Tables 1 and 2), even in cases where accuracy falls sharply.\n\n**Supplementary Table 6: CKA and corresponding accuracy drop ($\\Delta$) between single-task adapter and merged model.**\n\n|Method|Task|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|LoRA$_{(r=8)}$|CKA|0.78|0.39|0.58|0.75|\n||$\\Delta$|-6.48|-60.34|-30.15|-13.04|\n|FlyLoRA$_{(k=8)}$|CKA|0.85|0.53|0.71|0.84|\n||$\\Delta$|-2.02|-43.05|-21.81|-4.27|\n\n**W5&Q5: More ablation studies between ST+Trn and ST+Frz**\n\nWe further conducted ablation studies on the GSM8K, ScienceQA, and HumanEval datasets, with results presented in Supplementary Table 7. The results show that ST+Trn does not outperform ST+Frz, with almost no difference in accuracy. Previous papers have already demonstrated that, in LoRA, updating matrix $A$ is not necessary [7,8]. Our results further validate this finding in the MoE-based LoRA setting. Thus, ST+Trn offers no clear advantage in this context, while ST+Frz can help reduce both interference during merging and the memory footprint of activation values [7].\n\n**Supplementary Table 7: Ablation study of FlyLoRA variants using Llama-3.1-8B.**\n\n|Method|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|\n|ST+Trn|94.10±0.47|58.68±0.52|36.81±1.36|\n|ST+Frz|94.15±0.36|58.76±0.74|36.88±1.91|\n\n**References**\n\n[1] The fast Johnson-Lindenstrauss transform is even faster. ICML2023.\n\n[2] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR2023.\n\n[3] Sparse Low-rank Adaptation of Pre-trained Language Models. EMNLP2023.\n\n[4] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. Neurips2024.\n\n[5] Model merging with SVD to tie the Knots. ICLR2025.\n\n[6] Similarity of Neural Network Representations Revisited. ICML 2019.\n\n[7] LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. Arxiv2023.\n\n[8] Asymmetry in Low-Rank Adapters of Foundation Models. ICML2024."}, {"review_id": "Bng11vcRCs", "reviewer": "Reviewer_ruPf", "summary": "This paper introduces FlyLoRA, a novel PEFT) method inspired by the fly olfactory circuit. FlyLoRA addresses the limitations of existing LoRA methods, specifically parameter interference and computational inefficiency, through implicit rank-wise expert activation and frozen sparse random down-projections. The method theoretically decouples intra- and inter-task interactions, leading to enhanced decorrelation in both single-task instruction tuning and multi-task model fusion. Experimental results demonstrate consistent performance improvements over existing PEFT methods and significantly reduced performance degradation in multi-task merging scenarios.", "questions": "Please refer to weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. FlyLoRA proposes a unique approach to PEFT by incorporating implicit rank-wise expert activation and frozen sparse random projections, inspired by biological neural structures, which is a compelling and innovative design.\n2. By eliminating the need for explicit router parameters and leveraging inherent sparsity, FlyLoRA offers improved computational efficiency compared to other MoE-based LoRA variants.\n\nWeaknesses:\n1. The paper does not sufficiently explain why there is such a significant difference in results between merged and non-merged scenarios. A deeper analysis of this discrepancy would enhance the understanding of FlyLoRA's behavior in diverse contexts.\n2. The evaluation results, while promising, are not comprehensive enough. The paper would benefit from testing on more challenging and common benchmarks such as GPQA, LiveCodeBench, and advanced Math reasoning datasets, to thoroughly assess FlyLoRA's robustness and scalability.", "was_revised": true, "final_justification": "The paper is quite solid and interesting, and below are the final justifications.\n1. The paper introduces FlyLoRA, which is a compelling and innovative design.\n2. The paper explains why there is such a significant difference in results between merged and non-merged scenarios and addresses my concern.\n3. The evaluation results are comprehensive enough, which include more challenging and common benchmarks such as GPQA, LiveCodeBench, and advanced Math reasoning datasets.", "rebuttal": "We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.\n\n**W1: Analysis of the performance gap between merged and non-merged scenarios**\n\nIntuitively, the four tasks—general knowledge understanding (MMLU), scientific question answering (ScienceQA), mathematical reasoning (GSM8K), and code generation (HumanEval)—represent significantly different distributions, so merging their adapters is prone to substantial conflicts.\n\nEmpirically, following [1], we use centered kernel alignment (CKA) [2] to quantify the alignment between the output representations of each single-task adapter and the merged adapter. A higher CKA indicates better output alignment, which is likely the inherent reason, and therefore, results in a smaller accuracy drop after merging. Supplementary Table 1 reports both CKA and accuracy drop ($\\Delta$) on Llama-3.1-8B. Since there's no apparent difference between LoRA$(r=8)$, LoRA$(r=32)$ and Split-LoRA$(4 \\times 8)$ in model merging, we use LoRA$(r=8)$ as a representative to compare with FlyLoRA$(k=8)$. We observe that tasks with lower CKA (especially ScienceQA and GSM8K) suffer the largest accuracy drops. FlyLoRA consistently yields higher CKA than LoRA, which aligns with its consistently smaller $\\Delta$. This micro-level analysis corroborates why FlyLoRA outperforms LoRA in heterogeneous-task merging. All of this new analysis will be included in the final version.\n\n**Supplementary Table 1: CKA and corresponding accuracy drop ($\\Delta$) between single-task adapter and merged model.**\n\n|Method|Task|MMLU|ScienceQA|GSM8K|HumanEval|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|LoRA$_{(r=8)}$|CKA|0.78|0.39|0.58|0.75|\n||$\\Delta$|-6.48|-60.34|-30.15|-13.04|\n|FlyLoRA$_{(k=8)}$|CKA|**0.85**|**0.53**|**0.71**|**0.84**|\n||$\\Delta$|**-2.02**|**-43.05**|**-21.81**|**-4.27**|\n\n**W2: Evaluating on more challenging benchmarks**\n\nAs suggested by the reviewer, we further evaluated single- and multi-task performance on LiveCodeBench (using CodeAlpaca checkpoints), GPQA (using MMLU checkpoints), and MATH (for advanced math reasoning, using GSM8K checkpoints) with Qwen-2.5-7B. The results in Supplementary Tables 2 and 3 indicate that FlyLoRA remains robust even on more challenging test sets.\n\n**Supplementary Table 2: Single-task accuracy comparison (Pass@1 for LiveCodeBench) using Qwen-2.5-7B.**\n\n|Method|Param(%)|GPQA|MATH|LiveCodeBench|\n|:-:|:-:|:-:|:-:|:-:|\n|LoRA$_{(r=8)}$|0.26|34.9±0.6|45.7±0.4|12.4±0.1|\n|LoRA$_{(r=32)}$|1.05|36.8±0.2|47.7±0.1|14.8±0.2|\n|LoRA-Split$_{(4\\times 8)}$|0.33|36.2±0.3|46.9±0.3|14.5±0.1|\n|**FlyLoRA$_{(k=8)}$**|**0.13**|**37.4±0.3**|**48.1±0.4**|**15.2±0.1**|\n\n**Supplementary Table 3: Multi-task accuracy comparison (Pass@1 for LiveCodeBench) using Qwen-2.5-7B. We report the accuracy drop relative to single-task performance using weight averaging technique.**\n\n|Method|GPQA|MATH|LiveCodeBench|\n|:-:|:-:|:-:|:-:|\n|LoRA$_{(r=8)}$|-0.5|-4.1|-3.8|\n|LoRA$_{(r=32)}$|-1.6|-4.4|-3.4|\n|LoRA-Split$_{(4\\times 8)}$|+0.4|-5.2|-3.5|\n|**FlyLoRA$_{(k=8)}$**|**+1.3**|**-2.4**|**-1.8**|\n\n**References**\n\n[1] Model merging with SVD to tie the Knots. ICLR2025.\n\n[2] Similarity of Neural Network Representations Revisited. ICML 2019."}]

Author rebuttal:

Meta-review: In their paper, the authors introduce FlyLoRA, a new version of LoRA based on an implicit Mixture of Experts (MoE) model. FlyLoRA improves on the original method by: (i) Using an implicit routing system to activate experts for each rank in the up-projection matrix; (ii) Replacing the standard dense, updatable down-projection matrix with a fixed, sparse random projection.

All the reviewers are positive about the contributions of the papers, including: (1) the novelty of the proposed FlyLoRA; (2) the experiments are sufficient to support the claims about the interpretability and performance of the FlyLoRA; (3) the paper is written and presented clearly. After the rebuttal, most of the concerns of the reviewers were addressed, and all the reviewers are happy with the current stage of the paper.

In my opinion, the contributions and originality of the proposed FlyLoRA are sufficient for acceptance at NeurIPS. Therefore, I recommend accepting it in its current form. I encourage the authors to address the reviewers’ suggestions and integrate their feedback into the camera-ready version of their paper.
Decision: Accept (poster)