Expert Merging in Sparse Mixture of Experts with Nash Bargaining
Authors: Dung Viet Nguyen
Anh Nguyen Thi
Minh Hoang Nguyen
Luc Nguyen
Shiqi Jiang
Ethan Fetaya
Linh Duy Tran
Gal Chechik
Tan Minh Nguyen
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture of Experts
Game Theory
Primary area: foundation or frontier models, including LLMs
TL;DR: NAMEx uses Nash bargaining and complex momentum to merge experts more fairly and efficiently, outperforming prior methods across tasks.

Abstract: Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modeling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx’s scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings.

Published as a conference paper at ICLR 2026
EXPERT MERGING IN SPARSE MIXTURE OF EXPERTS WITH NASH BARGAINING
Dung V. V.Nguyen' Anh T. Nguyen2
Minh H. Nguyen'
Lue Q. N.guyen?
Shiqi Jiang'
Ethan Fetaya' Linh Duy Tran
Gal Chechik44
Tan M. Nguyen"
'Department of Mathematics, National University of Singapore
Viettel AI, Viettel Group
Faculty of Mathematics and Informatics, Hanoi University of Science and Technology 1 "Bar Iln University, Israel
$ AA Imaging Team, Data Solution Department, FPT Software Japan (dungnv, shiqijiane Bu.nus edu, tanmn nus edu. sg (ethan. fetayn, gal. chechik abiu.ac.il. mhanhhhh23331Mee
hust edu.vn vn [anhnt21, luenql @evietell... com. 1inhtd328 fpt.com. com
ABSTRACT
Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependen or input-independent averaging of expert parameters, but often lack principled weighting mechanism. In this work, reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based this perspective, of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modeling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEX'S 's scalability by applying it to large-scale systems, including Qwenl.5-Men effective in both shro-shot and tie-ttnnng settings. The code is publicly available at: httph:tt////t g thub. com/ anh 1orh1447NNMA /NAMEX.
introduce Nash Merging
(14B) and DeepSeck-Men (16B), where it proves
INTRODUCTION
Scaling up neural networks without proportional increases in computational cost key goal in modern deep learning. Sparse Mixture of Experts (SMoE) architectures offer powerful solution: they selec- tively activate only a subset of expert modules for each input, thereby maintaining high capacity while preserving computational efficiency. Building on the classical Mixture of Experts (MoE) framework (Ja- cobs et al., 1991), SMoE leverages dynamic gating mechanism to determine which experts participate in processing given input. This sparsity allows extremely large models to be trained efficiently and has shown promise across natural language processing (Shazeer et al., 2017; Liu et al., 2024a; Yang et al., 2024), and computer vision (Ruiz et al., 2021; Puigcerver et al., 2024a; Nielsen et al., 2025) applications.
core component of SMoE is the routing mechanism, which dynamically determines expert assignments. Significant efforts have focused on improving routing stability, load balancing, and expressiveness. For example, StableMofF (Dai et al., 2022) introduces a two-stage strategy to reduce routing variance; SMEAR (Muqeeth et al., 2024) proposes soft parameter merging via weighted averaging to bypass discrete selection; and HyperRouter (Do et al., 2023) uses hypernetworks to generate router parameters. Mcanwhile, SoftMoE (Puigcerver et al., 2024a) blends sparse and dense routing, and patch-level routing (Chowdhury et al., 2023) improves sample efficiency in visual tasks. Beyond these, Switch Transformer (Fedus al., 2022) simplifies routing to top-1 expert
"Co-first authors. CCo-last authors. Correspondence dungnv @ u.nus.edu & tanmn@ tanmn@nus.edu.sg

---

Published as a conference paper at ICLR 2026
selection with auxiliary load-balancing losses for stable large-scale training, GShard (Lepikhin et al., 2021a) introduces expert capacity constraints and scalable sharding strategies, and BASE Layers (Lewis et al., 2021) formulates routing as balanced assignment problem to improve expert utilization. Hash-based routing (Roller et al., 2021) replaces learned routers with deterministic hashing to reduce overhead, while DSelect-k (Hazimeh al., 2021) provides differentiable relaxation for sparse expert selection to improve gradient flow. Collectively, these approaches span hard, sparse routing with explicit balancing objectives to soft and differentiable formulations, reflecting broad design space aimed at enhancing stability, scalability, and specialization in SMoE systems. Beyond routing, a complementary yet underexplored direction is expert merging. Instead of selecting subset of experts per input, merging aims to combine all expert parameters into a unified model, either during training or at inference. This approach is especially appealing when deployment memory constraints demand single-cxpert representation. Merging is particularly valuable autore- gressive models (Zhong al., 2024) and cross-domain transfer settings (Chen et al., 2022). However, most current merging techniques, such as soft-merging (Muqeeth al., 2024: Zhong et al., 2024) and top-k aggregation (He et al., 2023: Li et al., 2024), rely on heuristic weighting schemes that ignore the intricate dynamics between experts.
Recent work has begun address this limitation. (Nguyen et al., 2025) introduces curvature aware merging scheme, namely Curvature- aware merging of experts (CAMEx), that uses natural gradients to account for non-Euclidean geometry parameter space. A variant, corresponding to the dynamic merging (Dynamic-Merg) mechanism in the CAMEx paper, which we refer to as CAMEX (EP-CAMEX), propagates base expert across layers to promote inter-layer communication. However, despite its elegance, EP-CAME: underperforms its static variant, likely due to insufficient coordination among expert question: Can we interpret expert merging as a structured interaction among experts, rather than just linear average?
Figure I: Cosine similarity of expert outputs in Swin-MoE (Liu et al., 2021) (top), Switch- Transformer (Fedus et al., 2022) (middle), and Qwen-MoE (Yang et al., 2024) (bottom). Swin-MoE shows stable mid-layer features, Switch- Trach-Transfmmer exhibits dynamic rout- ing Layer 8, and Owen-MoE yields robust final representations at Layer 9 h-highlighting diverse expert interaction patterns.
contributions. This motivates a deeper
Contribution. In this paper, we frame expert merging as cooperative-compeiin game among experts. Drawing inspiration from multi-task learning, we adopt the Nash Bargaining Solution (NBS) (Nash, 1950) to derive merging coefficients from first principles based on each expert's s contribution. Our method, named Nash Merging of Experts (NAMEx), treats expert domain vectors as utility functions in bargaining game. By solving for the optimal agreement point, NAMEx ensures fair and efficient merging process that reflects expert alignment and divergence.
To address the slow convergence of EP-CAMEX, we further integrate complex momenium (Lorraine et al., 2022) into the propagation process. This enhancement accelcrates convergence while preserving stability, especially when expert interactions include adversarial or conflicting dynamics. We theoret- ically prove the convergence of NAMEx under mild conditions and provide spectral radius- based bound for the convergence rate of NAMEx Our contribution is three-fold:
1. We develop NAMEx, new expert merging method that integrates the Nash Bargaining opti- mization framework of (Navon et al., 2022) into EP-CAMEx (Nguyen et al., 2025), improving expert propagation at each SMoE layer.
2. We incorporate complex momentum into our NAMEx to enhance the stability and convergence speed of expert propagation across layers and provide theoretical guarantees.
3. We demonstrate that quaternion momentum presents promising future direction for further improving expert merging.
Comprehensive experiments across diverse tasks- includiin WikiText- 103 language modeling (Merity 2017), GLUE text classification finetuning (Wang et al., 2018), and ImageNet- 1k image classifica-
2

---

Published as a conference paper at ICLR 2026
Seperd
Router
E
Figure 2: Architecture overview of (a) CAMEx (Nguyen et al., 2025), (b) Expert-rooaagation CAMEX (Nguyen al., 2025), and (c) our proposed merging method, NAMEx.
tion and zero-shot robustness under data corruption (Deng et al., our approach, achieving superior accuracy compared baseline methods while preserving advantages in computational efficiency. Moreover, establish NAMEx's scalability by deploying it large systems such as Qwen 1.5MMEE (I4B) and DeepSeck-MoE (16B), where it delivers strong performance in both zero-shot and fine-tuning scenarios.
the effectiveness of
Organization. Section reviews SMoE, CAMEX, and Nash Bargaining: Section introduces NAMEx and its momentum extension; Section 4 presents experiments with ablations; Section B discusses related work; and Section concludes with limitations.
BACKGROUND
2.1
SPARSE MIXTURE OF EXPERTS
The Mixture of Experts (MoE) framework enables modular neural computation by combining multiple specialized sub-networks (experts) through a gating function (Jacobs et al., 1991). The Sparse Mixture of Experts (SMoE) variant enhances scalability by activating only small subset of experts per input, significantly reducing computation during training and inference (Shazeer et al., 2017; Lepikhin et al., 202 lb; Fedus al., 2022).
Let Rd be an input and fx(x), denote expert outputs. such that:
gating network computes weights sx(x)
N
(x:0,) 00
Σ (x;00)))=1 F(x)
Σ 8,
(x)x
SMoE improves model capacity without linearly scaling compute, making a central design in recent large-scale architectures.
2.2
CURVATURE- AWARE MERGING EXPERTS
CAMEx uses natural gradients to align merged experts more closely with the geometry of the parameter space, enhancing both pre-training and fine-tuning processes (Nguyen et al., 2025). Hence, CAMEx generalizes popular expert merging methods such as SMEAR (Mugeeth et al., 2024) and Lory (Zhong al., 2024) and can be formulated the following natural gradient- like merging scheme:
E
+1 T7C
(1) M.
(s) +T
where E.!) E!! the domain- vector of the i-th expert, representing its deviation from the base expert. E E!! and ((() denote the weights of the i-th expert, the base expert, and the resulting merged expert that processes the input, respectively. Here, the base expert E!!!) is shared between tokens in layer I, just like in DeepSeek- V2 (Liu al., 2024a) and V3 (Liu al., 2024b). v) denotes the stepsize for updating the base expert, and M is the curvature matrix for the i-th expert. EP-CAMEx extension of CAMEx, in which the base expert (0) initialized at the first layer, and E!! and
b:(!) are updated at subsequent layers as follows:
N
((+1)
+
7
M
N
(3)
N
((+1)
(++1)
M (++1), (s (1-1)
+T (1-11)
+n

---

Published as a conference paper at ICLR 2026
Here. 7 > 0 denotes the step size for the propagation of the base expert Em In the first equation of system (3) above, if we view each domain vector-ecorr T; as a "gradient direction" attempting to pull the base expert toward the corresponding expert's domain, then the formulation can be interpreted as dynamical system that updates E using Multiple- Gradient Descent Algorithm (MGDA) (Désidéri, 2012) to minimize the distance between Em and i-th domain. Consequently, this can be framed as a multi-objective optimization or multi-task learning problem. We illustrate both CAMEX and EP-CAMEx in Figure 2(a) and (b).
2.3 NASH BARGAINING IN MULTI-TASE LEARNING
The Nash Bargaining Solution (NBS) (Nash, 1950) is a foundational concept in cooperative game theory, describing how multiple agents can reach fair and Pareto-optimal agreement. A bargaining problem is typically defined by agreement set of outcomes S C TR-V and disagreement point d e R which specifies the utility each player receives no agreement reached. The NBS selects an outcome u ES that maximizes the product of individual gains over the disagreement point:
N
u
argmna uES
II (tu -d:-d)).
(4)
The disagreement point in the Nash Bargaining Problem is the fallback outcome each player receives no agreement is reached. It serves as a baseline against which any cooperative agreement is measured, shaping the set feasible solutions. Players often consider their disagreement point strategically, as improvements to it can strengthen their bargaining position and influence the final outcome.
Recent work by (Navon et al., 2022) demonstrates that multi-task learning (MTL) can be naturally framed as a bargaining game. In this setting, each task corresponds to player, and the goal is to determine a shared parameter update direction AO that benefits all tasks. The agreement set is typically constrained to unit ball B. ( | S c). while the disagreement point is set to zero, indicating no parameter update. Each task provides a utility function
(5) where т; is the gradient of the task-specific loss with respect the model parameters. Under the assumption that these gradients are linearly independent, the NBS yields the optimal update direction:
uy ( =T; A6,
N
AO
OGTi where G T Ga=1/a,
with G G= and 1/a denoting element-wise reciprocals.
This formulation provides a principled way to resolve conflicting gradients, balancing cooperative and adversarial dynamics among tasks. In this paper, we leverage this framework to reinterpret expert merging in SMoE and particularly, CAMEx, a bargaining game among experts, where each domain (1) vector T _i= L....,NN plays the role of a task gradient.
NASH MERGING OF EXPERTS
Building the foundations of CAMEx and Nash Bargaining, we introduce NAMEX-a novel method for merging experts in SMoE via Nash Bargaining. Rather than treating expert merging as simple averaging task, NAMEx models it as a multi-agent bargaining game, where each expert proposes a directional update, i.e., its domain vector, and the merged expert is obtained through a principled aggregation reflecting both cooperation and competition. To address slow convergence in existing propagation methods like EP-CAMEX, we further introduce complex momentum into NAMEx, enabling faster and more stable propagation through SMoE layers. An overview of our approach is shown in Figure 2(c).
3.1
MERGING EXPERTS AS A BARGAINING GAME
Setting ( as an update direction for E!() of the I-th layer in the first equation of system (3), we adjust the expert- proprttpoppggatig updating step in EP-CAMEx as follows:
Em
((+1)
Em + (1))
((+1)
-E(++1) +1) >
N
M, (s)
(s/ +T (1+1)).
(1+1)
(7)
4

---

Published as a conference paper at ICLR 2026
Like CAMEx, we view the domain-vectors Em toward E," 's domain. However, different from the formulation of EP-CAMEx in system (3), we remove the curvature matrix in the first equation to align with the Bargaining Game given by Algorithm 1 in (Navon al., 2022). Our goal now is to find optimal update vector Ag(") which benefits all experts, i.c.. finding a (1) = [ai 2² ,..,, to aggregate the domain- vectors T into AE as in Eqn. 7. We hypothesize that experts in SMoE engage in mixed games comprising both cooperative and competitive dynamics.
E
Em analogous to gradient step that pulls
Layer-Wise Expert Interaction Dynamies. Following the analysis protocol described (Lo et al., 2025), we observe that expert behavior varies by layer and architecture (see Figure 1), revealing both cooperative and adversarial patterns. instance, in Swin-MoE (Liu et al., 2021), middle layers show high inter-expert similarity, while Qwen-MoF. (Yang et al., 2024) concentrates alignment in deeper layers. This motivates a dynamic, layer-wise approach to merging merging-exactly exactly what NAMEx provides. Please refer to Figure 8 and Figure 6 in Appendix G,4 for more analysis the dynamic of expert interaction. For comparison regarding expert interaction patterns under the impact of Load Balancing loss, please refer Figure 9 in Appendix G.4.
Adapting the bargaining game's formulation in (Navon et al., 2022), NAMEx solves the following problem:
[Bargaining of Expert Merging (BEM) Problem] Given an experts-merging, problem with the set of expert parameters Eeg.....,..E and the base expert's parameter Emn find an update vector ΣE within ball B. of radius centered at zero, i.e., B. [ |||
Inspired by (Navon et al., 2022), in this bargaining problem, we set the disagreement point to O, corresponding to not updating Em- Similar to Eqn. 5, the utility function for each expert is defined as u, (,EEE)= = AE, where is the domain- vecttor for expert i, representing its deviation from the base expert and capturing its unique contribution to the merging process. Here, AE is equivalent to AO in Eqn. 5. We have the following mild axiom the Nash bargaining solution.
Axiom 3.1 (Pareto optimality of Nash bargaining solution (Nash, 1950)). The selected agreement must be Pareto efficient, i.e. no other feasible outcome should exist that improves one player's utility without reducing the utility of at least one other player.
Under Axiom 3.1, the solution to the BEM Problem above is given by the following lemma.
Lemma 3.2 (Nash Solution of Expert Merging). Let denote the d matrix whose columns are the domain-vectors TÚ. The solution
N V log((E)) (AE
arg max AEEB.
(8)
is (up to scaling) ΣE -E OCTi where a E R. satisfies G T Ga = 1/a, with 1/a being the element-wise reciprocal.
Note that, under Axiom 3.1, it can be proven that the Nash solution to the bargaining problem is not dominated by other solutions. A proof sketch for Lemma 3.2 is provided in Appendix C.I.
3.2 NAMEX AS THE NASH SOLUTION OF EXPERT MERGING
We now formally define NAMEx as the Nash Bargaining Solution to the BEM problem.
Definition 3.3 (NAMEx: Nash Merging of Experts). Let (E)) (e be the expert parameters
and let (1) denote the base expert at layer I. Define the domain-vecton as T = -E!! -E(i) and
let G -(7 .....TXX [".,"""".., be the matrix formed by stacking these vectors. The NAMEx update direction
(1) (g(h)
is defined
AE (()
where ! E RN satisfies the Nash Bargaining equation: ( - 1/a with 1/ denoting the element-wise reciprocal. The NAMEx update then proceeds by plugging NAMEx update
5

---

Published as a conference paper at ICLR 2026
direction into Eqn. 7:
N
Em
Em
+7
(1)
N
Em ((++))
(L+1)
+1
M2
(s)
*T;
(111).
where y R step-size coefficients, M; is the curvature matrix expert i, and 8; routing weights at layer 1+1.
(1+1)
the
summarize the implementation of NAMEX Algorithm
Dissecting NAMEx. We now discuss the behavior of NAMEX by studying the Nash Solution of the
BEM Problem. First, all rj orthogonal, obtain 0j |ry
and =E= =E== == Ojjjj. which is
scale-invariant solution. When Ty are not orthogonal, we obtain
a111,,11 Σ EO
(10)
Lemma 3.2 allows us to calculate the optimal update direction AE for an ppperttroopagation step at 1-th layer as Ag = 1
Furthermore, assuming that EP-CAMEx obeys the update law in Eqn. 7 in (Désidéri, 2012), the norm ||T is (nearly) identical between domain vectors, we can view the expert update step in (Nguyen et al., 2025) as a trivial solution (with scaling factor) of Lemma 3.2, ignoring the interaction between experts. While they also apply curvature matrices the expert propagating step, the learned curvature matrices provide no additional information about other experts. Thus, the conclusion still holds.
Algorithm Expert Merging via Nash Bar- gaining
Algorithm NAMEX-Momentie
1: Input: 7 R B BBC, ((())
ECd E())
Initialize: Model M with L SMoE layers,
IRd
number of experts N, 7.77 R
2:
fo L- do
H BRBxSxN, router logits layer
3:
for fori=10 to do
T(() BBx5xD; token sequence at
4:
EE( E
layert
5:
end for
fort= L do
GU) - ( ,.... S
6:
5: for fori=1 to
N do
Solve a from: (G G
7:
end for
T;
(+)
- E
8:
Ag + &
-
T (1)
(Same update as
8:
G( -
[7
22 ....TN (t)
NAMEX)
n(++1) + B1
Solve for a: (G a= 1/0
10:
B(++1) <- ( + +
10:
((+1) (l0)
+7 E, T O2
(1)
11:
// Optionul: Add residual alignment or router term needed
11: (t+1), t E (1+1)
+
12:
end for
12:
end for
In Eqn. 10, we can consider Σns, CsT, T OsT Try as the interaction between the j-th expert and the other experts. If the sum is positive, the experts cooperate, and the other comain-vectoor aid the j-th expert. ay decreases in this case. If the sum is negative, the other experts hamper the j-th expert, ie., an adversarial behavior between experts, and therefore, ay increases to ensure that Eqn. 10 holds.
3.3 INTEGRATING MOMENTUM INTO EXPERT MERGING
We hypothesize that one reason for EP-CAMEx's 's inferior performance compared to CAMEx is its reliance fixed number of update steps, constrained by the model's layer count. This limitation hinders the convergence of the base expert in later stages, leading to suboptimal performance. To mitigate this, we introduce momentum accelerate convergence during optimization. In particular,

---

Published as a conference paper at ICLR 2026
adopt complex momentum (Lorraine et al., 2022), which has been shown be more robust and effective than standard first-order methods across a wide range of cooperative and adversarial games. By integrating complex momentum into expert merging, enhance the propagation of expert updates across layers and provide theoretical support for its improved convergence rate.
We present formal definition NAMEEx-Momentum below and summarize algorithm to implement it in Algorithm 2.
Definition 3.4 INAMEx-Momentume Nash Merging with Complex Momentum). Let {Ej,...,BB E.y] } be the expert parameters and Em the base expert at a given layer. Define the domain-vectors T; =E; Em
and the matrix G = [ TN. At each iteration j, the update direction Ag = =>i= computed where solves the Nash system:
ii=1 0;Ti is
GGa=1/aa
NAMEx-Momentum uses a complex momentum buffer ( E accumulate directional updates:
pG+1)
= =8p(1) AE())
Em b(+1)
-El! + +3(746+)1)
(11)
N
((++1)
E(+11) +n >nMM; M. (1+1)
+T, ( (1+1))
1=1
where B is the momentum coefficient, FERt is the step size, and R(.) denotes the real part.
We provide a convergence guarantee for Theorem C.3 in Appendix C. Their proofs are in Appendix C.3
update in Proposition 3.5 below and
Proposition 3.5 (Convergence rate of NA.MEx-Momentin There e 1, e E Algorithm
converges for NAMEx-Momentum.
EXPERIMENTAL RESULTS
We evaluate NAMEx and its variants against baseline methods (SMoE, CAMEx, and EP-CAMEx) across di- verse tasks: language modeling (WikiText-103 (Merity et al., 2017)), text classification (GLUE (Wang et al., 2018)), and image classification (ImageNet- 1K (Deng et al., 2009)). To assess robustness, we include evaluations on cor- rupted datasets: ImageNet-A, ImageNet-O,, and ImageNet- (Hendrycks et al., 202) Results, averaged over five random seeds, show that: (1) NAMEx, leveraging Nash bargaining, consistently improves performance on both vision and language benchmarks; and (2) complex momen- tum provides additional gains. For MLP-base experts, we follow standard practice and merge parameters layer- wise (Yadav et al., 2023; Yu et al., 2024; Matena & Raffel, 2022). Experiments are run on 8xA server. Additional details are available in Appendix D.
Accurary (36)
NAME: Hom
NAMEX
BP-CAMDSMMom
EP CAMEX
Epach
Figure 3: Top- Accuracy Evaluation of Swin Transformer Variants. Complex momentum enhances convergence speed and performance of both NAMEx and EP-CAMEx.
To match EP-CAMEX's training time, we fix the bargaining budget to 20 iterations per batch and evaluate two NAMEX variants: (1) NAMEX, which computes once at the first layer and reuses it, showing strong performance over naive averaging; and (2) NAMEX-Full, which distributes budget evenly across layers, inspired by (Navon et al., 2022). Update strategies further discussed in Section 5.
In the tables that follow, NAMEX-Full results are highlighted in grey, with the best and second-best scores shown in bold and underlined, respectively.
4.1
LANGUAGE MODELING
We adopt the experimental setup of (Pham et al., 2024) and (Teo Nguyen, 2024) for pre-training and evaluating the WikiText- 103 dataset. Table presents the results of our methods small-scale and medium-scall pre- treiirann tasks using WikiText- 103. For both small- and medium-seuu pre-trainin treiiiinn MAMEX-Full-Monn achieves the lowest validation/test perplexities, outperforming SMoE and CAMEx- based methods. NAMEx variants as well as momentum -equunpeeeeeuppedddeeeee variants consistently surpass their counterparts across scales, proving the efficacy of Nash bargaining and momentum integration.
7

---

Published as a conference paper at ICLR 2026
Table I: Validation and test perplexity WikiText- 103 for small- and medium-scale pretraining.
Model
Params
Small
Medium
Val PPL
Test PPL
Val PPL
Test PPL
(Top-1)
70M/216M
87.79, i
38.60 xM
40.51..mm
SMoE (Top-2)
70M/216M
84.26 12
84.81 20
33. 33.766..
35.55
SMEAR
70M/216M
85.56 a
36.15 17
37.42+
CAMEX
70M/216M
83.33 19
84.48 20
35.69h
36.53
w/O momertium
EP CAMES
70M/216M
$3.89
85.03+
35.78
36.55
NAMEX
70M/216M
83.30 as
84.12 20
33.14 19
36.404.ar
NAMEx Full
70M/216M
82.85
83.16
34.92:
36.21
v/ momentumm
EP-CAMEX-Mon
70M/216M
82.90 16
84.062 x
35.09 19
16
NAMEx Mom
70M 70M/216M
82.63
83.59
34.80
35.86+
NAMEx: Full-Mom
70M/216M
82.94.
34.25:
35.37
Table Performance of TS-ase variants on fine-tuning tasks for GLU All SMoP variants have experts per layer. Following (Devlin al., 2019), we conduct experiments on the GLUF henchmark.
Model
Params
SST-2
MRPC
CoL.A
STSB
RTE
QNLI
MNLI
Dense
220M
93.34
89.70...
58.06g..
89.06,
74.36
92.34:
86.36
SMoE (Top-1) 1)
1.0B
94.26 u
90.87 in
89.44, 56.78
70.75
92.07 is
86.38
SMoE (Tp-2)
1.0B
94.35+.1
91.04:1
a9.73
74.98
92.48:
a6.72
CAMEx
1.0B
93.80
91.16..1
58.57
89.47
74.72
92.60.
86,441.12
w/o momerntuu
EP-CAMEx
1.0B
.3.69...1 .
91.01:.1
58.29
89.92
92.17 15.8
86.941.14
NAMEx
1.0B
94.46...
92.01..44
58.81
90.12
92.86. +5.09
ir
86, 96.96
NAMEX-Full
1.0B
94.82
92 22.80...
20.63
90.27
77.83
90.23: is
87.23
w/ momention
EP-CAMEX-
1.0B
14.6
92.47 1.13 59.31 1.s
90.07 23
1..17
92.90 13
86.80
NAMEx-Mom
1.0B
93.02=
58.90
90.06
77.62z
93.11
87.02s
NAMEX-Full-Mar
1.0B
95.06..12
93. 27
60.13
90.63:
78.15s
93.31: 87.45
Table 3: Finetuning and zero-shot results ImageNet- Ik and corrupted variants.
Model
Params
Acc@1
Acc@
INet-O
INet-A
INet-R
SMoE
50M
83.15
43.34
23.72a.m
38.02 02
SMEAR
50M
83.15
96.91=.o
43.35
24.141.a6
38. 38.16..a2
CAMEx
50M
83.29
96.95-.15
50.69..as
25.45
38.37
w/o
EP-CAMEX
50M
83. 33.23+
96.93-..e
50.27
24.22
37.88+.a
NAMEx
50M
84.06.
97.19
50.30
25.32 25.74
38.56...
NAMEx-Full Full
50M
84.27
97.94g.14
50.66 m
38.70s
w/ momention
EP-CAMEX-Mom
50M
83.56
97.03
50.37
33.22
38, 22
NAMEx- -AMex-Mom
50M
84.28.
97.94-.12
51.22
35.05
38.82
F-MEx-Full-Mom
50M
84.52: 1M
98.11: m
51.34: sv
35.27
38.96.
4.2 TEXT CLASSIFICATION
We evaluate method on downstream text classification tasks using the GLUE dataset (Wang et al., 2018), with all models built on the T5-Base backbone. As shown in Table 2, NAMEx-Full-Monn achieves the best results on all tasks. NAMEx consistently outperforms SMoE (Top-1 and Top-2 routing), CAMEx, and EP-CAMEx. highlighting the effectiveness of the Nash bargaining solution. The momentum- based extensions, NAMEX -oo and CA-CCAEx-M,m, further enhance performance, demonstrating improved robustness and generalization.
4.3 IMAGE CLASSIFICATION
In this section, we evaluate our method on image classification tasks using the Swin-Transformer (Liu et al., 2021) and its MoE variant (Hwang al., 2023). Specifically, on ImageNet- 1k, training all models for 30 epochs with a batch size of 96. For each MoE layer, we perform Algorithm where apart from the first MoE layer that an E,,, expert is initialized, all experts
fine-tune Swin-MoE Small
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison actoss routing strategies and models on MMLU, GSM8K, and benchmanss.
Left: original results. Right: fine-tuned DeepSeek- MoE variants on SmolTalk.
Zero-Shot
Fine-tuned (Smoffallk)
Routing Strategy
Model
MMLUU
GSMSK
ARC
MMLU
GSM8K
ARC
Deepseek.N MolE
16.53
45.21
Lineur
EAPCA
44.85
16.63
49.26
45.33
17.24
49.62
NAMEX-Full disagreement point)
44,92
16.77
49.51
45.47
17.36
NAMEx-Full (mean disagreement point)
44.93
16.75
49.52
45.47
17.39
49.84
Deepseek-Mou
44.95
16.70
45.34
Cosine
EP-CAMEx
45.05
16.81
49.40
45, 15.45
17.32
49.73
NAMEx-Full disagreement point)
45.10
16.88
49.60
45.66
17.53
49.92
NAMEx-Full (mean disagreement point)
45.09
16.89
49.58
45.67
17.52
49.92
Deepseek-MoE
45,80
17.50
49.90
46.17
17.63
50.28
Stable-MoF:
EP-CAMEx
45.88
17.62
50.00
46:25
18.10
50.45
NAMEx-Full (O disagreement point)
45.95
17.70
50.15
46,42
18.23
50.64
NAMEx- (mean disagreement point)
45.92
17.68
50.19
46.40
18.23
50.63
merged into Em We further evaluate NAMEx another SMoE architecture ACMoE (Nielsen et al., 2025), we follow ACMoE training configurations, i.e., we train the NAMEX variants on top of the ACMoE backbone for 100 epochs with batchsize 512.
Table 3 shows NAMEx-Mom outperforming all baselines, with NAMEx close behind; even without momentum, NAMEX-Full matches NAMEX-Mom clean benchmarks, confirming the value of layer-wise Nash solutions. Across distribution shifts (ImageNet-A/CC A/Oe/ (Hendrycks et al., 202 aa;)); NAMEx-Mom achieves the best zero-shot accuracy, with momentum variants showing the strongest gains, especially on ImageNet-A.
In Table 15 of Appendix Appendix G.4, across all ImageNet variants, the NAMEx-based models consistently outperforms the ACMoE Top-I and Top-2 baselines. In particular, NAMEX-Full and NAMEx-Full-Mom set new best accuracies on both in-distribution metries (Acc@) and Ace@ and out-of-distribution benchmarks INet-O, INet-A, INet-R), This underlines the strong generalization ability NAMEx. Even with the same parameter budget, NAMEx variants deliver better robustness to corruptions and distribution shifts.
4.4 ZERO-SHOT AND FINETUNING ON DEEPSEEK- (1 6B) AND QWENI M5-MoE (14B)
We test NAMEX-Full at scale by integrating it into DeepScek-MoE (Liu al., 2024a) (16B parameters, I shared expert, and 63 routed experts) and Qwenl.5-Mof (14B parameters), evaluating in both zero-shot and SmolTalk fine-tuned settings. As shown in Table 4 below (for DeepSeek-Mon Table 10, 11 in Appendix F. (for Qwen M-MME), NAMEx-Full consistently outperforms the baselines and EP-CAMEx across routing strategies and benchmarks (MMLU (Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021) ) and ARC (Clark et al., 2018)), demonstrating robust and generalizable gains in expert collaboration.
and
5 EMPIRICAL ANALYSIS
Synthetic Example. To illustrate how NAMEX encourages balanced expert cooperation, we construct toy SMoE model with three experts per layer. Utility trade-offs are visualized in 3D space, where each axis represents the utility of one expert. Figure 11, Appendix G.4, shows that average- based expert merging may fail to reach the Pareto set, whereas NAMEx tends to produce more Pareto- efficient outcomes. This illustrates that NAMEx not dominated by EP-CAMEX or linear average merging.
Table 5: Impact of varying the argument of 3 on the performance of EP-CAMEX and NAMEX.
Model
STS-B
4/6
EA
NAMEs
94,27
94,83
92.82
92.50
89.68
89.77
76.17
76.42
m/12
BP-CAMEs
96,61
92.42
89,53
76 26.23
NAMES
92.69
EP-CAME<
93.45
92.24
89:51
72,20
NAMP
93.56
91.66
89.51
75.00
Number of Optimization Steps. Table shows that smaller step frequencies (Al 1,).5) often improve match baseline performance (Al L) but at the cost of higher runtime (0.69s - 4.70s), underscoring trade-off between accuracy and efficiency.
7/12
RP-CAMPs
93.56
91.91
89.53
76.03
NAMEX
92.60
90.15
75,57
EP-CAMES
-//6
NAMES
94.50
92.75
89:45
76.61
Impact of Momentum ju and Step size yy The results in Table 5 demonstrate the impact of varying the argument ò of (fixed modulus 0.9). All tasks show declines at & 0 (real momentum), suggesting non- zero arguments are critical. NAMEx consistently outperforms EP-CAMEx, reaffirming its robustness.
Optimal results require task-specific & tuning.
9

---

Published as a conference paper at ICLR 2026
Table 6: Impact of varying the frequency of merging weight update steps on performance of NAMEx.
Table 7: Performance comparison between complex momentum and quaternion momentum.
AI
SST-2
MRPC
STS-B
RTE
Runtime (sec)
Model
MRPC
STS-B
RTE
94.88
92.85
90.32
77.26
4.70
EP-CAMEx-Mon
92.47
90.07
76.17
94.95
92.38
90.37
76.89
2.29
NAMEX-Mon
93.02
90.06
77.62
5
95.18
92.09
90.13
77.98
1.14
EP-CAMEX-Q
92.52
90.35
77.12
L
94.46
92.01
90.12
75.09
0.69
NAMEX-Q
93.24
90.72
77.86
Finally, RTE exhibits higher sensitivity, peaking at Φ = xt x/11 and dropping sharply at Φ = 0 and
Φ= =-/ highlighting highhihinnnnnnn task-specific & dependencies. For analysis step size y, please refer Figure 10 in Appendix G.4.
Beyond Complex Momentum. Quaternions generalize complex numbers with richer 4D dynamics, enabling a quaternion momentum update + 1 = 32z + V (zx)). While more complex and harder to tune, quaternion momentum can better stabilize high-dimensional optimization and handle rotations. As shown in Table 7, choosing 3 = 0.8 0.31 31++ 0.3j+ 3-+-00 0.33k outperforms complex momentum, suggesting multi-buffer momentum is promising direction with careful hyperparameter tuning.
Table 8: Performance comparison across number of NBS solving iterations for the Linear router in NAMEx-Full config (Qwenl.3 M.-M and Keepseek-MoE) All results slightly improved while maintaining marginal gaps. Throughout experiments, we use 2 CCP iterations per layer for NAMEx Fuul (chosen config below).
Zero-Shot
Fine-tuned (SmolTalkk)
Model
No. Iterations
MMLU
GSM8K
ARC
MMLUU
GSMSK
(Chosen Config)
61.87
60.55
50.95
62.10
61.00
$1.35
61.70
60.55
50.94
62,20)
61.05
$1.31
Qwenl.S-MoE
20
61.94
60.57
50.96
62.14
61.03
51.34
61.92
60.62
51.01
62.22
61.05
51.46
60
61.81
60.48
51.08
62.15
60.98
$1.39
(Chosen Config)
45,0
16.86
49.58
45.63
17.47
49.92
5
44.84
16.86
49.57
45.73
17.51
49.88
Deepseck-Mel MoE
45.15
16.87
49.60
45.63
17.47
49.92
45.16
45.73
17.55
50.03
60
44.93
16.77
49.72
45.68
17.46
49.96
Number of CCP iterations. Tab. 8 reports the bargaining budget ablation. In the zero-shot setting, performance remains within narrow range: MMLU fluctuates between 44.8-45, (peaking at 40 iterations), GSM8K shows a slight increase up to 40 before dropping at 60, and ARC improves marginally with a best at 60. After SmolTalk fine-tuning, the curves flatten further-MMMLU tied at and 40 iterations, while GSM8K and ARC peak at 40 with minimal variation overall. Overall, 20-40 iterations slightly outperform 2-5 in some cases, whereas 60 provides no consistent benefit and can degrade performance. Given the small margins and likely variance, we recommend 2 iterations for efficiency and 20 when marginal gains are desired.
Roburtness to choices of disagreement point. In Tab. 4 and Tab. 17 in Appendix G.4, we tried "mean" (standard average merging) as the disagreement point and compared to Across Linear, Cosine, and Stable-MoF: the deltas are tiny (about 0.04 on any metric), with no consistent winner. This shows the gains come from the bargaining weights, not the fallback choice. We keep 0 as the default because it is conservative, stable, and casy to interpret, and it leaves compute unchanged.
LIMITATION AND CONCLUSION
In this work, we address expert merging through game theory by proposing NAMEX, method that integrates Nash Bargaining for equitable collaboration and complex momentum to accelerate convergence with theoretical stability guarantees. Experiments across diverse tasks demonstrate NAMEx's 's consistent superiority over existing methods, highlighting its adaptability to diverse tasks. While NAMEx could be extended to leken-level momentum based methods, such as (Teo & Nguyen, 2024) and (Puigcerver et al., 2024b), the computational cost of solving the Nash equilibrium per token remains challenge, leaving this avenue for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research / project is supported by the National Research Foundation Singapore under the Al Singa- pore Programme (AISG Award No: by the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2023) (A- (0000039-00-000 This research project is also supported by the NUS Presidential Young Professorship Award (1-0009807-01-000) and the NUS Artificial Intelligence Institute-Sed -Sed Funding (A-8003062-00-00),
This research/ / project is supported
8002040-00-00
We would like to thank FPT for their generous support in providing discounted GPU access on FPT AT Factory, which enabled the experiments in this work.
Ethics Statement. Given the nature of the work, we do not foresee any negative societal and ethical impacts of our work.
Reproducibility Statement. Source codes for our experiments are provided in the supplementary materials of the paper. The details of our experimental settings and computational infrastructure given in Section 4, Section 5, and the Appendix. All datasets that we used in the paper are published, and they are easy to access in the Internet.
LLM Usage Declaration. We use large language models (LLMs) for grammar checking and correction.
REFERENCES
Loubna Ben allal, Anton Lozhkov, Elie Bakouch, Gabriel Martin Blazquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Agustín Piqueres Lajarín, Hynek Kydlfček, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan Son NGUYEN, Ben Burtenshaw, Clémentine Fourrier, Haojun Zhao, Hugo Larcher, Mathieu Morlon, Cyril Zakka, Colin Raffel, Leandro Von Werra, and Thomas Wolf. SmolLM2: When smol goes big data-centric training of a fully open small language model. In Second Conference on Language Modeling, 2025. URL https: 4H.
//openrevs ew net fest/font
Robert Aumann and Sergiu Hart. Handbook of game theory with economic applications, volume 2. Elsevier, 1992.
Jackie Back and Vivek Farias. Fair exploration via axiomatic bargaining. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Curran Associates, Inc., 2021.
Processing Systems, volume 34, pp.
D Bertsekas. Nonlinear Programming. Athena Scientific, 2008
Ruisi Cai, Zhenyu Zhang, and Zhangyang Wang. Robust weight signatures: Gaining robustness as easy as patching weights? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 3495-3506. PMLR, 23-29 Jul 2023. URL httts::/ppoceedeee cai23f. html.
www.mm........eeemmmmmmmem.eeree v202/
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop Semantic Evaluation (SemEval-2011), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 18.13653/v1/S17. 2001.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http///ppeeeeeeeeeeeeeeeeeeeeeee net/ forum? id=MaYddM---oo Yd-MMa zugDmQV.
Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin- Yu Chen. Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 6074 61 PMLR, 23-29 Jul 2023. URL TRLLhtpp. https: : /pwcceddoeeeeeeoooooocooccccccce ngs. ml press v222 wwt.dhhrwwwde.ee.c html.
11

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 1r31v:1803.05457, 2018. BRLhtt/::// //tpxi//////vvv/////a arxiv org/ abs / /1003/80.000 05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint 0rXiv:2110.14106 2021. URL
https: / arxii. org/ abas 2210 14168.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognizing textual entailment challenge. In Proceedings of the First PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. StableMoE: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers), pp. 7095,7095, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/22222222222.llllllll long.489.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ationfor Computational Linguistics, 2019. URL https: //tp://apppp www.pp.saaaaa..................e www.nticscholar..ooooo
Corpus I:: 52967399.
Truong Giang Do, Le Huy Khiem, Quang Pham, TrungTii Nguyen, Thanh-Nam Doan, Binh T. Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, and Steven HOL. Hyperrouter: Towards efficient training and inference of sparse mixture of experts. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL https: http//ppeeeeveeeeee net/ forum? id= id=ELBAKDVELp.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
Jean-Antoine Désidéri. Multiple-gradien descent algorithm (mgda) for multiobjective optimiza- tion. Comptes Rendus Mathematique, 350(5):313 2012. ISSN 1631-073X. doi: https://doi.
URL https: www. wwweiiiieeieeeiee.oe.eeeeeeeeeee wttpe/eccccttceeeeeeeeeeeeeeeeee science/
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., (11,,, ISSN 1532-4435.
Simon Foucart. Matrix norm and spectral radius. https: www. www.math. mathh drexel. edu/ hww1111////01////111 6.f, 2012. Accessed: 2020-05-21.
-Foucart/Teachine http://aaaacaaniieeeeeee////////
Matsusaburo Fujiwara. Über die obere schranke des absoluten betrages der wurzeln einer algebraischen gleichung. Tohoku Mathematical Journal, 10:167-171, 1916.
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julis and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1802-1811. PMLR, 2019.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, and Rahul Mazumder. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Advances Neural Information Processing Systems (NeurIPS), 2021.
Shwai He, Run-Ze Fan, Liang Ding. Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improving computational efficiency of mixture of experts. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14685 14691, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18613 1/2023 1200://///222...mmmmmmi
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of duttr--distribution generalization. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Collin Burs,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021b. URL https: /ttp:///pppeeeieeeeeee ew net forum? d7KBjjmI 3GmQ.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021c.
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafacl Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive cexture--ffcxpertt at scale. Proceedings of Machine Learning and Systems, 5:269-287, 2023.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulat models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262-222777 2022.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991. doi: 10. wt..1162//cc.199 1991.3.1.79
Xisen Jin, Xiang Ren, Daniel Prootuuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. URL htts:: /ttp::///eeeeeeeeeeeeeeeeee net/ forum? 2 id=FCnohuR6Ank.
Ehud Kalai and Meir Smorodinsky. Other solutions to nash's 's bargaining problem. Econometrica: Journal of the Econometric Society, pp. 513-518, 1975.
Aran Komatsuzaki, Joan Puigcerver, James Lee- Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-expert from dense checkpoints. In The Eleventh International Conference on Learning Representations, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021a.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS hhard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021b. URL https: /ettps://opene http///ppeeeeeeeeeeeeee net teee/fforumee ttt///ormmmmmmmgmee 7b.TmYb.
Mike Lewis, Sharan Bhosale, Luke Zettlemoyer, and Marc' Aurelio Ranzato. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning (ICML), 2021.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, 2024. URL https: / net/ net/forum forum? i d=eFWG9Cy
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.044:: 2024a.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024b.
13

---

Published as a conference paper at ICLR 2026
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. A closer look into mixture-of-expent in large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguisties: NAACL 2025, pp. 4427-4444 447-4447, Albuquerque, New 195-7. URL
Mexico, April 2025. Association for Computational Linguistics. ISBN https: al antho ogy. oro.org////0 2025. fi ndi ngs- naasllnnaclll. 251/.
Jonathan P Lorraine, David Acuna, Paul Vicol, and David Duvenaud. Complex momentum for optimization in games. In International Conference on Artificial Intelligence and Statistics, pp. 7742-7766 PMLR, 2022.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertice in model merging. In The ehirty- Anmua Annual Conference on Neural Information Processing Systems, 2024. BRLhttps:/ https////pppeeeeeeeeeeeee net forum? id=811Iee 63TTn.
Michael Matena and Colin Raffel. Merging models with fisher- averaging. Advances in Neural Information Processing Systems, 35:17703- 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representutions, 2017. URL net/ / forum? i d= ]2udxx..
Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive rout- ing. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: Featured Certification.
//openreview.nee httf///ddddd11111mmmmm //
Abhinay Muthoo. Bargaining Theory with Applications. Cambridge University Press, 1999.
John Nash. Two-person cooperative games. Econometrica: Journal of the Econometric Society, pp. 128-140, 1953.
John F. Nash. The bargaining problem. Econometrica, 18(2):155-162, 1950. ISSN 00129682, 14680262. URL hattp://wwi hwt.:w..too......
1907266.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning bargaining game. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of o Proceedings of Machine Learning Research, pp. 16428-164H PML.,, 17-23 Jul 2022. URL htttttts//ppoceeeeee igg. mllr press v262 navon22a. html.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate o (1/k2). In Dokl akad nauk Sssr, volume 269, pp. 543, 1983.
Nghia Nguyen, Tan Minh Nguyen, Vo Thue Khanh Huyen, Stanley Osher, and Thieu Vo. Improving neural ordinary differential equations with nesterov's accelerated gradient method. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),.dvanccce in Neural Information Pro- cessing Systems, 2022a. URL ht http::///ppeeevieeeeeee http://eeeeoooooeooeeeeeeeeeeoo formm? id= OfK 3905hI.
Tan Nguyen, Richard Baraniuk, Andrea Bertozzi, Stanley Osher, and Bao Wang. Momentumrnn: Integrating momentum into recurrent neural networks. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin Acvaanc in Neural Information Processing Systems, volume 33, pp. 1924-1936. Curran Associates, Inc., 2020.
Tan Minh Nguyen, Richard Baraniuk, Robert Kirby, Stanley Osher, and Bao Wang. Momentum transformer: Closing the performance gap between self-attention and its linearization. In Bin Dong, Qianxiao Li, Lei Wang, and Zhi-Qin John Xu (eds.), Proceedings of Mathematical and Scientific Machine Learning, volume 190 of Proceedings of Machine Learning Research, pp. 189-204 PMLR, 15-17 Aug 2022b. URL https: http::/s/////cceeiii/e/ mlr. www.ppressiiiie nguyen22a html.
14

---

Published as a conference paper at ICLR 2026
Viet Dung Nguyen, Minh Nguyen Hoang. Rachel Teo, Luc Nguyen, Tan Minh Nguyen, and Linh Duy Tran. CAMEx: Curvature aware merging of experts. In The Thirteenth International Confer- Learning Representations, 2025. URL https:/ http//pppppeeeeeeeeeeeee net/ wttprrvviww.eee//ooummmmaaaee forum?id= nT220M00f8.
Stefan Nielsen, Rachel Teo, Laziz Abdullaev. and Tan Minh Nguyen. Tight clusters make specialized experts. In The Thirteenth International Conference on Learning Representations, 2025. URL attps://cpeenevaat
net/ forum? wdw.mmdddddPPd00-----
Satish Penmatsa and Anthony T Chronopoulos. Game-theoretic static load balancing for distributed systems, Journal of Parallel and Distributed Computing, 71(4):537-555, 2011.
Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho. Competesmoe: Effective training of sparse mixture of experts via competition. arXiv preprint arXiv:2402.02526, 2024. URL https: //axiv////xivv////a www.wiiwwi...ooooooooooooooooooe hbtrg//b///24/ 02526.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, 2024a. URL
https: /etps::/opennee /pwnreeieeeeeeeeeeeemmeemeeeoeee net/ / forum?id= jxpsA
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, 2024b. URL
https: httt::://openeevee
net/ / forum? jxxpss pwtpppss71tt
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
//jmlr. wwwperpppppr..aaaaooooooe..eerre 121/2021102000 html.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribttion generalization. ICML, pp. 28656-28679, 2023. URL https httpt:::s//prcceednnng mlr. pww.ppres//v/e v202////vv//////////// rame23a. html.
Mustafa Jahangoshai Rezaee, Milad Eshkevari, Morteza Saberi, and Omar Hussain. Gbk-means clustering algorithm: An improvement the k-means algorithm based on the bargaining game. Knowledge-Basse Systems, 213: 2021.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Carlos Riquelme Ruiz, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, An- dré Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttps:///ppattt ..=NNGPHH3vbAA.
net forum?
Aviv Shamsian, Aviv Navon. Neta Glazer, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Auxiliary learning as an asymmetric bargaining game. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 80689-30700 PMLR, 23 29 Jul 2023. URL httpp::/pproceeeeeeeee shamsian23a html.
mwr.press. v22222222222
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gatee -atted mixture-of-expert layer. In International Conference on Learning Representations, 2017. URL https: https:://ppereeeeeee //ppnreveee..... d=31 ckk1mmgg ckMDqlg.
15

---

Published as a conference paper at ICLR 2026
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631-1642, 2013.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In International conference on machine learning, pp. 1 1139-1147. PMLR, 2013.
Rachel Teo and Tan Minh Nguyen. MomentumSMoe: Integrating momentum into sparse mixture of experts. In The Thirty-eighti Annual Conference on Neural Information Processing Systems, 2024.
URL StRs:t::: htttt::s///ppeeevieeee www//eeruunnne wtw.ooruummdyyy99eeeeee
Rachel Teo and Tan Minh Nguyen. MoLEx: Mixture of layer experts for fine-tuning with sparse upcycling. In The Thirteenth International Conference Learning Representations, 2025. URL
btttps://openrevee/
.nt/ wforrmoormmaammm forrmmiddzrhe ierr 9
Viet-Hoang Tran, Van-Hoan Trinh, Khanh Vinh Bui, and Tan Minh Nguyen. On linear mode connectiv- ity of aarpr architectures. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: httttps://ppezvviiie wwt.//eeeevvieeeeeeeeeeeee net/ forum? did= 3mi SqdXa.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp. 353, 2018.
Bao Wang, Tan Minh Nguyen, Tao Sun, Andrea Bertozzi, Richard Baraniuk, and Stanley Osher. Scheduled restart momentum for accelerated stochastic gradient descent, 2021. URL https: // //oppreeviee www.eeiiewwnee net/ forum? 2
Bao Wang, Hedi Xia, Tan M. Nguyen, and Stanley J. Osher. How does momentum benefit deep neural networks architecture design? few case studies. Research in the Mathematical Sciences, 2022.
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxillary-loss-free load balancing strategy for mixture-of-experts, 2025, URL
id-yli0SczYpE.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pp. 7739,6739, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 11 1112-1122, 2018.
Hedi Xia, vai Suliafu, Hangjie Ji, Tan M. Nguyen, Andrea L. Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. In NeurlPS, New Orleans, USA, 2023. Proceedings of Machine Learning Research.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint 440v 00002447770676...0 2024.
16

---

Published as a conference paper at ICLR 2026
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: absorbing abilities from homologous models as a free lunch. In Proceedings of the 41st International Conference Machine Learming, (CML'24. JMLR.org, 2024.
Jian Zhang and Ioannis Mitliagkas. Yellowfin and the art of momentum tuning. arXiv preprint arXiv:1706.03471, 2017.
Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture-of- experts for autoregressive language model pre-training. In First Conference on Language Modeling, 2024. URL https: // //ppeneevewww....//////e wtwp://wwwwweeeteonnneeooe wedddd-EEEEE Lid-
17

---

Published as a conference paper at ICLR 2026
Supplement to "Expert Merging in Sparse Mixture of Experts with
Nash Bargaining"
Table of Contents
A Notation
19
Related Work
19
Proofs of the Main Results
20
C.I Lemma 3.2 Proof Sketch Sketch
20
C.2 Convergence guarantee for
20
C,3 Propostion 3.5 Proof Sketch
22
Additional Details on Datasets
23
D.I
WikiText-103 Language Modeling
23
D.2 GLUE Text Classification
D.3
ImageNet- 1k Image Classification
24
More experimental details
24
E.1
WikiTexx- 103 Language Modeling
24
E,2
GLUE Benchmark
Bee.............................
24
E.3
ImageNet- 1k and Corrupted Variants
................................
24
E,4
Implementation and Infrastructure
24
Additional Experimental Results
25
F.1
Zero-shot and Finetuning on Qwenl .5-MoE
Qwen1.55MoB: (14B parameters)
25
Additional Empirical Analysis
25
G.1 Overhead and Scalability
25
G.2 Convergence Analysis
26
G.3 Swin - Mon- S: 909 Epoch Fine - Tuning
26
G.4 Other Results
26
H Broader Impacts
28
18

---

Published as a conference paper at ICLR 2026
A
NOTATION
Table 9; Notation
a= ,2,,, ON.
The Nash coefficients for merging experts.
zyy, ... EC
Vectors
X,Y Z ... E..E
Matrices
X
The transpose of matrix X The identity matrix
I
(x)
The real imaginary component of EC
The imaginary unit. 2 E - 2 -=
2
The complex conjugate e
|=|:= V2 arg(=)
The magnitude or modulus of =C
The argument or phase of C = 2 == (=]exp((arg((
E ER"
Parameters of the base experts at the /-th layer of the network Parameters of the i-th experts at the 1-th layer of the network
E ER
=E EM! AE ERd
The domain-vector vector of the i-th experts the I-th layer of the network Aggregation of the domain-vector for updating the base expert.
E ER"
The initial base expert parameter at the first layer
YER+
The step size for the base expert propagation
NER+
The step size for creating the !!!! expert that responsible for processing input
BEC
The momentum coefficient
MECd
The momentum buffer
λCC
Notation for arbitrary eigenvalue
B RELATED WORK
Sparse Mixture of Experts. SMoE scales efficiently by activating only subset of parameters per token, favoring horizontal over deep expansion (Shazeer et al., 2017; Lepikhin et al., 2021b; Fedus al., 2022; Tran al., 2025; Teo & Nguyen, 2025), and improves Transformer efficiency without loss. In parallel, model merging has gained traction for combining open-source models (Ilharco et al., 2022; Matena & Raffel, 2022; Yadav et al., 2023; Ramé et al., 2023; Cai et al., 2023; Lu et al., 2024), with curvature-awane methods like Fisher Information (Matena & Raffel, 2022; Jin et al., 2023) improving quality but at high cost. But, most merging approaches assume shared initialization (Yadav et al., 2023; Ilharco al., 2022), conflicting with SMoE's independently initialized experts and making merging more difficult.
Nash Bargaining Game. Originally introduced by (Nash, 1950; 1953), the Nash bargaining frame- work has been widely studied (Kalai & Smorodinsky, 1975) and recently applied to multi-task learn- ing (Navon et al., 2022; Shamsian et al., 2023). It has also shown success in diverse domains such as multi-armed bandits (Baek & Farias, 2021). clustering (Rezaee et al., 2021 1), distributed comput- ing (Penmatsa Chronopoulos, 201 1), and economics (Aumann & Hart, 1992; Muthoo, 1999).
Momentum in Deep Learning, Momentum-based optimization has been widely studied, from its ori- gins in classical methods (Polyak, 1964; Nesterov, 1983) its adaptation for deep learning (Sutskever et al., 2013; Zhang & Mitliagkas, 2017; Nguyen et al., 2022b; Teo & Nguyen, 2024). Beyond optimization, momentum has also been examined from dynamical and architectural perspectives. Continuous-tim interpretations such as Heavy Ball Neural ODE (Xia et al., 2021) and Nesterov Neural ODE (Nguyen al., 2022a) connect accelerated methods second-order differential equations, while Nguyen et al. (2020) incorporates momentum dynamics directly into recurrent architectures. Scheduled Restart Momentum (Wang et al., 2021) studies restart schemes for accelerated stochastic gradient descent, and recent empirical analyses (Sutskever et al., 2013; Wang et al., 2022) investi- gate how momentum shapes representation learning and architectural behavior in deep networks. In game-theoretic settings, Gidel et al. (2019) explored negative momentum for games, and Lorraine et al. (2022) introduced complex momentum, extending momentum methods to differentiable games using complex-valued updates.
19

---

Published as a conference paper at ICLR 2026
C PROOFS OF THE MAIN RESULTS
Assumption C.1. We assume SMoE architecture of infinite SMoE
layers
with
{E.. ( _
"..,,E"] j being the epxerts parameters at /-th layer.
Assumption C.2. The norm of experts parameters is bounded, that is:
I (1,2..,)... ViE (m,1,.,..
(12)
C.I LEMMA 3.2 PROOF SKETCH
Proof. The derivative of the objective function
AE
T5
For all AE such that AE T; " for all the utilities increase monotonically with the norm of AE Hence, by Nash's Pareto optimality axiom, the optimal solution must lie on the boundary of B.. At the optimal point, the gradient
N
T3
AE
must be in the radial direction, i.e.,
N
Ti x AE
AE
Ti
Equivalently, there exists À 0 such that
Σ
T == AAE. AE
Since the gradients T; linearly independent, express AE AE -i= 0.7. Substituting this into the alignment condition, we obtain
Aa; Vi.
AE
This implies AE T:= Aas, gives the direction of AE. Thus, finding the Nash bargaining solution reduces finding a E T such that
1. As AE TO >.>0 0 for descent direction, we deduce À > 0. Setting À = 1
N Caj
1
T. T= AET
Vi.
Oi
j=1
This is equivalent to solving G Ga= 1/a, where 1/a is the wlement-wise reciprocal.
C.2 CONVERGENCE GUARANTEE FOR NAMEX-MOMENTUS
We first have that:
N
N
Ag
Σ
0
E
(13)
i=1
i=1
Given the analogy between expert merging and gradient descent, we apply the formulation of momen- tum into Eqn. 7:
(1+1)
-E!
+~ +3(E!! -E!!! 1))
N
(14)
(1+1))
-E(++)
M (s) (1+1)
*T; (+
(111).
Expanding the parameter updates with the Cartesian components of and 3 is key for Theorem C.3, which characterizes the convergence rate:
((+1) = 3r (A
R( -1(3) - +
N
N
= - +
Σ aj
(15)
m
1=1
(16)
20

---

Published as a conference paper at ICLR 2026
E E(++1) E. (1
(17)
-E E
+ A 10) +
m
N
N
E m
(1)
+
Σas aii
E
-
(18)
1=1
N
Setting o
0i, we have
1=1
R(3)
-3(3
-al R(3)
N
N
R=
3(8)
0
q== and q
Σ ar E
a E
(19)
R
-3(18)I
I I-qal
Our parameters evolve with expert-propagation merging via:
Ri 3, ). - 3, (1) ".E"T (OT
(20)
We can bound convergence rates by looking at the spectral radius of R with Theorem C.3.
Theorem C.3 (Consequence of Prop. 4.4.1 (Bertsekas, 2008)), tfe spectral radius p(R) < 1, then, for Em] in neighborhood of Ju BEi.h, the distance of ",,", converges at a linear rate ((pp R. + VeVE0
[p ,E(0] to the stationary point (p' .Em)
Proof. We have:
Riull) (4a()) E(!)
((llt))
= R
+q
(21)
E((+1)
By telescoping the recurrence for the 8th layer:
R(p ((u
Ri (0))
= R
((
R'q
(22)
B(0)
1-1
We can compare μ' and
Σ
R' q
with the values fe and q" they converge to which exists if R
contractive. We do the same with E. Because μ ="= +Ru"+q"== RR 4 Σ Td (
R)) R((x) (()()) ((())) E(D E-
1-1
R
(()(0)) - -3(1w)) E E²
> R'q (1) Σ R'q (8)
(23)
By taking norms:
R( R(4))
R0) R(a)) 3 3u 3(x)
((m0)) 3(4))
Rd
Σ R'q i
(24)
E -
E(O) - E'
i=1
R) R(())
R( R(x)) (1) 1 -3(x))
((20)) ((4)))
R
+
Σ
R
(25)
E E*
((0) -E" R( -H((()) 3 (1 -S(())))) E(0) -(0)_E* E"
R
Σ
R B
(26)
With Lemma 11 from (Foucart, 2012), we have there exists matrix norm Ve such that:
R D(p(R)+e))
(27)
21

---

Published as a conference paper at ICLR 2026
We also have
2
0
>
R
<C
(28)
if R is contractive. Combining Equation (27) and Equation (28) we have:
) R(())) (((0)) - (11))) E
R(((())) -R((")))
( (D(p(R)+e)
( -
+BC
(29)
E E
So, we have:
) ((x()) (1x)))
Thus, we converge linearly with rate of =O((p(B)+c))
(30)
E E
C.3 PROPOSTION 3.5 PROOF SKETCH
Proposition 3.5(Conveeeence rate of NAMEx- There exist EERR Algorithm 2
converges for Expert-Propagatioon NAMEx Momentum.
Proof. We want to show that we can select > 0 and BE C so that R contractive, That is, the spectral radius R is less than Recall that,
R(3)I
I(B)
-al
R=
I(3)I R(3)
(31)
R(SS)I
3((8)
I
Set 3et3rr+uus ui, we have:
d
r- x -u -t
r-x
-u
det Ret(R-xII=dess
0
=det
T-x
0
7/
-u
1-7a ---0 r
u
7
x2
(32)
P- x.
0
= 2 + 2oou= - +2rr
yr
1-70 -
(33)
33 +- + 2r-w" - xr) 2 (a -ox 1)u?
(34)
2 to get the following polynomial,
We can further simplify this by choosing
(35)
Using Fujiwara's 's bound (Fujiwara, 1916) we can determine one condition for p( RR) < 1, ih
2 2+((+0 -
| < 2max
11 81, /l.V -2 Arr 2r 2
<1
(36)
22

---

Published as a conference paper at ICLR 2026
15
-
1.0
<rd (1 a
0.5
0.0
-0.5
-10
-1.5
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
Figure 4: Graph of system of inequality (37) when 0.5 and y= 7
1
< 1+ 2
-1 <x² -fr++2+++
1
(37)
1 <2 +(1 11+0 9)
1
4
1 -
1
<r 2
-
27
<
-
for
2r
(38)
4
- <u 1-
Ita-9
a+a
We can consider the case when a and 7 = Figure 4 shows the region of (r,u) that satisfies inequality system (37).
D ADDITIONAL DETAILS ON DATASETS
This section outlines the datasets and evaluation metrics employed in the experiments discussed in Section 4.
D.1
WIKITEXT- 103 LANGUAGE MODELING
The WikiText- dataset contains collection of Wikipedia articles designed capture long-range contextual dependencies. It includes training set with 28,475 articles, amounting to around 103 million words. The validation and test sets consist 217,646 and 245,569 words, respectively, distributed across 60 articles per set. set.
Model and baselines We use the small and medium size Transformer as the baseline SMoE models. Our implementation is based on the codebase developed by (Pham et al., 2024) and (Teo & Nguyen, 2024). All model variants- MSoE, CAMEx, and NAMEx NAMEX-employ employ 16 experts per layer, with SMoE utilizing top-1 (k=1) expert selection for each input. The models share unified sparse routing mechanism. consisting a linear layer to process the input, followed by Top-K K and Softmax functions. Training is performed over 60 epochs for small models and 80 epochs for medium and large SMoE models.
23

---

Published as a conference paper at ICLR 2026
D.2 GLUE TEXT CLASSIFICATION
The GLUE benchmark includes SST-2 (Socher ct al., 2013) for sentiment analysis, MRPC (Dolan & Brockett, 2005) for paraphrase detection and sentence similarity, CoLA (Warstadt et al., 2019) for evaluating grammatical acceptabilityy, STS-B (Cer ct al., 2017) for sentence similarity measurement, RTE (Dagan et al., 2006) for logical reasoning. QNLI (Wang et al., 2018) for question-answer classification, and MNLI (Williams et al., 2018) for assessing entailment between sentence pairs.
Model and baselines We scale up T5 (Raffel al., 2020) using SMoE upcycling (Komatsuzaki et al., 2023). For each task, we conduct a comprehensive hyperparameter search, exploring batch sizes (8, 16, 32, 64) and learning rates (3e-44 4, 1e-4, 3e-5, 1e-5) to identify the optimal fine-tuned configuration.
D.3 IMAGENET-IE K IMAGE CLASSIFICATION
ImageNet-Ik, introduced by (Deng al., 2009), is widely used benchmark dataset comprising 1.28 million images for training and 50,000 images for validation across 1,000 categories. Performance is evaluated using top-1 and top-5 accuracy metries.
For robustness evaluation, we utilize several specialized subsets. ImageNet-A (Hendrycks et al., 2021c) focuses on 200 challenging classes from ImageNet-IK, specifically curated fool classifiers, highlighting their vulnerability to real-world ad adversarial examples. ImageNet-O) (Hendrycks et al., 2021c) contains out-o--distribution (OOD) samples derived from ImageNet-22k, carefully selected as instances that ResNet-50 model misclassifies with high confidence. The primary evaluation metric for ImageNet-O is the area under the precision- recall curve (AUPR). Lastly, ImageNet-R (Hendrycks et al., 202 consists of 30,000 artistic renditions representing 200 classes from ImageNet- Ik, designed to assess model robustness to non-standard visual representations.
Model and baselines For each MoE layer, we use Algorithm 2 to merge all experts into base expert, except in the first MoE layer, where a base expert is initialized instead. Training configurations follow Swin-MoE (Liu et al., 2021), and the code is publicly available on http://ggihuu..oommmoroooo Transformer/. For NAMEX variants, we start with checkpoints pretrained on ImageNet-224 and fine-tune them on ImageNet- 1k for 30 epochs.
win-
E
MORE EXPERIMENTAL DETAILS
This section provides additional details on the experimental setup, including model configurations, dataset processing, and training strategies used in our evaluation.
E.1
WIKITEXT- 103 LANGUAGE MODELING
We follow the setup from (Pham et al., 2024) and (Teo & Nguyen, 2024), using both small and medium- scale Transformer architectures with 16 experts per layer. All variants (SMoE, CAMEx, EP-CAMEE NAMEx) use Top-1 routing and share the same sparse gating mechanism. Training is conducted for 60 epochs (small) and 80 epochs (medium) with Adam optimizer and cosine learning rate scheduling.
E.2 GLUE BENCHMARK FINE-TUNING
For text classification, we fine-tune T5-base models upcycled with SMoE layers. We conduct grid searches over batch sizes 8, 16, 32, 64 and learning rates 3 x 10-5, 1 10-1, 3x 10-4. 4 Each result is averaged over five seeds ensure statistical stability. All SMoE variants employ experts per layer and share the same routing logic.
E.3 IMAGENET- 1 AND CORRUPTED VARIANTS
For vision experiments, we fine-tune Swin-MoF-Smaal on ImageNet- 1 for 30 epochs using batch size 96 and learning rate 1 x 10-4, NAMEx variants initialize Em in the first MoE layer and perform merging across all others via Algorithm I or Algorithm 2. For robustness, we evaluate zero-shot generalization on ImageNet-A, ImageNet-O, and ImageNet-R. All reported results are averaged over three runs.
E.4 IMPLEMENTATION AND INFRASTRUCTURE
Experiments are implemented in Py Torch and trained on 4-8 A100 GPUs depending on model size. We use automatic mixed precision (AMP) for memory efficiency. Allhyperrarameterss. data augmentations, and merging schedules are described in Appendix D.
24

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 ZERO-SHOT AND FINETUNING ON QWENI.5-MM 15-MOE (14B PARAMETERS)
To assess scalability, integrate NAMEX-Ful Full into Qwenl.5-Mon (14B) and evaluate it three challenging benchmarks: MMLU (Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021), and ARC (Clark al., 2018) 8) in both zero-shot and fine-tuned settings. In the fine-tuned setting, all models fine-tuned on the SmolTalk (allal et al., 2025) dataset before evaluation. As reported in Tables 10 and 11, NAMEx-Fulll consistently outperforms both the baseline and EP-CAMEx across routing schemes and tasks, highlighting its robustness, scalability, and architectural generality.
Table 10: Zero- shot results for Qwen 1.5 . MoE variants.
Routing Strategy
Model
MMLU
GSM8K
ARC
Linear
Qwen1.5 MooE
61.28
60.12
50.77
EP CAMEX
60.23
50.83
NAMEX Full
61.87
60.55
50.95
Cosine
Qwen 1.- MoE
61.10
59.88
50.60
EP CAMEX
61.40
60.00
50.68
NAMEX Full
61.85
60.52
50.93
Stable Moo
Qweal.5 15 MMoE
61.35
60.22
50.81
EP CAMEX
61.60
60.35
50.89
NAMEX Full
61.90
60.60
50.96
Table 11: Results after fine- tieettnnng on Tall
Routing Strategy
Model
MMLU
GSMSK
ARC
Qwenl @wenl.5-Mon 1.--MoE
61.50
60.52
51.12
Linear
EP-CAMEx
61.74
60.63
$1.23
NAMEx-Full
62.10
61.00
51.35
61.30
60.28
50.95
Cosine
EP-CAMEX
61.60
60.50
51.10
NAMEx-Full
62.05
60.95
51,30
Qwen1.5.on Mo.
61.60
60.65
$1.20
Stable-MoE Routing
EP-CAMEx
61.85
60.80
51.30
NAMEx-Full
62.15
61.10
51.45
G ADDITIONAL EMPIRICAL ANALYSIS
G.1 OVERHEAD AND SCALABILITY
Table 12: Impact of NBS update frequency (At). Per hatch wall clockk.
Af
-
MRPC
STS
RTF
Mean Batch
NBS Time
94.88
92.85
90.32
77.26
4,70
85.96%
94.95
92.38
90.37
76.89
2.29
71.18%
95.18
92.09
90.13
77.98
1.14
42.11%
L (first layer)
94,46
92.01
90.12
75.09
0.69
4.35%
Table 13: Training compute and throughput. Inference is unchanged relative to baselines.
Model
Train TILOPs
Infer TFLOPs
Truin Throughput (tok/s)
SMoE
13.95
4.65
22,236
SMOE (Top-2)
18.32
7.44
17,898
SMEAR
13,95
4.65
22,236
CAMEX
14,30
4.65
21,982
BP- CAMEX
14.25
4.65
21,982
NAMEX
14.25
4.65
21,995
MAMEX -Ful
14.25
4.65
21,897
EP CAMEX MMm
14.25
4.65
21.872
NAMEX Full-
14.25
4.65
21,783
In Tab. 12, provide detailed runtime cost analysis, "Mean Batch Runtime (see)" includes the entire forward/backwann pass and NBS step. "% NBS Occupied" isolates the share time spent solving NBS.
25

---

Published as a conference paper at ICLR 2026
In Tab. 13, provide comparison of training TFLOPs, inference TFLOPs, and training through- put across baselines and our proposed variants in Table 2 below. Notably, NAMEx-Ful achieves competitive throughput (21,897 tokens/sec), closely matching CAMEx and EP-CAMEx despite the added complexity of solving the Nash system. While NAMEX and NAMEX-Mon show slightly lower throughput due the large number of NBS iterations (20 iters), NAMEX-Full-Mon restores much of the efficiency by reducing the number of NBS iterations (2 iters per layer). Currently, the main overhead arises from transferring data the CPU for solving the Nash Bargaining update step. However, this implementation detail is orthogonal to the algorithm itself and can be optimized via GPU-native solvers or batching strategies. Overall, the results demonstrate that NAMEx introduces minimal overhead and remains practical for large-scale MoE training, supporting its applicability to future deployments involving more experts per layer.
G.2 CONVERGENCE ANALYSIS
NAMEX-Mom
NAMEX
EP-CAMEX
45
40
L2 distance
35
30
25
Step
Figure 5: 1.2 distance between expert updates across training steps (T5-Base, 12 MoF layers). Lower values indicate better stability. The figure shows that NAMEx converges faster and more stably than EP-CAMEx.
To validate the motivation for complex momentum, we provide empirical convergence analysis in Fig. which tracks the L2 distance between updates base experts across training steps (TS-Base, 12 MoE layers). As illustrated, NAMEX-We decline update distances, indicating faster convergence and more stable expert updates compared to EP-CAMEX. This directly supports our hypothesis that complex momentum enhances convergence stability and efficiency during expert merging.
without
a noticeably steeper
G.3
SWIN MoE S: 90 EPOCH FINE TUNING
Table 14: ImageNet- 1K Top- from epochs 86-90. Our best final performance is 85.046 %.
Epoch
NAMEX Mom
NAMEX EP-CAN EP-CAMEX- Mom
NAMEX- EP- CAMEX
NAMEX Full Mom Swin MoE
84.466
84.264
83.862
82,238
84.722
85.022
83.435
87
84,504
84.252
83.868
82,286
84.728
85.028
83,400
84,502
84.240
83.854
82.234
84.734
85.034
83,379
84,540
84.228
83.860
82.282
84.740
85.040
83.413
90
84.518
84,216
83.806
82.230
84,746
85.046
83.415
Tab. summarizes the t accuracy on ImageNet- I from epochs 86 to 90, these results confirm that our models continue to improve with more training, and that Namex-full-Mon reaches final top- accuracy of 85. ..46.. This demonstrates both competitive final performance and strong convergence behavior. Note that due to the difference in number of GPU being used, it seems that we could not reproduce the 84.5% result as reported by the official repo of Swin-MoE.
G.4 OTHER RESULTS
Figure presents the evaluation perplexity on WikiText- 103 during training. NAMEx-Mon achieves the lowest validation and test perplexities in both small- and medium-scale pre-training, outperforming
26

---

Published as a conference paper at ICLR 2026
Lays
Lewe's
Figure Cosine similarity between expert outputs Switch-Treee Srrtih--ranformmrs.
CAMEX
EP-CAMEX Mom
EP-CAMEX
NAMEX
Perpleskyy
Figure 5-Period Moving Average of Perplexity different Transformers medium variants on WikiText-
Figure 8: Cosine similarity between expert outputs in Swin-Mot
SMoE and CAMex-hased methods. Across all settings, Nash variants, including NAMEx and NAMEx- Mom, consistently surpass their counterparts, demonstrating the effectiveness of Nash bargaining and momentum mechanisms.
Figure 6 and Figure 8 visualize the cosine similarity between experts ouput at all SMoE layers indicating a complex dynamic of how experts at progressive layers interact with each other. This observation suggests that the behavior experts cannot be captured optimally by using simple averaging of the previous work. Instead, a more effective strategy for determining the merging coefficients should account for the experts' dynamics at each specific layer.
27

---

Published as a conference paper at ICLR 2026
Layer
5
with Bloss
Lose-free belarding
Figure compared expert interaction patterns under three settings: with Load Balancing loss, without Load Balancing loss, and loss-free balancing (in the sense of (Wang et al., 2025)).
30
MORDX
2.0
25
2.5
Figure 10: Impact of different values step-size y on NAMEx and NAMEX Mom performance. Overall, the optimal setting lies within the range [0.5.2). For > 2, the performance drops significantly, which may indicate an overshooting phenomenon.
Table 15: Pretraining and zero-shot results for NAMEx vs. ACMoE Top-1/Top-2 on ImageNet- 1k and corrupted variants.
Model
Params
Acc@1 1
Acc@5
INet-O
INet-A
INet-R
ACMoE-Top
280M
75.39
92.56
18.45
7.13
30.85
ACMoE-Top2
280M
76.31
93.14
19.55
9.42
32.35
NAMEx
280M
76.85
93.40
20.11
9.90
32.93
NAMEx-Full
280M
77.42
93.85
20.69
10.46
33.44
NAMEX-Full-ae
280M
78.15
94.23
21.16
11.02
33.95
As shown in Figure 9, the cooperative/comperiivee dynamics (as reflected in the off-diagonal correlation values) are much more visible when expert load is balanced-either through Load Balancing loss or loss-free mechanisms. In contrast, when training without Load Balancing loss, many experts appear less specialized, and the interaction patterns become weaker and less structured.
One hypothesis is that, without balanced token routing, some experts may be underused or even become inactive, which diminishes the emergence of meaningful cooperative or competitive behavior. Therefore, balanced expert load is not only important for preventing dead experts but also plays crucial role in making such dynamics observable and analyzable.
In Table 15, across all ImageNet variants, the NAMEx -AMex-based models consistently outperform the ACMoE Top-1 and Top-2 baselines. In particular, NAMEX-Full and WAMEX-FFll-Ma set new best accuracies on both in-distribution metrics (Acc@) @ and Aco@5) and out-of-distribution benchmarks (INet-O, INet-A, INet-R). This underlines the strong generalization ability of NAMEx. Even with the same parameter budget, NAMEx variants deliver better robustness to corruptions and distribution shifts.
H BROADER IMPACTS
NAMEX proposes a principled, tthertheereetc approach to expert merging in Sparse Mixture-of- Experts (SMoE) models, addressing key limitations of heuristic and curvature-bass based methods. By
28

---

Published as a conference paper at ICLR 2026
Parzto Surface
Nash Soluton
Lincar Average Solution
Layer
Layer7
Layer 30
Layer
Layer
Figure 11: Visualization expert utility trade-offs accross multiple SMoE layers. Each subplot corresponds to a different layer. with arrows indicating merging directions in the 3D utility space. The blue arrow represents the Nash Bargaining solution (NAMEx), while the red arrow denotes the wverage-based merging direction. The hemispherical surface is uniformly sampled illustrate the feasible utility region. Across layers, the Nash direction consistently stcers toward more balanced expert cooperation compared to the naive average.
Table 16: Performance comparison across number of NBS solving iterations Linear router NAMEX-F config (Qwen1 1 .5-MoE), All results to be filled; we keep the same marginal -gap setup. Throughout experiments, use CCP iterations per layer for NAMEx-Full (chosen config below).
Zero-Shot
Fine-tuned (Smol'alkk
Model
No. Iterations
MMLU
GSMBK
ARC
MMLU
GSM8K
ARC
2 (Chosen Config)
61.87
60.55
50.95
62.10
61.00
51.35
61.70
60.55
50.94
62.20
61.05
51.31
Qwen1.5-Moi
60.57
62.14
61.03
40
61,92
60.62
51.01
62.22
61.05
51.46
60
61.81
60.48
51.08
62.15
60.98
51.39
Table 17: Performance comparison across routing strategies and models MMLU, GSM8K, and ARC bench-
marks Qwenl.5 MoE variants. Left: zero-shot results. Right: fine-tuned NAMEX Full variants on SmolTalk.
Zero-Shot
Fine-tuned (SmolTalk)
Routing Strategy
Model
MMLU
GSM8K
ARC
MMLU
GSM8K
ARC
Qwent MoE
60.12
50.77
61.50
60.52
51.12
EP- CAMEX
61.54
60.23
50.83
61.74
60.63
51.23
NAMEX Full (0 disagreement point)
6187
60.55
50.95
62.10
61.00
51.35
NAMEX Full (mean disugreement point)
61.78
60.57
51.23
61.67
61.04
51.25
Qwent MoE
59.88
61.30
60.28
50.95
EP- CAMEX
61.40
60.00
50.68
61.60
60.50
51.10
Cosine
NAMEX -Ful (0 disagreement point)
61855
60.52
50,93
62.05
60.95
51.30
NAMEX Full (mean disugreement point)
61.86
60.45
50.77
62.01
60.81
$1.37
Qwent Moll
EP CAMEX
61.60
60.35
50.89
61.85
60.80
$1.30
StableMMoE
NAMEX -Ful (0 disagreement point)
61.90
60.60
50,96
62.15
61.10
51.45
NAMEX Full (mean disugreement point)
60.64
51.03
62.15
61.11
51.35
leveraging Nash Bargaining, it enables more balanced and interpretable parameter integration, par- ticularly settings with conflicting or specialized expert knowledge. This has direct implications for scalable deployment, as NAMEx can reduce the memory and compute footprint of large SMoE models while preserving performance. The addition of complex momentum enhances convergence stability during expert propagation, offering robust framework for layered expert interaction. These contributions may prove valuable for future research in modular deep learning, federated optimization, and transfer learning, where efficient and fair expert combination is critical.
29


Reviews:
[{"review_id": "FGISlSK7Gn", "reviewer": "Reviewer_q8Yj", "summary": "This paper proposes a token-wise expert merging strategy of MoE, mainly to replace the conventional sparse routing for better performance. The paper adopted the Nash Bargaining Solution (NBS) to enhance the collaboration among experts during the expert merging process. The paper provides theoretical and empirical support to justify the claim.", "questions": "Can the authors clarify whether the empirical advantage of expert-merging arises from extra training FLOPs or from the proposed design? A training FLOPs equivalent result can be a good way to clarify that.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposal of NBS in expert merging seems novel\n2. The paper provides theoretical justifications behind the proposed method\n3. The method improves performance over previous related expert-merging methods (e.g., SMEAR, CAMEx, EP-CAMEx)", "weaknesses": "1. The main advantage of the MoE-based models is their training efficiency. It has been established in the literature that MoE models generally achieve similar performance with significantly lower training FLOPs due to their sparse training. However, the proposed expert-merging method (and possibly previous methods also) doesn't employ the sparse routing. For example, in equation (7) of the paper, the update of the base expert uses all $N$ experts and their curvature matrices. Therefore, there is an uncertainty about whether the proposed expert-merging process sacrifices the training-efficiency advantage of MoE. As we can see, in Table 12 of the Appendix, the SMoE has lower training FLOPs than NAMEx. Therefore, it is uncertain whether the advantage of NAMEx appears for extra training FLOPs or from the delicate design.\n\n2. Another advantage of SMoE is its capability of maintaining a constant inference FLOPs with the increase of the number of experts. It is not clear whether the advantage remains in the proposed expert-merging method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnFrAGFGxa", "reviewer": "Reviewer_zXWC", "summary": "The paper introduces \"Nash Merging of Experts\" (NAMEx), a novel method for merging experts in Sparse Mixture-of-Experts (SMoE) models. It reframes expert merging as a game-theoretic problem, using the Nash Bargaining solution to calculate merging weights. This principled approach models the complex cooperative and competitive dynamics between experts.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Applies game theory (Nash Bargaining) to expert merging.\n\n- Consistently outperforms baselines across multiple experiments.\n\n- Deeply investigates key components like layer-by-layer bargaining and momentum.", "weaknesses": "- What impact does this 20-step bargaining budget have on the quality of the solution? If the budget is increased, will the performance improve further?\n\n- In the second line of Eq. 9, why not also use Nash Bargaining to guide the calculation of $\\hat{E}_m$?\n\n- It should be compared with more vision MoE models on ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZdpwPfS7Pc", "reviewer": "Reviewer_2Qyz", "summary": "This paper introduces Nash Merging of Experts (NAMEx), a novel framework for expert merging in Sparse Mixture of Experts (SMoE) architectures. Diverging from conventional input-dependent or averaging strategies, NAMEx reinterprets expert merging through the lens of game theory, specifically leveraging the Nash Bargaining Solution (NBS) to derive principled merging coefficients. This approach models expert merging as a cooperative-competitive game, using expert domain vectors (deviations from a base expert) as utility functions to ensure a fair and efficient optimal agreement.\nA key extension, NAMEx-Momentum, integrates complex momentum to accelerate convergence and enhance stability during the base expert propagation across SMoE layers, addressing observed slow convergence issues in prior methods like EP-CAMEx. The authors provide theoretical guarantees for the convergence of NAMEx-Momentum.", "questions": "Please provide a more detailed theoretical or empirical justification for the removal of the curvature matrix $M_i$ in the calculation of the propagation update $\\Delta E^{(l)}$ in NAMEx (Eqn 7) compared to EP-CAMEx (Eqn 3). Does the Nash Bargaining process implicitly account for the necessary geometry, or is this simplification merely an artifact of aligning with the existing bargaining framework?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is exceptionally well-written and clear. The authors effectively motivate the need for a principled merging strategy by highlighting the competitive and cooperative dynamics observed across different SMoE layers and architectures (Figure 1, Figures 6, 8, 9). Key concepts like the Nash Bargaining Solution and its adaptation to multi-task learning are concisely reviewed. The definition of NAMEx (Definition 3.3) and the algorithms (Algorithm 1 and 2) are clearly presented. The comprehensive experimental results are organized logically into tables, with the best performing variants (NAMEx-Full results) highlighted clearly.\n\nThis paper offers a highly original contribution to the Sparse Mixture of Experts:\n- Novel Game-Theoretic Framework: NAMEx introduces the first game-theoretic interpretation of expert merging, leveraging Nash Bargaining to move beyond heuristic or input-independent averaging schemes. This principled method for balancing cooperation and competition among experts is highly original.\n- Enhanced Convergence via Complex Momentum: The successful integration of complex momentum into the expert propagation mechanism (NAMEx-Momentum) accelerates convergence and provides needed stability, particularly addressing limitations found in EP-CAMEx. The theoretical convergence guarantee further enhances this contribution.\n\nThe central claims—that expert merging can be framed as a bargaining game and that NBS provides a principled solution—are adequately supported with both theoretical derivation and comprehensive empirical evidence.\nThe paper formalizes NAMEx as the Nash solution to the Bargaining of Expert Merging Problem, derived from the Nash product maximization objective. The authors provide a proof sketch, which establishes the Nash Bargaining equation $G^\\top G \\alpha = 1/\\alpha$ for computing the optimal update direction. Furthermore, the introduction of complex momentum is backed by a convergence guarantee.\nThe empirical methodology is thorough, testing NAMEx and its momentum variants against baselines across four distinct domains. The consistent performance superiority shown across small, medium, and large-scale models confirms the effectiveness and scalability of the method. The supplementary analysis showing NAMEx yields faster and more stable convergence compared to EP-CAMEx (Figure 5) validates the motivation for using complex momentum.\n\n**Originality and Significance**: The core innovation of framing expert merging as a bargaining problem solved via the Nash Bargaining Solution is highly original and offers a principled alternative to existing heuristics. This perspective enables a derived weighting mechanism ($\\alpha$) rather than a heuristic one.\n\n**Quality of Results**: NAMEx variants consistently deliver superior performance across various benchmarks.\n\n**Clarity and Insight**: The paper is clear in its definitions and provides insightful empirical analysis, such as visualizing how NAMEx steers outcomes closer to the Pareto surface than linear averaging (Figure 11). The investigation into complex and quaternion momentum further pushes the boundaries for stability in expert propagation.\n\n**Theoretical Foundation**: The theoretical proof of convergence for NAMEx-Momentum underpins the method's stability and provides a foundation for future analysis.", "weaknesses": "I don't find any major weakness in the paper. \n\nHere are two typos. Please fix\n- Line 365: \"present\" -> \"presents\"\n- Line 1187 \"hese” -> \"these\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3xyb0YTFr", "reviewer": "Reviewer_eQrG", "summary": "This paper proposes a novel framework, Nash Merging of Experts (NAMEx), for merging Sparse Mixture of Experts (SMoE) models into a single dense model for efficient deployment. Departing from traditional heuristic-based averaging, NAMEx models expert merging as a multi-agent cooperative game. It applies the Nash Bargaining Solution (NBS) from game theory to compute fair and Pareto-optimal merging weights that reflect each expert's contribution. Furthermore, to address the slow convergence of prior expert-propagation methods, the authors integrate complex momentum, resulting in the NAMEx-Momentum variant. The method's effectiveness is demonstrated through extensive experiments on language, text, and vision tasks, as well as on large-scale models like DeepSeek-MoE 16B and Qwen1.5-MoE 14B.", "questions": "1.  **Overhead and Hyperparameters:** For the key results in the main paper (e.g., Table 2), how many NBS iterations were *actually* used for the best-performing model, NAMEx-Full-Mom (was it 2 or 20)? What is the concrete wall-clock time overhead of your final, optimized model during training compared to the EP-CAMEx baseline?\n2.  **NAMEx vs. NAMEx-Full:** The superior performance of NAMEx-Full (recomputing $\\alpha$ at each layer) over NAMEx (reusing the first layer's $\\alpha$) seems to confirm that layer-wise dynamics (as hinted in Figure 1) are critical. Does this not imply that the basic \"NAMEx\" variant is a methodologically flawed ablation, as it ignores the very dynamics the paper observes?\n3.  **Choice of Disagreement Point:** The paper follows prior work in setting the \"disagreement point\" to 0 (no update). Did you consider other, perhaps more natural, disagreement points for this specific problem, such as \"standard average merging\" (the main heuristic baseline)?\n4.  **Role of the Curvature Matrix $M_i$:** Why was the curvature matrix $M_i$ explicitly removed from the NBS optimization step (Eq. 9, line 1) but kept in the final expert update step (Eq. 9, line 2)? If the game-theoretic solution itself ignores curvature, what role does $M_i$ play in the final update?\n\n\n----\n\nReferences\n\n\n\n[1] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017, 2022. \n\n[2] Viet Dung Nguyen, Minh Nguyen Hoang, Rachel Teo, Luc Nguyen, Tan Minh Nguyen, and Linh Duy Tran. CAMEx: Curvature-aware merging of experts. In The Thirteenth International Conference on Learning Representations, 2025.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The primary strength lies in reframing expert merging as a Nash bargaining game. This provides a principled, non-heuristic solution for combining experts, borrowing a sophisticated tool from multi-task learning to solve a practical problem in model deployment.\n\n The paper identifies a key weakness in prior work (EP-CAMEx), namely its slow convergence and suboptimal performance. The introduction of complex momentum is a targeted solution that demonstrably improves convergence speed and stability, as shown in the empirical analysis (e.g., Figure 5).\n\n* **Comprehensive and Strong Validation:** The experimental validation is extensive and a significant strength. The method is tested across multiple modalities (language, vision), on robustness benchmarks (ImageNet-A/O/R), and scales up to 16B parameter models, showing consistent improvements over baselines. This broad validation builds strong confidence in the method's generalizability and practical utility.\n* **Clarity:** The paper is well-written and clearly organized. Figure 2, for instance, provides an excellent visual comparison between NAMEx, CAMEx, and EP-CAMEx, making the architectural contribution easy to understand.", "weaknesses": "W1.  The core technical contribution is a clever *combination* of two very recent works: the expert propagation framework from (EP-CAMEx and the NBS) [1,2] . optimization from a multi-task learning paper. While highly effective, this might be seen as a successful application rather than a fundamental theoretical invention.\n\n\nw2.The NAMEx method requires iteratively solving the NBS equation, which introduces significant computational overhead. The appendix (Table 11) shows a 6.8x increase in runtime for a per-layer update. While the final model (NAMEx-Full-Mom) seems to mitigate this by using fewer iterations (as per Appendix F.1), this trade-off between accuracy and cost is a notable weakness.\n\nw3.The \"bargaining budget\" (number of NBS iterations) is a critical new hyperparameter, but its treatment is confusing. The text mentions \"20 steps\" in one place and \"2 iters\" in another. The paper lacks a clear ablation study on how this budget (e.g., 2 vs. 5 vs. 20 iterations) affects both final performance and computational cost.\n\nw4.The paper justifies its choice of utility function by drawing an analogy between the domain vector $\\tau_i$ and a task gradient. This is intuitive but remains an analogy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jqqSck3zq", "reviewer": "Reviewer_kguY", "summary": "This manuscript introduces Nash Merging of Experts (NAMEx), a novel, theoretically-grounded framework for compressing Sparse Mixture of Experts (SMoE) models. The core innovation is the reinterpretation of the merging process as a cooperative game, utilizing Nash Bargaining to determine the optimal, balanced parameter weighting for merging two or more experts. The authors claim this approach, coupled with a mechanism for enhanced convergence (complex momentum), leads to significant reduction in the SMoE's memory and compute footprint while preserving performance. The work is highly relevant to the scalable deployment of large sparse models.", "questions": "* Could the authors explicitly detail the mathematical expression of the utility function, $u_i(\\theta)$, for a single expert $i$ within the merging context? How is this function derived from the expert's parameters or its performance on a held-out dataset?\n* Please provide a clear, detailed equation for the \"complex momentum\" and explain how it differs from standard momentum methods (like Nesterov or Adam momentum). Does it specifically counteract issues arising from the non-convexity introduced by the merging objective?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The most significant strength is the introduction of a game-theoretic perspective (Nash Bargaining) to expert merging.\n* The work directly addresses the scalability and deployment challenge of large SMoE models by enabling significant model compression. The results demonstrating reduced footprint while maintaining performance are very compelling.", "weaknesses": "* While the memory/compute reduction is a strong result, the paper must clarify the computational overhead of the NAMEx merging process itself. Since Nash Bargaining requires solving an optimization problem to determine the merge point, how does the time taken for merging compare against simpler methods like magnitude pruning or basic parameter averaging? A clear complexity analysis or runtime comparison is needed.\n* The core of Nash Bargaining relies on the definition of the experts' utility functions and the disagreement point (status quo). The manuscript needs to dedicate more detail to explaining why the chosen utility function accurately captures an expert's \"value\" or \"specialized knowledge,\" and how it relates to the loss landscape or data distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)