On Linear Mode Connectivity of Mixture-of-Experts Architectures
Authors: Viet-Hoang Tran
Van-Hoan Trinh
Khanh Vinh Bui
Tan Minh Nguyen
Venue: NeurIPS 2025 oral ()
Keywords: linear mode connectivity
mixture-of-experts
Primary area: theory
TL;DR: We investigate Linear Mode Connectivity (LMC) in Mixture-of-Experts (MoE) architectures by analyzing their underlying permutation symmetries and proposing expert-matching algorithms that align independently trained MoEs to reveal LMC.

Abstract: Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapes
of neural networks, wherein independently trained models have been observed to
be connected—up to permutation symmetries—by linear paths in parameter space
along which the loss remains consistently low. This observation challenges classical
views of non-convex optimization and has implications for model ensembling,
generalization, and our understanding of neural loss geometry. Inspired by recent
studies on LMC in standard neural networks, we systematically investigate this
phenomenon within Mixture-of-Experts (MoE) architectures—a class of models
known for their scalability and computational efficiency, which combine traditional
neural networks—referred to as experts—through a learnable gating mechanism.
We begin by conducting a comprehensive analysis of both dense and sparse gating
regimes, demonstrating that the symmetries inherent to MoE architectures are
fully characterized by permutations acting on both the expert components and the
gating function. Building on these foundational findings, we propose a matching
algorithm that enables alignment between independently trained MoEs, thereby
facilitating the discovery of LMC. Finally, we empirically validate the presence of
LMC using our proposed algorithm across diverse MoE configurations—including
dense, sparse, and shared-expert variants—under a wide range of model settings
and datasets of varying scales and modalities. Our results confirm the existence
of LMC in MoE architectures and offer fundamental insights into the functional
landscape and optimization dynamics of deep learning models.

On Linear Mode Connectivity of Mixture-of-Experts Architectures
Viet-Hoang Tran*t Department of Mathematics National University of Singapore hoang. tranviettu.c.. nus edu
Van-Hoan Trinh"
Department of Mathematics Technical University of Munich
vanhoan. trinh@tum. de
Khanh Vinh Bui Independent Researcher Ho Chi Minh City, Vietnam com
Tan M. Nguyen
Department of Mathematics National University of Singapore tanmnOnus .du..uuuuu sg
Abstract
Linear Mode Connectivity (LMC) notable phenomenon in the loss landscapes of neural networks, wherein independently trained models have been observed to be connected-up permutation symmetries-by linear paths in parameter space along which the loss remains consistently low. This observation challenges classical views of non-convex optimization and has implications for model ensembling, generalization, and our understanding of neural loss geometry. Inspired by recent studies on LMC standard neural networks, we systematically investigate this phenomenon within Mixture-of off-Exerts (MoE) architectures-2 class of models known for their scalability and computational efficiency, which combine traditional neural networks-referre to experts experts-through learnable gating mechanism. We begin by conducting a comprehensive analysis of both dense and sparse gating regimes, demonstrating that the symmetries inherent to MoE architectures are fully characterized by permutations acting on both the expert components and the gating function. Building these foundational findings, propose matching algorithm that enables alignment between independently trained MoEs, thereby facilitating the discovery of LMC. Finally, empirically validate the presence of LMC using our proposed algorithm across diverse MoE configurations includdin dense, sparse, and shared-exe exxerr variants under wide range of model settings and datasets of varying scales and modalities. Our results confirm the existence of LMC in MoE architectures and offer fundamental insights into the functional landscape and optimization dynamics of deep learning models. The code publicly available at https: /ttpbbb..tttttttg//gg//g.g http/M/Reeearch... caw.aac--e.....m.e..me.m
Introduction
Despite the high-dimensional and non-convex nature of deep neural network (DNN) training, stochas- tic optimization methods-such as stochastic gradient descent (SGD) and its variants-consistently find solutions that generalize well. This empirical success contrasts with the theoretical complexity of the loss landscape, which contains many local minima. A growing body work has uncovered surprising phenomenon known as mode connectivity, wherein minima found by independent SGD runs can often be connected by continuous low-loss paths.
"Co-first author
i Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mode Connectivity. Related work on mode connectivity includes [34, 45, 67, 85, 61, 76, 91, 94]. Empirical studies demonstrate low-loss connections between independently trained models on MNIST and CIFAR-10 [31, 33, 21], and have shown that nearly any two solutions can be linked by low- error curve [32]. The implications of mode connectivity extend beyond theoretical curiosity. provides insight into why weight-space ensembling techniques, such as Stochastic Weight Averaging. yield improved generalization [39, 63, 89]. Moreover, it has proven useful in studying adversarial robustness [93], generalization theory [63, 44, 55], and the geometry of loss landscapes [35, 87, 56]. more restrictive and analytically convenient variant of this phenomenon is linear mode connectivity (LMC), where two models are connected by linear interpolation in parameter space along which the loss remains low [30, 24].
Permutation Invariance. A central challenge in observing LMC stems from the permutation in- variance of neural networks: permuting neurons within a hidden layer leaves the network function unchanged [7, 23, 29, 9, 60). Consequently, two functionally equivalent models may appear discon- nected in parameter space unless one is suitably permuted. To address this, recent work considers LMC up permutation, wherein low-loss linear path exists after aligning hidden units [71, 2. 36]. Optimal Transport (OT) [86, 82, 78, 79, 80J methods have been proposed to compute soft matchings between neurons [71, 72], enabling applications such as model fusion in federated learning. Em- pirically, OT-based alignment achieves near-zero error barrier LMC between independently trained ResNets CIFAR- 10 [2], with connectivity improving with width and degrading with depth. Theo- retically, dropout-star stable networks have been shown to exhibit mode connectivity [48, 70]. and [24] demonstrate that LMC up permutation
already arise at initialization, especially in the NTK regime [41]. [28] further provide formal guarantees for LMC under OT alignment. These insights support the convexity conjecture [25]. which posits that the SGD solution set is approximately convex once symmetries are accounted for. This view is strengthened by [68], who propose simultaneous linear connectivity, where single model aligns linearly with multiple others. Additional studies explore the geometry of the solution space [3, 90] and identify star-shaped regions conducive to LMC [73].
Functional Equivalence. Prior work on LMC has primarily focused on feedforward and convo- lutional architectures, owing to the well-characterized permutation symmetrics in these models [11, 62, 13, 77, 88). In contrast, the symmetry structures of more modern architectures-sude as Transformers [84, 19, 12] and Mixture-of-Expers [40, 69, 52, 26)-remain relatively underexplored. The study of these symmetries falls under the broader concept of functional equivalence, which aims to characterize when different parameter configurations yield identical input-output behavior [37, 15, 27, 50, 6J. Recent work has extended this analysis to attention-bass -bsed models, with [83, 46] examining symmetries in attention mechanisms, particularly permutations of heads and group actions induced by the general linear group. A rigorous understanding of functional equivalence is essential for uncovering and formalizing LMC in such complex, structured architectures.
Permutation Alignment Methods. These methods align parameter permutations to establish LMC. [25] proposed a simulated manseling-based algorithm. [72] employed Optimal Transport, while [4] utilized the Wasserstein Barycenter. [2] introduced three methods: activation matching (using Estimator (which minimizes interpolation midpoint loss via gradients); all are based on solving the Linear Assignment Problem [49, 42, 17]. [36] developed Sinkhorn re-basin, differentiable method that improves alignment but struggles with residual connections due to layer-independent optimization.
intermediate activations), weight matching (in parameter space), and the Straight-Througg
Contribution. Motivated by this line of work, we extend the investigation of LMC to Mixture-of- Experts (MoE) architectures, which are related to traditional neural networks in that they aggregate outputs of multiple subnetworks via gating mechanism. The paper is organized follows:
1. In Section 3, we introduce the concept of the weight space of MoE architectures and define a group action on this space that preserves the functional behavior of MoE models.
2. In Section 4, we present two core results concerning functional equivalence in MoE models. We demonstrate that the proposed group action characterizes all inherent symmetries of the MoE gating mechanism, with rigorous theoretical justification.
3. In Section 5, we first observe that permutation invariance alone is sufficient to induce LMC. Building on this insight, we develop a Weight Matching algorithm that enables alignment between independently trained MoEs, thereby facilitating the discovery of LMC.

---

4. In Section 6, we provide empirical evidence of LMC across wide range of MoE configurations including dense, sparse, and shared-expert variants-evaluated under diverse model settings and across datasets of varying scales and modalities. Additionally, we assess the effectiveness of our proposed expert-matching algorithms, and conduct ablation studies analyze LMC at different layers within deep MoE. models.
Section offers background on LMC and MoE architectures. Table of notation, along with theoretical foundations, experimental details, and additional content, is provided the Appendix.
2
Preliminaries
This section provides an overview Linear Mode Connectivity and Mixture-of off-Exssts architectures.
2.1 Linear Mode Connectivity
Let f(::8) be a function parameterized by 0 E θ, where denotes the weight space. Given a non-negative loss function C(0), seek to minimize [(0) over O The notion of the loss barrier was originally introduced by [30] and later formalized by [24] as follows: for A,,, 0 E O, define
B0 = sup [C(10A + (1-1)0B) (IC(0A) + (1-0))((g))) xE(0.1]
(I)
A and O are said to be linearly mode connected if their loss barrier is negligible, i.e., B(O 0n) 0. In many architectures, the function f(: is invariant under certain permutations the weight space- namely, for permutation g and all 0 E O, we have f(:: 0) f(; go). Therefore, A and fB are said to be linearly mode connected up to permutation there exists a permutation g such that ss 0. Any such permutation g is referred to as a winning permutation [24].
2.2 Mixture-of-Experes Architectures
A Mixture-of-Expert (MoE) is neural network architecture that combines the outputs of multiple models into a single prediction through gating mechanism that assigns input-dependent weights. The two most commonly used gating strategies are dense and sparse gating. While we provide a brief overview these concepts here, a comprehensive and formal treatment is deferred to Appendix A.
Expert. Let d denote the imput dimension. We define an Expert as function E-; Rd R parameterized by E I", where e N represents the total number of trainable parameters of E. In this work, we consider each expert E:; 0) to be feedforward neural network with ReLU activation functions. Unless stated otherwise, all experts are assumed to share the same architecture.
Mixture-of-Be Exxprrs with dense gating, Given n denoting the number of experts, we define Mixture- of Ex-extsss with dense gating function D: Rd Rd such that
D(1; (W bi, 0.1
Σ
(.t (x)))
where sz(x) = Wa + b; determines the contribution of each expert to the final output. Here, 0; the parameters of the 7th expert. The function (81,...,8, is called the gating score, and is parameterized s(:; (W,, b, ) with (W,, b.) E Ed x R are the corresponding gating parameters.
Mixture-of-Experts with sparse gating. Given positive integer & < n denoting the number of activated experts, define the Top-k map by Top-k(z) = (i1,..., i for (#1,.... 2n) E R where in. f1.....y, is are the indices corresponding to the largest components of æ. In the event of ties, we select smaller indices first. We define a Mixture-of Experts with sparse gating (SMoE) as function S: Rdd Idd as follows. For & E Rd, let T(x) = Top=k(s(z)), then define:
(z: (.., by 0,)
Σ TET(T)
softmax; ((=:())) E(:; 0z)
(3)
In other words, the Top-k selects the & highest-scoring experts used compute the output.
3

---

3 Group Action on Weight Space of Mixture-of-Experts
In this section, we introduce the formal notion of the weight space associated with MoE models. Furthermore, we define group action on this space that preserves the functionality of MoE. A complete and rigorous exposition of these definitions and their implications is provided in Appendix A.
3.1 Weight Space of offtture-of-Eppert
The map D is parameterized by:
(4) Here, ((n) is called the weight space of a Mixture-off1--Exertt.. Varying the number of experts leads to general denoted by = Ln>) Note that the weight space of & coincides with that of S, since the map Top-k does not introduce any new trainable parameters.
(W,, b, 0.):=1.....n Φ(n) = x (R4 X R9)".
weight space that spans across expert sets of different sizes,
3.2 Group Action on Weight Space of Mixture-of-Experr
We define the group G(n) as the direct product G(n) 12 x IR Sn of the groups Rd, IR with addition, and the permutation group Sn. Each element E G(n) is of the form g = (cw, Cb T), where CW E Rd, Cr E IR and E Sn. The group (nn) acts on the weight space ((n) as follows. For
g E G(n) and o E ((n) presented as Equation (4), define:
gō (W+() (1) + cw, br()) cb, (.00)) E ((n).
(5)
The result below establishes that this group action preserves the MoE map. Proposition 3.1 (Weight space invariance of Mixture-of -f-upertst-...rs....... The MoE function is G(n)- invariant under the action of ((n) its weight space (n), i.e. D(: D(: g()
A proof of Proposition 3.1 is presented in Proposition A.3. An analogous invariance result holds in the case of the SMoE function S. However, since the Top-k selection map is generally discontinuous- primarily due tie cases in the gating acores-additional conditions are required ensure the validity of the invariance result. To address this, we focus on a subset of TRd where the Top-k scores are unambiguously defined. Specifically, for {W,, by 1n E (IR x IR), we define:
(6) The SMoE function S exhibits well-behaved properties on this domain. First, consider the case where pairwise distinct. In this case, the set 22( (W,, i. b=); is clearly empty. However, when the pairs (W,, by ) are pairwise distinct, the set 52 (, by ba); becomes an open and dense subset of Rd. Moreover, the SMoE function S is continuous on (([,, b, 1=))) These properties are formally established in Propositions A.1 and A.2. We now show that the invariance property of the SMoE map holds under restriction to this domain.
e Rd (W.c + b,)
pairwise distinct).
the pairs (W., 1
Proposition 3.2 (Weight space invariance of Sparse Mixture-of Ex-perrtss... Given the SMoE function S, as defined in Equation (3), Assume that [W b.} are pairwise distinct l..... nt. Then, the set Q(W,; i, by )=)= is invariant under the group action of G(n), i.e., for g (cw, ch, E G(n), we have O (WW bi) )=)) = S([Wr() + cw, br(i) + ct In Moreover, the function S, restricted to D([,,, b n 1= ). is invariance under the action of G(n) on its weight space ((n). i.e. S(=i))
((:g0) on Q (W,, 1
A proof of Proposition 3.2 is presented in Proposition A.4.
Remark 3.3. The permutation invariance of the summation operator and the translation invariance of the softmax function are the two primary sources of invariance for the functions D and S, as stated in Propositions 3.1 and 3.2. In the case of SMoE, these invariance properties are additionally upheld by the permutation and translation invariance of the Top-k operator.
4 Symmetries and Functional Equivalence in Mixture-of-Experts
In the remainder of this section, we let o ((n) and ø' 4(n') denote the parameters of two Mfxture-of-Exee Experts models with n and n' experts, respectively:
o (W,, b,, 0 et. (nn)) and o' = (W1, b/, (l 2(n').
(7)

---

4.1 Functional Equivalence in Mixture-of-Expert
We aim to characterize the conditions under which distinct parameter sets yield functionally equivalent MoE models. The central result of this section is that the symmetries of the gating mechanism fully characterized by the group action of G(n), as defined in Equation (5). Given the fundamentally different structural and analytical characteristics of the two gating mechanisms, independently in the analysis. In addition, we introduce a set of assumptions, the motivations for which are discussed in detail in the following subsection. We begin with the case of dense gating.
treat each
Theorem 4.1 (Functional equivalence in Mixture-ot of-Experts with Dense Gating). Suppose o of define the same MoE function, ie., D(5 Φ) D(; d'). Assume that the following conditions hold;
1. Both (E(;;:))]== and {E(:; ( yr' consist of pairwise distinct functions;
2. Both (W; - WWhhsksssn and (W Wjiisisisn' consist of pairwise distinct vectors in TRd.
Then nt, and there exists 9 (ew, T) E G(n) such that for all i = 1,.... ; we have W = W+ cw, by bz(1) + Cb. and E(;; ) = E(; (-())) on Rd.
For the case of sparse gating, we require the notion of the strongly distinct property. Specifically, two functions f and g defined on IRd are said be strongly distinct if the set {a Rd f(x) # g(x)3 is dense Rd. For instance, distinct polynomials are strongly distinct, whereas distinct ReLU networks are not strongly distinct in general. A formal definition of this property, along with illustrative examples, is provided Definition C.I and Example C.2.
We now state a result that parallels Theorem 1.1, adapted to the sparse gating regime with > 1, and established under set of assumptions that are stronger than those previously required.
Theorem 4.2 (Functional equivalence in Mixture-of-Expert. with Sparse Gating). Suppose of define the same SMoE function, i.e., S(; Φ) S(: &'). Assume that the following conditions hold:
1. Both {E(;; and {E(:; 0 consist of pairwise strongly distinct functionn;
2. Both { w. and W Wi are linearly independent subsets of Rdd
Then n², and there exists g (cw, cb. T) E G(n) such that for all i :,...: n, have : = WW(() + cw. by = by(1) Cb. and (x;; 8.) = Ex; (7(1)) for all x E 2((W,, b. )-m) such that r(1) E Top-k((W,a + +b, 1-)
The proofs Theorems 4. and 4.2 are provided in Appendix B and Appendix C, respectively. Both arguments build upon two essential ingredients: result establishing the linear independence of exponential functions, as formulated in Lemma B.3, and key observation on the piecewise affine structure of ReLU networks, discussed in Appendix B.
Remark 4.3. While Theorem 4.2 shares a conceptual parallel with Theorem 4.1, it is important to underscore that the sparse gating case presents significantly deeper mathematical difficulties. The core source of complexity lies in the discontinuous behavior of the Top-k operator, which induces nontrivial discontinuities in the gating function, thereby disrupting smoothness and complicating the functional analysis required establish equivalence. As result, standard analytical techniques used in the dense setting become insufficient, necessitating more delicate arguments to account for the combinatorial and piecewise structure inherent to sparse gating.
Theorems 4.1 and 4.2 provide rigorous characterization of functional equivalence in both dense and sparse MoE models, with particular attention given to the role and structure of the gating mechanism. Nonetheless, it is important to note that these results do not account for all potential symmetries inherent in the MoE and SMoE mappings defined in Equations (2) and (3). Notably, further symmetries may exist within the expert networks, especially when they are implemented as ReI.U neural networks. Given that this work is primarily concerned with the MoE architecture itself, our analysis restricted to the behavior of the gating mechanism. Consequently, we treat the experts as black-box functions and abstract away from their internal parameterizations.
5

---

4.2 Necessity and Implications of Technical Assumptions
At a conceptual level, symmetry analysis aims to identify universal invariances those that persist regardless of specific parameter values explicitly excluding singular symmetries, which emerge only under special. degenerate configurations of the model parameters. The assumptions introduced in our results are designed precisely to rule out such singularities. which do not represent inherent structural invariances of the architecture but instead arise from pathological or measure-7ero subsets of parameter space. We provide a brief overview of the motivation behind these assumptions here; a more detailed justification, along with illustrative examples, is given in Remarks 6.6 and C.8.
The case of dense gating, Assumption I is introduced to eliminate degenerate scenarios in which two experts compute identical functions and receive identical gating scores-situations where permuting the corresponding experts has no effect on the model's 's output. By excluding linear dependencies among the gating weight vectors, Assumption 2 ensures that expert activations remain distinguishable- thereby preventing the emergence of non-structural, symmetry -iike artifacts.
The case of sparse gating. Theorem 4.2 relies on stronger set of assumptions than those required in Theorem 4. 1. This added strength is necessary due to a key feature of sparse gating: an expert's behavior inputs where it is not selected is unconstrained, meaning that the expert act arbitrarily outside its region of activation. Consequently, different sets of expert functions may result in the same overall model output, provided their outputs agree where they are active.
The case of k = 1. In the particular case where = 1, the SMoE architecture employs a Top-1 gating mechanism that activates only the expert associated with the highest gating score. Consequently, the softmax output reduces to a one-hot vector, with a single component equal to 1. Under this regime, the SMoE map exhibits additional nontrivial symmetries, specifically invariance under the multiplicative group 1 That is, for any positive scalar e >> 0, the following identity holds:
(1;; (W; by, m-) (ax; (cW., cbi, 0-);=1)
(8)
This invariance holds because the argmax used for expert selection is unaffected by positive scaling, i.e. argmax;=1....p (W,æ + 1b)) = argmax (cW,r (.............rr + cb;). for all æ E S( by H= Addi- tionally, since only single expert contributes to the output, the model lacks any form of explicit
aggregation across experts. Due to the analytical challenges posed by these additional invariances, we do not consider in our primary results and instead defer its investigation to future work.
5 Linear Mode Connectivity of Mixture-of-Experts Architectures
This section outlines the theoretical foundations of LMC in MoE architectures. We first show that permutation invariance suffices to explain the existence of LMC in MoEs, then propose algorithm to identify such permutations, which we later employ to empirically assess LMC in MoE models.
5.1 Permutation Invariance Sufficiency in Linear Mode Connectivity of Mfxture-of-Experts
Theorems 4.1 and 4.2 show that the group action as in Equation (5) suffices to uncover LMC between two MoE models. Each group element g (ew, Ch, E G(n) consists of two components: translation term (cw, <6) E Ed x R applied to the gating function, and permutation that reorders both the experts and their associated gating scores. Observe that the translation component does not affect the barrier loss defined in Equation (1). Indeed, for any h (cw, cb, id,)) E G(n)- id,, denoting the identity permutation S,,-th loss barrier remains unchanged:
B(AA, dB) = B( hΦB).
This invariance, established in Proposition D.1, indicates that only the permutation component r influences LMC behavior. Combined with prior work showing that permutation symmetries are sufficient to induce LMC in standard neural networks [24, 28], this suggests that analyzing permutations alone suffices to capture LMC in MoE models.
5.2 Permutation Alignment Algorithm for Mixture-of-Experts
We propose the Permutation Alignment Algorithm to align MoEs. Inspired by the data-independent Weight Matching algorithm [3], our method operates the parameter space of two MoEs, enabling efficient permutation alignment tailored to MoE architectures in large-scale settings.

---

Table Configurations for experiments analyzing LMC in Transformers with dense MoE replacing the FFN in the first layer (refer to Table 5 for SMoE and DeepSeekMoE). For each dataset, number of layers, and number of experts, we visualize LMC curves for the three model pairs among three models in the referenced figures. Datasets denoted as A - B indicate pretraining on dataset A, fine-tuning on dataset B, and evaluating on B.
Dataset
Layers
Experts
Figure
Dataset
Layers
Experts
Figure
MNIST
[2, 4]
[5, 6]
12
[2, 4,
[18, 19, 20]
2
[2. 4]
[7, 8]
ImageNat=2Ik-CIFAN
12
[2, 4, 6]
[21, 22, 23]
CIFAR-10
[2
[9, 10, II]
ImageNett-l
2,
[24, 25, 26, 27]
6
[2, 6]
[12, 13, 14]
WikiText103
12
6,
[28, 29, 30, 31]
CIFAR-100
6
[2, 6]
[15, 16, 17]
One Billion Word
12
[2, 4, 6, 8]
[32, 33, 34, 35]
CIFAR-100
ImageNet21k
ImageNet1k
Billion
Xalv WM
Test Looss
L5
4.5-
4.0-
100
3.5
Modal
Nodal 2
Modeal
Mode Model
Model Model
Modal
Figure I: LMC curves for VIT (subplots 1-3) and GPT-2 (subplot 4) with a 4-expert MoE replacement ImageNet-Ie Ik, and One
at the first Transformer layer, on CIFAR-100, ImageNe2IK--CIFA--DI Billion Word datasets, respectively. Plots show consistent low-loss linear interpolation paths between fine-tuned models, indicating strong linear mode connectivity.
Proposed Algorithm. In MoEs, equivalent functionality arise from different expert orderings and internal neuron configurations. To address this, we propose a two-stage alignment method that first reorders experts to match their roles, then aligns the internal weights of corresponding experts. Both steps are formulated as Linear Assignment Problems (LAPs) [10]. An LAP assigns tasks to N agents using cost matrix C' E R N where each entry Cij represents the cost of assigning task to agent.. j. The objective is to find permutation æ that minimizes the total cost: min Σ Cz,x()) This is efficiently solved using the Hungarian algorithm in ((N3) time [49, 42, 17].
Suppose we have (W;, b,, O.)=i...... and d' (W/ bg. (1) representing the parameters of two distinet MoEs with experts. In practice, each expert is implemented as an MLP with one hidden layer [26, 52, 22]. We denote the parameters of expert i in á and expert j in o' as 0; {Ai, uy B3, v. and 0% = {A', u/y, By, v/J), respectively, where A, E gxx u4 E Rhh B, Raxh es Rdd, with h as the hidden dimension. Denote A, [Ar, u.] E (dxx(d-1), B, [B vi] (dx(1++),, and similarly for A' and B/, as concatenated matrices.
Matching Order of Experts. In MoEs, experts and their gating can be reordered without affecting functionality. We propose two methods to align the expert order between á and ø', using LAP.
Method Gating Weights. To address the wranslatonninvariinne of the softmax function in the gating, we center the weights and biases of the gating. For Φ, these are W; W; Σ Wm and b, b, Em= bm per expert i, and similarly for Φ'. The cost matrix with respect to gating weights is defined C {Cr.hhssssnnssnn E paxz, where
C₁2 Ctj= - W1 83)3)*
(10)
measuring dissimilarity between centered gating parameters of expert in & and expert in ø'. Method Internal Weights. Experts exhibit permutation invariance in hidden neuron orderings. We use Gram matrices for a permutation-invariant invariant representation (see Appendix D.2). The cost matrix
7

---

with respect to expert internal weights is defined as C = [Ci ER where
Ci. (1) 1
2 BBBB
(11)
quantifying dissimilarity between weight representations of expert in Φ and expert j in ø'. For both methods, the optimal expert ordering is determined by solving the LAP: arg mis. Σi=1 Ci,r(1) O(n") time.
Aligning Expert Internal Weights. After finding the permutation T to align experts between and 6', align the internal weights of each matched expert pair (t)) using established MLP alignment methods [2, 36, 25, 72], specifically the Weight Matching algorithm [3]. This LAP-based algorithm aligns MIPs with one hidden layer of size h in (h²) time, ensuring efficient scaling.
Applying this to each pair (i. r()) guarantees consistent neuron ordering across aligned experts.
5.3 Weight Matching Algorithm
We propose a Weight Matching Algorithm designed align MoEs by addressing permutation invariance in expert ordering and internal parameters, as detailed in Algorithm 1. The algorithm employs gate-based and expert-based methods to optimally permute experts. with both performing comparably well in loss barrier evaluations during model interpolation (see Section 6.3). Operating solely parameter space, the algorithm avoids data data-dependent computational overhead, with complexity of On + nh²), ensuring scalability for large-scale MoE alignment tasks.
Algorithm 1 Weight Matching for Mixture-of-FER Ex-errs
Input: MoE model weights (W;, s; b,, 0.)==1...... o' (W7, (4,8))....... Output: Permutation 7 for experts, and permutations {P: for hidden units % Step 1: Match experts' order using two methods for method in (gate, expert) do
Compute cost matrix C
Solve LAP to obtain expert permutation Tmethod end for
% The two candidate expert orderings Tgate and Texpert are obtained % Step 2: Align internal weights of matched expert pairs for method in (gate, expert) do
for = 1 to n do
Compute P. by applying Weight Matching to 0g and 6'
end for end for return
(Teane: ((Pi) )gme).
(Texpent: (Texpent((PP11 ((P.)
Experiments
This section provides empirical validation of LMC across three MoE variants, including dense MoE, sparse MoE (SMoE), and DeepSeekMoE, detailed in Section 6.1. By analyzing interpolation paths between independently fine-tuned models-each subject permutation invariance-low-lle connections are consistently observed, indicating shared solution basins in the optimization landscape.
6.1 Experimental Setup
Experimental Design. We investigate LMC by replacing the Feedforward Network (FFN) in the Transformer [84] layer with randomly initialized MoE, based on empirical evidence from Section 6.4, Appendices G.2, and G.3 indicating lower perturbation sensitivity when replacing deeper FFN layers with MoEs. Only the MoE parameters are fine-tuned using multiple random seeds for each experiment. LMC evaluated by linearly interpolating between all checkpoint pairs, measuring model performance the test set at 25 evenly spaced points along the interpolation

---

Table 2: Comparison of the two expert matching methods for 12-layer ViT-MoE models with 4 experts
on CIFAR-IV 10 and CIFAR-100, showing Rank (out of 24 permutations) and L Lrueheld Lugl x 10
Treaise Tugi
across layer placements, averaged over 10 checkpoint pairs. Please refer to Tables 8 and 9 for SMoE and Mee variants.
Layer replaced
Expert Woight Matching
Gute Weight Manching
Dusaset
Layer replaced
Expert Waight Matching
Weight Matching
Rank
1
L1
Rank
L.
Rank
L
Bank
L1
2.50
2:12= DA2
+ 1.00
0.55
CIFAR-100
0.40
0.25
290 +
273
2,10
1.04
2.00 092
134 0.44
3.60 1.20
1.15 0.55
2.00
203
2.70 0.46
0.60 = 0.27
2,00 0:30
0.74 0.17
3.30 1: 0.78
0.67 0:144
3.20 1:07
1:13
1:66
0.03
0.03
+.089
011
CIPAR-10 -10
CIFAR-10
CIFAR-100
CIFAR-100
2,5
0.54
2.0
0.14
0.01
Test Loss
1.5-
0.12
0.74
1.0
3.10
0.8
3.08
0.5
3.06
0.4
Model
Figure 2: Loss curves for 12-layer ViT-MoE models with a 4-expert MoE replacement in either the first layer (subplots 3) the last layer (subplots 2, 4), on CIFAR-10 and CIFAR-100. The curves compare two Expert Order Matching methods across 24 permutations, with Weight Matching applied post-reordering all permutations. Corresponding accuracy metries are presented Figure 96.
path. We examine three MoE variants: dense MoE [40, 43], SMoE with top-2 routing (k = 2) [26]. and DeepSecKMoE with top-2 and one shared expert (k 2, 8 = 1) (see Appendix D.3 for formal formulations) [54].
Datasets and Models. We use VIT [20] for image classification (MNIST [51]. CIFAR-10/100 [47], ImageNet [18]) and GPT-2 [64] for language modeling (WikiText105 [59] and One Billion Word [14]). Hyperparameters such as batch size, optimizer. number of experts, and hidden size are fixed, while the learning rate is tuned per setting. Full details are provided in Appendix F.
6.2 Linear Mode Connectivity Verification
Experimental Objective. We analyze LMC in pretrained Transformer models where the FFN of a Transformer layer is replaced with an MoE. Our study focuses on two configurations: replacing the FFN in the first, last and all Transformer layers. For each configuration, we fine-tune three independently initialized models. We vary the number of experts (e.g., 2, 4, 6, 8, 16) and the number of Transformer layers to assess LMC under different architectural scales. LMC is evaluated by linearly interpolating between all model pairs using our proposed algorithm (Algorithm I).
Results. Table shows results for first-layer MoE replacement, with full details for three MoE variants in Table 5. The results for both the last-layer and all-layer configurations Appendix G.2 and G.3. LMC persists in all cases, but is less pronounced in the last layer due to greater stability and reduced influence on convergence. Early layers strongly shape the solution basin, making first-layer changes more revealing of connectivity. This depth-dependent effect is also demonstrated in Section 6.4 and Appendix Figure presents representative LMC curves across our testing cases.
provided in
6.3 Expert Order Matching Method
Experimental Objective. We evaluate two Expert Order Matching methods (Step 1, Algorithm 1)- Expert Weight Matching and Gate Weight Matching-using 12-layer ViT models with 4-expert MoE replacements. The methods are assessed by ranking the selected permutation among all 24 possible expert permutations after Weight Matching (Step 2), with MoE integrations at layers 1, 4, 8, or 12 on CIFAR-10 and CIFAR-100 100, repeated five times. We report the rank and scaled metric

---

Table Ablation Study on varying the number of Transformer layers and MoE layer replacement in ViT for CIFAR-10. 10. The ratios of metrics (loss barrier and AUC, accuracy barrier and AUC) compare Algorithm to naive interpolation. For all MoE variants, kindly refer Tables 11, 12, and 13.
Lagars
Layar reploced
Lass Barrir
AUC )
Accumay Barter
Accuracy AUC
Layers
Layer replaced
Lasa Barnir
Losa AUC
Accuracy Barner
Accunacy AUC
834 A: 1.53
8.13 4 1.68
8.39 1.42
5.1 =: 1.39
4.16 =: 1.47
4,96 2.51
3.64 2 1,00
4.13 1.55
2.04
883
8,96
4
8.29 + 1.63
5.08 + 1,59
7.41-144 1.45
6.43 1.09
9.69 + 4.92
8.62-544
9.03++3.3
8,14-1.5 3,57
2
10.19 +: 3.43
10.19 3.16
10.21 5.25
9.08 5.35
10.83 =: 5.61
12.01 2,42
8.2
10,00 7.30
11.72
5.12 +
4.60 0.65
4.17-166 1.68
4.58 + 1.01
6,90 + 3.54
9.67 6.99
7.16 3,98
9.69 3.11
i. Lencchot Lowpl x 102, averaged over ten checkpoint pairs, where methhod Lopt and naive denote Laua Lisp!
loss barriers of our methods, the best permutation, and the naive interpolation, respectively.
Results. Table shows both methods achieve low ranks and near-zero L. indicating near-optimal matching. Full results are presented in Tables (loss) and 9 (accuracy). Figure visualizes the representative performance of our methods across all 24 permutations, highlighting the importance of correct expert order matching, poor ordering can yield performance close to the naive interpolation. To further validate the importance of expert alignment, we extend our analysis to the large-scale One Billion Word dataset. As shown in Table 10, the proposed Total Weight Matching consistently yields lower loss barriers compared to the Seappinn-Expert-OOOn variants, confirming that incorrect expert ordering substantially hampers interpolation smoothness. These results demonstrate that accurate Expert-Order Matching is crucial for revealing true Linear Mode Connectivity between MoE models, as even minor permutation mismatches can disrupt the underlying shared representation space.
Matching baseline across all MoE
6.4 Ablation Study on Number of Layers
This study investigates the impact of the number of Transformer layers on LMC in MoE models on CIFAR-10. We evaluate models with 2, 4, and layers, incorporating MoE at all possible layer positions. Employing Algorithm 1, we compute ratios for four metrics-loss barrier, loss Area Under the Curve (AUC, relative to the straight line connecting the metrics of the two endpoint models), accuracy barrier, and accuracy AUC, relative to linear interpolation-accne evaluating ten model pairs per configuration. Table 3 reports results for MoE, showing significant reduction ratios across all configurations. Corresponding results for SMoE and DeepSeckMoE are provided in Table 11, with detailed loss and accuracy metrics presented in Tables 12 and 13.
five repeated experiments,
7 Conclusion
This paper presents a systematic analysis of LMC in MoE models. We show that functional equiva- lence via permutation symmetries is sufficient to construet low-loss linear phen they exist- between independently trained models. To enable this, we introduce an algorithm for identifying such permutations, Empirical evaluations confirm the presence of LMC across wide range of MoE configurations-inclee
dense, sparse, and shared-expert variants-under diverse hyperparameter settings and across datasets of varying scales and modalities. While our method does not provide theoretical bounds on the loss barrier, this remains a common limitation across prior LMC studies. Overall, our work offers principled foundation for extending LMC analysis to other architectures. such as Transformers and State Space Models.
10

---

Acknowledgments and Disclosure of Funding
This research project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: D0GG2-TC-2023--12--SGL., This research / project is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier This research / project is also supported by the NUS Presidential Young Professorship Award (A-0009807-01-00) and the NUS Artificial Intelligence Institute-Seed Funding (A-8003062-00-00),
(FY2023) (A-8002040-00-00) A 1-8002039-00-000
References
[1] Lars V. Ahlfors. Complex Analysis. McGraw-Hill New York, 3rd edition, 1979.
[2] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04886 2022.
[3] Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re- Ra-Basin: Merging Models modulo Permutation Symmetries, December 2022. arXiv:2209.04836 [cs].
[4] Aditya Kumar Akash, Sixu Li, and Nicolás Garefa Trillos. Wasserstein barycenter- burycenter-based model fusion and linear mode connectivity of neural networks. arXiv preprint arXi::2210.0661 2022.
[5] Francesca Albertini and Eduardo D Sontag. For neural networks, function determines form. Neural networks, 6(7):975 1993.
[6] Francesca Albertini and Eduardo Sontag. Identifiability of discrete-time neural networks. In Proc. European Control Conference, pages 460-465. Springer Berlin, 1993.
[7] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on machine learning, pages 242-252. PMLR, 2019.
[8] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for web of open data. In international semantic web conference, pages 722-735. Springer, 2007.
[9] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):5544--15854 2019.
[10] Dimitri Bertsekas. Network optimization: continuous and discrete models, volume Athena Scientific, 1998.
[11] Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint
2019.
[12] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[13] Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of relu networks. In 8th International Conference Learning Representations, 2020.
[14] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 131..3005. 2013.
[15] An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural computation, 5(6):910 (16):90-9227, 1993.
[16] John B. Conway. Functions of One Complex Variable. Springer, New York, 2nd edition, 1978.

---

[17] David F Crouse. On implementing 2d rectangular assignment algorithms. JEEE Transactions on Aerospace and Electronic Systems, 52(4):1679-1690) 2016.
[18] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, leee, 2009.
[19] Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805. 2018.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[21] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1309-1318. PMLR, 10-15 Jul 2018.
[22] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Efficient scaling of language models with mixture-of-exppers, In International conference machine learning. pages 5547-5569, PMLR, 2022.
[23] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[24] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint 10.00221 2021.
[25] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur: The Role of Permutation Invariance Linear Mode Connectivity of Neural Networks, July 2022. arXiv: 1 10.06296 [cs].
[26] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[27] Charles Fefferman and Scott Markel. Recovering a feed-forward net from its output. In Advances in Neural Information Processing Systems 6, /7th NIPS Conference, Denver, Colorado, USA, 1993), pages 335-342. Morgan Kaufmann, 1993.
[28] Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport. In International Conference on Artificial Intelligence and Statistics, pages 1863-3861. PMLR, 2024.
[29] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint DxXiv:/8033.3666 2018.
[30] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Leurning, pages 3259-3269. PMLR, 2020.
[31] C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimiza- tion. arXiv preprint arXiv: 16/ 1.01540, 2016.
[32] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, October 2018. arXiv:1802. 10026 [cs, stat].
12

---

[33] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances neural information processing systems, 31, 2018.
[34] Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint
2014.
[35] Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Using mode connectivity for loss landscape analysis. arXiv preprint 1xxx::8806009777 2018.
[36] Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit Sinkhorn differentiation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20237- 20246, Vancouver, BC, Canada, June 2023. IEEE
[37] Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-135. Elsevier, 1990.
[38] Marcus Hutter. The human knowledge compression contest. URL http:/prrie... hutterl. net, 6, 2012.
[39] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407. 2018.
[40] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991.
[41] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[42] Roy Jonker and Ton Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. In DGOR/NSOR: Papers of the 16th Annual Meeting of DGOR in Cooperation with NSOR/Vorträge der 16. Jahrestagung der DGOR zusammen mit der NSOR, pages 622-622. Springer, 1988.
[43] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[44] Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. arXiv preprint arXiv:2205. /2411, 2022.
[45] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 17609004336 2016.
[46] Boris Knyazev, Abhinav Moudgil, Guillaume Lajoie, Eugene Belilovsky, and Simon Lacoste- Julien. Accelcrating training with neuron interaction and nowcasting networks. arXiv preprint arXiv:2409.0444 2024.
[47] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny i200).. 2009.
[48] Rohith Kuditipudi, Xiang Wang. Holden Lee, Yi Zhang. Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32, 2019.
[49] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[50] Vera Kurkova and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3):543-558, 1994.
13

---

[51] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))227-225 1998.
[52] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Wang, Yonghui Chen, et al. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[53] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[54] Aixin Liu, Bei Feng. Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-expert language model. arXiv preprint arXiv:2405.0445: 2024.
[55] Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965-23004. PMLR, 2023.
[56] James Lucas, Juhan Bae, Michael R Zhang, Stanislav Fort, Richard Zemel, and Roger Grosse. Analyzing monotonic linear interpolation in neural network loss landscapes. arXiv preprint arXiv2214.110444 2021.
[57] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142- 150, 2011.
[58] Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-332 1993.
[59] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 011..66699007333 2016.
[60] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of parametrizzaoo in generalization of neural networks. arXiv preprint 1rXiiv1800: 1006, 2018
[61] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
[62] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08766 2018.
[63] Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning, pages 17759-17781. PMLR, 2022.
[64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[65] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gal- linari, and Matthieu Cord. Diverse weight averaging for out-of-distributiio generalization. Advances in Neural Information Processing Systems, 35: 15:11081-1088 2022.
[66] Walter Rudin. Real and Complex Analysis. McGraw-Hill, New York, 3rd edition, 1987.
[67] Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04444 2017.
[68] Ekansh Sharma, Devin Kwok, Tom Denton, Daniel M Roy, David Rolnick, and Gintare Karolina Dziugaite. Simultaneous linear connectivity of neural networks modulo permutation. arXiv preprint 2xXiv:2404.064900 2024.
14

---

[69] Noam Shazeer, Azalia Mirhoseini. Kelsey Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gand gated mixture-of-experts layer. arXiv preprint arXiv: 7701006588 2017.
[70] Alexander Shevchenko and Marco Mondelli. Landscape connectivity and dropout stability of sgd solutions for over- parammeteereeed neural networks. In International Conference on Machine Learning, pages 8773-8784. PMLR, 2020.
[71] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
[72] Sidak Pal Singh and Martin Jaggi. Model Fusion via Optimal Transport, February 2021. arXiv:1910.05653 [es, stat].
[73] Ankit Sonthalia, Alexander Rubinstein, Ehsan Abbasnejad, and Scong Joon Oh. Do deep neural network solutions form a star domain? arXiv preprint arXiv::403.07968, 2024.
[74] Elias M. Stein and Rami Shakarchi. Complex Analysis, volume 2 of Princeton Lectures in Analysis. Princeton University Press, Princeton, NJ, 2003.
[75] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers, arXiv preprint arXiv:2106. 2021.
[76] Norman Tatro, Pin-Yu Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Op- timizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 13:15300- 2020.
[77] Hoang Tran, Thieu Vo, Tho Huu, An Nguyen The, and Tan Nguyen. Monomial matrix group equivariant neural functional networks. In Advances in Neural Information Processing Systems, volume 37, pages 48628-48665. Curran Associates, Inc., 2024.
[78] Hoang Tran, Thanh Chu, Minh-Khoi Nguyen-Nhat, Huyen Trang Pham, Tam Le, and Tan Minh Nguyen. Spherical tree-sliced Wasserstein distance. In The Thirteenth International Conference on Learning Representations, 2025.
[79] Hoang V. Tran, Minh-Khun Nguyen-Nhat, Huyen Trang Pham, Thanh Chu, Tam Le, and Tan Minh Nguyen. Distance-basee based tree-sliced Wasserstein distance. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
[80] Hoang V. Tran, Huyen Trang Pham, Tho Tran Huu, Minh-Khoi Nguyen-Nhat, Thanh Chu, Tam Le, and Tan Minh Nguyen. Tree-sliced Wasserstein distance: A geometric perspective. In Forty-second International Conference on Machine Learning. 2025.
[81] Hoang- Viet Tran, Thieu N Vo, Tho Tran Huu, and Tan Minh Nguyen. A clifford algebraic ap- proach to e (n)-equivariant high-order graph neural networks. arXiv preprint arXiv:2410.04692. 2024.
[82] Thanh Tran, Hoang V. Tran, Thanh Chu, Huyen Trang Pham, Laurent Ghaoui, Tam Le, and Tan Minh Nguyen. Tree-sliced Wasserstein distance with nonlinear projection. In Forty-second International Conference on Machine Learning, 2025.
[83] Viet-Hoang Tran, Thieu N Vo, An Nguyen The, Tho Tran Huu, Minh- Khoi Nguyen- -haa, Thanh Tran, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. arXiv preprint 24 042090 2024.
[84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems 30, pages 1998-6008 2017.
[85] Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, 20:133. 2019.
[86] C. Villani. Optimal Transport: Old and New, volume 338. Springer Science & Business Media, 2008.
15

---

[87] Tiffany Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? In International Conference on Machine Learning. pages 22325-22344 PMLR, 2022.
[88] Thieu N Vo, Viet-Hoang Tran, Tho Tran Huu, An Nguyen The, Thanh Tran, Minh-Khoi Nguyen- Nhat, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant polynomial functional networks. arXiv preprint arXhv:2410.04213, 2024.
[89] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Lontijo-Lopes, Ari S Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-23998. PMLR, 2022.
[90] Tim Xiao, Weiyang Liu, and Robert Bamler. compact representation for bayesian neural networks by removing permutation symmetry. arXiv preprint (tti::24001006111, 2023.
[91] David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
[92] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[93] Pu Zhao, Pin-Yu Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridg- ing mode connectivity in loss landscapes and adversarial robustness, arXiv preprint arXiv:2005.00060, 2020.
[94] Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity, arXiv preprint 2rxiv:330..08286, 2023.
16

---

Table of Notation
Ra
d-dimensional Euclidean space
gd-1
(d 1)-dimensional hypersphere
Euclidean norm
Frobenius norm
Expert function
D
Mixture-of-Experts with Dense gating
s
ofiirertss-- with Sparse gating The Top-k map
Top-k(-)
0,0'
parameter of Expert functions
weight space of Expert functions
0,
parameter of Mixture-of Mfxture-of-Expprtts weight space of Mixture-of----Exeerts permutation group order n element of permutation group elements of additive group of Euclidean space group act weight space of Mixture-of-n-Experts sets with specific purposes
(n)
Sn
ow
(n)
0, 92
B
barrier loss
C, Ci
cost matrix and its entries
A, U
weight and bias of Expert functions permutation matrix
P
boundary of set topological space rectifier activation function
o, ReLU
H
space of holomorphic functions space of meromorphic functions polynomial ring in complex variables field of rational functions polynomial
F
Clz), Ea.
C(x), C(T),,
Pi
17

---

Appendix of "On Linear Mode Connectivity of Mixture-of-Experts Architectures"
Table of Contents
On the Weight Spaces of Mixture-of-Experts Architecture
19
A.1
Weight Space of Mixture-of- of-peerss
19
A.2 Group Action Weight Spaces of Mfxture-of-Experts
20
Results on Mixture-of-Experts with Dense Gating
22
B.1 A Structural Property of Neural Networks with ReLU Activation
22
B.2 A Technical Lemma on Holomorphic Functions in C
22
B.3 Functional Equivalence in Mixture-of-Experts with Dense Gating
C Results Mixture-of- with Sparse Gating
29
C.1 Strongly Distinctness Property
29
C.2 Functional Equivalence in Eexture-of-Expett with Sparse Gating
30
D Technical Details for Sections 5 and
34
D.1 Proof for the Sufficiency of Permutation Invariance LMC of MoE ................................
34
D.2 Proof of the Cermutation-Invariinn Property for Equation (11).
35
D.3 Formal formulation of DeepSeckMoE
36
Impact of Feedforward Reinitialization on Pretrained Transformer Performance
36
F Experimental Details and Hyperparameters
39
Experimental Results
40
G.1 Verification of Linear Mode Connectivity across diverse configurations.
40
G.1.1 Dense Mixture-of-Experts
42
G.1.2 Sparse Mixture-of-Experts
52
G.1.3 DeepSeek Mixture-of-Experts
59
G.2 Linear Mode Connectivity Analysis: Last Layer
66
G.3 Linear Mode Connectivity Analysis: All Layer
73
G.4 Expert Matching Method
73
G.5 Ablation Study Number of Layers
76
H Broader Impact
79
18

---

A
On the Weight Spaces of Mixture-of-Experts Architecture
Denote the input token dimension as
A.1 Weight Space of Mixture-of-Expeee -fpprrsss
Expert functions. We study expert functions & Rd R4, which are realized by standard feedfor- ward neural networks employing the ReLU activation function. Specifically, each expert map {:; is represented as composition of affine transformations and ReLU nonlincarities:
{x; 0) fL 0o0 fL-1 0
1...00..0000 000 fi(x),
(12)
where each affine layer is defined by
f((x) xW; b,, for i 1,........ L.
(13)
Here, a denotes the ReLU activation function, applied component-wiset -wise, and 0 = [, b: represents the full set of learnable parameters. The weight matrix W; E -m and bias vector b, RT parameterize the i-th layer, where no = nL respectively. We have
the input and output dimensions,
L II - Xn RT)) =R' where (n-1174 n2).
0
(14)
Define the weight space of expert functions
L II
+
RR": 1884 R*:) R
(15)
Dense Mixture-of- Expertss Given positive integer n representing the number of experts, Mixture-of- Experts with Dense gating (MoE) is defined as the function: D : R - 12d defined by
n
D (x; (W, 0.
softmax, (WW, + b.)%))) (x;
(16)
The function D is parameterized as D(x; Φ) where
Φ Φ= ((W,, b.), i=l...
((R2 x R x ))""
(17)
Denote the weight space of a Mixture of n Experts
4(n) ( R)) x x))"
(18)
Varying the number of experts leads Mixture-of-Experte weight space that spans across expert sets different sizes, denoted by
(n) Li ((XD 1R) x x
(19)
Sparse Mixture-of-Experte Given positive integer A: S n, the Top-k map is defined by: for any vector = = (2,,..., Fn) E R",
Top-k(z) (i fi.........) in).
(20)
where ig.... ... is are the indices corresponding to the ie largest components of x. In the event of ties, we select smaller indices first. Using this, a Mixture-ort of Experts with Sparse gating (SMoE) is the function S: Rd - Rd defined by
S (x; (W, b, 0 0,)))))=
Σ (E)(2)
softmax;
(WWr bilser(e)) E (x;
(21)
19

---

where
T(x) - (T; (W, ( Top-k (W + :-))
(22)
The weight space for the Sparse Mixture-of-Experts is identical to that of the Dense Mixture-of- Experts, as the inclusion the Top-k operator does not introduce any additional trainable parameters. However, the SMoE function S is generally discontinuous, primarily due to the non-smooth nature of the Top-k selection selection-particulann particularly in scenarios where multiple experts share identical gating scores (i.e., ties occur). To circumvent this issue and ensure well-defined behavior, we restrict our analysis to subset of the input space Rdd where the top K gating scores are uniquely determined without
ambiguity. In particular, for (W; by ) (R4 x R)", we define the subset of Rd such that the gating scores all experts are pairwise distinct, i.c.,
Ω ((W,, 1 = {x E R (W,r + b.) are pairwise distinet).
(23)
We provide result characterizing the structure of this domain, and another result describing the behavior of the SMoE map when it is restricted to inputs within this domain. Proposition A.1. If [Wa, b.} are pairwise distinct for i = I,...,,, n, then 5 (WW,, bi) an open and dense subset of Rd
Proof. We have
Ω ((W,,,
( Wiz + b, is pairwise distinet for all i 1, I.....n] n
N
War + by + W5x bj} f
Isisisn
N MREE" War + b. W5x+
(24)
1sK4jsn
Note that, the set
E Wi + = +b) ( (W; W3) b; b.).
(25)
is either hyperplane (when W; # Wr) or empty (when W; = W, but 6. # by)). In both scenarios,
its complement in IRd forms an open and dense subset. By Equation (24), and using the fact that a finite intersection of open and dense subsets in IRd remains open and dense, it follows that the set Ω (W,, 1 is itself open and dense in Rdd
O
Proposition A.2. If (W., b.) are pairwise distinct for i == I....,n, n, then the SMoE map S, as given in Equation (21), is continuous over the domain Q. (,,,,,,,;) 6.))=)).
Proof. Let x E Ω (1W,, ((W,,,,, 1)
By the definition of this set, there exists an open neighborhood U C Rd containing a such that C 52 (( and
Top-k ((W, + b)) Top-k ((Way + b:)
(26)
for all U. This condition ensures that the sparse gating mechanism defined in Equa- tion (21) remains fixed within U, and consequently, the SMoE map is continuous on the domain 2 ((W,, 1)
Propositions A.1 and A.2 serve as essential building blocks in the proof of Theorem C.4.
A.2 Group Action on Weight Spaces of Mixture-of-Expert.
We define the group G = G(n) as
G(n)
(T& x RR S..
(27)
which the direct product of the additive groups Rd and R. and the symmetric group S, on elements. Each element E G(n) can be written in the form
g (cw, Cb. T), where cw E Idd, Co R, and E Sn-
(28)
20

---

The group G(n) acts on the weight space ((n) as follows: for g = (ow, 7) E G(n) and ø E (()) given in Equation (17), the action is defined by
g (Wr(t) + Cw, br()) cb. (r())) i=I...m (()).
(29)
This group action preserves the functionality of both the MoE and SMoE maps. The invariance arises from two key properties: the permutation invariance of the summation operator, and the translation invariance of the softmax function. We begin by establishing a result that characterizes the invariance of MoE maps under this group action.
Proposition A.3 (Weight space invariance of offturr-of-Epperts) The MoE map D is G(n)- invariance under the action of (()) on its weight space, i.e.
D(: Φ) = -D(:g
(30)
Proof. Given = (ow, Cb, -) E G(n). For all æ E Rd, we have
D(x:g0)=
softmax; ((( + cw) x (br(0) + +c)) E (2;
Σ softmax; [ b. b.())) )=)) (2))+(()
Σ
softmax; ((W,r + b;)))))) (x;
1
= D(;;0) 0).
(31)
Thus, the proposition is proven.
The analysis of the SMoE architecture requires additional assumptions due to the inherent discontinu- ity of the Top-k selection operator. In what follows, we show that the SMoE map, when restricted to an appropriate subset of its domain, remains invariant under the group action of G(n).
Proposition A.4 (Weight space invariance of Sparse defined in Equation (21), assume that the pairs {W,, bj} are pairwise distinct for = I.... n. Then, the set Ω ((W, be 1 is invariant under the group action of G(n), i.e.
offuure-of-Experte
Given the SMoE map
((W, b.) Ω (g (W 11)
(32)
Moreover, the SMoE map, restricted to
Ω ((W;,
(33)
is G(n))invariance under the action of G(n) on its weight space, i.e.,
S(::0) = (:: on Ω ((W,, bibit
(34)
Proof. Let (ow, Co r) E G(n). We first verify that the group action preserves the set (HW,, b.) Indeed,
22 b(WW,, = (I Rd Wia + b, is pairwise distinct for all i = I.....n}
=(x Id: (W-() + +ww x + (b-(0) + co) is pairwise distinct for all i 1,..., n}
= (g b.)...
(35)
Let
gT(x) Top- ((Wr() +
(br() + cs) )m}
cw) x +
(36)
For all E ((W, by ) we have gT(r) = (T(())) The desired invariance result for the SMoE map then follows by applying the same reasoning in Proposition A.3,
21

---

Remark A.5. While the group action on Mixture- of ofoture-of-Expetts (MoE) models is formally introduced in Equation (22), does not fully capture all symmetries inherent these architectures. In particular, each expert network possesses internal neuron permutations that preserve its functional behavior-a well-documented property in the literature on neural networks [7, 23, 29, 9, 60, 11, 62, 13, 77, 81, 88, 37, 15, 27, 50, 5, 6]. Nonetheless, since the distinguishing feature of MoE models is their input-dependent gating function, our focus centers on the symmetries associated with the gate itself, treating expert-level invariances as part of a well-established theoretical foundation.
B Results on Mixture-of-Exper Experts with Dense Gating
B.1 A Structural Property of Neural Networks with ReLU Activation
polytope is a geometric entity bounded by flat surfaces, which can be either finite (bounded) infinite (unbounded) in extent. We define the notion of local affineness follows: function ; IRd - RD is said to be locally affine if there exists partition of 1d into a fimite set of polytopes such that on each polytope, the function f agrees with an affine transformation from gd to R Remark B.1. While the term local affineness may take on different interpretations in other contexts, its usage here is unambiguous within the scope of this work.
We examine the local affineness property in ReLU neural networks. Consider a feedforward neural network f R" - R. constructed from affine maps interleaved with ReLU activations, given by
/ = Jt. o a o fT-1 ..... o ft, where each map fi Rdd-1 - Rd, is affine and has the form fi(x) W, + 6;, and denotes the ReLU function applied elementwise.
The combination of these affine transformations and ReLU activations partitions the domain Rd into finitely many convex polytopes. On each such region, the activation status the ReLU whether they transmit their input or suppress it to zero-remains fixed. Consequently, the network reduces to simpler form in which each ReLU behaves either as the identity or the zero function. Given that both ReLU and affine maps are piecewise linear and closed under composition, the overall function is affine within each activation region.
Hence, the network satisfies the following property:
[x) = A&r + b;, for all x E P;
(37)
where the domain is partitioned into polytopes {P.)m and A,, define the affine transformation active within region P..
Let OP; denote the boundary of the polytope P.. Then the set
m
de
(38)
is open and dense in Rd;;; in other words, the union of the interiors of the polytopes covers a set that is both open and dense.
Now consider finite collection of ReLU networks { Since the intersection of finitely many open dense subsets is itself open and dense, there exists subset Z C Rgdo such that, for every point x E Ω there a neighborhood around x on which all functions (M) act as affine maps.
B.2 Technical Lemma on Holomorphic Functions in C
A function f: C + C is said to be holomorphic on C if it is complex differentiable at every point in C". A function is called meromorphic on C" if can be locally written as quotient of two holomorphic functions, where the denominator is not identically zero. The collection of all holomorphic functions on C forms an integral domain, denoted by H. while the set of meromorphic functions on C forms field, denoted by F. In particular, F is the field of fractions of the integral domain H. Let C[x) C(z),..., x..] denote the polynomial ring in n complex variables, and let C(z) C(ri, ... ...
) denote the corresponding field of rational functions. Then C(r) is subring of H and remains an integral domain, while C(x) is subfield of and represents the field of fractions of C[z].
22

---

Remark B.2. For any polynomial p E C|2)], the exponential e" defines holomorphic function on ol E D.
Since C(z) C F, we can view F as vector space over C(z). The following lemma addresses the linear independence exponential functions polynomials this vector space setting.
Lemma B.3. Let Pi.....PP PN be polynomials in C[a] such that p; Pp nonconstant whenever i + j- Then the functions e ePN, viewed as elements of F, are linearly independent over C(r).
Proof. We proceed by induction N. The base case N 1 is immediate, since e² * 0 for any polynomial p. Assume the result holds for any smaller number of exponentials. Consider polynomials IN C[æ]] such that
T1 e +...+r eMM = 0.
(39)
We aim to prove that all r; O. Suppose not; then least one r; nonzero. Without loss of generality, assume IN # 0. Dividing through by rN ePN gives
r1 e PN
TN-1 PN-1 PPN---P PN
+1=0.
(40)
TN
Differentiating both sides with respect to x; for each i I,..., n,, we obtain:
N
Fj
+
Tj
(pj pN) -P:-PN = 0.
(41)
Ox
TN
TN
dz
Note that each term
Tj
Tj 0
+
(p; PN)
(42)
TN
TN
dz,
belongs to C(+). Now, the polynomials p1 - PPN....... .P...... PN-1 PN are pairwise distinct and nonconstant. Thus, by the induction hypothesis, the exponentials ePi-PN are linearly independent over C(x) for
j ===, 1,...., Therefore, from equation (41), we must have
i
Yj
r
(P) PN) = 0,
(43)
Oz,
TN
TN
Ox;
for all = 1,,,,.... 1,..., and = I,.... N- 1, which implies
à [j e PN = O.
(44)
Dx; INN
Hence, for each j 1....,N N 1, the function
r2
EPs PP = EC
(45)
FN
must be constant. If oj * O. then both * 0 and cP: PP Spr must hold. But this forces PP PN rj
to be constant, which contradicts the assumption that Py PN is nonconstant. Thus, = 0, which implies ry for all .,..., NV- 1 # 0. Therefore, all r, = O, completing the proof.
1,
Plugging back into equation (40) gives contradiction, as
Remark B.4. This lemma plays a key role and will be applied repeatedly the proofs of Theorem B.5 and Theorem C.4.
B.3 Functional Equivalence in Mixture-of-Expere with Dense Gating
The following result establishes the equivalence between two sets of weights that define the same MoE map. Certain assumptions are introduced for technical reasons, and their justification is provided in Remark B.6.
Theorem B.5 (Functional equivalence in Mixture-of-Experts with Dense Gating). Suppose d of define the same MoE function, i.e. o(=;)) = Ø') Assume that the following conditions hold:
1. Both {E(;; (.); and (E(:: ya' consist of pairwise distinct functions;
23

---

2. Both {W, - WWghsisssn and { - Wjhsisjen consist of pairwise distinct vector of IR..
Then n', and there exists 9 (cw, cb. T) G(n) such that for all i = ,,...,n,, n, have W = W x() + ow, b/ br() + cb, and [-;;,, (.) = E(; (r()) on R
Proof. To enhance clarity, we begin by outlining the main steps the proof at a high level:
1. We first expand the equality ((: Ø) = (+: o²) and introduce simplified notation to stream- line the subsequent derivations.
2. We then observe that each expert function can be locally characterized an affine map.
3. Next, we prove that n' and establish the existence permutation 7 and transforma- tion ow with the required properties.
4. We proceed to verify the equivalence between the two sets of experts.
5. Finally, we show that a transformation es satisfying the necessary conditions can be con- structed.
We now proceed with the detailed derivations and justifications corresponding to each of the five steps above.
Step 1. Given that ():;)) = =D(:;') we have
Σ
softmax; ((Wor b 0,) Σ softmax; (W + M1 8 (1;0)),
(46)
for all x E Rdd Define
E(() = (=;0,) and =(:)();)))
(47)
By expanding the softmax terms in Equation (46), we obtain
Σ==ı eW Exx) = e Wio+K
Wia+b.
Σ
i1 e W3 (z).
(48)
Multiplying both sides by the respective denominators yields
n'
Σ Woa+h E((O))
e W22thy
Σ
Wa+ M. {.((())
e((1))
(49)
which can be rewritten as
(W++W))+(bi+tt) - (((())-((())) =
(50)
Since each function E, and E/ is locally affine, it follows from the result in Appendix B. that there exists dense open subset Ω c Rd such that: for any point E Ω one can find open neighborhood U C Ω containing a on which all functions E, and E are affine. Consequently, each of these functions agrees with a polynomial on U. That is, there exists a family of open sets {U& K covering Ω so that
Step 2.
Ω Ω= U
(51)
ket
and for each set U Up in this cover, there exist polynomials pu,s, Po,,,ee, E [R[x]] such that
(x) (U)())), and E((x) = (x) for all a E U.
(52)
24

---

Substituting into Equation (50) yields:
(W++3)=++(u+b) (pu,(()) - 0 for all U.
(53)
Observe that the expression on the left-hand side above defines holomorphic function. By the Identity Theorem for Holomorphic Functions (see [1, 66, 16, 74]), conclude that:
n
(We+W))a(((+b) (pu,(()) - (x)) = 0 for a E CCd.
(54)
According Assumption 2, the sets (Wi and (Wz) composed of mutually distinct elements. As a result, there exists direction
Step 3.
a E sd-1 = for E Rd ||l|2 = 1},
(55)
such that the projected values {W j= and (Wja}jz are comprised of n and n' distinct real numbers, respectively. Without loss of generality, we may relabel the indices so that:
Wia < W2a .... < a and Wia < Wiaa .... < Wa
(56)
Furthermore, observe that the problem setting and the preceding equations are invariant under translations of the form W + W + ow for a constant vector cw E R Hence, we can assume without loss of generality that W1 = W1. Under this assumption, we aim demonstrate that n n' and W; W for all i 1.... ... Toward that goal, we begin by showing that W; W/J for each i I..... min(n, n, n'} using mathematical induction.
Base case. By assumption, we have W1 Wi, so the base case holds trivially.
Auxiliary step for induction. For every index pair (i.j) * (1, have the inequality:
W1a + Wia < Wa Wja.
(57)
Thus, the sum W1 W1 differs from W. + Wj for all (i,j) + (1, Applying Equation (54) in conjunction with Lemma B.3, we conclude that
P P0.1
(58)
Inductive step. Assume that W; W/ holds for all < k, where is integer such that 1 < k < min n, n') Suppose, for contradiction, that W, * W/. We analyze the terms Wi + W and Wa W1. Under our assumption, these quantities must differ. Without loss of generality, we may assume that
W1a + Wia S Waa + W/
(59)
For all index pairs (i.j) with i 2 k, we have
Wia + ai S W& + Wia & W. + Wjo.
(60)
Equality holds if and only if (i. = (k, 1). Moreover, since W1 + W and W + W/ are distinct, it follows that W1 + W differs from W. + W/ for all (i.j) with i k.
For all (i.j) with j 2 k we have
Wia - Wia S Wa + Wja.
(61)
Equality occurs only when (i.j) = k1. Therefore, W1 + W/. is distinct from W; + W for all (1 * (1, with j 2 k.
For all (i.j) such that i,j < k. we claim that W, + W/ does not equal W, + Wy. Suppose, for contradiction, that
W1 + W W + w
(62)
25

---

for some pair (i.j) with j,j < k. Then by the induction hypothesis,
W1 + W = W + W
(63)
Rearranging terms, obtain
W1 W W W%
(64)
which contradicts the assumption that all such differences are pairwise distinct, since (1,5) * (i,
These observations collectively show that W1 + W% differs from W; + Wj for all (i. + (1, k). Applying Equation (54) together with Lemma B.3, we conclude that
PV,1 = P
(65)
Additionally, from Equation (58), already have
p P
(66)
This implies that Ej = Eg on U. Since this holds for every open set U in the covering {U* it follows that & = & on Ω. Because Ω is dense in R and each Ej is continuous, we conclude that = on all of Rd. This contradicts the assumption that the E functions pairwise distinct. Therefore, initial assumption must be false, and conclude that W1 + W W W7, which implies W₂ = W.
Conclusion. By induction, we have established that W; = W for all i= 1,... mn, J. It remains to show that n = n'. Suppose, for contradiction, that n < n'. Consider the sum W1 W" We claim that this quantity is distinct from every W, + Wj with (i.5) & (1,n'.. Assume otherwise, that
W1 + W = W + W for some pair (i.5) (1,n'). Using the inductive assumption that W; W/ for all i S n, deduce
(67)
W/ + W W + Wj,
(68)
which implies
W Wj W W
(69)
This leads to a contradiction, as it violates the assumption that the differences W - W are pairwise distinct. Hence, W1 + W", must be distinct from all other sums W; + W3 where (i.j) + (1, Then, by Equation (54) and Lemma B.3, it follows that
L PU
(70)
In addition, from Equation (58), we already know that
P = PP
(71)
Consequently, obtain & E, on U. Since this holds every open set U in the covering {Ux hiee it extends to Ω and by continuity, all of Id. This contradicts the assumption that the expert functions E/ are pairwise distinct. Therefore, our assumption must be false, and we conclude that n'. Finally, the reordering of indices and the translation applied to the set (=1 throughout the argument confirm the existence of a permutation 7 E S,, and a translation vector cw E Rdd
Step 4. We now establish that E = El on Rd for each = I,..., From Step 3, we already have that n' and W, Wi for all indices in this range. Consider any pair (f.j). If it holds that W, + W = W + Wy. then (f',j)) must be either (f,j) or (j.i). In particular, this implies that W; + W is distinct from W, + W/ for all (j,b) + (i, i). Invoking Equation (54) together with
Lemma B.3, we obtain
PU. Pu..
(72)
This argument parallels that used in Step 3, and applying the same reasoning, we conclude that ε E on Rdd Since this holds for all i = 1,... n, the result follows.
26

---

Step 5. It remains to prove the existence of a constant Co R such that
b/ b; + cb for all n. Recall from Step 4 that if W; + W = We + W',, then it must be that (V' = (i,j) or (j, 2). Using this structural property, along with Equation (50), Lemma B.3, and the identity & Eg established Step 4, we derive the following equality:
(73)
(W++W))++(++t))) (E,(1) Ej(z) + e (W),w:) (E)(2) & (z)) 0,
(74)
valid for all index pairs (i,j). Since & + & whenever + there exists point To E for which * (x0)
(2o). Plugging x = zy into Equation (74) and simplifying by factoring out the common nonzero difference, we find:
bitt', =
(75)
which implies
b. + 1/ = by + 1/4
(76)
and hence
6; 6/ bj b).
(77)
This shows that the difference b; b/ remains constant for all i. Letting == 1/4 - -b, we obtain
R b; + Gb for all ,,
(78)
This completes the proof of Theorem B.5.
Remark B.6 (Rationale behind the assumptions in Theorem B.5). In modeling architectures, it is important that the symmetry group arises from the structure of the model itself rather than from specific, possibly degenerate, parameter choices. That is, the symmetry group should act globally and consistently across the entire weight space. This requirement motivates the following conditions in Theorem B.5:
1. Both (E(;0,)) and (E(50)) consist of pairwise distinct functions;
2. Both (W; Wih and {W? - Wihisseee consist of pairwise distinct vector of IRd.
We now elaborate on the purpose of these assumptions.
Assumption I. If this condition is violated-for instance, if two experts compute the same function and are assigned identical gating values- then permuting these experts leaves the model output unchanged. Such permutations introduce additional, non-essential elements into the symmetry group, which we refer as spurious symmetries. These do not correspond to genuine structural invariances but rather arise from degenerate parameter settings that represent singular points in the model's parameter space.
Assumption 2. Assumption 2
addresses a more nuanced issue: it rules out cases where linear dependencies among gating weight vectors may cause multiple experts exhibit indistinguishable gating behavior. Although such situations may not be as immediately intuitive as those excluded by Assumption they too can artificially enlarge the symmetry group beyond its intended form. To illustrate this, we present an explicit example. Let d = 1 and n = n' = 3, and consider parameter configurations é and ø' such that:
W1 W 1, W2 = W = O. and W3 = W/ =
The parameters 01. 02 03 04, 062, 0/2 are chosen so that all six experts are constant functions.
Let define:
E(:01) A1, E(:0) = A2
E(;02) B1, 8(:0%) = B21
(79)
E(10g) CC,, E(:0%) = C2.
27

---

We now select the biases b,, by and constants A,. By. CT such that the models satisfy D(=:0) Di-: d'), even though there exists no transformation described in Theorem B.5 that maps ø d'. Our goal is to ensure that
e
e
A1 + e- + t + e
B1 +
C1
e e
+ th + er
e e + chz + t
2
e
Ag + e--th + e + ez
B2 + e- + c + cx
C2. (80)
e e + 2 + c
For convenience, we introduce the following shorthand:
en X1,
e¹ X2.
e Yi,
e Y2,
(81)
e Zı,
c² Zz
With this notation, Equation (80) becomes:
e
Y1
1" e +Y + et Z
A + e + +Y + t
B
Ci
e - +Y
8X2
Y2
e eL22
A2 e=xX² +
B2
C22, (82)
- + 12 + 6 Zz
e- + Y2 + e0 22
which is equivalent to:
X + B + e" Z1C()
= ( *X2A2 + Y2B2 + e" Z2C2) (e "X1" + Y1 + +
(83)
By equating the coefficients of 22 e²², we derive the following system:
2x
X1X2A1
X1X2A2,
2x
Z,Z2C1
= Z1Z202,
Y2Z2B, + 2Y201
Y17200 + Y2222
(84)
e-
YiXBBı + +XY2A
= Y1X2 + X1Y2B2,
X,724 + 4x201 + 1
X1Z2C2 + Xx + Y2Y2B2.
Setting A A2 = A and Ci = C2 = C satisfies the equations involving e -22 and 20 automati- cally. Removing those, we simplify the system to:
Y2 + ZY.
Y1ZCC + B22
6- Y, + 4
+XX24+X
(85)
1 X,Z24 + Z12.C Y1YB1
X,Z.C + ZxX2A+ Y2222..
Solving the equations for A and C, assuming Z,Y2 + Y12, and X,Y2 # gives:
gives:
4 XjY,B2 YiX3Bi
X,Y, - N.Y,
(86)
C ZY2B2 Y2
2iY2 Y1Z2
Substituting into the constant term equation in (85) gives:
Y2 - B2) = =C 2
(87)
Now computing A - C:
A - = X1Y8B, - Y18,B1
ZYYB, Y1ZBB
X1Y - -YX,
Z1Y2 Yı22
28

---

- (2) 2112)
(88)
- V K122)
Plugging into Equation (87), we find:
(B - B2) Y2 ( - (X1 Z1
(89)
(X1V² - xxx2) 12222
Assuming B1 # B2 and Y1 # O, divide both sides to obtain:
(X, - 12x2) - 21 = - -2
(90)
While this equation be solved in general, it suffices exhibit concrete solution. Consider:
(x1,
(1,
(Y,, Y2)
(3, 5)
(91)
(Zi, 22)
(2,3).
With these values, B1 and B2 may be freely chosen. These assignments define parameter configura- tions @ and o' for which (): ((; d'), yet no transformation described Theorem B.5 maps to d'.
C Results on Mixture-of-Experts with Sparse Gating
C.1 Strongly Distinctness Property
The following definition formalizes the notion of strong distinctness, which will be used in the statement of Theorem C.4.
Definition C.1 (Strongly distinct). Two functions f,g: X - Y are said to be strongly distinct if the set far E X f(x) & g())} is dense in X.
Example C.2. We present several examples illustrate the concept of strong distinctness.
Two distinct polynomials on R. or C are strongly distinct.
Two distinct holomorphic functions are strongly distinct.
In contrast, two distinct locally affine functions are not necessarily strongly distinct. For instance:
Let S2, R R be defined
0 if < <0
[1(x)
f2(x) =
(92)
x
if x
Then fi and fa are strongly distinct.
Let g1, :2: R be defined as:
if < 0, if x 2 0,
91
(x)))=
92(x) = 0.
(93)
I
Here, 91 and g2 are distinct but not strongly distinct, since they coincide on (-oc,)) 0).
We now define a certain class of subsets of Rd. Given
(W b.). E (IR x R)),
(94)
we define the set
Ω ((W,, == {x : ( b, bi are pairwise distinct).
(95)
The following result provides a sufficient condition on the gating weights under which the Top-k
operator is capable of selecting any subset of & experts.
29

---

Proposition C.3. Suppose that the collection {Wi).= satisfies the condition that the set {W(G,i1)-
W Jiz2 is linearly independent in R Then, for every subset A S (1....,n] of size k, there exists a point x E Ω (( b=)==)) such that
Top-k ((W,x + bb)) A.
(96)
Proof. Without loss of generality, assume A = .1,..., ,i:.. To prove the existence of x E Ω such that
Top-k ((W, bi) (1, ;l....,hh kk)
(97)
suffices to find x Idd such that
Wix bi > W2x b2 >...> Wnx + bn.
(98)
We can strengthen this require:
(Wix + b1) (W2x + b2)
1,
(Wax + b2) - (Wyr + by)
1,
(99)
(Wn-10 + bn-1) - (W,, b..) 1.
This is equivalent to the following system of linear equations:
(W, W2)x
1 (b) - -2),
(W2 - Ws)a
= 1 (b2 - -bs).
(100)
(W Wn)x = 1- (b,-1 bn).
Since the vectors {W;-1- W are linearly independent by assumption, this system has unique solution. Hence, there exists Id satisfying Equation (100).
Proposition C.3 will be invoked in the proof of Theorem C.4. A justification for the linear indepen- dence assumption is provided in Remark C.8.
C.2 Functional Equivalence in Mixture-of-Experts with Sparse Gating
We establish functional equivalence theorem for the Sparse Mixture-of-Eaie off-xxs (SMoE) architecture, in parallel with the result for MoE given in Theorem B.5. However, our analysis is limited to the case 1, as the k 1 setting leads singularities that disrupt the general structure of equivalence. A detailed explanation for this exclusion is provided in Remark C.9.
Theorem C.4 (Functional equivalence in Mixture-of-Exppeee
with Sparse Gating). Suppose of define the same SMoE function, i.e. S(;; Φ) = S(;; d'). Assume that the following conditions hold:
1. Both (E); and {E(:; )'' consist of pairwise strongly distinct functionn;
Both [W,-1- - Wi and { - Wi are linear independent subsets of IRd.
Then n n', and there exists = (ew, (6.
$0. 7) G(n) such that for all i 1,.... n, we have W! W Wr() cw, by b-(1) cb and E(x; 0%) E(x; 0-(1)) for all z E RQ(,, b.), such that T(1) E Top=k((W,, b,)
Before presenting the proof of Theorem C.4, we begin with two remarks.
Remark C.5. Suppose n n' and there exist T Sny OW IRd, and es R such that for every
i i= 1, n1, we have
W W7() + tww, 8/. = br(0) + cb.
(101)
Then the two sets Ω ((W,, b, )R and Ω ((W7 ())))) coincide. Furthermore, for any x in this set, the condition r(i) Top-k((W,z b,)) holds if and only if e Top-k((W/z 6)))))
30

---

Remark C.6. It is easy to verify that Assumption 2 in Theorem C.4 implies Assumption 2 in Theorem B.5.
Proof. To enhance clarity, we begin by outlining the main steps of the proof at a high level:
1. Formulate the identity S(:: 6) S(:: d') explicitly and introduce simplified notation to streamline the presentation.
2. Partition the input space into regions where the Top-k operator selects the same index set, and within which all expert functions are affine.
3. Demonstrate the desired equivalence for fixed subset of experts. The core technique is invoke the MoE equivalence result given in Theorem B.5.
4. Generalize the argument to establish the equivalence across all experts.
We now proceed with the detailed derivation and proof' for each of the outlined steps.
Step
Given that S(; Φ) ((::))) '"), follows that
Σ
softmax, (W + be bihherapp E (x;
TTTop deTop-k(((Wae) (((Wa++b,)) m
Σ
softmax; (W/x + + (ilbeenpp (1; 07),
(102)
sETop (
for x E Rd. To simplify notation, define
{-) = & (+:6), and g(() = (::6)).
(103)
Using this notation, rewrite Equation (102) more compactly
softmax; (WWrr + b beo Ex(x)
(eToopk((Watb)333)
Σ
softmax, (W,/ b() (11 (x)
(104)
Step We begin by highlighting two key observations:
Assumption 2 guarantees that the parameter pairs {W, b,} are pairwise distinct for i 1...., n, and likewise, {W?, 81} are pairwise distinct for 1....,n'. By Proposition A.1, the set
2 ((W, 6.)7 un (1W,
(105)
is open and dense in Rd For any x E Ωn the values {Wer + ba} and (W + K}} are pairwise distinct. Moreover, for each Ωi there exists open neighborhood around in which both Top-k selections remain fixed.
From the analysis in Appendix B. there exists a set Q2 C Rd also open and dense, such that for every x E 92 all expert functions & and Ej are affine in neighborhood of ar.
Taking the intersection Ω = O2 n Ω2, we obtain a set that is still open and dense. Within Q, both the Top-k selections and the expert functions remain locally constant and affine, respectively. Consequently, there exists collection of open sets {Usher that cover Q such that
Ω U NET
(106)
and each U., the expert functions E, and E are affine, and the Top-k selections do not vary.
31

---

Step 3. Let U be an arbitrary open set from the cover described in Equation (106), Without loss of generality, we may relabel the indices such that both Top-k maps are constantly equal {1...., k} throughout U. Under this reindexing, Equation (104) simplifies to
softmax, (1Waz+ + bi)( (x)
softmax, (1W/ + +b))=a)))))))))))) E((x) for all x E U.
(107)
According to Assumption 1, the expert functions &, are strongly distinct, which implies that they remain distinct on the open set U. The same conclusion applies to the functions E.. Therefore, the assumptions of Theorem B.5 are satisfied on U, and Equation (107) falls within its scope. As a consequence, there exist constants ow E Rd and cb IR such that, up to reindexing, we have for all i -.....,k ki
W = Wi + ew, U/ = be + Cb.
(108)
and furthermore, & = E on U.
Step 4. For any index E {3, .,... n the cover in Equation (106) such that both indices 1 and appear in T(1)). Restricting Equation (104) to V1 and applying Theorem C.4, we conclude that there exist indices s t1, 81 S n' such that
we invoke Proposition C.3 to select open set V12 from
W1 W m- W W
(109)
t2, 82 S nt' such that Applying the same reasoning with indices 2 and m, we obtain 1 12,
W2 Wm m W W
(110)
Subtracting Equation (110) from Equation (109) yields
W - W Wi W₂ (W1 W.. (W2 Wm) (W2, W", (W7 - W) (111)
Due to the linear independence guaranteed by Assumption 2, this identity can only hold if = (2 = 2, and 81 = 82. Denoting this shared index by r(m), i.e., (m)) = 81 = 82, we then have
Wi W., W W [mm))
(112)
which implies
W (mm) - Wm = W1 - W1 = CW.
(113)
Similarly, we deduce
b' ('((m) bm = bj - bj = Cb.
(114)
Since m ranges over {3, 4,.., n). the corresponding values of (m) must be distinct. If not, suppose that r(m) T(m') for some m + m', which would imply
W.,
Wm W"(m)
W Wr = O,
(115)
contradicting Assumption 3.
Applying a symmetric argument to the parameters of S(:;0'), we conclude that n = n Therefore, there exists permutation r of (1...., such that
w Wr() + tww, 6/. br(1) + co.
(116)
Finally, the analysis above shows that for any x 2 ((W, )im))) with T(i) E Top-k((W,r + 6,)... -i..., when index i is selected by the Top-k operator in S we have
(x) = (x).
(117)
This completes the proof of Theorem C.4.
32

---

Remark C.7. While Theorem C.4 is conceptually analogous to Theorem B.5, is crucial to recognize that establishing the result for SMoE involves substantially greater technical complexity. The main challenge arises from the Top-k operator, which introduces discontinuities by dynamically changing the subset of active experts a manner that depends intricately on the input. This input-dependent behavior complicates the analysis and makes the theoretical treatment significantly more delicate.
Remark C.8 (Rationale behind the assumptions in Theorem C.4). We begin by restating the two assumptions made in Theorem C.4:
1. Both (=). and { H consist of pairwise strongly distinct functions;
2. Both {W,-1 Wi and {W wi are linear independent subsets of E
These assumptions are strictly stronger than those required in Theorem B.5. We now discuss their necessity and implications greater detail.
Assumption 1. This condition arises primarily from the behavior of the Top-k operator, which induces input-dependent expert selection. As result, the functional contribution of expert is restricted to regions where it is actively selected by the gating mechanism. Outside these regions, the expert can behave arbitrarily without influencing the output. Therefore, if the experts are merely pairwise distinct-rather than pairwise strongly distinct-it becomes possible for different sets of expert functions to yield identical overall behavior when restricted their respective activation domains. This ambiguity underscores the necessity of strong distinctness to ensure functional identifiability in the SMoE setting.
Assumption 2. In practice, the number of experts n is typically much smaller than the input (token) dimension D. As a result, the collections (W(G,i-)) W and {W*(G,--)) - W are generically linearly independent. However, when linear dependence arises, it can prevent certain expert pairs from ever being simultaneously selected by the gating mechanism across all possible inputs. This limitation introduces singular symmetries: different parameter configurations that yield functionally identical outputs but cannot be related via the equivalence structure defined in Theorem C.4.
To illustrate this phenomenon concretely, consider an example with n = 4 and k: = 2. and let E,, Eg, Eg, E4 denote arbitrary expert functions. Define two SMoE models S₁ and S2. whose gating logits are (-2x, -x, x, x, 2x) and ( -3x, 2xx 2x, 3x), respectively. The resulting functions take the form:
(x) + softmaxz(-2n -x) Eg(r) softmax, (--xx 2x. -x)
if < 0, softmax (x,2r)) (xx + softmax2 2r) Ex)x if 0,
(1(z)
(118)
and
softmaxi (-3x, -2x) (x) + softmax2(-330 -2x) ) E2(2) softmax, (2x, 3x) (x) + softmax, (2x, 3x) (xx
if < 0, if >
S2(x)= =
(119)
It is easy to verify that (1 S2(x) for all x E RR {0). where the gating logits are pairwise distinct and the Top-k selections remain constant. However, no transformation of the form specified in Theorem C.4 maps one configuration to the other. This demonstrates how singular symmetries can arise in the SMoE architecture, even when functional outputs coincide on an open dense subset the input domain.
Remark C.9 (The of k 1). In the special where k 1, the SMoE function in Equation (21) simplifies to
S (x; (We, bi, 0.),7 (x;
(120)
where the index is determined by
i = argmax (Warr + b:).
(121)
In this setting, the Top-1 gating mechanism selects only the expert with the highest score, and the softmax reduces hot distribution with single nonzero entry equal to 1.
33

---

Beyond the standard (n)) symmetry acting on the expert parameters. the SMoE architecture with exhibits an additional, nontrivial invariance under the action of the multiplicative group Ry0- Specifically, for any scalar c <> O, we have
S (2; (W, bi, 0=);=1) (a; (cW,, cbi 0z) (=))))
(122)
as the argmax used to select the active expert remains invariant under uniform positive scaling:
argmax (W + b,) argmax (cW,z + cb;),
(123)
i=1,..,
for all x E Q ((W,, )=)) Furthermore, since only one expert is active at any given input, the architecture does not involve explicit interactions between experts. This leads to more complex symmetry structure, including hidden and continuous transformations, which complicates theoretical analysis. For this reason, we exclude the case k from main results and leave its investigation to future work.
D Technical Details for Sections and
D.1 Proof for the Sufficiency of Permutation Invariance in LMC of MoE
The following result establishes that translation transformations do not affect the barrier loss between two parameter configurations.
Proposition D.1. Let = (cw, Sw,,cbiin id,) E G(n), where id, E Sn, denotes the identity permutation. For any A ΦB (nn), the barrier loss remains invariant under translation:
B(dA, ΦB) = B( hδB).
(124)
That is, the translation components do not contribute to the barrier loss as defined in Equation (1).
Proof. Recall that
B( sup rE(0.1]
[L(ti + (1 - ())B) ((C(01) + (1 f)((()))
(125)
Fort E [0, 1]. denote h4 (tow, tc, id,) E G(n). For Φ ΦB E ((n) such that
ΦΛ (W:A. b, 0 )=)
(126)
ΦB = (W B 6 0 ;=
(127)
we have:
tœ^ + (1 - ()(hnn)
t(WA, bBttt 0/4)==1.... + (1 - tW kw, bg + cb, 00) )
(tW,A (1 t) B (1 t)ew, tb + (1 -) (1 t)cs. t)ct, + (1 (1-t)07 1
(hI-.) A + - t)))).
(128)
Since the group action of G(nn) on (n)) preserves the functionality of the MoE function D, we have
D ((:ón) -(::
(129)
and
D(;ldo + (1-1)1)0 (1-1)(non)
= =D (: (toa +(1 t)oB))
=D(:t6A (1-t)BB).
(130)
These observations lead to
[(0B) = L(h0B),
(131)
<(10 + (1 ())(())))))))))))))))))))))))() L _(11A + 1))m).
(132)
34

---

Thus, for E [0, 11, we have
B
[[ttA + (1- (1-
sup
tefo.1]
[(tóA (1 (1- t)68) - + + -[
sup
te[0,1]
OB)
(133) 1
Therefore, the proof is finished.
D.2 Proof of the Permutation-Invarian Property for Equation (11)
Proposition D.2. The cost function defined in Method of Section 5.2 is permutatioon with respect to the hidden units of the experts in the MoE. models.
Proof. For each expert i in the first coxture-of-Expeeeee (MoE) model Φ, let A; E Rxxd and u; E R be the weight matrix and bias for the first layer, and B, Rdxh and v; E F be the weight matrix and bias for the second layer. Define the augmented matrices A, = [A,, ui] E (xx((+1) and B; = [Br, vi] E Rdx((+1), Similarly, for each expert j in the second MoE model d', define AG. u/y, Byy oyy, and the augmented matrices A/, B3j accordingly. We define the Gram matrices (A,) T A, E R g(d+1)x(d+1) and (B.) E ddd for each expert i in o, and similarly for d'.
To establish permutation invariance, consider permutation matrix P E R applied to the hidden units of expert i in p. For the first layer, the permuted augmented weight matrix is PA, For the second layer, the permutation affects the columns of B; within B, = [B,, uz]. so the transformed P
matrix is B, 0
leaving the bias e, unchanged. We now verify the invariance of the Gram matrices under this permutation. For the first layer:
(PA.) (PP))) = (A.) (A, (A.) A.,
(134)
since PTP p= = I, the h x h identity matrix. For the second layer:
PT 0 B
B
PT
1 9) B
PT
0
P
B.
0
0
1
= B
PTP
P
0
B.
0
1
B
B.
B.B.
(135)
where i 1 the (l+1)) x (h 1) identity matrix. This confirms that both (Ā,) A, and (B.)
invariant under permutations of the hidden units.
The cost matrix C e Ruxn for the Linear Assignment Problem (LAP) is C (C..h where
Crj (1((A)) A, - A' B.(.) - BJ(B))
(136)
Since the Gram matrices (Ā,) Ā, and (B,) (and their counterparts in σ') are permutation- invariant, their differences and the Frobenius norms of these differences are also invariant. Conse- quently, Cjý remains unchanged under permutations of the hidden units in either Φ'.
Thus, the cost function is coomuutationinvvarian with respect to the hidden units of the experts, completing the proof.
35

---

D.3 Formal formulation of DeepSeekMoE
For completeness, we present the notation and formulations of three variants of MoE models: Dense MoE, Sparse MoE (SMoE), and DeepSeekMoF.
Given d denoting the input dimension. We define Expert as function ((; Rd Rd parameterized by E IR with e E N is the total number trainable parameters of &. In this work, consider each expert {1; () to be feedforward neural network with ReLU activation functions. Unless stated otherwise, all experts are assumed to share the same architecture.
Expert.
Mixture of Expert with dense gating. Given denoting the number of experts, define Mixture- of ef-xxperts with dense gating as function D: IR gd such that
(x; (W,, by, 0:1
softmax, (s)(2), ................................ Sn (x)))
(137)
where s,(x) War + b, determines the contribution of each expert to the final output. Here, 0, are the parameters of the 7th expert. The function 8 ($1,.... .n.) is called the gating score, and is parameterized as s(; (Wi, b=))))) with (W,, b.) E R x R are the corresponding gating parameters.
cetture-of-Expert with sparse gating. Given positive integer k S denoting the number of activated experts, define the Top-k map by Top-k(2) = ..k for & = (Z1,.... 2n) E R", where i1. .1...., ig are the indices corresponding to the l largest components of z. In the event of ties, we select smaller indices first. We define a Mixture-of- Experts with sparse gating (SMoE) as function S: Rd - Rd follows. For x E Rdd let (x) = Top-k(s(x))) then define:
(W bi 0.1
Σ (aT(a)
softmax; ((=(())) E(x;0,)
(138)
In other words, the Top-k selects the k highest-scoring experts used compute the output.
DeepseekMoB with shared and routed experts. The DeepseekMoE architecture extends the sparse Mixture-ot of Experts by incorporating both shared experts that are always active and routed experts that are sparsely activated based on the input. Given positive integers n, and n, denoting the number of shared and routed experts, respectively (n n, + n. as the total number of experts), and a positive integer k, S n, denoting the number of activated routed experts, the DeepseekMoE layer is defined as function M: Rd - Rd such that
12 (W., b, 0 M(1; 10 12
Σ 8 (2; 0 ((0 +
(x)))((;)))))
(139)
where:
(::0(()) Rd - Rd are the shared experts, parameterized by 0!()) elT (3:(")): Rd - Rd are the routed experts, parameterized by 0!*)
The gating values (x) are computed as follows:
5(x) softmax (z), (I.,...,, Sn (I)) E R""
(xxx) { (x))), if k,
0, otherwise
This formulation allows the model to leverage a combination of always-active shared experts and input-dependent routed experts, enhancing efficiency and performance in large-scale neural networks.
E Impact of Feedforward Reinitialization on Pretrained Transformer Performance
We empirically investigate the sensitivity of pretrained Transformer models to targeted parameter reinitialization within their Feedforward Network (FFN) modules. The FFN, critical component for
36

---

non-linear transformations and feature representation within each Transformer block, was selectively reinitialized layer by layer to assess its functional contribution and the robustness of the overall model architecture. This analysis aims to identify the extent to which performance is dependent on the learned parameters of individual FFNs and to potentially highlight layers that are more critical to maintaining task proficiency. To this end, experiments were conducted two representative model-task pairs:
1. Image Classification using Vision Transformer (ViT-Base, patch size 16, image resolution 224) pretrained on ImageNet.
2. Language Modeling using a GPT-2 model (12-layer, hidden size 768) pretrained on Wiki- Text103.
For each model, the parameters of the FFN subcomponent within a single Transformer block were reinitialized using a standard initialization scheme (e.g.. variance scaling), while all other model parameters, including attention weights and the FFNs in other layers, were kept frozen from the pretrained state. Following reinitialization, the models were evaluated on their respective tasks without any subsequent fine-tuning. This approach isolates the immediate effect of disrupting single FFN on pretrained performance.
Figures 3 and 4 illustrate the performance degradation observed when reinitializing the FFN at different layers for the VIT and GPT-2 models, respectively. A striking observation is the significantly larger performance drop (increase loss, decrease in accuracy) when reinitializing the FFN in the first layer (Layer 0) compared to any subsequent layer. While reinitializing FFNs in layers beyond the first does result in performance degradation, this impact is markedly less severe than at Layer The results further reveal non-uniform sensitivity profile among these subsequent layers, with FFN reinitialization in middle layers tending induce more pronounced performance decrement than in later layers. This pattern suggests unique functional importance sensitivity associated with the initial layer's i FFN, while the network's 's architecture, particularly the presence of skip connections, appears to enable greater degree of resilience to disruption in FFNs at deeper layers.
Validation Loss
Validation Accuracy (%)
100
Protzained: 0.8936
Pretroined: 01.67%
Accuracy (%)
60
Loss
20
0
7
10
13
3
8
9
10
11
Layer
Layer
Figure 3: Performance degradation in ViT-Base on ImageNet due to FFN reinitialization different layers.
key observation from these experiments, particularly for deeper and more complex models with skip connections. that reinitializing a single FFN (except potentially the very first one) often does not lead to catastrophic performance collapse. This phenomenon can be partially attributed to the presence of skip connections, which facilitate the unimpeded flow of information across layers, allowing features learned by other parts of the network to bypass the disrupted FFN. Furthermore, standard weight initialization schemes typically employ relatively small scales centered around zero, meaning the reinitialized FFN parameters introduce perturbation that is initially small and balanced compared to potentially large and specialized weights learned during pretraining. This combination of architectural properties (skip connections) and initialization characteristics helps the model maintain functionality, suggesting that the reinitialized state may remain within or close the original optimization basin found during pretraining. Previous work [53] has highlighted how skip
37

---

Validation Loss
Test Loss
Pretoussed:
Loss
Looss
0
3
8
1.0
11
2
3
5
7
8 100 11
Layer
Layer
Figure 4: Effect FFN reinitialization on GPT-2 perplexity across layers on WikiText103.
connections contribute to flatter minima and improve training stability in deep networks, which aligns with the observation that reinitialization causes less disruption than might intuitively be expected.
To further investigate the generality of this observation, we also conducted FFN reinitialization experiments on tasks commonly used for evaluating model robustness and optimization landscape properties, including text classification tasks (IMDB Review [57]. AGNEWS [92], DBPedia [8]) and additional language modeling tasks (EnWik8 [38], Penn Treebank (Word Level) [58]). Table 4 presents summary of the performance changes these datasets after reinitializing arbitrary single FFN layer compared to the original pretrained performance, and also shows performance after subsequent fine-tuning.
Table Effect of single FFN reinitialization on various tasks. "Pretrained" indicates performance of the original model. "Reinitialize" shows performance after reinitializing a single FFN layer. "Finetune" shows performance after fine- tne-tuning the reinitialized model. "Naive Loss Barrier" refers to the loss barrier observed between two fine-tuned models initialized with different random seeds and trained with different batch orders.
Dataset
Pretrained Loss
Pretrained Accuracy
Reinitialize Loss
Reinitialize Accuracy
Finetune Loss
Finetune Accuracy
Naive Loss
Barrier
IMDB Review
0.3724
87.50
0.6452
68.75
0.4284
87.50
0.0019
AGNEWS
0.2849
90.50
0.6006
76.64
0.2986
90.73
0.0007
DBPedia
0.1896
93.75
0.3017
87.50
0.2036
94.47
0.0003
EnWik8
0,9684
1,3053
0.9506
0.0082
Penn Treebank 4.5246
6.9699
4.4639
0.0454
As shown in Table 4, while reinitializing single FFN does lead performance degradation these tasks (e.g., loss increases by approximately 0.1 to 0.3), the magnitude of this change is relatively modest compared the significant loss increases observed in experiments involving more widespread reinitialization or on larger tasks like GPT-2 on WikiText103 or ViT on ImageNet (where full model reinitialization increase loss by several units). The relatively small performance drop suggests that for these specific datasets and this focused reinitialization strategy, the model's state remains close its original pretrained optimum. Consequently, metries used to assess mode connectivity between the original and reinitialized states, such as the Naive Loss Barrier (shown in Table 4), yield very low values. This indicates that simple linear path in parameter space between the original and reinitialized single-FFY models exhibits low loss values. However, due to the minor performance impact of the reinitialization itself, this low barrier value may primarily reflect the fact that the reinitialized state is already highly functional and located within the same or a very accessible optimization basin, rather than necessarily implying a broadly flat landscape between drastically different high-performing modes. Therefore, while consistent with observations of flat minima facilitated by skip connections [53], these specific reinitialization experiments on these datasets may not provide deep insights into complex mode connectivity far from the original optimum.
38

---

In summary, these findings highlight the heterogeneous functional roles of FFNs across different layers in Transformer models and demonstrate varying degrees of sensitivity to parameter reinitialization. The relative robustness observed for many layers, especially in deep models, is consistent with the structural benefits of skip connections and the properties of standard initialization. This layer-wise sensitivity analysis contributes to better understanding of Transformer architecture and has potential implications for model compression techniques (e.g., identifying less critical FFNs), efficient fine- tuning strategies, and the design of conditional computation mechanisms like Mixture-of-Exppe (MoE) routing.
F Experimental Details and Hyperparameters
We conduct a comprehensive evaluation of Linear Mode Connectivity (LMC) across broad suite of vision and language modeling benchmarks. For vision tasks, our study includes MNIST, CIFAR-10. 10. CIFAR- 100, ImageNet- 1k, as well as transfer learning scenarios from ImageNet-21k k to CIFAR-10 and CIFAR-100. For language modeling, we utilize WikiText103 and the One Billion Word dataset. All experiments are performed using pretrained Transformer-basee architectures, wherein all original model parameters are frozen and only the inserted oixture-of-Experr (MoE) layers are subject to fine-tuning.
For vision tasks, we adopt Vision Transformer (ViT) backbones, while GPT-2 serves as the backbone for language modeling. Model configuratioos-incllee architecture depth, width, and tokenization context-are adjusted per dataset to reflect task-specific complexity and scale. Each expert module is architecturally aligned with the original feedforward network (MIP) layers the pretrained model. To promote stable convergence, learning rates are independently tuned for each setting within the range of 10-4 10-!
MNIST We utilize the original MNIST grayscale images, each resized into a grid of non-overlapping patches with patch size of A lightweight Vision Transformer (ViT) model is employed, configured with an embedding dimension of 32 and a depth of to 2 Transformer layers, depending on the specific setup. Each self-attention layer uses 4 attention heads to process the patch embeddings. Dropout is applied at a rate of 0.0 and the activation function throughout the network is gelu. The model optimized using stochastic gradient descent (SGD) during both the pretraining and fine-tuning stages.
CIFAR-10. We utilize the original CIFAR-10 images, each divided into non-overlapping patches with a patch size of A Vision Transformer (ViT) model is employed, configured with an embedding dimension of 128, 8 self-attention heads, and depth ranging from 2 to 6 Transformer layers. Dropout is applied at a rate of 0.0, and the activation function throughout the network is gelu. The model is optimized using the Adam optimizer during pretraining and stochastic gradient descent (SGD) during fine-tuning.
CIFAR-100. We resize CIFAR-100 images to 224 x 224 and divide them into non-overlapping patches with a patch size of 16. A Vision Transformer (ViT) model is employed, configured with an embedding dimension of 384, 6 self-attention heads, and depth ranging from to 12 Transformer layers. Dropout is applied at a rate of 0.0, and the activation function throughout the network is gelu. The model is optimized using the Adam optimizer during pretraining and stochastic gradient descent (SGD) during fine-tuning.
Imagenet2 -Eell--CFFFA CIFAR1O. We adopt the UTT-Small-Patch16-22 model [75], pretrained on ImageNet-2 1k and subsequently fine-tuned on CIFAR-10. The model consists of 12 layers, a hidden size of 384, an MLP size of 1536. and attention heads, resulting in approximately 22.2M parameters. It employs a patch size and stride 16. Dropout disabled (set to 0.0), and the activation function is gelu. Stochastic Gradient Descent (SGD) is employed during fine-tuning.
We adopt the ViT-Small-Patch16 122ch6624 model [75], pretrained on ImageNet-21k Ik and subsequently fine-tuned on CIFAR-100. The model consists of 12 layers, hidden size of 384, an MLP size of 1536, and attention heads, resulting in approximately 22.2M parameters. It employs a patch size and stride of 16. The attention probability dropout and hidden layer dropout are set to 0.0, and the activation function is gelu. The SGD optimizer employed during fine-tuning.
Imagenet2lk-CIFARR
39

---

ImageNet-1k. We use the PBase-Patchh6- model [20], consisting of 12 Transformer layers, each with a hidden size of 768, an MLP hidden dimension of 3072, and 12 self-attention heads. The total parameter count is approximately 86M. The encoder employs patch size and stride of 16, and the gelu activation function is applied throughout. Both the attention dropout and hidden dropout rates are set to 0.0. The model is pretrained for 300 epochs with linear warm-up of 10,000 steps, followed by cosine decay scheduling. Fine-tuning is performed for 30 epochs without warm- up. Optimization is carried out using the Adam optimizer during both pretraining and MoE-layer fine-tuning stages with identical hyperparameter settings.
WikiTextt03. We adopt GPT-2 model architecture consisting of 12 layers with 12 attention heads and an embedding dimension of 768. The context length and maximum position embeddings are set to 1024. The dropout rate is set to 0.1, and the activation function is gelu. The vocabulary size 50,257. The model uses the standard GPT-2 initialization and layer normalization settings. The Adam optimizer is employed during both pretraining and fine-tuning.
One Billion Word. Similar to WikiText103, the GPT-2 model comprises 12 transformer layers, 12 attention heads, and an embedding dimension of 768, but with a reduced context and positional length of 256. The dropout rate and activation function remain identical to those in the WikiText103 setup. The vocabulary size is expanded to 793470 to capture the dataset's linguistic diversity. The model first pretrained for 500000 steps using the Adam optimizer, followed by a fine-tuning phase of 100000 steps with a 2000-step linear warm-up up schedule. Both stages share the same optimization hyperparameters unless otherwise noted.
All experiments are executed on a single NVIDIA H100 GPU with 80GB of memory, except for the One Billion Word task, which utilizes two H100 GPUs Due the use of the JAX framework, approximately 75% of GPU memory (around 60GB) ) is pre-allocated by default. The number of CPU workers used in data loading and preprocessing is kept less than or equal to 10 across all experiments. For smaller-scale tasks such as MNIST, CIFAR-IO, CIFAR-100, and transfer learning from ImageNet-21k,, each experiment completes in under 30 minutes. For larger-scale tasks, the fine-tuning durations are approximately 15 hours for ImageNet- 1k, 3.5 hours for WikiText103, and 4 hours for One Billion Word.
G Experimental Results
G.1 Verification of Linear Mode Connectivity across diverse configurations
To rigorously evaluate the robustness and generality of Linear Mode Connectivity (LMC) in expert- based Transformer architectures, conducted systematic experiments across wide range of Mixture-of-Experts (MoE) configurations. Our study includes three representative variants-vanilla MoE, SMoE, and DeepSeckMoE-whichh differ in both architectural structure and expert routing mechanisms. In all cases, we substituted the forw-fo network (FFN) of the first Transformer layer with an MoE module, while leaving the rest of the model architecture unchanged. All models were fine-tuned from identical random initializations to ensure consistency in comparison.
Table 5 summarizes the experimental conditions, covering diverse model depths (2, 6, 12 layers), task domains (including vision benchmarks such as MNIST, CIFAR-10, ImageNet21k--CIFF.. language modeling with Wiki and expert configurations. SMoF employs sparsity level of active experts per token, whereas DeepScekMoE further incorporates a shared expert (s = 1) to encourage parameter reuse and stability. Across nearly all configurations, LMC consistently emerges: linear interpolation between independently fine-tuned models (initialized identically) yields smooth loss trajectories, with no significant barriers. This suggests that expert-based models, even with dynamic routing and varying capacities, tend to converge to connected optima under matched training setups. The corresponding loss curves, linked in Table 5, reinforce this observation.
and
These results provide strong empirical evidence that LMC is robust and general phenomenon in expert architectures, supporting the hypothesis that expert specialization and routing do not disrupt the connected geometry of the optimization landscape when alignment in initialization and architecture is preserved.
40

---

Table 5: Experimental configurations for LMC analysis across MoE, SMoE, and DeepSeekMoE variants, evaluated on multiple datasets and settings. Datasets of the form A B denote pretraining on A and finetuning on B. SMoE uses k active experts, while DeepSeekMoF: uses k 2 with an additional shared expert (s = 1). In all cases, the MLP in the first Transformer layer is replaced with an MoE. module.
Method
Datasct
No. layers
No. experts
Figure
MoE
MNIST
[2, 4]
[5. 6]
[2, 4]
[7, 8]
CIFAR-10
2
[2, (2
[9, 10, 11]
[2,4,6]
[12, 13, 14]
CIFAR-100
[2
[15, 16, 17]
ImageNet-21k -CIFAR-10
12
[2,4,6]
[18, 19, 29]
ImageNa:-2RR-CTFARRR
12
[2,4,6]
[21, 22, 23]
ImageNet-Ik
12
12,4,6
[24, 25, 26, 27]
WikiTex:103
6,
[28, 29, 30, 31]
One Billion Word
12
6,
[32, 33, 34, 35]
SMoE (k
2)
MNIST
[4]
[36]
[4]
[37]
CIFAR-10
2
[4, 8]
[38, 39]
[4, 81
[40, 411
CIFAR-100
[4, 8]
[42, 43]
ImageNct-2IK-CIFFRR-10
12
[4, 8]
[44, 451
ImageNe+2II-CIFARR-100
12
[4, 8]
[46, 47]
ImageNet-1k
[4, 8, 16]
[48, 49, 50]
WikiText103
[4, 8, 16]
[51, 52, 53]
One Billion Word
12
[4. 8, 16J
[54. 55, 56]
DeepSeeKMoE
MNIST
[4]
[57]
(k 2, 1)
2
[4]
[58]
CIFAR-10
[4. 8]
[59, 60]
[4,
[61, 62]
CIFAR-100
[4, 8]
[63, 64]
ImageNet-21k- -CC-A---
12
[4, 8]
[65, 66]
ImageNNet22IK-CIAARRRRO00
12
[4. 8]
[67, 68]
ImageNet-Ik
[4, 8, 16]
[69, 70, 71]
WikiText103
12
[4, 8, 16]
[72, 73, 74]
One Billion Word
12
[4, 8, 16]
[75, 76, 77]
41

---

G.1.1 Dense Mixture-of-Experr
0.35
Naave
0.30
0.30
WX
Test Loss
0.25
0.20
0.20
0.15
0.15
0.10
0.10
Model
Model
Model
Model 3
Model 2
Model
Test Accuracy
0.96
0.96
0.96
0.94
0.92
0.92
0.92
Figure 5: Linear Mode Connectivity for ViT-MoE MNIST with 1 layer and 2 experts
0.30
0.25
Tast Loss
0.15
0.15
0.10
0.10
Test Accuracy
0.96
0.96
0.96
0.94
0.94
0.92-
0.93
Model1
Model
Model
Model
Modal
Figure 6: Linear Mode Connectivity for ViT-MoE MNIST with I layer and 4 experts
1.5
2,0
Tast Loss
1.5
1.0
1,0
Test Accuracy
0.9
0.9
0.8
0.9
0.7
0.7
0.6
Modal
Modul2
Model
À
Model 3
Model2
Modal 3
Figure 7: Linear Mode Connectivity for ViT-MoE on MNIST with layers and 2 experts
42

---

1.25
Navv
1.5
WS44
1.00
2.0
Test Loss
1.0
0.50
1.0
0.5
0.25
Model
Model
Model
Model
0.0
Model
Modal
0.35
Tast Accuracy
0.9
0.30
0.9
0.8
0.8
0.7
0.7
0.75
0.6
0.70
Figure Linear Mode Connectivity for ViT-MoE on MNIST with layers and experts
1.3
Natinn
WM
Test Loss
Model
Model
Mdddel
Model
Model
Test Accuracy
0.625
0.825
0.800
0.575
0.550
0.550
Model
Model
Model
Model
Model
Figure 9: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 2 layers and experts
1.3
1.3
Test Loss
1.2
1.2
1.1
1.0
1.0
0.650
0.650
0.650
Test Acouracy
0.625
0.625
0.600
0.600
0.600
0.575
0.550
0.550
0.550
Model
Model
Model
Model
Model
Model 3
Figure 10: Linear Mode Connectivity for ViT-MoE on CIFAR- 10 with 2 layers and 4 experts
43

---

1.10-
Test Loss
1.2
1.05
1.1
Model
Model
Model1
Model 3
Model 2
Modal 3
0.68
0.650
Test Accuracy
0.65
0.625
0.625
0.64
0.600
0.600
0.63
0.575
0.575
0.550
Model
Model
Model
Figure 11: Linear Mode Connectivity for ViT-MoE on CIFAR-I0 with 2 layers and experts
Natve WM
1.2-
Test Loss
1.1
1.00
0.55
0.9
0.9
0.90
Model
Model 2
Model 1
Model 3
Model 2
Model 3
0.750
0.74
Teat Accuracy
0.725
0.700
0.70
0.70
0.675
0.68
0.66
0.630
0.64
0.625
Model
Model
Model
Model
Mocel 2
Figure 12: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and experts
1.3
1.10
1.2
Test Loss
1.05
1.1
1.00
1.0-
0.95
Model
Model
Modal
Model
0.750
0.75
0.75
Test Test. Accuracy
0.70
0.70
0.700
0.65
0.675
0.60
Model
Model 2
Model
Model
Model
Model 3
Figure 13: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and 4 experts
44

---

1.3-
Teat Loss
1.2
1.2
1.1
1.1
1.0
1.0
1.0
0.9
Model 1
Model
Model
Model 3
Model 2
Model 3
0.75
0.75
Test Accuracy
0.70
0.63
0.65
0.60
Model
Model
Figure 14: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with 6 layers and experts
Neviv WM
3.0
3
3-
2-
1.5
Mode12
Test Accuracy
0.6
0.6
0.7
0.6
0.6
0.4
0.2
0.2
Model
Modul2
Model
Model 3
Model 2
Model 3
Figure 15: Linear Mode Connectivity for ViT-Moe on CIFAR-100 with layers and 2 experts
5-
3.0
Tast Loss
3-
1.5-
2-
0.7
Test Accuracy
0.6
0.6
0.6
0.5
0.4
0.4
0.2
0.2
Model
Model2
Model
À
Model 3
Model2
Modal 3
Figure 16: Linear Mode Connectivity for ViT-Moe on CIFAR-100 with 6 layers and 4 experts
45

---

Navvv
5-
WM
5-
1-
Model
Mode12
Model
Model
34ode122
Model
Test Accuracy
0.4
0.4
0.4
0.2
0.2
0.0
Model2
Figure 17: Linear Mode Connectivity for ViT-Moe on CIFAR-I00 100 with layers and 6 experts
2.5
2.5
1.5
2.0-
2.0
Test Loss
1.00
1.5
1.5
1.0
1.0
0.5
0.5
0.5
0.0
0.0
Modal
Model
Model
0.9
Tast Accuracy
0.7-
0.4
0.4
0.2
0.2
Model
Model2
Model
Model
Modal
Figure 18: Linear Mode Connectivity for ViT-MoE 2 experts
with 12 layers and
2.0
2.0
Test Loss
1.5
1.5-
1.5
1.0
1.0
1.0-
0.5
0.5
Model
Mode12
Model
Model 3
Modal 2
Modal 3
1.0
1.0
1.0
Tast Accuracy
0.0
0.8-
0.6
0.6
0.6
0.6
Modal
Figure Linear Mode Connectivity for ViT-MoE on 1mageNet-21k--CHE 10 with 12 layers and 4 experts
46

---

Navv
WS4
3-
2-
Model
Mode12
Model
Model
Model22
Model
Test Accuracy
0.6-
0.6
0.6
0.4
0.2-
0.2
0.0
Model
Model2
Figure 22: Linear Mode Connectivity for ViT-MoB on ImageNet-2IK--ITARR-IIIII with 12 layers and 4 experts
2.5
Naava WIM
2.5
2.5
2.0
2.0
Test Loss
1.5-
0.0
Model
Mode12
Model
Model
Model
Modal
1.0
1.0
Test Tast Accuracy
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
Model
Model2
Model
Model J
Model 2
Model 3
Figure 20: Linear Mode Connectivity for ViT-MoE on 2 100 with 12 layers and experts
Nuvve VS4
4-
2-
Model
Modul2
Model
Model
3ddde1
Tast Accuracy
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.0
Model
Modal
Figure 21: Linear Mode Connectivity for VYT-MoE on ImageNet-21k- 100 with 12 layers and 2 experts
47

---

Nocty
3-
3-
Model
Model2
Model
Model
Model2 2
Model
0.8
Test Accuracy
0.6
0.6
0.6
0.2-
0.2
0.2
Model
Model2
Figure 23: Linear Mode Connectivity for ViT-MoE with ImageNet22KK-CIFAR 100 with 12 layers
and 6 experts
Validation Loss
2.0
2,0
1.5
1.5-
1.0
Validation Accuracy
75
65
60
50-
55
1odel
Model
Model
Model
Modal
Figure 24: Linear Mode Connectivity for ViT-MoE ImageNet-Ik with 12 layers and 2 experts
Validation Loss
2.0
2,0
1.5
1.0
Model
Model
Model
Model
Model
Model
80
80
75-
75
70
Valdatton
65
65
60
58
55
Modal
A.
Model
Model
Model
Model 2
Model 3
Figure 25: Linear Mode Connectivity for VIT-MoE on ImageNet-lk with 12 layers and 4 experts
48

---

Norin
Validation Loss
2.0
2.0
2.0
1.3-
1.5-
1.5
Model
Model
Model
Model 3
Model 2
Model
00
Validation Accuracy
60-
60
50
3
Model
Figure 26: Linear Mode Connectivity for ViT-MoE on ImageNet-Ik with 12 layers and 6 experts
Validtton Loss
2.0
Narin WM
2.0
2.0
1.5-
1.5
1.0
1.0-
Model 1
Model
Model 1
Model 3
Model 2
Model 3
80
00-
00
Validation Acouracy
70-
70
50
50
Model
Model
Model
Model 3
Model 2
Model
Figure 27: Linear Mode Connectivity for ViT-MoE on ImageNet-Ik with 12 layers and 8 experts
5.0
Naoe WIM
Valxtatton Loss
4.0
4.0
3.5
3.5-
3.5
3,0
5.0
4.5
4.5
Test Loss
4,0
4.0
3.5
3.5
3,0
3.0
Model 1
Model
Model 1
Model 3
Model 2
Model 3
Figure 28: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 2 experts
49

---

Validation Loss
Norin
4.0
2.75
Model
Model
Model
Model
Model 2
Test Loss
3.5-
3.5
3.25
3.0
3.00
2.75
Model
Model
Figure 29: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and experts
3.6
3.6
Validation Lass
3.75
3.50
3.2
3.2
3.25
3.00
2.0
2.8
2.75
Model
Model
Modo)
Model
Model
Model
4.00
3.6
3.6-
3.75
Test Loss
3.4
3.4
3.2
3.2
3.0
3.0
3.00
2.75
2.8
2.8
Model
Model
Model
Model
Model
Model
Figure 30: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 6 experts
3.6
3.75
Validation Loss
3.50
3.2
3.0
3.00
2.8
2.75
Modal
Model
Model1
Model
Model
Modal
3.75
3.6
Test Loss
3.50
3.4
3.25
3.2
3.00
3.0
2.75
2.8
2,8
Model
Model
Model
Model 3
Model 2
Model 3
Figure 31: Linear Mode Connectivity for GPT2-MoE on Wikitext103 with 12 layers and 8 experts
50

---

5.5
6.0
5.5
Test Loss
4.5
4.0
3.45
3.5
3.5
Mode]
Model
Model 1
Model
Model
400
Perplextt
250
200
300
150
Figure 32: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and experts
3.52
Nasion
5.5
4.5
Test Loss
3.50
3.40
4.0
0.5
3.46
3.5
Model
Model
Model
Model
Model
Model
34
120
250
Test Perplexity
200-
39
32
50
Model
Model
Model 1
Model
Model
Model 3
Figure 33: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and experts
5.0
3.48
4.0
Test Loss
4.5
3.47
4.0
3.45
3.6-
3.40
Model
Model
Test Perplesity
60
75
4.0
31.5
50
30
Figure 34: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and experts
51

---

Novve
4.50
3.52
5.0
0.25
3.50
6.00
3.48
4,0
3.46
3.50
3.5
3.44
Model
Model
Model
Model
Mode)
250
Test Perplexity
100
200
80
33
150
60
100
40
Figure 35: Linear Mode Connectivity for GPT2-MoE on One Billion Word with 12 layers and experts
G.1.2 Sparse Hiixture-ofExpeere
0.225
0.30
0.25
Tast Loss
0.175
0.20
0.150
0.15
0.10
0.10
0.100
Model
Model
Model
Model 2
Modal 3
Test Test,Acoon Accuracy
0.97
0.96
0.96
0.95
0.92
0.94
Model
Model
Model
Model
Model
Modal
Figure 36: Linear Mode Connectivity for ViT-SMoE (A: 2) MNIST with layer and 4 experts
1.5
Tast Loss
1.0
1.0-
0.5
0.5
Model
Test Accuracy
0.9
0.9
0.30
0.85
0.0
0.7
0.80
0.75
Model
Modal
Model1
À
Model3 3
Modal
Modal 3
Figure 37: Linear Mode Connectivity for VIT-SMoE (A: = 2) on MNIST with 2 layers and 4 experts
52

---

1.3
Test Loss
1.2
1.1
1.0
Model
Model
Model
Model
Model2
Model 3
0.650
0.64
Tast Aocuracy
0.600
0.00
0.575
0.575
0.58
0.550
0.550
0.525
Model
Figure 38: Linear Mode Connectivity for ViT-SMoE (k = 2) on CIFAR-10 with layers and experts
1.3-
Test Loss
1.2
1.2-
1.0
1.0
1.0
Model1
0.650
0.850
0.850
Test Acouracy
0.625
0.625
0.625
0.800
0.800
0.575
0.575
0.595
0.550
0.550
0.525
0.525
Model
Model
Model
Model 3
Model 2
Model 3
Figure 39: Linear Mode Connectivity for ViT-SMoE (k 2) CIFAR-10 with 2 layers and experts
1.2-
1.2
Test Loss
1.1
1.1
1.1
1.0
0.9
0.9
Model
Model
Model 1
Model 3
Model
Model 3
0.750
Test Accuracy
0.725
0.700
0.675
0.65
0.625
Figure 40: Linear Mode Connectivity for ViT-SMoE (k 2) on CIFAR-10 10 with layers and experts
53

---

1.3-
Natve
WM
1.2
Test Loss
1.2
1.1
1.1
1.0-
0.9
0.9
0.9
Model 1
Model
Model 1
Model 3
Model
Model 3
0.75
0.750
Test Accuracy
0.725
0,70
0.70
0.675
0.65
0.65
0.630
0.60
0.60
Model
Figure 41: Linear Mode Connectivity for ViT-SMoE (k = 2) on CIFAR-10 with layers and experts
3.0
Tast Loss
2.5
2.0
3-
1.0
0.7
Test Accuracy
0.6-
0.6
0.8
0.5
0.4
0.4
0.2
0.2
0.3
Model 1
Model2
Model
Á
Model 3
Model2
Modal 3
Figure 42: Linear Mode Connectivity for ViT-SMoE (k 2) on CIFAR-100 with 6 layers and experts
5-
4-
Test
Model
Model2
Model
Model 3
Midee2
Model 3
0.7
Test Accuracy
0.6
0.5
0.4
0.4
0.34
0.2
0.0
Model2
Figure 43: Linear Mode Connectivity for ViT-SMoE (I: 2) on CIFAR-100 with layers and experts
54

---

2,0
Noxon
Test Loss
1.5
1.5
1.5
1,0
1.0
0.5
0.5
0.0
0.0
Model
Model2
Model
Model 3
Model
Model 3
1.0
1.0
1.0
Test Accuracy
0.8
0.6
0.4
0.4
0.4
Model2
Figure 44: Linear Mode Connectivity for ViT-SMoB (A: = 2) on ImageNet-2Ik -II-AR---00 with 12 layers and 4 experts
Natio
2.5
2.5
Test Loss
1.5
2,0
2.0
1.5-
1.5
1.0
1.0
1.0
0.5
0.5
0.0
Model
Model
Model
Model
Model
Modal
1.0
Test Accuracy
0.8
0.8
0.9
0.6
0.6
0.6
0.2
0.2
Model
Modal2
Model
Á
Model 3
Model2
Modal 3
Figure 45: Linear Mode Connectivity for ViT-SMoE (k: 2) ImageNet-2Ik -II------III with 12 layers and experts
3
2-
Model
Model2
Model
Model 3
Midde1
Model 3
Test Accuracy
0.8
0.6
0.6
0.5
0.4
0.2
0.2
0.2
Model2
Figure 46: Linear Mode Connectivity for ViT-SMoE (i = 2) on 1mageNet-21k-CIIFA. with 12 layers and 4 experts
55

---

Nocon
2-
Model
Model2
Model
Model
Model2 2
Model
0.8-
Test Accuracy
0.6-
0.6
0.4
0.2
0.2
0.0
Model
Model2
Figure 47: Linear Mode Connectivity for ViT-SMoE (k = 2) on ImageNet-2Ik- with 12 layers and 8 experts
1.6
Validation Loss
1.4
2.0
1.2-
1.5
1.0
0.8
Validation Accuracy
60
Model
Model
Model
Model 3
Model 2
Model 3
Figure 48: Linear Mode Connectivity for ViT-SMoE (k = 2) on ImageNet-Ik Ik with 12 layers and experts
Validation Loss
3.0
2.5-
1.5
2.0
2.0-
1.5
1.5
Model
Model
Model
Model
Model
Model
Vabdation Accuracy (%)
75
70
50
Figure 49: Linear Mode Connectivity for VIT-SMoE (k 2) on ImageNet-Ie 1k with layers and experts
56

---

Nakan
2.5
Validation
2.0
1.25
Mode]
Model
Model
Model
Model
3
Validation Accuracy (%)
70
00-
70-
30-
40
Figure 50: Linear Mode Connectivity for ViT-SMoE (A: 2) on ImageNet-IK with 12 layers and 16 experts
4.00
Validation Loss
3.4
3.75
3.75
3.2
3.50
3.50
3.25
3.00
2.8
2.75
Model
Model
Model
Model3
Model
Modal
4.00
4.00
Test Loss
3.4-
3.78
3.75
3.50
3.50
3.2
3.25
3.0
3.00
2.8
Model
Model
Model
Model
Model
Model 3
Figure 51: Linear Mode Connectivity for GPT2-SMoE (k: 2) Wikitext103 with 12 layers and experts
4.00
Natve WS4
3.75
3.50
3.25
3.25
3.00
2.5
2.75
Model
À
Modal
Mcdel1
A
Model
Modal
Modal 3
4.00
3.75
3.5
3.25
3.5
3.00
3.0
2.75
Model
Figure 52: Linear Mode Connectivity for GPT2-SMoE (R: 2) on Wikitext103 with 12 layers and experts
57

---

Nari
5.0
0.00
Validation Loss
WM
4.5
4.0
3,5-
3.0
3,0
3.00
2.75
Model
X
Model
Model 1
Model 3
Model 2
Model 3
4.5
5.0
Test Loss
4.-
3.75
3.50
4,0
3.5-
3.5
3.00
3,0
2.75
Model
Model
Figure 53: Linear Mode Connectivity for GPT2-SMoE (A: = 2) on Wikitext103 with 12 layers 16 experts
Nadio
Test Loss
3.48
3.6-
3.46
3.5-
3.40
Model
Model2
Model
Model
Model 2
Model
1500-
Test Parplexity
33.0
32.5
37.5
1000
35.0
500
31.5
0
Model
Model 2
Model
Model 3
Model
Model 3
Figure 54: Linear Mode Connectivity for GPT2-SMoE (k 2) on One Billion Word with 12 layers and experts
Natoo
3.65
3.8-
Test Loss
3.60
3.7
3.55
3.5
Model
Model 2
Model
Model
Model
3R
45
38
36
Test
34
32
Model
Figure 55: Linear Mode Connectivity for GPT2-SMoE (I: 2) on One Billion Word with layers and 8 experts
58

---

3.65
3.60
3.8
Test Loss
3.55
3.50
3.8
3.50
3.45
Model1
Perpledty
36-
40
34
32-
32
Model
Figure 56: Linear Mode Connectivity for GPT2-SMoE (A: = 2) on One Billion Word with layers and 16 experts
G.1.3 DeepSeek Mixture-of-Experts
0.25
Test Loss
0.25
0.20
0.20
0.15
0.15
0.15
0.10
Model
Model
Model 1
Model 3
Model 2
Model
Tast Tast.Accurae Accuracy
0.97
0.96
0.96
0.95
0.95
0.92
0.94
Figure 57: Linear Mode Connectivity for ViT-DeepScekMoE s 1) on MNIST with layer and 4 experts
1.5
0.8
Test Loss
1.00
1.0
0.75
Modal
Modal
Model1
Model3
Modal
Model 3
0.35
Test Accuracy
0.9
0.90
0.0
0.0
0.80
0.7
0.7
0.75
Model
Figure 58: Linear Mode Connectivity for ViT-DeepSeekMoE (I: 2, 1) on MNIST with layers and 4 experts
59

---

1.3-
1.3
Teat Loss
1.2
1.2
1.1-
1.1
1.0
Model 1
Model
Model
Model 3
Model 2
Model 3
0.650
0,630
0.630
Test Accuracy
0.625
0.600
0.600
0.575
0.575
0.573
0.550
0.550
0.550
0.525
0.525
Model1
Model2
Figure 59: Linear Mode Connectivity for ViT-DeepSeckMoE (A: 2, 3 = 1) on CIFAR-10 with layers and 4 experts
Test Loss
1.2
1.2-
1.2
1.3
1.1
1.1
1.0
Model
Test Acouracy
0.650
0.650
0.650
0.625
0.625
0.600
0.575
0.575
0.550
0.559
0.550
Model 1
Model
Model 1
Model 3
Model 2
Model
Figure 60: Linear Mode Connectivity for ViT ViT-DeepSeekMoB (k: 2, 1) on CIFAR-10 with layers and 8 experts
1.3
1.3
1.4
1.3
Test Loss
1.2
1.2
1.1-
1.0-
1.0
0.9
0.9
0.9
Model 1
Model
Model
Model 3
Model 2
Model 3
0.75
0.75
Test Accuracy
0.70
0.70
0.65
0.65
0.60
Figure 61: Linear Mode Connectivity for ViT-DeepSeekMoE (I: 2, 1) on CIFAR-10 with layers and 4 experts
60

---

1.2
1.2-
Test Loss
1.1
1.2
1.1
1.0
1.0
0.9
0.9
0.9
Model 1
Model
Model
Model 3
Model
Model 3
0.75
0.75
Test Accuracy
0.725
0.700
0.70
0.678
0.65
0.65
0.50
0.625
0.60
Figure 62: Linear Mode Connectivity for ViT-DeepSeckMoB (A: = 2, = 1) on CIFAR-10 with layers and 8 experts
5-
4-
Test
3
3-
2-
Model
Model2 2
Model
Model 2
Test Test Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
Model
2
Model2
Model
Model 3
Model2
Modal 3
Figure 63: Linear Mode Connectivity for ViT-DeepSeekMoE (k: 2, 1) on CIFAR-100 with layers and experts
5-
2-
Model
Model2
Model
Model 3
36dee1
Model 3
Test Accuracy
0.6-
0.4
0.2
0.2
Model2
Figure 64: Linear Mode Connectivity for ViT-DeepSeekMoE (k: 2, 1) on CIFAR-100 with layers and 8 experts
61

---

2.0
Noxon
1.5
Test Loss
1.0-
1,0
0.5-
0.5
0.5
Model
Mode12
Model
Model
Model 2
Model 3
1.0
1.0
1.0
Test Accuracy
0.8-
0.6
0.6
0.6
0.4
0.4
Model2
Figure 66: Linear Mode Connectivity for ViT-DeepSeekMoE (A: 2I--CIRA--10 with 12 layers and 8 experts
2, 8 = 1) on ImageNet-
Novto WM
2-
1-
1-
Model
Mode12
Model
Model
Mode12
Model
Test Accuracy
0.8
0.8
0.6
0.6
0.4
0.2
0.2
Model
Model2
Model
Model 3
Model
Model 3
Figure 67: Linear Mode Connectivity for ViT-DeepSeekMoB (I: 2I--CIFAR-100 with 12 layers and 4 experts
2, s = 1) on ImageNet-
2.0
Nave VIM
2.0
1.5
Test Loss
1.5
1.5-
1.0
1.0
0.5
0,5
Model
Model2
Model
Model 3
Model 2
Model 3
1.0
1.0
Test Accuracy
0.9
0.8-
0.8
0.6-
0.6
0.4
0.4
0.4
Modal2
Figure 65: Linear Mode Connectivity for ViT-DeepSeekMoE 2IK-CIFAR-10 10 with 12 layers and 4 experts
8 1) on ImageNet-
62

---

Nocon
2-
1-
1-
1-
Model
Model2
Model
Model
Model2
Model
Tast Accuracy
0.8-
0.8
0.8
0.6
0.6-
0.4
0.2
0.2
Model2
Figure 68: Linear Mode Connectivity for ViT-DeepSeekMoE (A: 2, 8 = 1) on ImageNet- 21k 2IFAAIR--0000 with 12 layers and 8 experts
Nazw
Validation Lows
1.75
VS4
2.0
2.0
1.50
1.5-
1.25
1.00
1.0-
0.75
Model 1
Model
Model
Model
Model 2
Modal
80
80
Validation Accuracy
75
75
65
60
60
58
Model
X
Model
Model 1
A
Model 3
Model 2
Model 3
Figure 69: Linear Mode Connectivity for ViT-DeepSeekMof (k 2,=2,, 1) on ImageNet-1k with 12 layers and 4 experts
Validation Loss
1.75
1.50
1.25
1.5-
1.00
1,0
1.0
Model
Model
Model
Model
Model 2
Model
Vabdation Accuracy (%)
80
65
60
60
50
58
Model
Figure 70: Linear Mode Connectivity for ViT-DeepSeekMoE (k: 16=2, 1) on ImageNet- l with 12 layers and 8 experts
63

---

Nari
2.00
Validation Loss
2.0
2.0
1.3
1.5
1.0-
Model 1
Model
Model
Model 3
Model 2
Model
00
Validation Accuracy
75-
65
60
50
Model
Model
Figure 71: Linear Mode Connectivity for ViT-DeepSeekMoE (k = 2, 1) on ImageNet-IK with 12 layers and 16 experts
Nation
Validation Loas
3.75
3.50
3.25
3.0
3.00
2.8
2.8
2.75
Model 1
Model
Model
Model
Model 2
Model 3
3.6
3.6
3.75
Test Loss
3.4
3.50
3.25
Model
Model
Model
Model 3
Model 2
Madel 3
Figure 72: Linear Mode Connectivity for ViT-DeepSeekMoE (A: 2, layers and experts
1) on Wikitext103 with 12
Validation Loss
3.75
3.50
3.5-
3.25
3.0
3,0
3.00
4.5
4.00
4.0
3.75
Test Loss
3.50
3.5
3.5
3.25
3.00
3,0
2.75
Figure Linear Mode Connectivity for ViT-DeepSeekMoE (h: 2, = 1) on Wikitext103 with 12 layers and 8 experts
64

---

4.5
Validation Loss
Nari
WM
4.5
4.0
3.5
3.5
3.0-
Model
Model
Model
Modal 3
Model 2
Model 3
4.5
4.5
4.5
Test Loss
4.0
4.0-
3.5
3.5
3.0
3,0
Model
Model
Figure 74: Linear Mode Connectivity for ViT-DeepSeekMoE (A: = 2, 1) on Wikitext103 with 12 layers and 16 experts
Narto
Test Loss
3.55
3.8
4.0-
3.50
3.6-
3.6
Model
Model
Modo)
Model
Model
Model
Test Perplexitty
60
80-
32
4.-
Model
Model
Model
Model
Model
Model 3
Figure 75: Linear Mode Connectivity for ViT-DeepSeekMot (k 2, 1) One Billion Word with 12 layers and experts
3.60
Test TestLos Loss
5.0
3.7
3.55
3.50
4.0
3.5
3.45
Model
Model
Modal
Model
45.0
Perplexity
42.5
36
200
40.0
150
37.5
32.5
Figure 76: Linear Mode Connectivity for ViT-DeepSeekMoF (k: 2, 1) on One Billion Word with 12 layers and experts
65

---

Table In all configurations, the MLP component of the last Transformer layer is replaced by an MoE component.
Method
Dataset
Number of layers
Number of experts
Figure
MoE
CIFAR-10
6
[2, 4]
[78, 79]
ImageNet-2IK--IIARR-IIII
12
[2, 4]
[80, 81]
ImageNet22IK-CIFAR-1000
12
[2, 4]
[82, 83]
SMoE (k 2)
CIFAR-10
6
[4, 8]
[84, 85]
ImageNet-2IK--IFAAR-00
12
[4, 8]
[86, 87]
ImageNee:21k-CITAAR-1100
12
[4, 8]
[88, 89]
DeepSeckMoE
CIFAR-10
6
[4, 8]
[90, 91]
2,8 = 1)
ImageNet-21k--CIFAARR
[4, 8]
[92, 93]
ImageNe+21IK-CIFAR-1100
12
[4, 8]
[94, 95]
3.70
3.52-
3.60
3.65
Test Loss
3.55
3,40-
3.55
3.46
3.45
Model 1
Model
Model
Model 3
Model 2
Model 3
34.0
Perplesty
33.5
36
33.0
34
34
32.0
32
31.5-
32
Model
Model
Model 1
Model 3
Model
Medel3
Figure 77: Linear Mode Connectivity for ViT-DeepSeekMoE (k: 2, 1) One Billion Word with 12 layers and 16 experts
G.2 Linear Mode Connectivity Analysis: Last Layer
We investigate Linear Mode Connectivity (LMC) in settings where the feed-forward network (FFN) of the last Transformer layer is replaced with a Mixture-of-Experts (MoE) module. This extends earlier analysis of first-layer substitutions by examining connectivity the terminal depth of the architecture, offering a complementary view on model plasticity and the modular structure of expert-based designs.
Following the procedure outlined Section I. all pretrained weights were frozen, and only the MoE module in the last layer fine-tuned. For each configuration, three independent fine-tuning runs were conducted with different random seeds. LMC was assessed by evaluating the loss along linear interpolation paths between pairs of and overlap of their respective solution basins.
models, providing insight into the connectivity
Table summarizes the experimental settings for last-layer FFN replacement across dense MoE, SMoF, and DeepSeekMoF variants. Unless otherwise stated, all figure references correspond to lasr-layer experiments.
66

---

0.905
Natio
0,900
0.900
Test Loss
0.900
0.095
0.893
0.895
0.890
0.800
0.890
0.885
0.885
Model
Model
Model
Model
Model 2
Model 3
0.745
Test Accuracy
0.740
0.260
0.740
0.735
0.735
0.735
0.725
0.725
0.725
Model
Model
Model
Figure 78: Linear Mode Connectivity for ViT-MoE on CIFAR-10 with layers and 2 experts.
0.905
Nation
1.000
0.900
Test Loos
1.805
0.090
0.885
0.865
0.065
0.880
0.880
0.880
0.745
0.745
Test Acouracy
0.740
0.740
0.235
0.735
0.730
0.730
0.725
0.725
0.720
Model
Model
Model 1
Model 3
Model 2
Model
Figure 79: Linear Mode Connectivity for ViT-MoE CIFAR-10 with 6 layers and 4 experts.
0.066
Test Loss
0.063
0.065
0.062
0.050
0.063
0.055
Model
Model
Model
Modal
Modal
Model 3
0.985
Test Accuracy
0.500
0.975
0.975
0.975
0.370
0.963
Model
Figure Linear Mode Connectivity for ViT-MoE on 11ageNet-21k--CCI 10 with 12 layers and 2 experts.
67

---

0.075
Test Loss
0.070
0.065
0.060
0.06
0.06
0.055
Model
Model
Model
Model
Model
Model
Tast Accuracy
0.980
0.960
0.900
0.975
0.975
0.975
0.965
0.965
0.963
Model
Figure 81: Linear Mode Connectivity for ViT-MoE on on ImageNe+2IK--IFAR-II with 12 layers and 4 experts.
Xatos
0.475
0.42
0.475
Test Loss
0.050
0.450
0.425
0.375
0.915
0.915
Test Accuracy
0.910
0.910
0,910
0.905
0.905
0.900
0.900
0.895
0.895
0.895
Model1
Model
Model
Model
Model 2
Model 3
Figure 82: Linear Mode Connectivity for ViT-MoE ImageNet-2 with 12 layers and experts.
0.06
0.60
0.00
Test Loss
0.44
0.55
0.45
0.45
0.40
0.00
0.36
Model
Model 2
Modal
Modal
Modal
Model3 3
0.915
0.915
Test Accuracy
0.910
0.905
0.905
0.90
0.900
0.900
0.895
0.895
Model3
Figure 83: Linear Mode Connectivity for VIT-MoE on 1 100 with 12 layers and 4 experts.
68

---

0.905
0.905
Teat Loss
0.900
0,900
0.900
0.890
0.895
0.885
1.900
0.890
0.885
0.885
Model 1
Model
Model
Model 3
Model 2
Model 3
0.745
0.245
0.745
Test Accuracy
0.740
0.260
0.740
0.735
0.730
0.725
0.725
0.725
Model1
Model 2
Model
Figure 84: Linear Mode Connectivity for ViT-SMoE (k 2) on CIFAR-I0 with layers and experts.
0.906
Natue
0.900
Test Loss
0.895
0.885
0.890
0.890
0.745
0.745
0.745
Test Accuracy
0. 740
0.740
0.738
0.730
0.730
0. 0.725
0.725
Model
Model
Model
Model
Model
Model 3
Figure 85: Linear Mode Connectivity for ViT-SMoE (k 2) on CIFAR-10 with 6 layers and experts.
0.08
Tast Loss
0.070
0.065
0.060
0.06
0.055
0.055
Mdddel
Modal
Model
Model 3
Model 2
Modal 3
0.985
0.985
Test Accuracy
0.980
0.980
0.975
0.975
0.970
0.970
0.965
0.965
Figure 86: Linear Mode Connectivity for ViT-SMoE (I: 2) on ImageNet-21k - with 12 layers and 4 experts.
69

---

0.075
0.075
Test Loss
0.079
0,070
0.085
0.063
0.060
0.053
0.055
0.055
Model
Model
Model
Model
Model
Model
0.985
Tast Accuracy
0.980
0.960
0.980
0.975
0.975
0.570
0.990
0.965
0.965
0.965
Model
Figure 87: Linear Mode Connectivity for ViT-SMoE (A: 2) on ImageNet-2Ik- - with 12 layers and 8 experts.
Natte
0.60
Test Loss
0.53
0.42
0.50
0.50
0.80
0.45
0.38
0.40
0.35
0.35
Model
Model
0.915
0.915
0.915
Test Accuracy
0.910
0.905
0.966
0.900
0.900
0.900
0.895
Model
Model 2
Modal
Modal
Modal
Model3
Figure 88: Linear Mode Connectivity for ViT-SMoE (k on ImageNet-2IK-CIFAR with 12 layers and 4 experts.
0.46
0.60
0.00
Test LosS
0.04
0.55
0.45
0.45
0.40
0.00
0.36
Model
Model 2
Modal
Modal
Modal
Model3 3
0.915
0.915
0.915
Test Accuracy
0.905
0.905
0.90
0.960
0.900
0.900
0.836
Model3
Figure 89: Linear Mode Connectivity for VIT-SMoE (k 2) on ImageNet-21k 2n-keNeet21K-CIARR- with 12 layers and 8 experts.
70

---

0.900
Teal Loss
0.895
0.895-
0.890
0.890
0.805
0.885
Mddde1
Model
Model
Model
Model
Model 3
0.745
0.745
Test Accuracy
0.7740
0.740
0.735
0.730
0. 0.730
0.725
0.725
0.725
Model
Model2
Figure 90: Linear Mode Connectivity for ViT-DeepSeckMoE (A: (k= 2, = 1) on CIFAR-10 with layers and 4 experts.
0.006
0.900
0.900
Test Loss
0.895
0.805
0.885
0.800
0.890
0.885
0.880
0.880
Model
0.745
0.745
Test Accuracy
0,740
0.740
0.740
0.735
0.735
0.730
0.730
0.730
0.725
0.725
Model
Model
Model 1
Model 3
Model 2
Model 3
Figure 91: Linear Mode Connectivity for ViT-DeepSeckMoE (k: 2, 1) on CIFAR-10 with layers and 8 experts.
0.075
0.070
0.070
0.070
Test Loss
0.085
0.065
0.060
0.060
0.060
0.050
0.050
Model
Modal
Model
Model
Model2
Model 3
0.905
0.305
0.905
Test Accuracy
0.975
0.975
0.975
0.970
0.370
Model
Modal
Modul2
Figure Linear Mode Connectivity for ViT-DeepSeekMoE 21k-CIFAR-10 10 with 12 layers and 4 experts.
8 = 1) on ImageNet-
71

---

0.075
Test Loss
0.070
0,070
0.065
0.060
0.065
0.055
0.055
Model
Model
Model
Model
Model
Model
0.585
0.985
Tast Accuracy
0.900
0.980
0.975
0.975
0.075
0.900
0.970
0.965
0.965
Model 2
Figure 93: Linear Mode Connectivity for ViT-DeepSeekMoE (A: 2Ik-CIFA--10 with 12 layers and 8 experts.
2, = 1) on ImageNet-
0.070
0.070
Test Loss
0.065
0.065
0.060
0.060
0.055
0.055
0.055
0.050
0.050
0.905
0.385
0.985
Test Accuracy
0.980
0.980
0.980
0.970
0.970
0.970
0.965
0.965
0.965
Model
Model
Model
Model 3
Model2
Model 3
Figure 94: Linear Mode Connectivity for ViT-DeepSeekMoE (k 2, on ImageNet- 20K-CIFAR-100 with 12 layers and experts.
0.46
0.60
0.00
Test LosS
0.04
0.55
0.50
0.45
0.45
0.40
0.00
0.36
Model
Model 2
Modal
Model
Modal
Model3 3
0.915
0.915
0.915
Test Accuracy
0.905
0.905
0.90
0.960
0.900
0.900
0.836
Modul
Figure Linear Mode Connectivity for ViT-DeepSeekMoE 11k-CIFAR-100 100 with 12 layers and 8 experts.
8 1) on ImageNet-
72

---

G.3 Linear Mode Connectivity Analysis: All Layer
Table 7: Loss and accuracy barriers for weight matching interpolation versus naive interpolation, alongside test loss and accuracy values, across MoF, SMoFt (k = 2), and DeepSeekMoF (I: = 2, 1) variants of VIT models on MNIST, CIFAR-10/100 datasets, and of GPT-2 on the One Billion Word dataset. Barriers (lower is better J) are averaged over pairs of models from a pool of 4 checkpoints, while metric values are averaged over the model checkpoints. All metries are scaled up by 10 to improve readability.
MoE variont
Dataset
No.
No. experts
WM Loss
Naive Lass Barrier
Loss value
WM Anc.
Naive Acc.
Acc. value
layers
Barrier
Barrier 1
Bartie
MoE
MNIST
1,20 = 0.32
11.62 i 2.44
10.20 +: 2.04
2.12 0.21
25.04 + 3.75
97.32 = 0.73
1,31 0.21
11.77
3.29
8.45 2.53
2.34 + 0.30
26.62 + 4.22
97.01
1.01
CIFAR-10
4,24 0.72
37.24
3.25
95.06 1: 0.42
1.52 1: 0.35
14.21 =: 1.2
66.57
0.89
4,52 = 0.42
36.66
4.01
95.44 1: 0.84
1.74 0.50
16.01 = 1.02
65.93 = 1.20
5,42 = 0.74
47.47 8.02
90.25 + 195
2:21 0.52
24,02 1: 2,66
74.61 2,13
91.31
22.45 3.21
74.49
CIFAR-100
2,43
23.33
4.01
One Billion
56.82 3.51
400.11 50.54
348.60 0.05
Word
68.76 4.455
435.65
34.92
34.57 0.04
78.44 3,96
476.65
29.58 455.82
342,44 11444+00.00 0.03
93.23 8.84
38.10
341.29 d: 0.32
SMoR (8- 2)
MNIST
1,55 + 0.23
18.26 +
11.21 + 2.02
2.14 0.31
24.92 + 3,56
97.22 0.63
1.62 =: 0.54
19.19
2.27
10.09 : 1.03
2.30 a 0.34
26.24
4.41
97.06 1.00
CIFAR-10
4,44 =: 0.52
37.44 3.35
95.10 :: 0.44
1.50 0.37
14:33 :: 1.32
66.62
2:62::000 0.69
4.61 0.46
36.56 3.87
95.74 + 0.92
1.64 + 0.54
17.04 + 1.32
65.97 = 1.27
5.57 0.84
47.75 8.12
90.33 1: 1.85
2.19 0.42
24.12 2:44
74.63
2.03
5.26
46.61 5.36
91.35 + 2.14
2.31 0:11
23,66 + 4.21
74.51 1.25
CIFAR-100
0.9
75.35
2.67
24.39
0.13
75.72
One Billion
72.31 0.28
$22.22 41.40
0.00
Word
79.86 4.81
618.23 67.93
342.31 0.11
98.68 18.59
492. 42.23
340.57 0:05
DeepSeckMMEE
MNIST
1,35 0.64
17.46 2.42
12.32 à: 2.02
2.31 0.41
25.12 a 3.85
97.42 0.93
1.52 0.62
16.72 +:
11.27 + 3.24
2.04 0.32
27,02
+ 4.36
97.17
1.31
(R - 2)
CI
4,22 = 0.42
38.64 3.44
95.11 + 0.33
1.66 + 0.45
17.31 = 1.63
66.77
1.04
4.61 : 0.52
38.86 1 1.11
95.36 1: 0.77
1.68 1.522
16.92 1 1.15
65.99
1.11
5.44 0.92
$2.27 + 5.32
V0.05 + 1.64
2.12 + 0.6
26.11 + 2.72
74.63
2.33
5,47 = 1.22
49.34 6:24
91.21 2.12
2.22 + 0.63
25.25 4.21
74.88 = 1.25
CIFAR-100
2,43 = 0.50
1.3.1.5.66 23.23
95.22 3:15
0.23 1 0.07
3,02 0.61
75.36 = 1.73
2,72 0.71
24.55
75.76
Billion
64.06 4.06
58.07
Word
28.63
0.11
72.94 2.12
623.63
G.4 Expert Matching Method
CIFAR-10
CIFAR-10
CIFAR-100
CIFAR-100
1.0
Test Accuracy
0.810
0.8.
0.960
0.6
0.573
0.4
0.902
Daper:
Gasia
0.874
Modiel
Modill
Modal
Modal
Model
Modal
Modal
Moda12
Figure 96: Accuracy curves for 12-layer ViT-MoE models with a 4-expert MoE replacement in either the first layer (subplots 1, 3) or the last layer (subplots 2, 4), on CIFAR-10 10 and CIFAR- 100. The curves compare two Expert Order Matching methods across 24 permutations, with Weight Matching applied post-reordering to all permutations. Corresponding loss metrics are presented in Figure 2
73

---

Table 8: Comparison of expert matching methods (Expert Weight Matching and Gate Weight Matching) across three ViT-MoE model variants evaluated on the CIFAR-10 and CIFAR-100 test sets, measuring loss. Metrics reported include rank and L, defined in Section 6.3, computed across 24 permutations for 10 checkpoint pairs. All models consist of layers and experts.
Layer replaced
Expert Weight Matching
Gate Weight Matching
Method
Dataset
Rank
1
Rank
MoE
CIFAR- 10
1
2.50 + 1.50
2.12 0.42
3.00 + 1.00
3,42 + 0.55
4
2.10 + 0.50
1.04 + 0.32
2.60 + 0.92
1.54 + 0.44
8
2.70 + 0.46
0.60 0.27
2.90 0.30
0.74 + 0.17
12
4.60 + 2.00
0.13 0.05
3.80 + 1.66
0.09 + 0.03
CIFAR-100
1
2.80 0.40
3.17 0.25
0.70
2.73 + 1.03
3.60 1.20
1.15 0.55
2.70
2.03 0.93
8
3.30 + 0.78
0.67 = 0.14
3.20 0.87
1.13 = 0.55
12
3,40 + 0.92
0.07 0.03
4.20 + 0.89
0.11 + 0.04
SMoE (k: =
CIFAR-10
1
3.00 + 1.00
3.80 1.18
3.20 + 0.98
3.46 + 2.94
4
2.80 + 0.98
1,73 0.49
2.40 + 1.56
1.64 + 0.86
8
2.60 + 0.49
0.40 + 0.46
2.60 + 0.43
0.36 + 0.83
12
4.00 + 2.45
0.06 0.04
2.80 + 1.66
0.22 + 0.19
CIFAR-100
1
2.80 0.40
3,29 - 3.18
2.10
2.00 + 2.02
4
2.60 + 1.20
1.20 0.43
3.20
1.91 + 1.05
8
3.10 + 0.83
0.13 0.07
0.90
0.11 0.09
12
2.40 + 0.92
0.07 = 0.13
2.00 + 0.89
0.03 + 0.12
DeepSeekMoF
CIFAR-10
1
3.10 + 2.12
5.06 1.22
3.60 + 0.83
4.28 + 2.92
(k: 2,
4
2.60 + 0.77
3.32 0.38
3.10 1.52
2.80 + 0.88
8
2.30 4 0.51
0.39 0.48
2.40 0.37
0.78 1.00
12
3.60 3.55
0.08 + 0.04
3.60 + 1.48
0.23 + 0.17
CIFAR-100
1
4.20 + 0.60
4.45 -
3.70
4.21 + 2.60
4
2,50 + 0.78
3.10 1.34 0.46
2.98 + 1.46
8
1.90 + 0.60
0.13 0.08
3.10
0.13 + 0.16
12
3.30 + 1.26
0.04 0.11
2.70
0.02 0.20
Table 10: Loss barrier comparison between Total Weight Matching and
Matching across MoE variants the One Billion Words dataset. Each value is averaged over independent checkpoint pairs. The last column reports the interpolated loss value at the midpoint between matched models.
Loss Barrier L
MoEl Variant
No. Layers
No. Experts
Loss Value
Total Weight Matching
Skipping Expent Onder Matching
12
4
0.0053 + 0.0006
0.1384
0.0960
3.4399 0.0002
Moli
8
0.0066 0.0008
0.1728 0.1144
3.4394 0.0001
0.8579 1.1412
3.4403
SMoE (k
8
0.0051 + 0.0005
0.2202
0.1477
3,4400
0.0000
12
4
0.0047 1 0.0001
0.3223
0.3817
3,4395 0.0003
DeepSeckMoE
8
0.0044 + 0.0004
0.4005
0.3666
3,4395 0.0002
74

---

Table 9: Comparison of expert matching methods (Expert Weight Matching and Gate Weight Matching) across three ViT-MoE model variants evaluated on the CIFAR-10 and CIFAR-100 test sets, measuring accuracy. Metries reported include rank and î., defined in Section 6.3, computed across 24 permutations for 10 checkpoint pairs. All models consist of 12 layers and experts.
Layer replaced
Expert Weight Matching
Gate Weight Matching
Method
Dataset
Rank |
Ż |
Rank
MoE
CIFAR-10
1
4.90 + 2.59
1.80 = 0.78
4.40 + 1.80
2.00 + 0.44
3.20 + 0.40
0.95 0.97
2.90 0.70
1.64 0.98
8
3,00 + 1.80
0.03 = 0.05
3.80 + 1.83
0.05 + 0.05
12
4.00 + 1.45
0.02 0.03
2.80 - 1.40
0.04 + 0.06
CIFAR-100
1
3.20 + 1.47
2.66 1.03
4.20 = 1.00
1.51 + 1.16
2.70 0.78
1.87 1.17
2.90 + 0.83
1.39 1.06
8
4.70 + 1.42
0.08 = 0.02
3.80 + 1.47
0.05 + 0.03
12
2,40 + 0.92
0.05 0.02
3.20 + 0.87
0.02 + 0.02
SMoE (A: =
CIFAR-10
1
2.30 2.06
3.09 1.21
3.20 2.75
2.24 + 0.90
2.90 0.59
1.16 1.00
2.80 0.83
1,75 + 1.00
8
3.40 + 1.47
0.07 + 0.03
3.00 + 1.74
0.04 + 0.02
12
3.60 + 1.21
0.12 0.08
4.70 2.05
0.07 + 0.03
CIFAR-100
1
2.10 +: 1.85
3.52 2.26
2.40 + 0.93
2.05 + 2.95
4
2.00 + 0.67
1.81 1.28
2.90 + 0.73
1.02 + 2.41
8
4.30 + 1.60
0.09 + 0.03
3.10 + 1.02
0.11 + 0.08
12
2.60 + 1.03
0.11 0.08
2.70 0.89
0.03 0.06
DeepScekMoE
CIFAR-10
1
2.80 + 2.20
2.95 1.15
3.30 + 2.40
2.60 + 0.85
(k: = 2=
4
3.10 4 0.80
1.40 1.13
2.90 0.90
1.60 + 1.30
8
3.60 + 1.32
0.09 + 0.04
3.40 + 1.10
0.07 + 0.03
12
3.20 1.00
0.06 0.02
3.00 0.87
0.05 0.02
CIFAR-100
1
2.54 + 1.90
3.20 2.00
2.80 2.00
2.90 1.80
4
2.90 + 0.70
1.60 + 1.20
1.10
1.40 + 1.00
8
3.80 + 1.43
0.10 0.03
3.50
0.08 + 0.04
12
2.70
0.90
0.07 0.02
2.90
0.06 0.03
75

---

G.5 Ablation Study on Number of Layers
Table 11: Ablation Study on varying the number of Transformer layers and all MoE variants Integration in VIT for CIFAR-10. The ratios of metrics (loss barrier, loss AUC, accuracy barrier, and accuracy AUC) compare Algorithm I to naive interpolation. For full table results (non-raio), kindly refer to Tables 12 and 13
Number of layers
Layer replaced
Loss barrier ratio (%)
Loss AUC ratio (%)
Acc. harrier ratio (%: |
Acc. AUC ratio (%
Method
MoE
8.54 1.53
8.73 1.68
8.39 1.42
8.01 1.39
9.33 2.04
8.94 1.92
9.10 2.19
8.83 2.22
8.29 + 1.63
8.08 = 1.59
7.41 + 1.45
6,43 + 1.09
10.19 + 3.43
10.19 = 3.16
10.21 + 5.26
9.08 + 5.33
8.50 1,82
7.83 = 1.89
9.82 + 2.58
8.70 + 2.58
5.12 + 0.67
4.60 = 0.65
4.17 = 1.68
4.58 # 1.01
4.16 1.47
4.96 2.31
3.64 + 1.00
4.13 + 1.55
6.78 + 4.80
7.71 3.09
7.27 + 5.73
8.98 + 2.94
9.69 4.92
8.62 5.46
9.03 3.31
8.14 3.57
4
10.83 +
12.01 6.42
8.92 2.21
10.00 + 7.30
9.56 + 4.52
11.72 5.59
8.72 + 2.89
11.01 + 3.83
6,90 + 3.54
9.67 6.59
7.16 + 3.98
9.69 + 3.11
SMoE (k
8.93 0.92
9.34 1,23
9.36
8.84 1.40
7.55 1.39
1.35
9.10
7.29 2.78
8.96 + 1.54
8.83 = 1.64
8.66 + 2.07
8.21 + 1.91
2
9.62 2.68
10.07 = 3.50
9.41 + 3.04
9.67 + 3.52
13.31 + 1.49
12.44 = 2.06
11.37 + 3.01
13.63 + 3.17
9.17 + 1.40
9.36 = 1.48
10.97 + 2.77
12.52 + 2.57
2.37 1.92
2.65 1.34
2.25 - 2.71
4.67 +:: 2.38
10.09 3.11
9.26 3.19
8.35 3.37
8.36 + 2.38
11.22 6.25
10.17 8.52
12.10
12.80 10.77
8.76 + 2.06
12.16 2.22
13.09 5.49
10.85 - 7.21
14.39 7.82
14.66 9.10
11.82 = 9.19
12.45 + 7.85
6.38 5.01
9.56 3.79
11.96 + 11.93
10.93 15.06
Mot
8.94 1.95
9.12 2.10
7.96 1.59
7.88 + 1.90
2
9.25 0.37
= 0.44
12.00 1.35
11.14 1.73
4
8.65 + 1.01
8.56 1.19
7.17 + 0.99
6.46 + 0.88
2
10.80 +: 3.77
10.88 3.69
10.45 + 4.91
9.92 + 4.83
7.90 + 1.79
8.82 = 1.87
10.21 + 2.44
9.02 : 2.52
8.35 + 1.20
8.29 = 1.39
9.29 + 2.75
7.04 + 3.14
6.29 5.00
6.17 3.48
8.07 5.34
8.35 5.65
8.03 1.79
9.16 2.10
7.88 3.66
8.54 3.93
9.27 4.25
9.29 3.19
12.93 2.18
11.56 5.58
10.80 2.18
12.72 1.52
7.34
8.99 5.10
13.77 2.27
13.03 7.01
11.14 4.14 5.47
13.28 + 3.32
9.82 2.99
8.38 4.81
8.92 + 4.22
8.14 4: 3.48
76

---

Table 12: Ablation study results on CIFAR-10 for LMC in VIT with layer replacements of all MoE variants, reporting test set loss. We evaluate the loss metrics for linear interpolation using our proposed Algorithm against naive linear interpolation. Metrics include the loss barrier and loss Area Under the Curve (AUC), with AUC computed relative to the straight line connecting the two model endpoints. Each experiment is repeated five times, and LMC performed on ten model pairs. Additionally, report loss values across five models to provide enhanced context. To improve readability, all metrics are scaled up by a factor 10P.
Method
Number layers
Layer replaced
WM loss barrier
Naive loss barrier
loss
Naive loss AUC |
Loss value
AUC
MoE
2
1
3.14 +: 0.55
34,22 + 1.25
1.45 + 0.28
16.11 +: 0.74
99.04 +: 0.92
2
2.84
0.44
33.54 + 1.12
1.38 = 0.15
17.53 + 0.57
104.52
4
9.48 1.36
115.56 10.19
4.25 = 0.68
53.07 + 4.54
97.00
2
6.59 2.12
64.73 + 2.05
3.71 1.11
36.39 + 0.96
91.74 0.87
3
4.20 +: 0.90
48.23 1 0.92
2,21 = 0.53
28.23 1 0.50
88.06 + 1.30
2.76
0.22
31.01
86.15
1.42 0.44
37.47 8.47
0.74 0.30
18.73 3.30
90.35 1.15
2
0.47 0.32
7.09 + 0.60
0.27 0.10
3.53 = 0.30
88.73 0.39
3
0,33 0.17
3,42 +: 0.22
0.13 + 0.08
1.60 + 0.14
88.26 0.50
0.14
1.36 0.33
0.07 0.04
0.64 164:0.00
86.48
5
0.14 0.06
1.68 1: 0.44
0,08 = 0.03
0.3:
85.92 + 0.46
6
0.13
2.46 + 0.29
0.12 = 0.04
1.26 0.21
88.61 0.33
SMoE
2
2.95
33.17 + 1.01
1.39 = 0.16
14.95 + 0.51
98.88 0.86
(k
2
2.52 0.46
33.46 1 0.67
1.20 1 0.25
18.12 0.48
105.59 0.37
4
1
10.45
1.99
116.54 1 8.34
4.75 1.01
53.65 3.97
96.73 0.85
2
8.31 = 0.83
86.88 + 3.25
4.73 = 0.57
47.40 = 1.91
90.80 = 1.02
4.73 0.62
35.49 + 1.34
2.59 = 0.49
20.78 + 0.80
86.29 0.24
4
4.48 0.70
48.84 + 0.68
2.68 0.44
28.62 0.44
82.33 0.37
6
1
2.12 0.67
34.23 + 7.22
1.28 = 0.35
12.59 + 2.67
90.25 = 1.00
2
0.75 1:: 0.29
7.28 1 0.83
0.34 0.15
3.59 1: 0.43
88.99 = 0.65
0.34
3.28
0.06
1.16
87.69 0.52
4
0.14
2.78 0.38
0.14 0.07
1.19
86.08 0.40
5
0.11
2.50 1: 0.04
0.11 + 0.05
1.25 11 0.02
86.08 0.34
0.13
3.63 1 0.17
0.13 0.08
1.93:00010
88.38 0.36
DeepScekMoE
2
0.60
33.63 + 1.30
1.37 0.27
15.14
98.41 1.06
(k = 22= 2,
2
3,07 0.21
33.16 + 1.26
1.50 = 0.11
18.18 + 0.72
105.92 + 0.53
4
1
10.57 + 1.57
122.13 10.50
4,91 + 0.84
57.32 + 4.89
97.71 + 1.26
5.85 2.14
54.13 1: 2.27
2.90 = 1.26
30.17 + 1.35
90.89
3
2.67 0.65
33.36 1 1.47
1.35 1 0.40
19.48 11 0.89
86.37 0.43
4
4.11 0.61
49.18 + 0.95
2.37 = 0.41
28.49 + 0.55
82.26 = 0.39
6
1
2.25 + 0.82
39.42 + 6.23
1.14 = 0.38
18.88 + 3.32
91.32 + 0.25
1: 0.57
28.53 1.26
0.32
1:
89.50
3
0.56 0.12
7.90 + 0.18
0.26 = 0.06
3.90
88.22 0.57
4
0.16
3.07 1 0.34
0.12 = 0.03
1.53 0.14
86.09 1 0.40
0.32 0.17
3.39 0.31
0.16 0.08
1.63 +
86.10 0.51
6
0.23 + 0.08
2.61 + 0.53
0.14 0.04
1.43 +
89.27 + 0.57
77

---

Table 13: Ablation study results on CIFAR-10 for LMC in VIT with layer replacements of all MoE variants, reporting test set accuracy. We evaluate the accuracy metrics for linear interpolation using our proposed Algorithm against naive linear interpolation. Metrics include the accuracy barrier and accuracy Area Under the Curve (AUC), with AUC computed relative to the straight line connecting the two model endpoints. Each experiment is repeated five times, and LMC performed on ten model pairs. Additionally, report accuracy values across five models to provide enhanced context. To improve readability, metrics are scaled up by a factor of 10².
Number
Layer replaced
WM acc. barrier L
Naive acc. harrier J
WM acc. AUC
Naive acc. barrier |
Method
Acc. value
of layers
MoE
0.91 + 0.28
12.33 1.28
0.44 0.15
5.89 0.54
66.53 3.30
2
0.76 +: 0.21
7.85 + 2.31
0.33 0.08
3.77 + 1.11
63.65 0.47
1
2.10 = 0.42
28.36 1.85
0.86 0.16
13.43 0.99
72.48 0.68
2
1.41 = 0.71
13.83 = 0.53
0.75 0.41
7.88 = 0.29
73.43 +: 0.61
3
0.67 + 0.17
6.84 + 0.24
0.35 = 0.09
4.00 = 0.12
73.64 + 0.20
4
0.23 = 0.09
5.48 + 0.19
0.14 0.03
3.05 = 0.14
73.58 0.19
1
0.50 1 0.13
13.95 1: 2.20
0.24 = 0.07
5.71 1 0.93
74.29 0.35
0.35 + 0.10
2.49 + 0.36
0.12 = 0.04
1.23 0.21
74.65
3
0.14 0.05
1.48 0.28
0.05 0.02
0.72
74.72 0.46
4
0.05 = 0.08
1.33
0.03 0.05
0
74.46 1.4600.112
5
0.05 = 0.03
0.59 = 0.05
0.02 = 0.01
0.22
74.62 2 0.08
6
0.06 + 0.04
0.61 + 0.12
0.03 0.02
0.30 0.05
74.25 0.21
SMoB
1
1.07 = 0.15
11.42 + 0.22
0.46 = 0.07
5.20
65.27 +: 0.69
(A: 2)
2
0.58 + 0.17
6.41 + 0.24
0.26 0.10
3.63 0.13
62.58 0.16
1
2.48 1 0.66
28.55 1: 1.55
1.12 - 0.29
13.52 1 0.81
72,26 0.60
1.78 0.23
0.71
0.97 0.15
0.36
73.12 0.40
3
0.87 = 0.17
4.98 0.32
0.38 0.10
2.78 0.22
73.58 + 0.33
4
0.79 +: 0.14
5.29 1: 0.31
0.40 0.09
3.15 = 0.15
73.60 4: 0.44
0.30 + 0.38
12.61 + 2.27
0.28 0.11
5.11 0.86
74.09 + 0.28
2
0.29 1 0.16
2.85 1 0.37
0.13 0.08
1,48 1 0.22
74.81 1. 0.47
3
0.18 1 0.14
1.36 0.18
0.10 0.06
0.95 + 0.10
74.97 0,30
4
0.18 0.07
1.35 - 0.05
0.08 = 0.02
0.76 0.03
74.60
5
0.19 0.09
1.45 + 0.08
0.09 0.04
0.77 0.05
74.72 + 0.24
6
0.19 = 0.08
1.60 + 0.14
0.08 0.02
0.89 = 0.09
74.15 0.28
DeepScekMoE
1
0.92 0.18
11.64 + 0.48
0.42 0.10
5.29 0.25
65.43 0.30
(k = 2, 1)
2
0.78 + 0.11
6.45 = 0.31
0.41 = 0.08
3.67 = 0.21
62.55 + 0.27
4
1
2.07 + 0.31
28.85 + 1.64
0.90 0.13
13.93 0.91
71.57 + 0.84
2
1.77 +: 0.56
12.14 1 0.50
0.67 0.32
6.74 0.24
73.70 +: 0.14
3
1.01 1 0.14
9.80 11 0.28
0.55 0.08
4.91 1 0.17
73.72 1 0.20
4
0.52 + 0.15
5.60 = 0.14
0.23 = 0.10
3.22 0.11
73.89
2.43 + 0.73
30.80 + 2.43
0.99 0.31
14.06 + 0.90
74.26 +: 0.19
0.95 0.19
12.63 0.43
0.47 0.10
7.37 0.18
0.18
74.51 0.14
3
0.39 0.13
4.85 0.14
0.14
0.20 0.06
2.44 0.09
74.98 0.199
4
0.14 0.07
2.29 1 0.08
0.10 0.04
74.70 1.13 0.03
5
0.12 + 0.03
1.39 0:08
0.08 = 0.02
0.66 0.04
74.62 0.14
0.16 + 0.07
1.74 + 0.12
0.06 = 0.04
0.88 - 0.05
74.31 0.49
78

---

H
Broader Impact
The investigation into Linear Mode Connectivity (LMC) within Mixture-of-Experts (MoE) architec- tures presents both potential positive and negative societal implications. On the positive side, this work could lead to more efficient training methods for MoE models, enhancing their scalability and computational efficiency. Such advancements may democratize access to advanced AI technologies, enabling researchers and developers with limited resources leverage powerful models for innova- tive applications. Furthermore, the insights gained into the optimization dynamics and functional landscape of neural networks could contribute to the development of models that generalize more effectively, thereby improving the reliability and robustness of AI systems in critical domains such as healthcare, finance, and autonomous systems. Additionally, this research enriches the broader AI community's 's understanding of neural network loss landscapes, fostering further theoretical and practical advancements.
It is important to note that while these potential impacts are significant, the primary contribution of this work lies in its theoretical and foundational insights. The actual societal ramifications will largely depend on how these insights are applied and governed in practical settings. As such, responsible development and deployment practices, coupled with ongoing research into bias mitigation and ethical AI, are essential to maximizing the benefits and minimizing the risks associated with advancements MoE architectures.
79

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are clearly stated in the Contribution the Introduction. We provide mathematical contexts Section 2, which provides background
LMC and MoE architectures. We introduce the concept of the weight space MoE architectures and define a group action on this space that preserves the functional behavior of MoE models, aligning with the claim of defining such space and action in Section We present two core results concerning functional equivalence in MoE models, demonstrating that the proposed group action characterizes all inherent symmetries of the MoE gating mechanism in Section 4. We develop a Weight Matching algorithm that enables alignment between independently trained MoEs inSection 5. We provide empirical evidence LMC across a wide range MoE configurations and additional experiments to support our work in Section 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
80

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results in the paper are given together with the full set of assumptions and complete/correct proofs (See Sections 3, 4, and Appendices A, B, C, D in our manuscript).
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the experiment details in the Experiment Details Section (Ap- pendix F) the Appendix of our manuscript. We also provide the source code that the results in the paper can be easily reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
81

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the source code in the Supplemental Materials so that the results in the paper can be easily reproduced. We verify our proposed methods using public benchmarks (See the Experimental Results Section, i.e., Section in our manuscript)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ mublic/guides/
sttg:idddee//oeeeumiisssssss wwwriimmisssoooololoooo mmPliccy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwtppcccccgguueecueeccccceccccpt http///ieee//eee// httee//ooSuuummisseeeeee
wwiniiissiiiiiciiiii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We specify all the training and test details necessary to understand the results in the Experimental Results Section (Section 6) in the maintext and the Experiment Details Section (Appendix F) in the Appendix of our manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
82

---

7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report enorr bars suitably and correctly defined the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula. call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources for all experi- ments in our Experimental Results Section (Section 6) and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics https: h/tpps:://////ss////// http://uuurrppsccppppplee
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
83

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix H.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
84

---

Answer: [Yes]
Justification: We cite the githubs we use and the baselines we compare with in our manuscript. All the assets used in the paper properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include details about training and implementation Appendix F, and code in supplementary materials. The new assets provided in the paper are well documented, and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
85

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The methodological contributions of this research does NOT rely LLMs in any any important, original, or non-standard way.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ /tttrr//eneur...eee cc/Confferences/2225 for what should or should not be described.
86


Reviews:
[{"review_id": "ZhCM4LnVcO", "reviewer": "Reviewer_tznV", "summary": "In this work, the authors mathematically characterize the symmetries of Mixture-of-Expert (MoE)\nand Mixture-of-Experts with sparse gating (SMoE) layers. When considering the MoE layers in\nisolation, the authors prove that only two types of parameter symmetries are introduced by these\nlayers: permutation of experts and shifting of gating parameters. Informed by their theoretical\nresults, they propose an algorithm to best match two networks trained from different initial values\nto align according to the studied symmetries, a procedure akin to modern linear mode\nconnectivity (LMC) results. The authors claim to be the first to report the presence of LMC\nbetween MoE layers in various setups.", "questions": "Curious to see if more symmetries may arise when considering interactions between MoE layers and other layers. Can the authors speculate on this?\n\nDid the authors try to align two networks trained from scratch with their current methods? Can LMC be found?", "limitations": "See weaknesses above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThis work provides the first complete mathematical characterization of parameter symmetries in MoE and SMoE layers. While permutation and gating shift symmetries have been observed previously, the authors' proof that these constitute the only symmetries present is a non-trivial theoretical contribution. \n\nThe mathematical framework is well-developed, and the proofs are clearly presented.\n\nDespite showing experiments only on fine-tuned MoE (and not alignment on networks trained entirely from scratch, which can be costly), the experimental evidence is extensive and provides clear support for the MoE-centered theory introduced.\n\nThe literature review section is well-exposed, complete, and thorough.\n\n**Weaknesses**\n\nThe authors should make it clearer that the symmetries they prove to be the only ones present are proved without taking into consideration the rest of the layers in the network. \n\nWhile I doubt that other downstream or upstream layers can introduce more symmetries (aside from trivial linear\nones), the authors do not prove their non-existence. This point should be discussed in a limitations paragraph. Another point to add to the limitation paragraph is that experiments were conducted only by replacing some layers with MoE and only fine-tuning MoE layers, non-attentive readers may miss this point.\n\nThe description of the experimental setup could use some more detailed explanations. I would suggest using the extra page to improve the limitations and experimental setup sections.", "was_revised": true, "final_justification": "This work makes a solid theoretical contribution, supported by adequate experimental validation. While the analysis is limited to isolated MoE layers and fine-tuning scenarios, the mathematical rigor and comprehensive experiments justify acceptance. The author's rebuttal provided helpful clarifications that addressed my concerns.", "rebuttal": "We address the concerns raised by the Reviewer in the Weaknesses and Questions sections as follows.\n\n---\n\n**Response to Weaknesses.** \n\n*Theoretical Aspect.* In this work, we focus exclusively on analyzing the symmetry structure within the Mixture-of-Experts (MoE) architecture. We do not address potential symmetries arising from components outside the MoE layers. Modern deep learning models are composed of heterogeneous stacks of layers, making the characterization of global symmetry across such architectures highly challenging. While we believe we have clearly specified the exact structure of the MoE architecture under consideration - namely, as defined in Equations (2) and (3) - we acknowledge that inattentive readers may misinterpret our scope as encompassing all possible symmetries in the full network. As suggested by the Reviewer, we will revise the manuscript to clearly articulate the scope of our analysis and avoid any potential misunderstanding about the extent of our theoretical contribution.\n\n*Experimental Aspect.* We limited our experiments to fine-tuning the MoE layers due to the significant complexity of identifying Linear Mode Connectivity (LMC) across the full model. This would require understanding the symmetries of preceding MHA layers and developing alignment algorithms for such composite structures. While some recent works have explored MHA symmetries, no general alignment method currently exists. Given the scope of this submission, we focused on MoE alignment and leave broader alignment strategies for future work.\n\nRegarding the experimental setup, in addition to the details and the experimental results provided in Appendices E, F, and G, we will include further clarifications on the setup in the revised version, as suggested by the Reviewer.\n\n**Q1. Curious to see if more symmetries ... on this?**\n\n**Answer Q1.** Yes, additional symmetries may emerge from interactions between MoE layers and preceding components, such as MHA layers. For example, the attention mechanism admits symmetries under the general linear group: given query and key matrices $Q$ and $K$, the expression $QK^\\top$ is invariant under the transformation $Q \\mapsto QA^\\top$ and $K \\mapsto KA^{-1}$ for any invertible matrix $A$. Extending symmetry alignment to such composite structures would require a more general algorithm.\n\nTo speculate further on interactions between MoE layers and other layers, we provide an intuitive perspective on how symmetries can emerge when composing two general layers. Consider two parameterized layers, $f(\\cdot;\\alpha) : \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_1}$ and $g(\\cdot;\\beta) : \\mathbb{R}^{d_1} \\to \\mathbb{R}^{d_2}$, which we compose as $g \\circ f: \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_2}$. In practice, these layers may admit certain forms of symmetry. Specifically, let $P$ be a $d_1 \\times d_1$ permutation matrix, representing a reordering of the indices of vectors in $\\mathbb{R}^{d_1}$. Suppose there exist modified parameters $\\alpha'$ and $\\beta'$ such that $f(\\cdot;\\alpha') = P \\circ f(\\cdot;\\alpha)$ and $g(\\cdot;\\beta') = g(\\cdot;\\beta) \\circ P^{-1}$. Then, the composite function remains invariant: $g(\\cdot;\\beta) \\circ f(\\cdot;\\alpha) = g(\\cdot;\\beta') \\circ f(\\cdot;\\alpha')$. Typically, $\\alpha'$ corresponds to permuting the coefficients of $\\alpha$ according to $P$, and similarly for $\\beta'$. For intuition, one may think of $f$ and $g$ as two feedforward layers, and $P$ as a permutation of neurons at the output of $f$ and the input of $g$. Depending on the choice of $f$ and $g$, the transformation $P$ may extend beyond permutations to generalized permutation, orthogonal, or even arbitrary invertible matrices. This reflects a broader class of symmetries that can arise when stacking layers. While we believe these capture many relevant cases, a full understanding would require a general and rigorous framework, an important direction for future research on symmetry in deep learning architectures.\n\n**Q2. Did the authors try to ... Can LMC be found?**\n\n**Answer Q2.** As in **Q1**, aligning two networks trained from scratch requires handling symmetries across the entire architecture, including interactions between MoE layers and components like MHA. Our decision to focus solely on the MoE layer (as defined in Equations (2), (3)) is driven by the substantial complexity of identifying LMC across the full model. This task would require a detailed understanding of the symmetry structure of the MHA layers that typically precede MoE blocks, along with the development of a general alignment algorithm capable of handling such composite modules. While some recent works have explored MHA symmetries, to the best of our knowledge, no general method exists for aligning stacked components. We experimented with approaches for aligning composite (Attention + MoE) layers, but due to time constraints, their empirical performance was limited. As noted in **Q1**, additional nontrivial symmetries may arise when stacking layers across the full model. Given the scope of this submission, we focused on MoE alignment and leave model-wide symmetry and LMC for future work.\n\nTo address this further, we believe that investigating LMC in models where MoE layers are initialized from scratch across all layers - rather than only the first layer as in our current experiments - would be more meaningful, as it would strengthen and provide additional context to our findings. We have conducted this experiment, and the results are presented in the following table.\n\n---\n\n*(Due to the new policy, we cannot provide the corresponding figures.)*\n\n**Table 1.** Loss and accuracy barriers for weight matching interpolation versus naive interpolation, alongside test loss and accuracy values, across MoE, SMoE ($k=2$), and DeepSeekMoE ($k=2,s=1$) variants of ViT models on MNIST, CIFAR-10/100 datasets, and of GPT-2 on the One Billion Word dataset. Barriers (lower is better ↓) are averaged over 6 pairs of models from a pool of 4 checkpoints, while metric values are averaged over the 4 model checkpoints. All metrics are scaled up by $10^2$ to improve readability.\n\n|MoE variant|Dataset|No. layers|No. experts|Weight Matching Loss Barrier ↓|Naive Loss Barrier ↓|Loss value ↓|Weight Matching Accuracy Barrier ↓|Naive Accuracy Barrier ↓|Accuracy value ↑|\n|-|-|-|-|-|-|-|-|-|-|\n|MoE|MNIST|2|2|1.20 ± 0.32|11.62 ± 2.44|10.20 ± 2.04|2.12 ± 0.21|25.04 ± 3.75|97.32 ± 0.73|\n||||4|1.31 ± 0.21|11.77 ± 3.29|8.45 ± 2.53|2.34 ± 0.30|26.62 ± 4.22|97.01 ± 1.01|\n||CIFAR-10|2|4|4.24 ± 0.72|37.24 ± 3.25|95.06 ± 0.42|1.52 ± 0.35|14.21 ± 1.42|66.57 ± 0.89|\n||||8|4.52 ± 0.42|36.66 ± 4.01|95.44 ± 0.84|1.74 ± 0.50|16.01 ± 1.02|65.93 ± 1.20|\n|||6|4|5.42 ± 0.74|47.47 ± 8.02|90.25 ± 1.95|2.21 ± 0.52|24.02 ± 2.66|74.61 ± 2.13|\n||||8|5.27 ± 1.26|46.47 ± 5.44|91.31 ± 2.32|2.44 ± 0.43|22.45 ± 3.21|74.49 ± 1.35|\n||CIFAR-100|6|4|2.32 ± 0.51|20.11 ± 4.34|94.04 ± 2.02|0.21 ± 0.07|3.22 ± 0.71|75.25 ± 1.22|\n||||8|2.43 ± 0.62|23.33 ± 3.33|93.43 ± 1.15|0.27 ± 0.12|4.01 ± 1.21|75.66 ± 1.30|\n||One Billion Word|12|2|56.82 ± 3.51|400.11 ± 50.54|348.60 ± 0.05|\n||||4|68.76 ± 4.45|435.65 ± 34.92|344.57 ± 0.04|\n||||6|78.44 ± 3.96|476.65 ± 29.58|342.44 ± 0.03|\n||||8|93.23 ± 8.84|455.82 ± 38.10|341.29 ± 0.32|\n|SMoE ($k=2$)|MNIST|2|2|1.55 ± 0.23|18.26 ± 2.32|11.21 ± 2.02|2.14 ± 0.31|24.92 ± 3.56|97.22 ± 0.63|\n||||4|1.62 ± 0.54|19.19 ± 2.27|10.09 ± 1.03|2.30 ± 0.34|26.24 ± 4.41|97.06 ± 1.00|\n||CIFAR-10|2|4|4.44 ± 0.52|37.44 ± 3.35|95.10 ± 0.44|1.50 ± 0.37|14.33 ± 1.32|66.62 ± 0.69|\n||||8|4.61 ± 0.46|36.56 ± 3.87|95.74 ± 0.92|1.64 ± 0.54|17.04 ± 1.32|65.97 ± 1.27|\n|||6|4|5.57 ± 0.84|47.75 ± 8.12|90.33 ± 1.85|2.19 ± 0.42|24.12 ± 2.44|74.63 ± 2.03|\n||||8|5.26 ± 1.32|46.61 ± 5.36|91.35 ± 2.14|2.31 ± 0.41|23.66 ± 4.21|74.51 ± 1.25|\n||CIFAR-100|6|4|2.24 ± 0.51|21.25 ± 2.05|94.24 ± 2.13|0.22 ± 0.08|3.36 ± 0.91|75.35 ± 1.11|\n||||8|2.67 ± 0.52|24.39 ± 1.44|93.83 ± 4.25|0.23 ± 0.13|4.12 ± 1.02|75.72 ± 1.20|\n||One Billion Word|12|4|72.31 ± 0.28|522.22 ± 41.40|345.02 ± 0.00|\n||||8|79.86 ± 4.81|618.23 ± 67.93|342.31 ± 0.11|\n||||16|98.68 ± 18.59|492.16 ± 42.23|340.57 ± 0.05|\n|DeepSeekMoE ($k=2, s=1$)|MNIST|2|2|1.35 ± 0.64|17.46 ± 2.42|12.32 ± 2.02|2.31 ± 0.41|25.12 ± 3.85|97.42 ± 0.93|\n||||4|1.52 ± 0.62|16.72 ± 2.33|11.27 ± 3.24|2.04 ± 0.32|27.02 ± 4.36|97.17 ± 1.31|\n||CIFAR-10|2|4|4.22 ± 0.42|38.64 ± 3.44|95.11 ± 0.33|1.66 ± 0.45|17.31 ± 1.63|66.77 ± 1.04|\n||||8|4.64 ± 0.52|38.86 ± 4.11|95.36 ± 0.77|1.68 ± 0.52|16.92 ± 1.15|65.99 ± 1.11|\n|||6|4|5.44 ± 0.92|52.27 ± 5.32|90.05 ± 1.64|2.12 ± 0.62|26.11 ± 2.72|74.63 ± 2.33|\n||||8|5.47 ± 1.22|49.34 ± 6.24|91.21 ± 2.12|2.22 ± 0.63|25.25 ± 4.21|74.88 ± 1.25|\n||CIFAR-100|6|4|2.43 ± 0.50|23.23 ± 5.62|95.22 ± 3.15|0.23 ± 0.07|3.02 ± 0.61|75.36 ± 1.73|\n||||8|2.72 ± 0.71|24.55 ± 4.07|93.89 ± 2.17|0.24 ± 0.11|4.00 ± 1.02|75.76 ± 1.21|\n||One Billion Word|12|4|64.06 ± 4.06|426.11 ± 58.07|343.27 ± 0.05|\n||||8|117.42 ± 37.94|390.59 ± 28.63|340.21 ± 0.11|\n||||16|72.94 ± 2.12|623.63 ± 147.42|338.25 ± 0.21|\n\nWhen replacing and initializing MoEs across all layers, as opposed to our current configuration of substituting only the first layer, the naive interpolation barrier increases substantially across all experimental settings. In contrast, the weight matching barrier exhibits a far more modest rise, thereby preserving the relative percentage improvement over the naive barrier - or even enhancing it slightly in certain cases - and underscoring the efficacy of our proposed method. Overall, these findings highlight the robust scalability of our weight matching approach when applied to deeper MoE replacements.\n\n---\n\nWe thank the Reviewer for the constructive feedback and thoughtful suggestions. We will incorporate the proposed clarifications to improve the clarity and precision of our work. If our responses adequately address the concerns, we kindly hope that the evaluation may be adjusted to reflect this. We remain open to further discussion during the next stage of discussion."}, {"review_id": "vgQMamGQxE", "reviewer": "Reviewer_HzV1", "summary": "This work introduces the investigation of Linear Mode Connectivity for Mixture-of-Experts based architectures. They propose group actions which account for all symmetries of the Router Network parameters and show that permutation invariance is sufficient to induce LMC. Based on these insights they develop a weight matching to align independently trained MoEs which they empirically evaluate extensively.", "questions": "See weaknesses.", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Extension of Weight Matching Algorithm to MoE bases architectures\n- Theoretical analysis and thorough empirical evaluation.\n\n\nWeaknesses:\n- Since Linear Mode Connectivty is already given early in pre-training [1], and this work only exchanges the FFN layer/s with randomly initialized MoEs, it is somewhat intuitive that LMC is given. How does this change when Transformer-based MoEs get initialized from scratch (all layers)?\n- Since Mixture-of-Experts based architectures may vary the number of experts, how could aligning two independently trained networks, that started from the same pre-trained transformer. and got extended to MoEs, look like?\n\n[1] Frankle, Jonathan, et al. \"Linear mode connectivity and the lottery ticket hypothesis.\" International Conference on Machine Learning. PMLR, 2020.", "was_revised": true, "final_justification": "This work extends the analysis of linear-mode connectivity on the well-established MoE architecture. \nThe work does so by thorough theoretical analysis as well as empirical validation. I recommend accepting this paper.", "rebuttal": "We address the concerns raised by the Reviewer in the Weaknesses and Questions sections as follows.\n\n---\n\n**W1. Since Linear Mode Connectivty is already given ... get initialized from scratch (all layers)?**\n\n**Answer W1.** There are several key points we would like to address in response to the Reviewer’s comments, as outlined below:\n\n(i) **Existence of LMC in MoE.** \n\nWe appreciate the Reviewer’s suggestion that, given the existence of Linear Mode Connectivity (LMC) in feedforward networks (FFNs) as shown in [1], one might intuitively expect similar behavior in MoE architectures. However, we contend that MoE models introduce fundamentally different challenges. In particular, the presence of a non-linear softmax gating mechanism sets MoEs apart from classical FFNs. This gating leads to conditional computation and sparse expert activation, resulting in highly non-uniform training signals and expert specialization. Consequently, the optimization landscape becomes significantly more fragmented, making alignment across models substantially more difficult. From the perspective of symmetry, this gating component introduces additional operators that give rise to new types of symmetry. Our work rigorously characterizes the full symmetry group of the MoE layer and provides formal proofs. These results show that identifying LMC in MoE only requires accounting for the specific symmetries we define. We view this characterization as a critical and novel contribution of our work.\n\nOur ablation studies on the Expert order matching method in section 6.3 underscore the importance of each stage in our alignment method for uncovering LMC in MoEs, highlighting in particular the novelty of stage 1 (expert order alignment) in our matching method. These results demonstrate that expert order alignment is a crucial step in aligning two MoEs, while stage 2 (weight matching) aligns the FFN models, with its importance being well established in prior work. We observe that removing or modifying any stage - such as skipping expert permutation alignment or omitting weight matching - significantly degrades the quality of the interpolated models, often resulting in pronounced loss barriers. These findings indicate that our approach is not simply a straightforward extension of existing techniques, but rather a carefully crafted pipeline specifically designed to address the unique challenges inherent to MoE architectures.\n\n(ii) **Why not find LMC in the full Transformer-based MoE?**\n\nAs is well known, modern deep learning models are composed of heterogeneous stacks of layers, making it extremely challenging to characterize global symmetry across the entire architecture. Our decision to focus solely on the MoE layer (as defined in Equations (2) and (3)) is driven by the substantial complexity of identifying LMC across the full model. This task would require a detailed understanding of the symmetry structure of the Multi-Head Attention (MHA) layers that typically precede MoE blocks, along with the development of a general alignment algorithm capable of handling such composite modules. While some recent works have explored MHA symmetries, to the best of our knowledge, no general method exists for aligning stacked components. We attempted several approaches for aligning composite (Attention + MoE) layers, but due to time constraints, their empirical performance was limited. Additional nontrivial symmetries may also arise when stacking layers across the full model, as discussed in the next point. Given the scope of this submission, we focused on MoE alignment and leave model-wide symmetry and LMC for future work.\n\n(iii) **Which kinds of symmetry can arise when stacking layers?** \n\nWe provide an intuitive perspective on how symmetry can emerge when composing two general layers. Consider two parameterized layers, $f(\\cdot;\\alpha) : \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_1}$ and $g(\\cdot;\\beta) : \\mathbb{R}^{d_1} \\to \\mathbb{R}^{d_2}$, which we compose as $g \\circ f: \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_2}$.\nIn practice, these layers may admit certain forms of symmetry. Specifically, let $P$ be a $d_1 \\times d_1$ permutation matrix, representing a reordering of the indices of vectors in $\\mathbb{R}^{d_1}$. Suppose there exist modified parameters $\\alpha'$ and $\\beta'$ such that \n$f(\\cdot;\\alpha') = P \\circ \nf(\\cdot;\\alpha)$ and $g(\\cdot;\\beta') = g(\\cdot;\\beta) \\circ P^{-1}$. \nThen, the composite function remains invariant:\n$g(\\cdot;\\beta) \\circ f(\\cdot;\\alpha) = g(\\cdot;\\beta') \\circ f(\\cdot;\\alpha')$. \nTypically, $\\alpha'$ corresponds to permuting the coefficients of $\\alpha$ according to $P$, and similarly for $\\beta'$. For intuition, one may think of $f$ and $g$ as two feedforward layers, and $P$ as a permutation of neurons at the output of $f$ and the input of $g$.\nDepending on the choice of $f$ and $g$, the transformation $P$ can be generalized beyond permutations, e.g., as generalized permutation matrices, orthogonal matrices, or even arbitrary invertible matrices. This illustrates one type of symmetry that can emerge when stacking layers. Intuitively, we believe this covers a broad class of possible symmetries, though to the best of our knowledge, no existing work has fully addressed these scenarios. Developing a complete understanding of the full symmetry structure would require a general, conceptual, and rigorous framework - an important and promising research direction for deepening our understanding of deep learning architectures through the lens of symmetry.\n\nIn response to the Reviewer’s suggestion to investigate LMC in models with MoE layers initialized from scratch at all layers (rather than only the first, as in our original setup), we conducted this experiment and present the results in **Table 1**. Due to space constraints, the table is provided at the end of the response to **Reviewer tznV**.\n\n**W2. Since Mixture-of-Experts based architectures ... and got extended to MoEs, look like?**\n\n**Answer W2.** Thank you for raising this insightful point and for suggesting further development of our work in this direction.\n\nWhen two networks share the same pre-trained Transformer backbone but are independently extended into MoE architectures with differing numbers of experts, their alignment becomes considerably more complex, and may not fully capture or reflect the theoretical significance of LMC. This complexity arises because the models' loss landscapes diverge due to differences in capacity and routing dynamics, potentially leading them to converge in distinct basins.\n\nTo explore alignment in such scenarios, we considered extrapolating the smaller MoE model (with fewer experts) to match the larger one's structure. Specifically, given an MoE with $n$ experts defined as a function $\\mathcal{D} : \\mathbb{R}^d \\to \\mathbb{R}^d$ such that\n\n$\\mathcal{D}(x; \\\\{W_i,b_i,\\theta_i\\\\}\\_{i=1}^{n}) = \\sum_{i=1}^{n} \\text{softmax}_i(s_1(x), \\ldots, s_n(x)) \\mathcal{E}(x;\\theta_i),\n$\n\nwhere $s_i(x) = W_i x + b_i$ are the gating scores parameterized by $\\{W_i, b_i\\}_{i=1}^n$ (with $(W_i, b_i) \\in \\mathbb{R}^d \\times \\mathbb{R}$), and $\\theta_i$ are the parameters of the $i$-th expert $\\mathcal{E}(\\cdot; \\theta_i)$. We duplicate experts to create an equivalent MoE with $2n$ experts as follows:\n\n- Set $\\phi_{2i-1} = \\phi_{2i} = \\theta_i$ for all $i \\in \\{1, 2, \\ldots, n\\}$.\n\n- Set $T_{2i-1} = T_{2i} = W_i$ and $a_{2i-1} - u = a_{2i} - v = b_i$, where $u, v$ are constants. Then,\n\n$\nr_{2i-1}(x) = T_{2i-1} x + a_{2i-1} = W_i x + b_i + u = s_i(x) + u, r_{2i}(x) = T_{2i} x + a_{2i} = W_i x + b_i + v = s_i(x) + v.\n$\n\nThis yields\n\n$\\mathcal{D}(x; \\\\{T_i, a_i, \\phi_i\\\\}\\_{i=1}^{2n})$\n\n$= \\sum_{i=1}^{2n} \\text{softmax}\\_i(r_1(x), \\ldots, r_{2n}(x)) \\mathcal{E}(x; \\phi_i)$\n\n$= \\sum_{i=1}^{n} \\frac{e^{s_i(x) + u} + e^{s_i(x) + v}}{\\sum_{j=1}^n (e^{s_j(x) + u} + e^{s_j(x) + v})} \\cdot \\mathcal{E}(x; \\theta_i)$\n\n$= \\sum_{i=1}^n \\frac{(e^u + e^v) e^{s_i(x)}}{\\sum_{j=1}^n (e^u + e^v) e^{s_j(x)}} \\cdot \\mathcal{E}(x; \\theta_i)$\n\n$= \\sum_{i=1}^n \\text{softmax}\\_i(s_1(x), \\ldots, s_n(x)) \\mathcal{E}(x; \\theta_i) = \\mathcal{D}(x; \\\\{W_i, b_i, \\theta_i\\\\}_{i=1}^n).$\n\nIn preliminary experiments, we extended a shared pre-trained Transformer (e.g., ViT on CIFAR-10 and CIFAR-100) into MoEs with varying expert counts, such as 2 experts for the smaller model and 4 (or more) for the larger one. After independent fine-tuning, we extrapolated the smaller model using the above method and attempted alignment with the larger model. Initial alignments yielded low-barrier loss curves, though not as low as those observed between two large MoEs of comparable size (as reported in the paper). Continued training of the extrapolated model improved alignment in some cases, approaching the performance levels seen in the paper; however, results fluctuated across pairings (e.g., 2 vs. 4, 2 vs. 8, 4 vs. 8, 4 vs. 16). Due to time constraints, we were unable to draw a decisive claim from these explorations.\n\nCrucially, LMC is expected to hold only between converged checkpoints with similar performance and within the same basin. Even if LMC emerges after retraining the extrapolated smaller model, it pertains only to a modified, retrained version - not the original. This weakens the theoretical significance of LMC in this context, as it fails to reflect alignment between the original, independently trained small and large models.\n\n-----\n\nWe thank the Reviewer for the constructive feedback and thoughtful suggestions. We appreciate the recommendation to explore LMC in fully initialized Transformer-based MoEs and across models with varying expert counts, which enabled us to conduct additional experiments and gain further insights into the robustness of our approach. We will incorporate the necessary revisions and results. If our responses address the concerns raised, we kindly hope that the evaluation may be adjusted to reflect this. We remain open to further discussion in the next review phase."}, {"review_id": "yCCj8XbFwT", "reviewer": "Reviewer_8bxH", "summary": "This paper investigates Linear Mode Connectivity (LMC) in Mixture-of-Experts (MoE) architectures. The authors provide a theoretical characterization of the symmetries in MoEs, which stem from expert permutation and gating function translations. Based on this theory, they propose a two-stage alignment algorithm to find low-loss linear paths between independently trained MoE models. The work is supported by extensive experiments showing that LMC is a robust phenomenon across various MoE types, model backbones, and tasks.", "questions": "* The exclusion of the k=1 case for sparse MoEs is justified by its additional invariances under the multiplicative group. Could you provide more intuition on why this scaling invariance makes the functional equivalence analysis so much more challenging? Does it introduce continuous families of equivalent solutions that are not captured by the discrete permutation group?\n\n* The assumptions for Theorem 4.2, particularly the linear independence of {W_{i−1} −W_i}, are noted to be generically true when the number of experts is small compared to the input dimension. Have you empirically verified how often this assumption holds for the models trained in your experiments? A brief check could help bridge the gap between the theory and the practical results.\n\n* Your work convincingly shows that aligning models reveals low-loss linear paths. A natural next step is to leverage this for model merging, such as creating \"model soups\". Did you perform any experiments to see if averaging the weights of two aligned MoE models produces a better-performing model than simply ensembling their outputs or averaging unaligned models? This could be a powerful demonstration of the practical utility of your alignment algorithm.\n\nIn the experiments, the feed-forward network (FFN) in a single layer was replaced with an MoE module. How do you expect the LMC results to change if MoE layers were present at multiple, or even all, layers of the Transformer? Would the alignment problem become significantly harder, and would you expect the loss barriers to increase?", "limitations": "The authors have adequately addressed the limitations. In the conclusion, they explicitly state that their method does not provide theoretical bounds on the loss barrier, which is a common limitation in this line of work. They also clearly state the exclusion of the k=1 sparse MoE case from their main analysis and provide a rationale for it in both the main text and the appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* This is the first systematic study of LMC in MoE architectures, a timely and increasingly important class of models.\n* The paper offers rigorous proofs for the functional equivalence of MoE models, providing a solid foundation for the empirical work.\n* The claims are validated across a wide range of MoE variants (dense, sparse, DeepSeekMoE), tasks (vision, language), and model configurations, demonstrating the generality of the findings.\n\nWeakness:\n* The proposed alignment algorithm is an effective, but novel combination of existing methods (e.g., Weight Matching, Hungarian algorithm) rather than a fundamentally new approach.\n* While important to verify, the existence of LMC in MoEs can be seen as a non-trivial but expected extension of the same phenomenon observed in standard networks.\n* The theoretical analysis excludes the k=1 sparse MoE case and relies on strong assumptions for the sparse case (Theorem 4.2), whose practical prevalence is not empirically verified.", "was_revised": true, "final_justification": "I have reviewed the author rebuttal and the full discussion. The authors successfully addressed the primary shared concern regarding the work's novelty. Their new experimental data, which demonstrates that the proposed expert-order alignment is essential for finding LMC, was decisive. This convincingly refutes the idea that the contribution is a simple extension of prior methods.\n\nGiven that these clarifications led other reviewers to raise their scores, I am confident in maintaining my original \"Accept\" rating and have increased my confidence score accordingly.", "rebuttal": "We address the concerns raised by the Reviewer in the Weaknesses and Questions sections as follows.\n\n---\n\n**W1. The proposed alignment algorithm ... a fundamentally new approach.**\n\n**Answer W1.** We acknowledge that our alignment algorithm builds upon existing techniques such as Weight Matching and the Hungarian algorithm. However, its novelty lies not in introducing a fundamentally new primitive, but in adapting and integrating these methods effectively for the Mixture-of-Experts (MoE) setting - a setting where alignment is particularly challenging due to the massive number of parameters and dynamic expert routing.\n\nIn contrast to many recent approaches that rely on data-driven optimization to learn expert correspondences, our method is data-free, fast, and efficient, which is crucial in the MoE context. MoE models are often used in conjunction with extremely large datasets, making data-dependent matching methods computationally expensive and often impractical for exploring Linear Mode Connectivity (LMC).\n\nDespite these constraints, our algorithm performs well and consistently discovers meaningful alignments that reveal interesting connectivity patterns across fine-tuned MoE models. We believe this demonstrates that even with limited assumptions and no data usage, effective alignment in large-scale MoE models is possible - highlighting both the practicality and the contribution of our method.\n\n**W2. While important to verify, the existence of LMC in MoEs ... in standard networks.**\n\n**Answer W2.** We appreciate the Reviewer’s suggestion that, given the existence of LMC in feedforward networks (FFNs) as shown in [1], one might intuitively expect similar behavior in MoE architectures. However, we contend that MoE models introduce fundamentally different challenges. In particular, the presence of a non-linear softmax gating mechanism sets MoEs apart from classical FFNs. This gating leads to conditional computation and sparse expert activation, resulting in highly non-uniform training signals and expert specialization. Consequently, the optimization landscape becomes significantly more fragmented, making alignment across models substantially more difficult. From the perspective of symmetry, this gating component introduces additional operators that give rise to new types of symmetry. Our work rigorously characterizes the full symmetry group of the MoE layer and provides formal proofs. These results show that identifying LMC in MoE only requires accounting for the specific symmetries we define. We view this characterization as a critical and novel contribution of our work.\n\nOur ablation studies in Section 6.3 highlight the importance of both stages in our alignment method for uncovering LMC in MoEs, particularly the novelty of Stage 1 (expert order alignment). We find that removing or altering either stage - expert permutation or weight matching - significantly degrades interpolation quality, often introducing loss barriers. These results demonstrate that our method is not a trivial extension of prior work, but a tailored solution to the unique challenges of MoE alignment.\n\n[1] Frankle, Jonathan, et al. \"Linear mode connectivity and the lottery ticket hypothesis.\" ICML 2020.\n\n**W3. The theoretical analysis excludes the k=1 ... is not empirically verified.**\n\n**Answer W3.** We kindly refer the Reviewer to our responses to **Q1 and Q2** below.\n\n**Q1. The exclusion of the k=1 case for ... discrete permutation group?**\n\n**Answer Q1.** The reason the case of $k=1$ behaves differently in the sparse regime lies in the structure of the proof. At a high level, we begin with two MoE models that represent the same function and aim to identify the correspondence between their respective sets of parameters. Given $n$ total experts, the goal is to match them one by one. When $k = 1$, each input activates only a single expert, allowing us to match just one expert pair at a time - without gaining any information about the remaining unmatched experts. In contrast, when $k>1$ (e.g., $k=2$), each input activates multiple experts, enabling us to match at least one expert pair and still retain information about the remaining $k-1$ experts. This intuition is reflected in the proof of Theorem 4.2, particularly in Step 4 (line 1000), where the condition $k > 1$ is essential.\n\nWe would also like to share that, during the development of the proof for Theorem 4.2, we did not initially expect to exclude the case of $k = 1$, as we did not anticipate the emergence of a new symmetry action by the multiplicative group in this setting. It was only at a later stage - once we reached the point of needing this specific condition - that we recognized the additional difficulty posed by the $k = 1$ case and the necessity of excluding it due to the appearance of this nontrivial symmetry.\n\n**Q2. The assumptions for Theorem 4.2 ... the gap between the theory and the practical results.**\n\n**Answer Q2.** \nRegarding the assumptions of Theorem 4.2 concerning the linear independence of ${W_{i-1} - W_i}$, we note that this condition holds generically when the number of experts is small relative to the ambient dimension. That is, the set of matrices violating this condition forms a measure-zero subset in the parameter space. Indeed, in our experiments, we have conducted empirical evaluations on many of our checkpoint instances and found all of them to exhibit full rank.\n\nFrom a numerical perspective, exact linear dependence is unstable under perturbations: even infinitesimal changes - arising from stochastic gradient updates, floating-point arithmetic, or randomness in data sampling - are sufficient to restore full rank. This phenomenon is well-documented in matrix perturbation theory and underscores the fact that exact rank deficiency is not robust in practical machine learning systems.Consequently, empirically checking for exact linear independence is of limited interpretability, as it is highly sensitive to numerical precision and implementation details. \n\n**Q3. Your work convincingly shows that ... of your alignment algorithm.**\n\n**Answer Q3.** We appreciate the suggestion to explore model merging, such as model soups [Wortsman et al., 2022], which leverages LMC to average aligned model weights for improved performance.\n\nIn our experiments, we applied model soups within the LMC pipeline by aligning MoE layers from fine-tuned checkpoints of the same pretrained Transformer, considering both single-layer and full-layer MoE replacements. However, these attempts yielded only marginal improvements over the best original checkpoint. Visualizations reveal that the midpoint loss along both naive and aligned loss barrier curves remains the highest (though notably reduced with alignment), indicating limited benefit from simple averaging in this MoE-only setup.\n\nWe hypothesize that model soups would be more effective in flexible, generalized frameworks extending beyond MoE layers alone. Our current algorithm aligns only MoE components, which renders full-model fusion less meaningful. Extending alignment to the entire Transformer architecture introduces complex symmetries, such as those in Multi-Head Attention (MHA) layers (e.g., symmetries under invertible transformations on query/key matrices: $Q \\cdot K^\\top = (QA^\\top)(KA^{-1})^\\top$). To provide intuition on how symmetries can emerge in composite layers, consider two parameterized layers, $f(\\cdot;\\alpha) : \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_1}$ and $g(\\cdot;\\beta) : \\mathbb{R}^{d_1} \\to \\mathbb{R}^{d_2}$, composed as $g \\circ f: \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_2}$. In practice, these layers may exhibit symmetries. For instance, let $P$ be a $d_1 \\times d_1$ permutation matrix representing a reordering of indices in $\\mathbb{R}^{d_1}$. Suppose there exist modified parameters $\\alpha'$ and $\\beta'$ such that $f(\\cdot;\\alpha') = P \\circ f(\\cdot;\\alpha)$ and $g(\\cdot;\\beta') = g(\\cdot;\\beta) \\circ P^{-1}$. Then, the composite function remains invariant: $g(\\cdot;\\beta) \\circ f(\\cdot;\\alpha) = g(\\cdot;\\beta') \\circ f(\\cdot;\\alpha')$. Typically, $\\alpha'$ involves permuting the coefficients of $\\alpha$ according to $P$, and similarly for $\\beta'$. For example, one may think of $f$ and $g$ as feedforward layers, with $P$ permuting neurons at the output of $f$ and input of $g$. Depending on the layers, $P$ can generalize beyond permutations to include generalized permutation matrices, orthogonal matrices, or arbitrary invertible matrices. This illustrates one form of symmetry that arises when stacking layers.\n\nCharacterizing global symmetries across heterogeneous Transformer stacks is challenging, as no general method exists for aligning such composites. Due to scope and time constraints, we focused on MoE alignment and deferred full-model experiments. We are actively developing a comprehensive alignment algorithm for entire Transformer blocks, including MoEs, and plan to demonstrate model fusion applications in future work. This could enable practical benefits such as efficient merging of large models for improved generalization or resource savings.\n\n**Q4. In the experiments, the FFN in a single layer ... expect the loss barriers to increase?**\n\n**Answer Q4.** Due to space constraints, we kindly refer the Reviewer to our response to **W1 for Reviewer 8bxH** above, which also references **Table 1** provided at the end of the response to **Reviewer tznV**.\n\n---\n\nWe thank the Reviewer for the insightful feedback and valuable suggestions, which have helped strengthen our work. We especially appreciate the recognition of our contributions in characterizing MoE symmetries and demonstrating LMC, as well as the suggestions to extend our analysis. We will incorporate the recommended revisions, including empirical checks and additional clarifications. If our responses address the concerns raised, we kindly hope that the evaluation may be adjusted to reflect this. We remain open to further discussion in the next review phase."}, {"review_id": "pR1zYcZHwk", "reviewer": "Reviewer_7X3K", "summary": "This paper provides an affirmative answer to the question of whether MoE networks exhibit LMC when utilizing the symmetries inherent to MoE architectures. Specifically, it addresses not only the permutation symmetries of FNNs used in existing research, but also the translation invariance of the softmax function. The authors prove functional equivalence theorems for both dense gating and sparse gating, propose a Weight Matching algorithm, and empirically confirm the existence of LMC across diverse datasets and architectures.", "questions": "**Q1. Validity of theoretical assumptions**: To what extent are the strongly distinct assumptions in Theorem 4.2 satisfied in actual ReLU networks? Could partial results under weaker assumptions be obtained, or validation on real data be provided?\n\n**Q2. Scalability and practicality**: Does the proposed method scale to larger actual MoE models (such as Switch Transformer)? How can the discovered LMC be utilized for practical model fusion applications?\n\n**Q3. Generalization potential**: Equation (9) demonstrates that continuous symmetries (translation) do not affect loss barriers while discrete symmetries (permutation) are essential for mode connectivity in MoE architectures. This finding may represent a more general principle: continuous transformations preserve connectivity within the same basin, while discrete transformations enable connectivity between different basins. Have the authors considered whether this continuous vs. discrete framework could be extended to other architectures (e.g., attention heads in Transformers, convolutional filters)? Establishing this as a general principle could significantly broaden the theoretical impact of this work and provide a unified understanding of symmetry roles in neural network loss landscapes.", "limitations": "The main limitations of the paper are: (1) unverified practicality of strong theoretical assumptions in sparse gating, (2) experiments limited to relatively small-scale models and datasets, and (3) unclear concrete pathways to practical applications. While the authors acknowledge some of these limitations in the conclusion, more detailed analysis and presentation of future research plans would be desirable.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strrengths\n\n**S1. Novel theoretical framework**: The unified treatment of permutation symmetries and translation invariance represents a valuable theoretical contribution. The complete characterization proofs of functional equivalence for both dense gating (Theorem 4.1) and sparse gating (Theorem 4.2) appear to be correct.\n\n**S2. Comprehensive and systematic experimental design**: The authors validate three MoE variants (Dense MoE, SMoE, DeepSeekMoE) across both vision (MNIST, CIFAR, ImageNet) and language (WikiText103, One Billion Word) domains. Systematic ablation studies varying the number of experts (2-16) and layers (1-12) demonstrate the universality of LMC.\n\n**S3. Practical algorithmic contribution**: The Weight Matching algorithm for expert permutation alignment with O(n³ + nh³) computational complexity is efficient and implementable, utilizing permutation-invariant representations through Gram matrices.\n\n## Weaknesses\n\n**W1. Strong theoretical assumptions**: Sparse gating (Theorem 4.2) requires very strong assumptions such as \"strongly distinct\" expert functions and linear independence of {Wi−1−Wi}, but the analysis of how well these assumptions are satisfied in actual MoE models is insufficient.\n\n**W2. Limited experimental scope**: The experiments concentrate mainly on FFN replacement, confirming LMC only for parts of actual MoE networks. Additionally, the model sizes are limited to relatively small scales.", "was_revised": true, "final_justification": "I acknowledge the novelty of this paper's theory that considers MoE. I believe that the extension of symmetry to MoE is a steady and important step toward identifying the symmetry of the entire transformer in the future.", "rebuttal": "We address the concerns raised by the Reviewer in the Weaknesses and Questions sections as follows.\n\n---\n\n**W1. Strong theoretical assumptions: Sparse gating (Theorem 4.2) requires very strong assumptions such as \"strongly distinct\" expert functions and linear independence of {Wi−1−Wi}, but the analysis of how well these assumptions are satisfied in actual MoE models is insufficient.**\n\n**Q1. Validity of theoretical assumptions: To what extent are the strongly distinct assumptions in Theorem 4.2 satisfied in actual ReLU networks? Could partial results under weaker assumptions be obtained, or validation on real data be provided?**\n\n**Answer to W1 and Q1.** Regarding the concerns related to the set of assumptions in the two main theorems of the paper, namely Theorem 4.1 and Theorem 4.2, we would like to note that we have already provided clarifications on both the necessity and the implications of these assumptions in Section 4.2. These are further elaborated in Remarks B.6 and C.8 in the Appendix. In particular, we explain why these assumptions are typically satisfied in practice and provide examples of scenarios where they fail - referred to as singular cases - under which the theorems may no longer hold. We kindly invite the Reviewer to revisit those sections, as we believe they contain a clear and thorough explanation.\n\nAdditionally, we would like to elaborate more on the “strongly distinct” condition in Remark C.8, which may initially appear to be a restrictive or unrealistic assumption. In fact, this condition is almost surely satisfied in practice. Specifically, if one randomly selects a set of ReLU experts, they will, with probability one, be strongly distinct. While a full proof of this claim may be lengthy, the intuition is straightforward: each ReLU expert can be viewed as a locally affine map, and within each affine region, it is parameterized by a vector in $\\mathbb{R}^d$ for some fixed $d$. Under this perspective, the strongly distinct condition reduces to a statement akin to: “A finite set of randomly sampled vectors in $\\mathbb{R}^d$ are almost surely pairwise distinct.” We emphasize that this observation is not intended to serve as a formal proof; however, due to its intuitive nature and practical relevance, we believe it provides a satisfactory justification to address the Reviewer’s concern.\n\n\n**W2. The experiments concentrate mainly on FFN replacement, confirming LMC only for parts of actual MoE networks. Additionally, the model sizes are limited to relatively small scales.**\n\n**Q2.  Scalability and practicality: Does the proposed method scale to larger actual MoE models (such as Switch Transformer)? How can the discovered LMC be utilized for practical model fusion applications?**\n\n**Answer to W2 and Q2.** We appreciate the reviewer's suggestion to explore model fusion. To address this, we experimented with **model soups** [Wortsman et al., 2022], which leverages LMC to average unaligned model weights for improved performance.\n\nIn our experiments, we applied model soups within our experimental setup using the LMC pipeline. Specifically, we aligned MoE layers from fine-tuned checkpoints of the same pretrained Transformer, testing configurations with MoE replacement at the first layer as well as across all layers. However, these efforts yielded only insignificant improvements over the best original checkpoint in the pool. A key insight from our visualizations is that the midpoint loss along both naive and aligned loss barrier curves remains the highest (though significantly reduced post-alignment), suggesting limited gains from simple averaging in this MoE-only setup.\n\nWe hypothesize that model soups would be more effective in flexible, generalized frameworks extending beyond MoE layers alone. Our current algorithm aligns only MoE components, which renders full-model fusion less meaningful. Extending alignment to the entire Transformer architecture introduces complex symmetries, such as those in Multi-Head Attention (MHA) layers (e.g., symmetries under invertible transformations on query/key matrices: $Q \\cdot K^\\top = (QA^\\top)(KA^{-1})^\\top$). To provide intuition on how symmetries can emerge in composite layers, consider two parameterized layers, $f(\\cdot;\\alpha) : \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_1}$ and $g(\\cdot;\\beta) : \\mathbb{R}^{d_1} \\to \\mathbb{R}^{d_2}$, composed as $g \\circ f: \\mathbb{R}^{d_0} \\to \\mathbb{R}^{d_2}$. In practice, these layers may exhibit symmetries. For instance, let $P$ be a $d_1 \\times d_1$ permutation matrix representing a reordering of indices in $\\mathbb{R}^{d_1}$. Suppose there exist modified parameters $\\alpha'$ and $\\beta'$ such that $f(\\cdot;\\alpha') = P \\circ f(\\cdot;\\alpha)$ and $g(\\cdot;\\beta') = g(\\cdot;\\beta) \\circ P^{-1}$. Then, the composite function remains invariant: $g(\\cdot;\\beta) \\circ f(\\cdot;\\alpha) = g(\\cdot;\\beta') \\circ f(\\cdot;\\alpha')$. Typically, $\\alpha'$ involves permuting the coefficients of $\\alpha$ according to $P$, and similarly for $\\beta'$. For example, one may think of $f$ and $g$ as feedforward layers, with $P$ permuting neurons at the output of $f$ and input of $g$. Depending on the layers, $P$ can generalize beyond permutations to include generalized permutation matrices, orthogonal matrices, or arbitrary invertible matrices. This illustrates one form of symmetry that arises when stacking layers.\n\nCharacterizing these global symmetries across heterogeneous Transformer stacks is challenging, as no general method yet exists for aligning such composites. Due to scope and time constraints, we focused on MoE alignment in this work and deferred full-model experiments. We are actively developing a comprehensive alignment algorithm for entire Transformer blocks, including MoEs, and plan to demonstrate model fusion applications in future work. This could enable practical benefits, such as efficient merging of large models for enhanced generalization or resource savings.\n\n**Q3.  Generalization potential: Equation (9) demonstrates that continuous symmetries (translation) do not affect loss barriers while discrete symmetries (permutation) are essential for mode connectivity in MoE architectures. This finding may represent a more general principle: continuous transformations preserve connectivity within the same basin, while discrete transformations enable connectivity between different basins. Have the authors considered whether this continuous vs. discrete framework could be extended to other architectures (e.g., attention heads in Transformers, convolutional filters)? Establishing this as a general principle could significantly broaden the theoretical impact of this work and provide a unified understanding of symmetry roles in neural network loss landscapes.**\n\n**Answer to Q3.** In the context of MoE, we agree that continuous symmetries do not influence the loss barrier, and hence do not affect the observation of LMC. However, we believe this is largely due to the specific structure of MoE, considered in Equation (2) and (3). In general, various forms of continuous symmetry can be found in other architectures. A well-known example arises in the Attention mechanism, where there exists a continuous symmetry induced by the general linear group acting on the query and key matrices. Specifically, let $Q$ and $K$ denote the query and key matrices; for any invertible matrix $A$, we have $Q \\cdot K^{\\top} = (QA^\\top)(KA^{-1})^\\top$ demonstrating a continuous symmetry under the action of the general linear group on $Q$ and $K$.\n\nBuilding on this, our experiments to explore LMC in this setting reveal that considering only permutation matrices for $A$ was insufficient to observe LMC in Attention layers. This suggests that focusing solely on discrete symmetries may not be adequate for identifying LMC in architectures that admit richer symmetry structures.\n\nThe problems of characterizing functional equivalence in neural architectures and identifying LMC are both promising and foundational areas of research. A complete understanding of symmetry in deep learning models will likely require a general, conceptual, and rigorous framework - one that we believe holds great potential for advancing our understanding of network behavior through the lens of symmetry.\n\n\n---\nWe thank the Reviewer for their constructive feedback, thoughtful evaluation, and probing questions, which have significantly strengthened our paper by prompting deeper clarifications and highlighting opportunities for broader impact. We particularly appreciate the suggestions on validating assumptions, scaling to larger models, and exploring practical applications like model fusion. We will incorporate the recommended enhancements, including expanded discussions on the practicality of assumptions (e.g., via additional intuitions and remarks), scalability considerations, and future research directions for full-model alignment and generalization to other architectures, to further improve the paper's rigor and applicability. If the Reviewer is satisfied with our responses to the weaknesses and questions raised, we kindly hope that the evaluation may be adjusted to reflect this. We remain open to further discussion and would be happy to engage in the next phase of the review process."}]

Author rebuttal:

Meta-review: The paper investigates the presence of Linear Mode Connectivity (LMC) in Mixture-of-Experts (MoE) architectures by offering theoretical and empirical characterizations of their inherent symmetries. The paper proves that MoE and sparse-gated MoE layers admit only two fundamental parameter symmetries---expert permutation and gating-parameter translation---and establish functional equivalence theorems for both dense and sparse gating settings. Building on these results, they propose a weight-matching alignment algorithm that takes advantage of these symmetries to connect independently trained MoE models along low-loss linear paths. Through extensive experiments across multiple model backbones, datasets, and MoE variants, the paper confirms that LMC is a robust and general phenomenon in this setting, thereby extending the scope of mode connectivity research beyond standard feedforward networks.

The paper was reviewed by four referees who agree that it provides a valuable contribution to the community. All four reviewers emphasize the significance of the mathematical characterization of parameter symmetries in MoE and sparse-MoE models, providing a non-trivial advance in our understanding of LMC in MoE architectures. As several reviewers point out, this mathematical characterization is well developed and clearly presented. As noted by each reviewer, a key strength of the paper is its extensive experimental evidence that clearly supports the theory that is introduced. The reviewers initially raised a few relatively minor questions/concerns with the paper, which the authors thoroughly addressed in during the author-reviewer discussion phase, as the reviewers acknowledge in their follow-up comments.
Decision: Accept (oral)