Title: On Linear Mode Connectivity of Mixture-of-Experts Architectures
Venue: NeurIPS 2025
Accepted: yes
Keywords: linear mode connectivity

mixture-of-experts
Abstract: Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapes
of neural networks, wherein independently trained models have been observed to
be connected—up to permutation symmetries—by linear paths in parameter space
along which the loss remains consistently low. This observation challenges classical
views of non-convex optimization and has implications for model ensembling,
generalization, and our understanding of neural loss geometry. Inspired by recent
studies on LMC in standard neural networks, we systematically investigate this
phenomenon within Mixture-of-Experts (MoE) architectures—a class of models
known for their scalability and computational efficiency, which combine traditional
neural networks—referred to as experts—through a learnable gating mechanism.
We begin by conducting a comprehensive analysis of both dense and sparse gating
regimes, demonstrating that the symmetries inherent to MoE architectures are
fully characterized by permutations acting on both the expert components and the
gating function. Building on these foundational findings, we propose a matching
algorithm that enables alignment between independently trained MoEs, thereby
facilitating the discovery of LMC. Finally, we empirically validate the presence of
LMC using our proposed algorithm across diverse MoE configurations—including
dense, sparse, and shared-expert variants—under a wide range of model settings
and datasets of varying scales and modalities. Our results confirm the existence
of LMC in MoE architectures and offer fundamental insights into the functional
landscape and optimization dynamics of deep learning models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an affirmative answer to the question of whether MoE networks exhibit LMC when utilizing the symmetries inherent to MoE architectures. Specifically, it addresses not only the permutation symmetries of FNNs used in existing research, but also the translation invariance of the softmax function. The authors prove functional equivalence theorems for both dense gating and sparse gating, propose a Weight Matching algorithm, and empirically confirm the existence of LMC across diverse datasets and architectures.\n\nSTRENGTHS AND WEAKNESSES:\n## Strrengths\n\n**S1. Novel theoretical framework**: The unified treatment of permutation symmetries and translation invariance represents a valuable theoretical contribution. The complete characterization proofs of functional equivalence for both dense gating (Theorem 4.1) and sparse gating (Theorem 4.2) appear to be correct.\n\n**S2. Comprehensive and systematic experimental design**: The authors validate three MoE variants (Dense MoE, SMoE, DeepSeekMoE) across both vision (MNIST, CIFAR, ImageNet) and language (WikiText103, One Billion Word) domains. Systematic ablation studies varying the number of experts (2-16) and layers (1-12) demonstrate the universality of LMC.\n\n**S3. Practical algorithmic contribution**: The Weight Matching algorithm for expert permutation alignment with O(n³ + nh³) computational complexity is efficient and implementable, utilizing permutation-invariant representations through Gram matrices.\n\n## Weaknesses\n\n**W1. Strong theoretical assumptions**: Sparse gating (Theorem 4.2) requires very strong assumptions such as \"strongly distinct\" expert functions and linear independence of {Wi−1−Wi}, but the analysis of how well these assumptions are satisfied in actual MoE models is insufficient.\n\n**W2. Limited experimental scope**: The experiments concentrate mainly on FFN replacement, confirming LMC only for parts of actual MoE networks. Additionally, the model sizes are limited to relatively small scales.\n\nQUESTIONS:\n**Q1. Validity of theoretical assumptions**: To what extent are the strongly distinct assumptions in Theorem 4.2 satisfied in actual ReLU networks? Could partial results under weaker assumptions be obtained, or validation on real data be provided?\n\n**Q2. Scalability and practicality**: Does the proposed method scale to larger actual MoE models (such as Switch Transformer)? How can the discovered LMC be utilized for practical model fusion applications?\n\n**Q3. Generalization potential**: Equation (9) demonstrates that continuous symmetries (translation) do not affect loss barriers while discrete symmetries (permutation) are essential for mode connectivity in MoE architectures. This finding may represent a more general principle: continuous transformations preserve connectivity within the same basin, while discrete transformations enable connectivity between different basins. Have the authors considered whether this continuous vs. discrete framework could be extended to other architectures (e.g., attention heads in Transformers, convolutional filters)? Establishing this as a general principle could significantly broaden the theoretical impact of this work and provide a unified understanding of symmetry roles in neural network loss landscapes.\n\nLIMITATIONS:\nThe main limitations of the paper are: (1) unverified practicality of strong theoretical assumptions in sparse gating, (2) experiments limited to relatively small-scale models and datasets, and (3) unclear concrete pathways to practical applications. While the authors acknowledge some of these limitations in the conclusion, more detailed analysis and presentation of future research plans would be desirable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates Linear Mode Connectivity (LMC) in Mixture-of-Experts (MoE) architectures. The authors provide a theoretical characterization of the symmetries in MoEs, which stem from expert permutation and gating function translations. Based on this theory, they propose a two-stage alignment algorithm to find low-loss linear paths between independently trained MoE models. The work is supported by extensive experiments showing that LMC is a robust phenomenon across various MoE types, model backbones, and tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* This is the first systematic study of LMC in MoE architectures, a timely and increasingly important class of models.\n* The paper offers rigorous proofs for the functional equivalence of MoE models, providing a solid foundation for the empirical work.\n* The claims are validated across a wide range of MoE variants (dense, sparse, DeepSeekMoE), tasks (vision, language), and model configurations, demonstrating the generality of the findings.\n\nWeakness:\n* The proposed alignment algorithm is an effective, but novel combination of existing methods (e.g., Weight Matching, Hungarian algorithm) rather than a fundamentally new approach.\n* While important to verify, the existence of LMC in MoEs can be seen as a non-trivial but expected extension of the same phenomenon observed in standard networks.\n* The theoretical analysis excludes the k=1 sparse MoE case and relies on strong assumptions for the sparse case (Theorem 4.2), whose practical prevalence is not empirically verified.\n\nQUESTIONS:\n* The exclusion of the k=1 case for sparse MoEs is justified by its additional invariances under the multiplicative group. Could you provide more intuition on why this scaling invariance makes the functional equivalence analysis so much more challenging? Does it introduce continuous families of equivalent solutions that are not captured by the discrete permutation group?\n\n* The assumptions for Theorem 4.2, particularly the linear independence of {W_{i−1} −W_i}, are noted to be generically true when the number of experts is small compared to the input dimension. Have you empirically verified how often this assumption holds for the models trained in your experiments? A brief check could help bridge the gap between the theory and the practical results.\n\n* Your work convincingly shows that aligning models reveals low-loss linear paths. A natural next step is to leverage this for model merging, such as creating \"model soups\". Did you perform any experiments to see if averaging the weights of two aligned MoE models produces a better-performing model than simply ensembling their outputs or averaging unaligned models? This could be a powerful demonstration of the practical utility of your alignment algorithm.\n\nIn the experiments, the feed-forward network (FFN) in a single layer was replaced with an MoE module. How do you expect the LMC results to change if MoE layers were present at multiple, or even all, layers of the Transformer? Would the alignment problem become significantly harder, and would you expect the loss barriers to increase?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations. In the conclusion, they explicitly state that their method does not provide theoretical bounds on the loss barrier, which is a common limitation in this line of work. They also clearly state the exclusion of the k=1 sparse MoE case from their main analysis and provide a rationale for it in both the main text and the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This work introduces the investigation of Linear Mode Connectivity for Mixture-of-Experts based architectures. They propose group actions which account for all symmetries of the Router Network parameters and show that permutation invariance is sufficient to induce LMC. Based on these insights they develop a weight matching to align independently trained MoEs which they empirically evaluate extensively.

STRENGTHS AND WEAKNESSES:
Strengths:

- Extension of Weight Matching Algorithm to MoE bases architectures
- Theoretical analysis and thorough empirical evaluation.


Weaknesses:
- Since Linear Mode Connectivty is already given early in pre-training [1], and this work only exchanges the FFN layer/s with randomly initialized MoEs, it is somewhat intuitive that LMC is given. How does this change when Transformer-based MoEs get initialized from scratch (all layers)?
- Since Mixture-of-Experts based architectures may vary the number of experts, how could aligning two independently trained networks, that started from the same pre-trained transformer. and got extended to MoEs, look like?

[1] Frankle, Jonathan, et al. "Linear mode connectivity and the lottery ticket hypothesis." International Conference on Machine Learning. PMLR, 2020.

QUESTIONS:
See weaknesses.

LIMITATIONS:
None
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors mathematically characterize the symmetries of Mixture-of-Expert (MoE)\nand Mixture-of-Experts with sparse gating (SMoE) layers. When considering the MoE layers in\nisolation, the authors prove that only two types of parameter symmetries are introduced by these\nlayers: permutation of experts and shifting of gating parameters. Informed by their theoretical\nresults, they propose an algorithm to best match two networks trained from different initial values\nto align according to the studied symmetries, a procedure akin to modern linear mode\nconnectivity (LMC) results. The authors claim to be the first to report the presence of LMC\nbetween MoE layers in various setups.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis work provides the first complete mathematical characterization of parameter symmetries in MoE and SMoE layers. While permutation and gating shift symmetries have been observed previously, the authors' proof that these constitute the only symmetries present is a non-trivial theoretical contribution. \n\nThe mathematical framework is well-developed, and the proofs are clearly presented.\n\nDespite showing experiments only on fine-tuned MoE (and not alignment on networks trained entirely from scratch, which can be costly), the experimental evidence is extensive and provides clear support for the MoE-centered theory introduced.\n\nThe literature review section is well-exposed, complete, and thorough.\n\n**Weaknesses**\n\nThe authors should make it clearer that the symmetries they prove to be the only ones present are proved without taking into consideration the rest of the layers in the network. \n\nWhile I doubt that other downstream or upstream layers can introduce more symmetries (aside from trivial linear\nones), the authors do not prove their non-existence. This point should be discussed in a limitations paragraph. Another point to add to the limitation paragraph is that experiments were conducted only by replacing some layers with MoE and only fine-tuning MoE layers, non-attentive readers may miss this point.\n\nThe description of the experimental setup could use some more detailed explanations. I would suggest using the extra page to improve the limitations and experimental setup sections.\n\nQUESTIONS:\nCurious to see if more symmetries may arise when considering interactions between MoE layers and other layers. Can the authors speculate on this?\n\nDid the authors try to align two networks trained from scratch with their current methods? Can LMC be found?\n\nLIMITATIONS:\nSee weaknesses above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-20"}