Title: On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: mixture-of-experts

expressive power

approximation rate

compositional sparsity

low-dimensional manifold
Abstract: Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood.
In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity.
For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality.
For deep MoEs, we show that $\mathcal{O}(L)$-layer MoEs with $E$ experts per layer can approximate piecewise functions comprising $E^L$ pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks.
Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.


Peer reviews:
comments: SUMMARY:
The paper presents a theoretical analysis of expressivity of mixture-of-experts (MoE) networks in the context of complex tasks governed by two structural priors: low-dimensionality and sparsity.

STRENGTHS AND WEAKNESSES:
**Strengths**:
- The paper provides a solid theoretical framework with clear proofs outlining the approximation capabilities of both shallow and deep MoE networks.
- The work brings a novel angle to the study of network expressivity. This analysis bridges the gap between empirical successes of modern deep learning models and their theoretical understanding.
- The discussion on architectural design (e.g., nonlinear gating and alternating structures) is valuable for practitioners, as it provides insights into how theoretical improvements can translate into better network designs for real-world tasks.

**Weakness**: There is no major weakness in this work in my opinion. Some minor weaknesses are included in **Question** part.

QUESTIONS:
- What is the concrete definition of each MoE layer $h^{(l)}$, including the dimensionality of its inputs and outputs? Providing a clear and detailed definition would help readers understand the MoE model more easily.

- In practical applications, how large does the layer depth $L$ compare to the input dimension $d$ for a given target function in practice? Expanding on this point could help readers see how the theoretical result overcomes the curse of dimensionality.

- (minor, a typo?) Should the assumption in Theorem 5.3 be "Let the target function $f$ be of the form (5)" rather than "of the form (4)"?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this article, the authors illustrate the capacity of Mixture of Expert (MoE) model for approximating a function $f$ over a manifold $\\mathcal{M}$. Based on the known result for two-layer ReLU networks (Theorem 3.1) in \\cite{}, the main result (Theorem 4.8) shows that the MoE model can approximate with an error of $\\mathcal{O}(m^{-(\\kappa/d \\wedge 1/2)})$, where $d$ is the intrinsic dimension, $\\kappa$ control the smoothness of function $f$, and $m$ controls the width of network. \nThis result demonstrates the outstanding ability of the MoE model over shallow dense network, which approximate with error $\\mathcal{O}(m^{(-\\kappa/D\\wedge 1/2)})$, where $D$ is the ambient dimension of $\\mathcal{M}$. The author formularises the multiple tasks problem using piecewise functions and product of manifold, and derives the analogous result for this situation, which similarly shows the capacity of MoE to approximate a function with error depennds on intrinsic dimension.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1.  The result is nice overall and meaningful, which can significantly improve the known bound $\\mathcal{O}(m^{-(\\kappa/D \\wedge 1/2)})$ to $\\mathcal{O}(m^{-(\\kappa/d \\wedge 1/2)})$. This improvement shows the adaptability of MoE model in the case of low dimension. \n\n2. The author provides a significant suggestion for MoE mechanism, such as non-linear gating function, alternation between MoE network and dense network.\n\n3. The writing is clear and easy to read, the new concept is well-explained with several examples. \n\nWeaknesses: \n\n1. The paper should provide some experiments to support the result for the case of low dimension. \n\n2. Besides the algorithm that prove the existence of good enough MoE, the paper should provide an algorithm to find a feasible neural network. \n\n3. It appears that the model only consider one kind of gating function (ReLU), so it would be much better to consider other gating functions, like GeLU, softmax, etc. \n\nDespite of some experimental drawback in the paper, the theoretical result here is excellent and interesting, which provides a rigorous justification for the adaptability of MoE model.\n\nQUESTIONS:\n1. The article mentions the concept of reach, how can we use and how important this concept in this article? \n\n2. In equation 7, there may be some situations that $x$ belongs to two atlas, so what happens in this case?\n\n3.  As I mention in strength and weakness part, can we extend the model to the case of other activating function rather than ReLU (like softmax, GeLU, etc,...)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors analyze the expressive power of sparse MoE models when data (1) lies on a low-dimensional manifold or (2) exhibits compositional sparsity. They find that sparse MoEs can match the approximation power of comparable dense networks while evaluating only $1/E$ of the expert parameters per input in both the (1) low-dimensional manifold setting and (2) the compositional sparsity setting, ignoring the routing parameters and assuming the router can correctly route to the correct expert.

STRENGTHS AND WEAKNESSES:
### Strengths
* The paper is clearly written and presented
* Results seem interesting and plausible, though I have not carefully verified the results

### Weaknesses
* On line 319, should the expression currently written as 
$$ ( f_{1,i_1}(x_1),\cdots, f_{1,i_L}(x_L) ) $$
instead be:
$$ ( f_{1,i_1}(x_1),\cdots, f_{L,i_L}(x_L) ) $$
* The paper's claims may hinge on assumptions about the router, in particular that it picks the one correct expert for every input. 
* The paper contains no empirical comparisons or discussions of MoE scaling laws, although this is standard for expressivity papers.

QUESTIONS:
* Is there potentially an issue with how a Top-1 gate maintains a clear margin so that each input is routed to a single expert, especially near chart boundaries?
* In the deep MoE setting, have you looked at how small routing errors in one layer might propagate through the stack, and whether a bound on that accumulation is possible?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper draws the parallel between MoE networks with structured tasks, by showing an MoE model having representation capacity of a $E^L$-piece piecewise function class.

STRENGTHS AND WEAKNESSES:
### Strengths:
- Introduction, motivation, and organization is clear; with annotation in theorems/equations whenever available.
- Rigorous proof for a simple-sounding concept, specifically analyzing the un-simplified (i.e. close to the real implementation) linear MoE router.

### Weaknesses
- Difficult to follow notations.
  - Overloaded notation: $\alpha$ were used as softmax outputs (L105), then an integer permutation vector (L117)
  - Related, what is $\mathcal{A}$?
  - Similarly, $f|_{U_i}$ in Theorem 4.8 onwards were never formally defined.
- Albeit this is a purely theoretical work, experimental results would greatly benefit the persuasion of the paper.
  - The authors could _manually_ craft a theoretical 4-3-MoE to fit the toy problem in Fig. 3, then compare it to baselines such as VI.

QUESTIONS:
See Weaknesses (for lack of better naming).

LIMITATIONS:
Paper listed potential mismatch between theory and real-world problems, and whether SGD can train such a (theoretical) MoE model. No potential impact is listed.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts a systematic study on the theoretical foundations of Mixture-of-Experts (MoE) networks in modeling complex tasks. It demonstrates that shallow MoE networks can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoE networks, the paper shows that they can approximate piecewise functions with compositional sparsity, enabling the modeling of an exponential number of structured tasks. The analysis also provides insights into the roles of architectural components and hyperparameters in MoEs and offers practical suggestions for MoE variants.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:    \n(1) The paper provides a solid theoretical analysis of the expressive power of both shallow and deep MoE networks, filling a gap in the understanding of their capabilities, thus a valuable contribution to the community.\n\n(2) It offers practical suggestions for designing MoE architectures, such as incorporating nonlinearity into gating mechanisms and using low-dimensional expert networks via autoencoding.\n\n(3) The results apply to a wide range of structured tasks, demonstrating the versatility and potential of MoE networks in various domains.\n\n(4) The paper effectively compares the efficiency of MoE networks with dense networks, highlighting the computational advantages of MoEs.\n\n\n**Weaknesses**:  \n(1) I understand this paper is more of a theory paper. However, it would be great if the authors can design some synthetic tasks (e.g., low dimensional or piecewise functions) to better illustrate their theoretical results. It would strengthen the claims and provide practical evidence of the proposed methods.\n\n(2) The analysis focuses primarily on compositional sparsity, and other forms of sparsity such as temporal sparsity on the sequential data are not explored, which might be related to the real-world language understanding task where we have seen MoE's successful usage.\n\n(3) Some recent studies have shown that token routing strategy has significant effects on the MoE performance (see references 1-4 from the list below). Just wondering whether the theoretical results in this paper would be affected by using different routing strategies in any way. It would be great if the authors can include some discussions about it.  \n\n**Reference**:   \n[1] Nguyen, et al. \"Statistical Advantages of Perturbing Cosine Router in Sparse Mixture of Experts.\" arXiv preprint arXiv:2405.14131 (2024).  \n[2] Liu, et al. \"Gating dropout: Communication-efficient regularization for sparsely activated transformers.\" International Conference on Machine Learning. PMLR, 2022.  \n[3] Zuo, et al. \"Taming sparsely activated transformer with stochastic experts.\" arXiv preprint arXiv:2110.04260 (2021).  \n[4] Lewis, et al. \"Base layers: Simplifying training of large, sparse models.\" International Conference on Machine Learning. PMLR, 2021.\n\nQUESTIONS:\nSee my comments above.\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}