Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Authors: Huy Nguyen
Nhat Ho
Alessandro Rinaldo
Venue: NeurIPS 2024 poster (2024)
Keywords: mixture of experts
sigmoid gating function
softmax gating function
sample efficiency
Primary area: learning_theory

Abstract: The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due to its inherent structure. In response, the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically to achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, in fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as a mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case in which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. In both cases, we find that experts formulated as feed-forward networks with commonly used activation such as $\mathrm{ReLU}$ and $\mathrm{GELU}$ enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Huy Nguyen Nhat Ho*
Alessandro Rinaldo"
Department of Statistics and Data Sciences, The University of Texas at Austin
[huynm, minhnhat]@utexas edu,
utexas. edu
Abstract
The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. both cases, we find that experts formulated as feed-forward networks with commonly used activation such ReLU and GELU enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.
inherent structure. In response,
Introduction
Mixture of experts (MoE) 15 17 has recently emerged as powerful machine learning model that helps scale up the model capacity while requiring a nearly constant computational overhead [34] 8). In particular, it aggregates multiple sub-mode called experts based on gating network. Here, experts can be formulated as neural networks, and they specialize different aspects the data.
5 30 one expert might focus on syntax, another on semantics, whereas the others concentrate on context. Meanwhile, the gating network takes the input data and calculates a set of weights that determine the contribution of each expert. This gating mechanism guarantees that the most relevant experts are assigned more weight for a given input. In addition to language modeling, MoE has also been leveraged in several other applications, including computer vision 32
For instance, in large language models [16 31 19 40
20 33 multi-task learning [2] 10), multi-modal learning (T].
speech recognition
38
9
29 and reinforcement learning 3 28
*Co-last authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In the above applications of MoE, practitioners frequently utilize the softmax gating function to compute the mixture weights. However, the use of softmax function might introduce an unexpected competition among experts, which leads to the representation collapse issue 12 30 the weight of one expert increases, those of the others decrease accordingly. As consequence, some experts will dominate the decision-making process, and overshadow the contributions of others. Therefore, the ability of MoE models to incorporate the diversity of expert specialization is partially limited. To this end, Csordás et al. 4 propose using the sigmoid function in place of the softmax to remove the unnecessary expert competition, and empirically demonstrate that it is indeed a better choice of gating function. On the other hand, the theoretical guarantee for that claim has remained missing in the literature. The main objective of this work is to provide a comprehensive analysis of the sigmoid gating function from the perspective of the expert estimation problem. In particular, will show that sigmoid gating delivers superior sample efficiency for estimating the model parameters and allows for more general expert functions than softmax gating.
18 24
when
Related works. very recent line of research has focused on analyzing the convergence rates for expert estimation in Gaussian MoE models under a variety of assumptions and choices of gating functions. Assuming that data from an input-free gating Gaussian MoE, Ho et al. [14] demonstrated that the expert estimation rates for the maximum likelihood estimator vary with the algebraic independence among expert functions. Under the same setting but using instead softmax gating, Nguyen et al. [27] discovered that the expert estimation rates depend on the solvability of a system of polynomial equations resulting from an intrinsic interaction among gating and expert parameters. Subsequently, Gaussian MoE with Top-K K sparse softmax gate, designed to activate only a fraction of experts per input, was studied in (25). The convergence analysis in that work revealed that activating exactly one expert for each input would eliminate the previous parameter interaction, thus boosting the expert estimation rates. Despite many important insights provided in this line of works, the mixture setting of Gaussian MoE is still far from practical. To that effect, Nguyen et al. [26] introduced more realistic regression framework in which, conditionally on the features, the response variables are sampled from noisy realization of an unknown and deterministic softmax gating MoE-type regression function. They found that experts formulated as feed fcrd-forward networks with sigmoid or hyperbolic tangent activation functions enjoy faster estimation rates than those equipped with a polynomial activation. In this paper, we adopt the same regression setting and carry out an analogous analysis by considering instead sigmoid gating for the underlying regression function, as opposed to the more popular softmax gating considered in [26). As we will see, the choice of gating function turns out to be critical, requiring a different type of analysis and leading to markedly different conclusions.
Problem setup. We assume that the data Yi), (X2, Y2), ..... (Xn, Ya) E Rd x R follow standard regression model
Y = fo. +6, i=
where the features X1, X2,.... Xn are i.i.d. samples from a probability distribution μ Rd and él....;6o 6n. are independent noise variables such that [l]] 0 and Var(e:|X.) = v for all 1 i S n; for simplicity we further assume that they follow a Gaussian distribution. The unknown regression function fc. is formulated as a sigmoid-gan -gate MoE with k. experts, i.c.,
x E Rd fo. (z)
h(2, n)),
(2)
I+ ((8))) x - -ba)
where the expert function x + 1(x 11) is of parametric form, specified by parameter 17 E Rt. The regression function fc. is fully characterized by the unknown parameters (Por ni )i) in R x Rd R9, which can be compactly encoded using the associated mixing measure G. Σ I+expl 3, (B a weighted sum of Dirae measures 6.
Least squares estimation. To estimate the unknown parameters (3% 3r 77 k. (equivalently, the ground truth mixing measure G²), we deploy the least squares method [35] and focus on the estimator
G = arg min
Σ -Yi-Jche (K-so(x.))" i=1
(3)
G
2

---

where (+) = fG=
S k, (1i,,31.,1) E )) is the set of all mixing measures with at most k atoms. Since the true number of true experts h2 is typically unknown in practice, we will assume that the number & of fitted experts is at least as large, i.c. that k > k..
1eexpt-B..) (B).)))))))) :1 :
Technical challenges. The main challenge in analyzing MoE models with sigmoid gating lies in the convergence behavior of the mixture weights. Specficially, since we fit the ground-truth MoE model (2) with mixture of k > ki, there must be some true atoms ($is ni fitted by more than one component; we will refer to the corresponding parameters P1 as over-specified parameters.
To illustrate, suppose that ($1, (7)) (811, for i E {1.2). 2). in probability. Then, ensure convergence of fā. to Sc. in the L (j)-norm we must have that, for 1,2,
1
i=1 1 +
1 + exp(
as n + 0o, for (u-almost every z. Note that the above limit can be achieved only if 31 = 0a. As a result, we will consider the following regimes for the over-specified parameters 8ig
Regime 1. All the over-specified parameters 31, are equal to Oa;
Regime 2. At least one among the over-specified parameters Si is different from Oa.
It is worth emphasizing that the second regime presents additional technical challenges, the least squares estimator G, converges to mixing measure G E arg ( (e) Il - fc. [[))
that is in general different that the true mixing measure G., as in the first regime.
Contributions. In this paper, we carry out a convergence analysis of the sigmoid gating MoE under two regimes of the gating parameters. Our main objective is to compare the sample efficiency between the sigmoid gating and the softmax gating. The contributions of our paper are three-fold, and can be summarized follows:
(C.1) Convergence rate for the regression function. We demonstrate in Theorem I that the regression estimation fč, converges to its true counterpart fc. at the rate of order Op(n (/2), which is parametric on the sample size n. This regression estimation rate is then utilized for determining the expert estimation rates.
(C.2) Expert estimation rates under the Regime 1. Under the first regime, we first establish a condition called strong identifiability to characterize which types of experts would yield polynomial estimation rates. In particular, we find out that the rates for estimating experts formulated as feed- forward networks with popular activation functions such ReLU and GELU are of polynomial orders. By contrast, those for polynomial experts and input-indepedent experts are slower than any polynomial rates and could be of order (p(1/ log(n). Such expert convergence behavior is similar to that when using the softmax gating.
Under the second regime, the regression estimation fä. converge to a function taking the form of a sigmoid gating MoE which is different from fa.. From our derived weak identifiability condition, it follows that estimation rates for foed-forward expert networks with ReLU or GELU activation and polynomial experts are of orders (p(n- 1/2,, which are substantially faster than those when using the softmax gating (see also Table 1 Therefore, follows that the sigmoid gating is more sample efficient than the softmax gating.
(C.3) Expert estimation rates under the Regime 2.
Outline. The rest of the paper is organized as follows. In Section 2 we introduce some mild assumptions on the parameters, and then determine estimation rates for the regression function. Next, we establish convergence rates for expert estimation under both Regime 1 and the Regime 2 in Section 3 Then, we conduct some experiments empirically justify our theory Section Finally, we provide a discussion on the practical implications, limitations and future directions of our work in Full proofs of the results and additional experiments are deferred to the Appendices.
Section
Notation. For any n. E N, we denote [n] as the set = S, we refer to |S] as its cardinality. Next, for any vectors v : (U1, .2.....,0) v2) Rd and a : (a), .0,....,,oa)) E Nd. we let y = ii u ...u., ... |o| = in + 22 + ... + va and a! = an anlo!! .ollg!....ll oaa while ||||| stands for its 2-norm value. Lastly, for any two positive sequences (an)n21 and (2)),,1 we write on = O(b,) or on S bn if an S Cb, for all E N, where
nn). Additionally, for any set
some universal constant. The notation on (b(),) indicates that an/b, bb, is stochastically bounded.
3

---

Table 1: Summary of expert estimation rates (up logarithmic factor) under the MoE models equipped with the sigmoid gating (ours) and the softmax gating [26]. In this work, we consider three types of expert functions including experts network with RcLU, GELU activations; polynomial experts; and input-independent experts.
ReLU, GELU Experts
Polynomial Experts
Input- independent
Regime
Regime 2
Regime 1
Regime 2
Experts
Sigmoid
Op(n 1/4)
Op(n-1/2
([L/log(()) Op(n~1/2)
pp(/ (ol//))g())))))))))))))))))))l
Softmax
Op(n-14 1/4)
Op(1/log(n)
(o(())og(())
Preliminaries
In this section, we demonstrate the parametric convergence rate for the least squares estimator of the regression function Sa.- We begin by describing the assumption that will be used throughout the paper, which are overall very mild.
(A.1) Topological assumptions. To ensure the convergence of the least squares estimators, assume the parameter space O C IR Rd IR is compact, and the input space X C R bounded.
(A.2) Lipschitz experts. The expert function h(-,n) is assumed to be Lipschitz continuous with respect and bounded, p-almost surely.
(A.3) Distinet experts. The ground truth expert parameters vi..... me. are pair-wise different so that the experts h(r, mi), ... h(x, vh)) are also distinct from each other.
Given the above assumptions, we now ready to present the main result of this section. Recall that we divide our analysis into two following regimes:
* Regime 1. All the over-specified parameters 8%, are equal to Oa;
Regime 2. At least one among the over-specified parameters $1, is different from Os-
Let us begin with Regime 1. We show in Theorem I that the regression estimator fā., converges to the true regression function fa. the parametric rate.
Theorem 1. Under the Regime and with the least squares estimator G defined in equation 3} the regression estimator fö., admits the following rate of convergence fG.:
Ms.. - -Sc.ll =Op((oo(m)//n)))
The proof of Theorem iis in Appendix A.I The above result indicates that if we are able to design a loss function among parameters 'D that satisfies the lower bound life. - fc. (22(r) 2 D(G,, G.).
then we obtain parameter estimation rates via the bound D(G,, G.) Or(loo(m)/)))))))) Conse- quently, those parameter estimation rates lead
desired expert estimation rates.
Next, under the Regime 2, the true model is misspecified, i.c., the regression estimator fa, converges to the missepcified regression function fa rather than the true regression function fa.. where G E (+) := arg (e) (0) llfe fc. [l< By employing the same arguments
as in Theorem we also capture the parametric regression estimation rate (r-(log(m)/n())) for llfa. fa ll (z))) under this regime, which is stated in the following corollary.
Corollary 1. Under the Regime the regression estimator fä. admits the following rate of conver- gence to fz: inf infzez.()) Ilfa - -57l Or(log(n)/n))
3 Convergence Rates for Expert Estimation
In this section we will establish the expert estimation rates under both Regime and the Regime 2; see Section 3.1 and Section 3.2 respectively. Under each regime, will formulate appropriate

---

conditions, which we refer to as identifiability conditions, on the expert functions that will guarantee fast estimation rates. Furthermore, we will determine the (slow) estimation rates of some commonly used experts which fail to satisfy such identifiability conditions.
3.1 Regime of Gating Parameters
Recall that under the Regime 1, all the over-specifted parameters 3i,, ie., those fitted by at least two parameters, are equal to Og. Without loss of generality, we assume that Si....... S are over-specified parameters, where < k < k., The remaining parameters 0i = PI Oa- Meanwhile, Pi(a+1))-...,9991 are assumed to be fitted by exactly one estimator.
Following a strategy that has been successfully used for analyzing MoE and mixture models (see, e.g. 21 and (6)), in order to derive the expert estimation rates, it is sufficient to propose a Voronoi loss function D among parameters, and then prove the lower bound llfe. fc. lI 2 D(G G.). Given the parametric regression estimation rate in Theorem | we then obtain that D(Gn, G.) Op([og(n)/mw /n] Note that key step in deriving the previous lower bound is decompose the difference fc. (x) - fG. (x) into combination of linearly independent terms.
Towards that goal, we will Taylor expand to the product of mixture weight and expert functions F(I, B1, fo. n) == olz BI, 3o) (zx 1)), where we let axx S1, 86) == I+exp(- x-063' To obtain the desired decomposition of fā_ (x) fc. (r), we provide below in Definition a condition, which we refer to as a strong identifiability condition, that guarantees that the derivatives of the product F(z, B1, So. n) w.r.t its parameters are linearly independent.
Definition (Strong identifiability.. An expert function 1(x, n) is called strongly identifiable if is twice differentiable war. its parameter n, for p-almost all x and, for any positive integer and any pair-wise distinct choices of parameters {(Bor, Bis, (i),))) the functions in the classes
① galthal F (x,, (s, Box. n): : [4] (11:12) E N x N9, hl 122 S 2 037'0m
and
OF
03 0 Orpas (I, BIG, Bor, m.) : [4], (a1, a2, as) E NO |o =
are linearly independent, for p-almost all x.
Example. Consider experts formulated as two-layer neural networks, i.e., h(r, (a, sp(( (a xx where oo is activation function and (a, b) Rd x R. If Og, then the expert h(- (a, b)) is not strongly identifiable for any choices of the activation function yo. On the other hand, if a o Og and ip is one among popular activation functions such as ReLU and GELU, then h(, (a, 6)) is strongly identifiable expert. By contrast, if & is a polynomial, e.g. (o()) = 2", then the expert h(: (a, b)) does not meet the strong identifiability condition.
Intuitively, the strong identifiability condition helps eliminate potential interactions among parameters expressed in the language of PDE (see equations (8) and (11) where gating parameters 3, interact with expert parameters a). Such interactions are demonstrated to result significantly slow expert estimation rates presented in Theorem 3 and 4). From the technical perspective, key step in our proof techniques rely on the decomposition of the discrepancy fa_ (x) fc. (æ) into a combination of linearly independent terms. This can be done by applying Taylor expansions to the function F(x, B1, 3. v)) = a (3 + (z) v)) defined as the product of the sigmoid gating and the expert function h. Thus, the condition is to ensure that terms in the decomposition are linearly independent.
Next, following the strategy first introduced by Manole et al. 21 and then adopted in several convergence analyses of MoE (22 (23) we proceed construct a loss function among parameters based on the notion of Voronoi cells.
Voronoi loss. Given an arbitrary mixing measure G with K' < k atoms, distribute its atoms across the Voronoi cells (A₁ = A,(G). j E [k.]] generated by the atoms of G., where
A₁ = {i E [k]] ||w - will < ||w wi ll. Ve + j}.
(5)
5

---

with w/ == (B1s, na) and wj- == nf). In particular, the cardinality of the Voronoi cell Aj corresponding to the least squares estimator G, in (3) is the number of fitted components assigned (and, likely, converging) to the true atoms wj. Then, define the Voronoi loss function
k
G.( = Σ
Σ
+
|A ||
EEA, 1 (xi) 1 1+ex expp(
==1
j=liEA,
+ Σ
| + 1 + ||Ansl] 1=K+1+EA;
(6)
where we let A81új = 31s Fig. ABoe := So; - and Anj == % ?z. Above, if the Voronoi cell A, is empty, then the summation term becomes zero by convention. Additionally, since (G, G.) 0 if and only if G G., it follows that when Di (G, G.) is sufficiently small, the differences A 181 and Any are also small. This observation suggests that, though not symmetric its arguments and therefore not metric, (G, is nonetheless suitable loss function for capturing the discrepancy between the least squares estimator G, and the true mixing measures G.. Furthermore, it is worth noting that the Voronoi loss function Dj can be efficiently evaluated, as its computational complexity is of order O x k.).
In our next result, whose proof can be found in Appendix A.2 we show that the Voronoi loss between the least squares estimator and the true mixing measure is upper bounded, up to universal multiplicative constants, by the estimation error for the regression function.
Theorem 2. Let x + Mx,n)) be a strongly identifiable expert function.. Then the lower bound
|l5G JG. 2 DD(G,
holds true for any G E (O))). As consequence, this result together with the regression estimation rate in Theorem indicate that DD (G.. G.) Op((oo(n))/]))
The following remarks regarding the results of Theorem 2 are in order.
(i) follows from the formulation of the loss function D, that the estimation rates for the over- specified parameters I Mīg. where j E [.), are all of order (pol)o())//)) Note that as the expert 11., 17) is twice differentiable over a bounded domain, it is also a Lipschitz function. Thus, letting Gn := Σxn 1
Bus)) 6 (3t. we obtain that
sup |1(2, (7)) h12 n3) L1 |6 nill S Op(llog(n//m)))
for any E (,,),, where I, 2 is Lipschitz constant. The above bound indicates that the strongly identifiable expert h(, n7)) is over-specified, i.e. fitted by at least two experts, then its estimation rate is of order (log((()))n))
(ii) Secondly, for exact-specified parameters 37 77. where ++1 < < k., the rates for estimating them are faster than those of their over-specified counterparts Remark (i), standing at order
O Op( Op(1lo((())))) ([oog())/]]3 By arguing as in equation ( we deduce that the expert n(,) also enjoys the faster estimation rate of order Op(log(n)/m F), which is parametric on the sample size n.
(iii) Putting the above two remarks together, we observe that when the expert functions are formulated as feed-forward networks with a widely used activation, namely ReLU and GELU, their estimation rates under the sigmoid gating MoE matches exactly those under the softmax gating MoE: see Theorem 3.2 in (26).
Non-strongly identifiable experts. The strong identifiability condition of Definition iis far from technical requirement: it is in fact crucial assumption. To illustrate its importance, we investigate the estimation rates of ridge experts of the form {h(x, (a².b;)) = o p((a)) Tx bb)) E [k=]], where is a scalar function, that fail to satisfy the strong identifiability condition. For ridge experts, the strong identifiability will not be met in two cases: when at least one among over-specified parameters aj equals Oz,, for any arbitrary activation function yp (Scenario I) and when is a polynomial of the form (p()) = z", for p e N (Scenario II).

---

Scenario I. Without loss of generality, we may assume that aj Oa, which means that the first expert ((((f)) x + bi) becomes independent of the input z. This gives rise to an interaction among the gating parameter 31 and the expert parameter a via the partial differential equation (PDE)
OF
OF
(x; iūn Pon 61, Coj.9%
(z; Si Po1 ai,b])
(8)
03,
aa
oa;; Br, (a) TT b), and Coj is some constant depending on bi and 861 The above PDE can be verified by taking the derivatives of F(x; 31, So a, b) w.r.t 31 and a and using the fact that 31 = aj O2. Notably, the PDE (8) accounts for the violation of the strong identifiability of the expert (p((a)) x + bi) under the Scenario To this end, we propose the following Voronoi loss to analyze the effects of such parameter interaction the expert estimation rates in Theorem 3
where we recall that F(2; 31, 86, a,
&
(2, =
|| + || ||"
1 + exp(-
1 exp( exe((-800
1=1 iEA,
j= iEA,
+
A + 11 + || |I" + |Abbl"
(9)
j=k+1 rEA;
The following result establishes a minimax lower bound on the estimation error of G. the D2, loss. Theorem Suppose that the experts take the form cp(a" T x + b). Then, under the Scenario we have E 1 (D,,- (G.,. 2n
sup for any r 2 1, where Esc indicates that the expectation taken w.r. the product measure with fö and the infimum over all estimators taking values in Gb (θ).
inf (9) (e)
See Appendix A.3 for the proof details. We highlight some important implications of Theorem 3
(i) The estimation rates for the parameters Bi oz and bj are slower than (p(n21/2r), for any 1. This means that they are slower than any polynomial rates, and could be of order Or(1/ (o)/))))))).
(ii) Using the same reasoning described after equation we have
sup (T)(() x - B7)) <((a))T= x + b3)l S L2 (105 atll 03 b7)),
(10)
where Le is a Lipschitz constant. As a consequence, the rates for estimating experts c((aa) Tx 67) are no better than those for estimating the parameters aj and bj. and could also be slow as (p(1/ lo)( This result suggests that all the expert parameters ......, of, should be different from O₂ to hope for fast convergence rates. In other words, every expert of the form slaa x + b) in the MoE model should vary with the input value.
Scenario II Now, we consider the second scenario when y a polynomial of the form (fx) = zP, for p N. For simplicity, we will only focus on the setting of p = i... h(z, = (aj) T x + bj: the case of > 1 can be argued in a very similar fashion. The structure of the polynomial experis leads to a non-linear interactions among the parameters, expressed through the PDE
02 F 03,
F2F
(x; 3 b;) = dad3n (r, a; b;),
(11)
0b
where F(x; $1, ,9, a, b) := (2; 31 (a +). This interaction is the reason why the polynomial experts are not strongly identifiable. Ultimately, this dependence yields slow rates of convergence for expert estimation, just like in Scenario I, as revealed by the next result, which deploys the Voronoi loss D2, of equation (
Theorem 4. Suppose that the experts take the form (a T + 6)P, for some p N. Then, under the Scenario II and for any r 2 1,
inf G.EGG (9) GE
sup
E (2,r (Gn, G)] ス n
1/2 112
where Ei indicates the expectation taken w.r.t the product measure with fü.
7

---

The proof of Theorem 4 is in Appendix A.4 To sum up, when either the experts are independent of the input or they take polynomial form, their estimation rates could be as slow as (r(((o(()))))))
3.2 Regime 2 of Gating Parameters
Recall that under the Regime 2, at least one among the over-specified parameters 8i is different from Og. As discussed in Section 2 the least squares regression estimator fa.. in this case converges to
a regression function fg. where G E Gx (+) == arg ( (0) llfe - fa, [[)))))) That is, the estimators of the parameters specifying fä. converge to the parameters of fz. WLOG, we may
assume that G = Eisl I+exp(-Bo.) (3).
Similarly to Regime 1, we also provide in Definition 2a weak identifabilitty condition to characterize those expert functions that will have fast estimation rates under the Regime Weakly identifiable experts are required to satisfy only subset of the conditions imposed to strongly identifiable experts.
Definition (Weak identifiabilityy. An expert function + h(r, n) is called weakly identifiable if it twice differentiable war. its parameter n for p-almost all and, for any positive integer E and any pair-wise distinct choices of parameters {Boi, Bis ms) 1= the functions in the class
B1s, Boi, m)) : [e], (01, a2 0s) E No MPxxNxNN,, |o =1 100000mp Orpas (x, 0 0
OF
are linearly independent, for p-almost all x.
Example. Let take an expert network fu(2z, (a, b)) sp(a +) as example. It can be checked that if a * Oa and the activation function op is a ReLU or GELU or polynomial, then the expert h(x, (a, b)) is weakly identifiable. Conversely, if Oz, i.e. the expert does not depend the input, then the weak identifiability condition is not met, regardless of the choice of the activation.
We now describe the convergence rates for expert estimation under Regime 2. Since the analysis of input-independent experts can done similarly to Theorem 3 we will focus only on weakly identifiable experts. As it turned out, the appropriate Voronoi loss function is given by
Da = ΣE |8o: - Bosl + 1131+ - -Billl 11m 72ll
(12)
j=1 VEA,
Theorem 5. Let h(,, v) be a weakly identifiable expert. Then, for any mixing measure G ()),
inf GEG.(0)
I|ʃa -
As a consequence, we obtain that inf (e). Dg(G,n G) = Op(lo(())/n]))
The proof of Theorem 5 is in Appendix A.5 As a direct consequence of the bound and the definition of D3, the convergence rates of the estimators 3n. 94 and rj are of the same of parametric order (p(oog(()) ()) Furthermore, by equation 7 also establish that the convergence rate of the expert estimator f(z ff() is also of order Op (nngnnn/]]] Comparing these estimation rates with those established by 26 assuming instead softmax gating, there are two main observations:
(i When the expert is neural network with ReLU GELU activation, the above parametric expert estimation rate obtained using the sigmoid gate is faster than the rate guranteed by the softmax gate, which could be of order (p(loo(n))))))
(see [Theorem 3.2, [26]])
(ii) When the activation is polynomial, the gap between expert estimation rates when using the sigmoid gate versus when using the softmax gate becomes even more dramatic: Op((log())//)] compared to (1/ log(n)) (see Theorem 4.6 in (6)).
The above remarks highlight the considerable benefits of deploying sigmoid gating over softmax gating in MoE models: provably faster estimation rates not only for expert networks with popular activations like ReLU and GELU, but also for those with polynomial activation.
8

---

Table 2: True Parameters for Gating and Experts. The variance for the gating parameters is V. 0.01/d / and for the expert parameters is v. = 1/d, where d = 32.
Gating parameters
Expert parameters
[
NN(O,, Ugla)
1 Bb ~ ~N(, of ~ (N(( bg ~ ~(000)
[8
0a
==8 8
4 Numerical Experiments
In this section, we perform some numerical experiments to empirically demonstrate our claim that the sigmoid gating is more sample efficient than the softmax gating.
From Table 1 it can be seen that the sigmoid gating shares the same expert estimation rates as the softmax gating under the Regime 1. However, the former gating outperforms the latter under the Regime 2, particularly for ReLU experts and polynomial experts. Therefore, we will consider those experts under the Regime 2 in our subsequent experiments.
Data generation. In particular, we generate the data by first sampling X, ~ Uniform([-1, 11 for i l=,...., Then, generate Y, according the following model:
Y = 9G. (X;) + Ci,
;=
(13)
where the regression function gG. (.) take the form of a softmax gating MoE:
k-
gG. (x) Soltmax((11)) x (bx) &b ((at))x++) b(),
(14)
The input data dimension is 32. We employ k, 8 experts of the form Φ(a T x + b), where is either the identity function or the ReLU function. The variance of Gaussian noise is 0.01.
Experimental setup. We summarize the choices of the ground-truth parameters Büs. Big of and bg for 1 < 8 in T which satisfies the condition of the Regime 2.
Training procedure. For each sample size n, spanning from 10 to 10 we perform 20 experiments. In every experiment, we employ k = k +1= fitted experts, and the parameters initialization for the gating's 's and experts' parameters are adjusted to be near the true parameters, minimizing potential instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across 10 epochs, employing a learning rate of 1) 0.1 to fit model to the synthetic data. For each experiment, we calculate the Voronoi losses for every model and report the mean values for each sample size in Figure I Error bars representing two standard deviations are also shown.
Results. In Figure [ when employing the ReLU experts, that is, d is the ReLU function, the Voronoi loss for the sigmoid gating approaches zero at the rate of order O(n (0-51.) ) which nearly matches our theoretical results in Theorem (5) Mcanwhile, the loss for the softmax gating converges to zero at the slower rate O(n 00.24). On the other hand, when using the linear experts, that is, is the identity function, the vanishing rate of the Voronoi loss associated with the sigmoid gating is O(n (0.40.. ), while that for the softmax gating is significantly slower, standing at O(n (0011.... These observations empirically shows that the sigmoid gating is more sample efficient than the softmax gating.
5 Discussion
In this paper, we carry out convergence analysis of least squares expert estimation under MoE models with the sigmoid gating, which was found empirically to be robust to the representation collapse issue and to have a favorable performance in the MoE applications. We demonstrate that under both the gating regimes, the sigmoid gating requires smaller sample size than the softmax gating to reach the same expert estimation error: that is, the sigmoid gating is more sample efficient than its softmax counterpart. Furthermore, we also verify that experts formulated as feed-forward networks with popular activation functions such as ReLU and GELU are more compatible with the sigmoid gating than other types of expert functions.

---

37-
Softaee
G.)
17
G
0L1 m-t
120 20
10
L,00s
3.8
Edtt ir Cluctiyy
Honwi HME
Strwed ME
DvG2. G.)
011 A-
016 am
10
167
117
samppe size 12
samppe size
(a) ReLU Expert
(b) Linear Expert
Figure 1: Log-log scaled plots displaying the empirical averages of the Voronoi losses when using the sigmoid gating (blue line) versus when using the sofmax gating (green line) under the same data. The red dash-dotted lines illustrate the fitted lines for determining the empirical convergence rates.
Practical implications. Our theoretical findings support the following conclusions, which have considerable practical relevance.
(I.1) The sigmoid gate is more sample efficient than the softmax gate for expert estimation. It can be seen from Table that while the expert estimation rates obtained when using the sigmoid gate match those attained when using the softmax gate under the Regime 1, the former are totally faster than the latter under the Regime 2. Notably, Regime 2 is closer to practice since the gating values under this regime hinge upon the input while those under the Regime are input-independent. As consequence, we can claim that the sigmoid gate is more sample efficient than the softmax gate from the perspective of the expert estimation problem.
(I.2) The sigmoid gate is compatible with a broader class of experts than the softmax gate. It follows from the expert characterization for Regime 1 (resp. Regime in Definition [ (resp. Definition 2 that formulating expert functions as feed-forward networks (36) with commonly used activation functions such ReLU and GELU expert estimation rates, and thus, require a smaller sample size to reach the same tolerance of estimating experts compared to the softmax gate. Thus, our theories indicate that the sigmoid gating is compatible with a broader class of experts than the softmax gating. This implication is particularly useful when people employ mixture of fine-grained (shallow) expert networks
polynomial activation will lead to faster
Limitations and future directions. There are two main limitations of our current analysis:
1. We assume implicitly that the ground-trude parameters are independent of the sample size. Therefore, the expert estimation rates established the paper are point-wise rather than the desirable uniform rates. A potential approach to cope with this problem is using the techniques for deriving the uniform parameter estimation rates in mixture models (6) and in MoE models (37]. However, those techniques are only valid for input-independent gating MoE, and we believe that further technical
tools should be developed to adapt such framework to the sigmoid gating MoE. Thus, we leave it for future development.
The assumption that the true regression function belongs the parametric class of MoE models under sigmoid gating is, of course, quite restrictive, and likely to be violated in real-world settings. This issue can be alleviated by assuming the data are sampled from regression framework with an arbitrary regression function g, which is not necessarily formulated as mixture of experts.
Under that setting, the least squares estimator G, converges to G E arg ming (e) Ilfe =ll Nevertheless, it requires the comprehensive knowledge of the universal approximation power of the sigmoid function, which has been slightly explored in [ to determine the expert estimation rate. Therefore, we leave this potential direction for future work.
Acknowledgements
NH acknowledges support from the NSF IFMI. 2019844 and the NSF Institute for Foundations of Machine Learning.
10

---

References
[1] A. Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information Theory, 39(3):930 945, 1993.
[2] Z. Chi, L. Dong. S. Huang, D. Dai. S. Ma, B. Patra, Singhal, Bajaj. X. Song, X.-L. Mao, H. Huang, and F. Wei. On the representation collapse of sparse mixture of experts. In A. H. Oh, Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[3] Y. Chow, A. Tulepbergenov, O. Nachum, D. Gupta, M. Ryu, M. Ghavamzadeh, and C. Boutilier. Mixture-of-Expert Approach to RL-based Dialogue Management. In The Eleventh Interna- tional Conference Learning Representations, 2023.
[4] R. Csordás, K. Irie, and J. Schmidhuber. Approximating two-layer feedforward networks for efficient transformers. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 674 692, Singapore, Dec. 2023. Association for Computational Linguistics.
[5] D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng. X. Yu, Y. Wu, Z. Xic, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. Deepseckmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint 2rXiv:2401.044 .04088, 2024.
[6] D. Do, H. Nguyen, K. Nguyen, and N. Ho. Minimax optimal rate for parameter estimation in multivariate deviated models. In Advances in Neural Information Processing Systems, volume 36, pages 30096-30133. Curran Associates, Inc., 2023.
[7] N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. Yu, O. Firat, B. Zoph, I.. Fedus, M. Bosma, 7. Zhou, T. Wang, E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellssern, T. Duke, L. Dixon, K. Zhang, Q Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts In ICML, 2022.
[8] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:1-39, 2022.
[9] A. Gulati, J. Qin, C. Chiu, N. Parmar, Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: canolution-augmennt Transformer for Speech Recognition. In Proc. Interspeech 2020, pages 5036-5040, 2020.
[10] Gupta, S. Mukherjee, Subudhi, E. Gonzalez, D. Jose, Awadallah, and Gao. Sparsely activated mixture-of-experts are robust multi-task learners. arXiv preprint arxiv 2204.0768, 2022.
[11] X. Han, H. Nguyen, C. Harris, N. Ho, and S. Saria. Fusemoe: ofxture-of-experre transformers for fleximodal fusion. In Advances in Neural Information Processing Systems, 2024.
[12] H. Hazimeh, Z. Zhao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, R. Mazumder, L. Hong, and Chi. DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages Curran Associates, Inc., 2021.
[13] X. O. He. Mixture of million experts. arXiv preprint arXiv:2407.04153 2024.
[14] N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence rates for Gaussian mixtures of experts. Journal of Machine Leurning Research, 23(323):1-81, 2022.
[15] R. A. Jacobs, M. Jordan, Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3, 1991.

---

[16] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L.. Scao, Gervet, T. Lavril, T. Wang. TT. Lacroix, and W. F. Sayed. Mixtral of experts. arxiv preprint arxiv 2401.04088, 2024.
I..
[17] M. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6: 1:141224, 1994.
[18] J. Lee- hee-Thorp and J. Ainslie. Sparse mixers: Combining moe and mixing to build a more efficient BERT. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 58-75, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguisties.
[19] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Reprexentations, 2021.
[20] H. Liang, Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, and Z. Wang. VIT: Mixture-off of Experts Vision Transformer for Efficient Multi-task Learning with Model- Accelerator Co-design. In NeurIPS, 2022.
[21] Manole and N. Ho. Refiedd convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 14979-15006. PMLR, 17-23 Jul 2022.
[22] H. Nguyen, P. Akbarian, and N. Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML, 2024.
[23] H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the ICMI., 2024.
[24] H. Nguyen, Akbarian, T. Pham, T. Nguyen, S. Zhang, and N. Ho. Statistical advantages of perturbing cosine router in mixture of experts. arXiv preprint arXiv:2405. 14131, 2024.
[25] H. Nguyen, P. Akbarian, F. Yan, and N. Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations, 2024.
[26] H. Nguyen, N. Ho, and A. Rinaldo. On least square estimation in softmax gating mixture of experts. In Proceedings of the ICMI, 2024.
[27] H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. Advances in Neural Information Processing Systems, 2023.
[28] S. Obando Ceron, G. Sokar, T. Willi, C. Lyle, Farebrother, J. N. Foerster, G. K. Dziugaite, D. Precup, and P. S. Castro. Mixtures of experts unlock parameter scaling for deep RL. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 38520-3850 3854000 PMLR, 21-27 Jul 2024.
[29] F. Peng, R. A. Jacobs, and M. A. Tanner. Bayesian Inference in of--eertss and Hierarchical Mixtures-of--Exprts Models With an Application to Speech Recognition. Journal of the American Statistical Association, 91 91445))953,966, 1996.
[30] Q. Pham, G. Do, H. Nguyen, T. Nguyen, C. Liu, M. Sartipi, B. T. Nguyen, S. Ramasamy, X. Li, S. Hoi, and N. Ho. Competesmoe effective training of sparse mixture of experts via competition. arXiv preprint arXiv:2402.025500 2024.
[31] Puigcerver, C. Riquelme, B. Mustafa, and N. Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, 2024.
[32] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, Jenatton, A. S. Pint, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In Advances Neural Information Processing Systems, volume 34, pages 9583-8595. Curran Associates, Inc., 2021.
12

---

[33] C. Ruiz, J. Puigcerver. B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In NeurIPS, 2021.
[34] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra- geously large neural networks: The sparsely-gates mixture-of-experts layer. In In International Conference on Learning Representations, 2017.
[35] S. van de Geer. Empirical processes in M-estimation. Cambridge University Press, 2000.
[36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, I.. u. Kaiser, and
I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[37] F. Yan, H. Nguyen, D. Le, P. Akbarian, and N. Ho. Understanding expert structures on minimax parameter estimation in contaminated mixture experts. arXiv preprint arXiv:24 /0. 12.5258,, 2024.
[38] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech, 2021.
[39] B. Yu. Assouad, Fano, and Le Cam. Festschrift for Lucien Le Cam, pages 423 423-435, 1997.
[40] Y. Zhou, N. Du, Y. Huang, D. Peng, C. Lan, D. Huang, S. Shakeri, D. So, A. Dai, Y. Lu, Z. Chen, Q. Le, C. Cui, Laudon, and J. Dean. Brainformers: Trading simplicity for efficiency. In International Conference on Machine Learning, pages 42531-42542. PMLR, 2023.
13

---

Supplement to "Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts"
In this supplementary material, we present proofs for the main results of the paper in Appendix A while we study the identifiability of the sigmoid gating mixture of experts (MoE) in Appendix B
Lastly, we provide additional numerical experiments verify our theory in Appendix C
A Proof of Main Results
In this appendix, we provide proof' for the theoretical results introduced in Section 2 and Section of the paper.
A.1 Proof of Theorem
To streamline the arguments for this proof, we need to define some necessary notations. First of all, let (+) = (fa(x): G E (+)) stand for the of all regression functions in Gy (9). Next, for each > O, we define the (pb-lalll centered around the regression function fa, (x) and intersected with the set (+) as
F F
{ E (x(
Ilf fa. (())))))) s 8}
Furthermore, the size of the above ball is captured by the following integral as suggested in 35
JB(O,F((0,8)) Se 182/21 (1, (()(W) di 6,
(15)
in which (t. Fu (e. -11)2()) denotes the bracketing entropy 35 of Fe(O,t) under the (2( norm, and t V ; max(t, 8}. By arguing in a similar fashion to Theorem 7.4 and Theorem 9.2 (35) with adapted notations to this work, we achieve the following lemma:
Lemma 1. Let (6)) J&( such that (68) is non-increasing function of 8. Then, for some universal constant e and for some sequence (6,) that satisfies vni,,, 2 (V)) the following holds for any 5 2 on:
P(15a. - fc. > >6) cexp
nd² c2
General Picture. If we are able to demonstrate that the following bound holds for any & E (0,1/2):
(lll) $ log(1/e),
(16)
then it follows that
(B(0,F((0,0)) = /"
1/2 (t, F,,, f), (122))) v
82/213
log(1/t)dt V
(17)
62/213
By choosing (( = 8. [l0g(1/6))//2 then (8) non-increasing function 8. Moreover, equation 17 suggests that (5) 2 [5 (O,, Additionally, we set ó, (og(n)/nn n, and then get that vn 2 (8.,) for some universal constant c. According to Lemma we achieve the desired conclusion of the theorem. As consequence, suffices to establish the bound (16).
Proof of inequality (T)). Note that as the expert functions are bounded, we have that [fo(z)) S M for almost every where M > 0 is a bounded constant.
Next, let < and (G....., (N) } be the 7-cover under the L22 (1²( -nomm of the set Fx(O), where : NN || ||->) stands for the 7--oovrrig number of the metric space (O). |l..)). Now, we consider brackets of the form [L,(x), (z)). where we define
L((x) = (xx T,0).
U²(x)= == max(C((2) + T,, M),
14

---

for all i E [M]. Then, it can be verified that Fx(O) C U U and Ui(x) - L((x) -
min M.. Thus, we obtain that
||0 1 LillEA(H) = (U. & 47)dp(z) = 47²,
which indicates that ||U - Lillxa(0) S 2T. From the definition of the bracketing entropy, we arrive at (2))), (1^)()))) S log N log (T, l1r.)).
(18)
Consequently, is sufficient provide an upper bound for the covering number N. For that purpose, let us denote A == ((3., IR x R (So, 31 7) ) and f2 == (1) RR (So, 31.7) E (). Recall that a compact set, therefore, A and Ω are also compact. Then, there exist r-covers A and 22 for A and 92, respectively. Furthermore, it can be validated that
A - Op(T OP(T- (d+1)))
12+ S Op(r )
Given a mixing measure G = Ein exp( E (+()), we take into account two other mixing measures defined as:
k
G= i=1 exp(-Box) (Bmm)))
Σ
;= Σ
+ e Boi) 8 (BL..)))
1
Above, fis S2, such that ii is the closest to 7i in that set, while (Bor, 310) E Dr is the closest to (Bor, 914) in that set. follows from the above formulations that
1
[/ Salle= sup sE
[M(r,, m)) - M(x, 1.)
exp(-(811) -oo)
1
sup ZEA' 1 + expl exp(-((11)) Bos)
11(2,, ni) h(x,
k sup sEA
|h(z, m²) - h(x,7)1
iml
sup L ||m - 7ill
=EX
1=1
< S
Here, the second inequality occurs the sigmoid weight is not larger than one and the third inequality follows from the fact that the expert function (zx, .) is Lipschitz function with some Lipschitz constant L1 > O.
Subsequently, we have
llfa - folle sup xEX
1(x, me)
+ exp( ( (B11) Boi) 1 1+ exopp( (-3p--(11)] Tx. 30x))
1
1
sup
11(x, 7:)
x
+
-Bo))
+ + ((31s) - Box)
sup M' ZE
- exp(-(B11) Boo) 1 +xeepp-- exp(=((11)T_ Boi)
sup M' L2 - Brell fall- || + 18os - Boel)
1=1
< kM' ( BB-B)
Above, the second inequality is due to the fact that the expert function is bounded, that is, [M(z, n)) M'. The third one happens as the sigmoid function Lipschitz function with some Lipschitz constant L2 > 0, while the fourth inequality occurs since the input space is bounded.
15

---

By the triangle inequality, we obtain
llf - follm & llfa - falle> + life - fallr ST.
From the definition of the covering number, yields that
Fx)), ||-~ 2 A-| 2 S Op(n (d= x O(n ok) ) S O(n (((-((+++)))) (19)
Putting equations and together, we get that
-l(e))))))) $ log(1/7).
Let = =/2, then we reach the desired bound
HB(E, (1ll)2()) & log(1/s).
Hence, the proof is totally completed.
A.2 Proof of Theorem
In this proof, aim establish the following inequality:
inf |lfe fa. fc.lllaw 0.
(20)
(e)
For that purpose, we divide the proof of the above inequality into local and global parts as below. Local part: In this part, we focus only on demonstrating the following local inequality:
llfa - fa. fo.llla 5 GEG. (i).
lim
inf
(21)
Assume by contrary that the above claim does not hold true, then there exists sequence of mixing measures G, E op."" in GR (9) such that Drn == 'D (Gn, G.) 0 and
|lfG. - fc. (ccll/ + 0,
(22)
as n. + 00. Let us recall that
Din =
| 12 ||
1 exp(-
1 exepl -Būr)
j=1
E
A + | + |
iE,Aj
where A0gi = 8g, Р65- A = B Big and Anj == 1p n/.
Since DI, 0 bc, deduce that
For <is < lc EIEA, [+exp[-BBJ 1+exp(-P6)
and (5T n..) - (Big, 13), for any
i Aj;
For & + 1 1 kw: (8 (i.m.)) (96 Sig. m3) for any i E Ag.
16

---

Step 1- Taylor expansion: In this step, we decompose the term fc. fc. (2) using Taylor expansion. Firstly, let denote
k
h(2,nf)) x (i))
11(2,, ((z))17)
(x)
Σ
Σ
+
cxp(pppppp (-0o3)
j=1
s
h(
1(2, 17))
1 + expl ((3tt) (00
+ exp( (31,)² x - Bos
j +
iEA,
k
h(x,, n7,))
h(x, 07))
1 1xexp( (oypp-((81)) x 30.
1 1 exp(-
1
1
,h(2, n3)
1 exp(-
1 exp(-1t5)
iEAs
h(r,
h(2, n3))
1+ exp( (ff)) x -Pie) 1+ exp(
-= iEA,
= An (z) + B (x) + (z)).
(23)
Let us denote (x, (1, == I+exp(-84 z- Bo)* Then, by means of the second-order Taylor expan- sion, can decompose the term An (2) defined above
An(x) Σ
o (o) n/' (xx Oa, (o) nj)
j=1 iEAs
almlo 0,37
ghalh
Σ
(A3 (An)))))
(x, 0a. phx)
S1132 (x, nf) + Ri (x)
j=l iEA, |n|=1
2 ΣΣ
al o 08
ghrllh Onp
Th
jime
(z, Oo Pbi)
(xr, nT) + (I(
(24)
j=1 1E
where Ri (z) is Taylor remainder such that Rí (x) nn 0 Oas oc, and we define
1 i ( 1r1 (Am
Th Gama
for any j E [R), Aj. 7 E No and 72 E No such that $ Im + ha/ S 2.
Next, let us denote
WT. = Σ
1 exp(- (.)
1 explepp-0 :55))
SEA;
for any j E [K), then the term Bx(x) can be represented as
k
Ba(x)= = Σ Wh 1,,,
(25)
Finally, recall that [4,1= = for any k + +1 S k. we can decompose the term (x) by applying the first-order Taylor expansion as follows:
Cx(x)= =
oxx Sy. B Ma, m (zx Big M2 ni)
j=k+1 REA;
(x, Bi (60) 2 goas
(x, n2) + R2(x), (26)
03 0,30
1=b+1
|a|=1
17

---

where (2(2) is a Taylor remainder such that (2 - as a - oc, and we define
S ,3,1,,2,03 = Σ
SEA; ad (A
(A))))) )a (An)))
for any + k+1sisk, k. on E No, 02 E N and 0g E No such that (ayl + 02 |as
Combine the decomposition of the terms (r) B, (x) and (x()) in equations (4 25 and (26
rewrite the difference fc, (x) fc. (z)
k
an 0
Ohz h
fG, - -c.) (x)
Σ j=1 j= TEA, 1-|=1
T
Jamm
(r, Oa, Por) Or (0.
+
W M(x, nJ)
j=1
R.
glaltono
Glasih
ST 2,01,02.3
(x, Gig Pb,) Srpas (2, 1151) + (1) + R2(x).
(27)
03 030
j=k+1
|a/=1
Step Non-vanishing coefficients: In this step, we show that at least one among the ratios T 3.. /Di,, WT/ /Dinin and St 53.11.22.03/ / DDin does not go to zero n tends to infinity. Indeed, assume by contrary that all of them converge to zero, i.e.
T JAm.22
W
S jay,02,0 0
+0, 0,
0,
Din
Din
Din
as n + co. Then, it follows that
Din Eド=1 ΣiEA, 1+exp(-8a) Fa)
1
ΣS= WWT 0;
1+expl (6o1)
Din
Din Σj= EiEd, |
12 Di..
Σ/= EiEd, Ed=1 2e,,
O, where ed,e
is vector in Rd whose u-th entry is one while other entries are zero;
Din ΣS EieA, Angil Din
Ej EiEA, Σ
+ 0;
Dia k+ h+l Esed, || 1 Din E5 E
- 0:
Ik. Dia Ljzk+1 Σie A =
Dim Lj==+1 1, 0;
1. -
Lj=k+1 EiEA, | 1. Ej=k+1 E
1 0,0.0 - 0.
Din
By taking the summation of the above limits, we obtain that
1
1
+
| + || [1]
Din
1 (x)(-
+ cxp(-36,,)
j=1
EE
j=1 EE,As
k.
A Biyl
|
|| + An 11
O.
j=k+1 TEA,
Due to the topological equivalence between the 1-norm and the 2-norm, we deduce that
1
|| 1| 11221111
Din
1 exp(expppp )
I + expeep(p-p -Būj)
VEAs
j=1 TE
Σ
A +
|
11 |] O.
j=b+1
which is contradiction. As consequence, at least among the ratios Ty DDns WT/DDar and S),an,a / Din must not approach
-
Step 3 Application of Fatou's lemma: In this step, we use the Fatou's lemma to argue against the result in Step 2, and then, achieve the desired inequality in equation (1
18

---

In particular, let us denote by ma the maximum of the absolute values of T J,i.n.n2 / Din. W / 1n and 02.01 Dnn Since at least one among those ratios must not approach zero - oc, we get that 1/m,, A 0o as n2 + oc.
Recall from the hypothesis in equation (2 that [[,e,, So- (w + n co, which indicates that f.. - fc. (n)/2in + due the equivalence between L (n)-norm and (2) norm. By means of the Fatou's 's lemma, we have
0 lim |IG. - fc. NG.lIrm
flim
i lim inf Ifa, - fa. du(x) 2 0.
MnDin
mn
This result implies that [Sa, (x) Sc. (m)]//mm, + 0 for almost every
Let us denote
Tammna /m Din - f3..m.1771
WT mm/mmm Din - W/
S0.102003 mm Din In 8j,01,02,035
with note that at least one among the limits i, Wy and 53 is non-zero. Then, from the decomposition in equation (27). we deduce that
alloa 837
Σ
(x,, Oa, Boi) Orprz (1r, (3) + Σ my h(r,, 11}))
f=1iEA, hl=1
j=1
a0allta2
glash h Oypa
Σ
Sj
05 0500
(x, Bi. Pog)
(Iz, n3)) 0,
j=k+1 a|=1
for almost every x. Note that the expert function h(,, is strongly identifiable, then the above for any [k.]. a E N4, 02 and 72, 0' E No such that < ha + 17ll < 2 and (01|+a 02 + |oagl = 1. This contradicts the fact that least one among the limits $1,01,02.04 different from zero.
equation implies that lim,,2 Wj 81.01.02.03
Hence, we obtain the local inequality in equation (2T). Consequently, there exists some > 0 such that
inf [|a fccllutw//P((G..G 0.
GEGU (I.
Global part:
Given the above result, it suffices to demonstrate that
(. llfe fc.lle (G) 0.
inf
(28)
Assume by contrary that the inequality (28) does not hold true, then we can find sequence of mixing measures G"₁ (x( such that Dí (G, G.) g and
[|,e; fa.l) 0, DG G.)
lim 8-000
which indicates that |l,a; - -G. (1)2(w)) + +0 as n + xe. Recall that t a compact set, therefore, we can replace the sequence G¹, by one of its subsequences that converges to a mixing measure G' E (S)). Since Di(G", G.) > e', we deduce that D, (G', G.) > e'.
Next, by invoking the Fatou's lemma, we have that
0 lim llfa; fa. flimm 2 { limint fa; [im 2-330
fc. (x)) dp(z).
Thus, we get that fe fc. (x:) for almost every From Proposition 1 deduce that G' = G.. Consequently, follows that (G",", "G',,.., G.) = O, contradicting the fact that 2 (G', (G), > 56 > 0.
Hence, the proof is completed.
19

---

A.3 Proof of Theorem
Lemma 2. If the following holds for any r 2 1:
Ilfa - SG. (1 = 0,
lim e Ge
inf
(29)
D2,(G,
then obtain that
inf GNEGL (0) (9)
sup
E fo (2,r (G:: G)] 2 n1/2
(30)
Proof of Lemma 2 Indeed, from the Gaussian assumption on the noise variables et, we obtain that
YJ [fa.. (x)) for all E [n]. Next, the assumption in equation 29 indicates for
sufficiently small e > and a fixed constant C1 > which we will choose later, we can find a mixing
measure G², E G% (+) such that ( G.) 2e and Ilfa; fc. [[( I C1e. From Le Cam's
lemma 39 as the Voronoi loss function 22, satisfies the weak triangle inequality, we obtain that
inf (0) CE)
sup
Eja (2n, G)]
(2, G.)
2
exp( exp(-nExx X KL (T))
8
2 e ....0 exp( pnap--nlfa: - fc. l1?:2()))
e exp( (xxp=")nn"",
(31)
where the second inequality is due to the fact that
(So: (X) -c. (X))²
KL(N(fo: (X), N(fc.
20
By choosing = n 11/2, we obtain that E exp( - = n 1/2 exp( As consequence. we achieve the desired minimax lower bound in equation (30).
1/2,
Main proof. Following from the result of Lemma it is sufficient to show that the following limit holds true for any 2 1:
[|fa - fc. (l(a(N) = 0. (G,
lim
inf G.)
(32)
To this end, need construct a sequence of mixing measures (G₂,) that satisfies D2, (Gn G.) 0 and
||... Na.1l1a( + 0, D2,r(Gr G.)
as n + 0o. Recall that under the Case 2, at least one among parameters .i...., of is equal to 0g- Without loss of generality, we may assume that at Og. Next, let us take into account the sequence (G,) with k, + atoms in which
exp(- - -6 =
1 2 exppp-pppp (81) and B = P for any Sii k. + 1;
8n 812 1 0. and ST: Pi for any
< + 1;
af ag a O& and at -, for any 3 < $ k. 1;
6 bj + -. by bi
2c and 6 = b for any 3 < i < k. 1,
where c E will be chosen later. Consequently, we get that
c² (2c)"
(G,, Gx)
n2 n" = On)
20

---

Next, we demonstrate that fl fa. ll ( (G,, G.) - O. In particular, following from the decomposition of fc. (x)
fa. (x) in equation 23) and the above parameter setting, we have that
(((a,)) T x + bp)
(0((a)) x + b9)
(x) (c)
1 + c (59.)
1 cxp(ccppp -862
2
1
:p((af) T x b9)
+ exp(- (xp)
1+ exp(- -901)
k 1
(o((a))
Tx + b,)
((((;=1) Tx: + b(_1)
1 + exp( (PT:)
1 1eepp( (Pi( 1014
(o(b
sp(bi)
1 + exp( -B)))
+ exp(-9fi
(p((b)) (0())
+ 2 2exp(-80
Thus, we deduce that
2 fc. (x)
[2 + 2 exxp(- [fc.
Eip((;) ((b))).
When r is an odd natural number: By applying the Taylor expansion up to order r-th, we get that
- b1)a
|2+ 2exp(-P61 - (x) - fc.
Σ
(()) + (z
a!
i=
Σ
)a
(g(a)(b)) + R1 (x),
a!n
where Rí (x) is the Taylor remainder such that Rí (z)) (Gn: G.) + 0 as co.
Note that E.=. root of this polynomial, which leads to the fact that
(g (bi) cal is an odd-order polynomial of c. Thus, we can choose e as
alno
(x)
fa., (x) fx) = + 2+2exp(-
From the above results, we deduce that [fc,, (x) - fc. (x)) D2, G.) + 0 for almost every As consequence, we achieve that [l.cc. fa. ( DD,, (Gn, G.)
+ co.
When r is an even natural number: By means of the Taylor expansion of order (r + 1) 1)-th we have
(1 + 29)
12 + 2exp(-961)) [sc, - Sc. Σ
( + R2(x),
aln
where (2(x) is Taylor remainder such that (2 Gz) + 0.
Since E (n ca is an odd-degree polynomial of variable e, we can argue in
alea
similar fashion to the scenario when is odd to obtain that ||,c. fa. (aa.ll2(w)/// (Gn, G.) -
Combine results from the above two scenarios of r, we obtain the claim in equation 32)
A.4 Proof of Theorem
Following from the result of Lemma 2 it is sufficient to show that the following limit holds true for any 1:
IIaa - SG. (1(2()) = 0. D2. G.)
lim 8 8
inf DL
(33)
21

---

To this end, we need to construct a sequence of mixing measures (G,) that satisfies (2, G.) -
0 and
|l... - Sa. (a + 0, D2, G.)
as 7 - co. Recall that under the Case 2, at least one among parameters -i,-... oz is equal to 0g- Without loss of generality, we may assume that at Og. Next, let us take into account the sequence (Gn) with k, + 1 atoms in which
501
P such that
2
1
1
1 + exp( -5))
1 exe((p888 -Bi1)
n++1"
i=1
and P6 = Po(i-)) for any 3 Sis k. + 1;
8 0 -s1 Og and pi: = (i 1) for any 3 3 h: +
af az aj and a. a;- for any أ k + 1;
bf bj + L. b2 bi + and 6. = b for any 3 < i < k. +
Consequently, we get that
2
D2, (G., G.) = n=+1 12" = =0(
Next, we demonstrate that IlSa. fa. (l()(w)) / /D2, (G., G.) O. In particular, following from the decomposition of Saa, (I) - Sa. (2:) in equation 23 and the above parameter setting, we have that
(a.)
x
+ by
(at) x + bj 1+ exp(
(c.
1 + exp( (8%))
Tx
1
1
+
[[aa)] x 2++ bj]
+
1 exp(-3i)
k. 1
(a.1) T x + 6,
(a)_2) (T + bū
+ exp( (-(8): Tx Tx- B 1 expl -(8 ((i-)) ) Txx- 5
(at) x + b9
(at) T x + bj
+ n=+1 [(a)) T bil
1 + exp( -8p)
+ exp(-3p6
16 bil nt#I [(ai] x + +1
+ exp( -B1))
1=1
((a)) Tor + b7].
n++
From the above result, we deduce that [fc, (x) - fa.))). (x))//2 (Gn,G.) G.) - 0 for almost every
As consequence, we achieve that |lfa. fc. fc..ll (G G.) 0 as n - cc. Hence, we obtain the claim in equation 33
A.5 Proof of Theorem
Following from the result of Corollary L it suffices establish the following inequality:
inf GEG((O)
I|/e - Sellua
0,
(34)
for any mixing measure G E (+). For that purpose, we divide the proof of the above inequality into local and global parts as in Appendix A.2 However, since the arguments for the global part remain the same (up to some changes of notations) for the over-specified setting, they will be omitted.
22

---

Therefore, given an arbitrary mixing measure G : ==Em. 1 exp(-Bni) (Bmn)) E (+), we
focus only on demonstrating that
lim GES, (G,
inf
||aa - (oollllll/////(..)))
(35)
Assume by contrary that the above inequality does not hold true, then there exists a sequence of
1
mixing measures Gn Di= 1 eep(pxppppp0 (b6) (177.0)) in (c( such that D3n D3 G) 0 and
[G... - (a)ll> 0
(36)
oc. Let denote by AW = A:(G., Voronoi cell of G, generated by the j-th components of G. Since our arguments are asymptotic, we may assume that those Voronoi cells do depend on the sample size, i.c. A₁- = AA"
Additionally, since G, and G share the same number of atoms under the Regime 2 and D3n + O. the Voronoi cell A; has only one element for any E [k:]. WLOG, assume that A₁ (i} for all i E [k), which follows that ( PP: nT,") -> (Bos, PIG, la as n. - oo for any i E [A)].
Thus, the Voronoi loss Dan can be represented
Dan
| + | + |4
(37)
where we denote A&%
Pb: Bos. A PT B1s and Any == 17," n7c
Now, we divide the proof of local part into three steps as follows:
Step 1 Taylor expansion: In this step, we decompose the term fc. (x) fc(() using Taylor
expansion. Let us denote ozz $1, Bu) = 1+exp(-A -8] =-))) then we have
k
h(1,
h(x, 17c)
(())) =
exp( -(3%) xx B
1 exp(-((11)) Bus)
(6)) 02 (An )33 g
glaslh
)
(x, BIg, Buo)
Orpas (x, 17)) + (x)
05t 0.00
Jlaolh
35 0.30 (s,, Pi, Box)
(cr, Rí R1),,,
(38)
Drps
|a|=1
where Rí (1(x) is Taylor remainder such that Rí (r)/Dass 0 as xc, and
Sha .g. = Σ
1 al (A
)A (Ank))
for any i E [k), E N4, 02 E N and 03 E No such that |ay| + 02 + |0al 1.
Step 2 Non-vanishing coefficients: In this step, we show that least one among the ratios ST ST. 22. 'Dan does not go to zero as n tends to infinity. Assume by contrary that all of them
/Da converge zero, i.e. Siay.a 'Dnn -> 0 as - co, for any [k),, on E Nd. 02 E N and ag E No such that |a1l + +22 layl = 1. Then, it follows that
A
|1 = Dan
1. Σ Ed
S hea.a. 0.0, + 0;
Daa
Dan E S
Sh....1.0 0:
Dan
Ar7 ||1
Σ/= E1=1
15 Si,0a.0.em..
+ 0.
Dan
23

---

By taking the summation of the above three limits, we deduce that
1
A
|| + |1 0.
Dan
Due to the topological equivalence between the 1 1-norm and 2-norm, we achieve that
1
A 0z
+ ||
|| +|
O,
D3n
as n + oo, which a contradiction. Thus, at least one among the ratios S aan / Dan must not approach zero
DC.
Step 3- Application of Fatou's lemma: In this step, use the Fatou's lemma to argue against the result in Step 2, and then, achieve the desired inequality in equation (35).
In partieular, let denote by m, the maximum of the absolute values of ST. o2.00 'Dan- Since least among those ratios must not approach zero as n co, we get that 1/m,, A Do oo.
Recall from the hypothesis in equation (36) that |lSa. Sall /D3n indicates that [l, ( + due to the equivalence between L (n)-norm and (p))-
as - oc, which
norm. By means of the Fatou's lemma, have
llfa.. falla.
/ lim inf n=000
[aa, (2) fa(()) dp(x) 2 O.
lim 20000 mn D3n
m, Dan
This result implics that [fc, (I) fo(x))/(/////)/ Panl + for almost every æ.
/ Let us denote S o0.1.gg.0 mm, D3n + Si.an an .ag,ag - oo with note that at least one among the limits Si an,ay is non-zero. Then, follows from the decomposition in equation (38) that
alailt020
(0,B1i. Bus) Gl h (z. 17.)
01,01,02,05
0 0,002
Orps
for almost every æ. Note that the expert function h(, n) is ... then the above equation implies that $1,0,,02033 0, for any i [k]], a1 E Nd, 02 N and ag No such that |a + 02 + |ayl 1. This contradicts the fact that at least one among the limits 0i.0 is different from zero.
Hence, we obtain the local inequality in equation
35
B Identifiability of the Sigmoid Gating Mixture of Experts
In this appendix, we explore the identifiability of the sigmoid gating MoE model in Proposition ! Proposition 1. If the equation fa(x) = fc. (x) holds true for almost every then it follows that G
Proof of Proposition 7 From the assumption of this proposition, we have
k
Σ
1 exp(- Boo) M(x, mR)
Σ exp(- -Bs)
h(x, n1),
(39)
Since the expert function h(, 11) is identifiable, the set (h(x, 1/)) i [K]}, where ./...., mh are distinet vectors for some V.' N. is linearly independent. If h: * k., then there exists some
i E [k] such that * ni for any j E [R:.]. This implies that I#exp( 2-B0 = O, which is contradiction. Thus, must have that k h. As result,
[k]] = {{ exp(-31.) Tx 3%)
: E [k.1].
+ exp((311)
8ox)
for almost every WLOG, we may assume that
1
(40)
1 + exp(- BBx) Tx Bos)
+ (xpp(-3))) [T) )
24

---

for almost every for any i E [k+]. It is worth noting that the sigmoid function is invariant to translations, then equation (40) indicates that $1. -ii; + U1 and for - + to for some 01 E IRd and Vo E I. Nevertheless, due to the assumptions B1R = Bix = Og and Bok füx
we have that V1 = Oa and up = 0. Consequently, it follows that 31, = Pi, and for = 9o: for any i E [k.]. Then, equation 39) can be represented
k.
h(x, mR) =
Σ
h(x, n;),
(41)
1+ -310 x Bos)
1
x P())
for almost every z. Next, let us denote J1, J2,..., "h....,,m Im as a partition of the index set [k.], where m k, such that (Bus. 91z) = (Bar, (ix) for any i, E J, and j E [Rx]]. On the other hand, when and i' do not belong to the same set Jj, we let (Bos. 31s) + (Sor, (iv). Thus, we can reformulate equation as
1 j=iiE,; 1+ exp(-Bri) Boi)
(x)
1(2, n
j=lieJy 1+ exp(- Tx
Būs))
for almost every Recall that B1, = Bi, and fo: = Pox for any E [k.]. then the above leads to {n :n:iE J3} = {n, i Jjh.
for any j E [m]. As a result, we achieve that
G Σ 1+ cxp(- Bus)
m
1
d(emm))
Σ
m
+ exp( -3t)) ((prnnt) = G..
j=1iE,,
1=1 NEJ,
Hence, the proof is totally completed.
C Additional Experiments
In this appendix, we conduct a simulation study to empirically validate our theoretical results on the convergence rates of the least squares estimators under both the Regime and the Regime of the sigmoid gating MoF. All the subsequent experiments are conducted on MacBook Air equipped with an MI chip CPU.
Regime 1. We begin with numerical experiments for the Regime in which we sample synthetic data based the model described in equation (]). In particular, we generate {(X,, Y.) C Rd x IR by first sampling X₃ Uniform([-1,1) 1j4) for i I,..., Then, generate Y, according to the following model:
Y1 (X.) + t i= 1, I,...,,
(42)
where the regression function fc. (.) defined as:
fo. = 1 1 exp( ( BGg) - ((a))) b7)
(43)
The input data dimension is set 32. We employ k, 8 experts of the form yp ((a)) x x+++0 6t). where the activation function is either the ReLU function or the identity function. The variance of Gaussian noise 6, is specified as 0.01.
Tx
The ground-truie gating parameters PG, E are drawn independently from an isotropic Gaussian distribution with zero mean and variance Vg 0.01/d for 1 < 6, while set 8 = =... Six. 0a. Similarly, the true expert parameters, (a by) E TRd x R, are drawn independently of an isotropic Gaussian distribution with zero mean and variance V 1/d for all experts. These parameters remain unchanged for all the experiments (see also Table
Training procedure. For each sample size n, spanning from 103 to 10 we perform 20 experiments. In every experiment, we employ k k. + fitted experts, and the parameters initialization for the gating's and experts' parameters are adjusted to be near the true parameters, minimizing potential
25

---

Table Ground-truth Parameters for Gating and Experts. The variance for the gating parameters is V/ = 0.01/d, and for the expert parameters is ve = 1/d.
Gating parameters
Expert parameters
81
0a
Bor ~ (g) at NN(aa, bj ~ No)
Table 4: True Parameters for Gating and Experts. The variance for the gating parameters is va 0.01/d /d/dam and for the expert parameters is V. 1/d.
Gating parameters
Expert parameters
[8
((Os,Vyla)
1 isisn
Pb: ~ ~((, of ~ ((Ta, 0f ~ N(0,ve)
18 0g
i 8
Lawer mynts Du G.)
Eseer oppers
0.2
30
Looes
001
16U caperte
IxlU epente
G.
1yt 23 (12
1.4
sample size 12
somppe size R2
(a) Regime 1
(b) Regime 2
Figure 2: Logarithmic plots empirical convergence rates. Figures 2a and 2b illustrate the empirical averages of the corresponding Voronoi losses under the Regime 1 and the Regime 2, respectively. The blue lines depict the Voronoi loss associated with the ReLU experts, while the green lines correspond that of the linear experts. The red dash-dotted lines are used illustrate the fitted lines for determining the empirical convergence rates. See Appendix C for the experimental details.
instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across 10 epochs, employing a learning rate of r = 0.1 to fit model to the synthetic data.
Results. For each experiment, values for each sample size in Figure 2a] Error bars representing two standard deviations are also shown. In Figure 2a both the empirical convergence rates corresponding the ReLU experts and the linear experts are analyzed under the over-specified setting. It can be seen that the use of ReLU experts induces a rapid convergence rate of order O(n 00.66), while the linear experts lead considerably slower rate of order Onn -0.02) Those empirical rates match our theoretical results in Theorem 2 and Theorem respectively.
calculate the Voronoi losses for every model and report the mean
Regime 2. In the experiments for the Regime 2, we apply the same experimental setup as in those for the Regime except for the generation of true parameters 87.- More specifically, 81; drawn independently from an isotropic Gaussian distribution. N (( Ta) for 1 s S 8, where (see also Table 4).
0g 0.01/d, /d. while set Bi, 0₂ for
Results. can observed from Figure 2b that the uses of ReLU experts and linear experts both
lead to fast empirical convergence rates of orders -0.48.8 and O(n (ln-2.22,, respectively. Those empirical rates align with our theoretical findings in Theorem
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions and scope are reflected accurately the abstract and introduction.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Diseussion Section describe possible extensions and generalizations of our analysis.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Each theoretical result contains the full set of assumptions. All the proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
27

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
All the experimental details are provided in Section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No].
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ca cc/
for more details.
28

---

reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips ctiips.cc/uublice www.ggiddcsooooo Cttp///ooeeeeiseeeeeeeeeeee wwriiisssioooiiioiooc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the experimental details are provided in Section 4 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The error bars are reported in Figure [nd and Figure
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
29

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the information the computer resources are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have read and followed all the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given the theoretical nature of the paper, we do not think there are any positive negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
30

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is theoretical work, and we do not release any data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We do not use any existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppswiphhhtooooooooap thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
31

---

Justification:
We will release the code for our simulations after the review process.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "MiMXNJlrVP", "reviewer": "Reviewer_PmAq", "summary": "This paper presents a theoretical analysis of expert estimation in MoE models using sigmoid gating, in contrast to the more widely used softmax gating. The authors show that sigmoid gating leads to better sample efficiency compared to softmax gating for estimating expert parameters. In particular, the paper analyzes convergence rates for least squares estimation under two regimes: \n* Regime 1) when all over-specified gating parameters are zero\n* Regime 2) when at least one of the over-specified gating parameters is non-zero\n\nOne of the key findings is that experts formulated as NNs with common activations like ReLU and GELU obtain faster convergence rates under sigmoid gating compared to softmax gating. This work provides theoretical justification for the potential empirical benefits of sigmoid gating, showing it requires smaller sample sizes to achieve the same estimation error as softmax gating.", "questions": "Minor point:\n- misspecified is misspelled in line 175.dq", "limitations": "Two limitations concerning the assumptions used in the analyses are discussed in the limitations section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper provides rigorous proofs and establishes convergence rates under different conditions.\n- Sigmoid gating function has been less explored in the MoE field and analyzing its behaviour for expert estimation problem is interesting.", "weaknesses": "1) The structure of the paper is very confusing and could be much improved. There is way too much repetition in the paper. For example, the section Technical challenges in the introduction which defines Regime 1 and 2 (Lines 83-92) is largely copy pasted to lines 155-165 and is redundant. The various sections in the introductions particularly the contributions should be kept at a more high level and not just copy paste the method section. The authors should provide a more concise overview of the problem and main contributions. Remove technical details that are repeated in later sections.\n\n2) The recent insights into training Sparse MoEs demonstrate that \"the common practice of setting the size of experts in MoE to mirror the feed-forward layer (of a base dense network) is not optimal at almost any computational budget\" (Jakub Krajewski et al. 2024). If in practice, top-k combination of small fine-grained experts is proven to be more effective than a single large expert, are we dealing in a regime where all true experts are *over-specified* in practice? If this is the case what are the implications for Regime 1? Does it mean that none of the experts are input-dependent as all the over-specified parameters $β^*_{1i}$ are equal to $0_d$? Does Regime 1 hold in practice?\n\n3) It would be best if the authors could find some practical implications of their findings. The theoretical work presented in the paper does not establish any relationship to any of the SoTA MoE models in the literature. It is unclear to me how impactful the convergence analysis with the considered regimes is in practical MoE research works.\n\n4) At present, the content and organization of the paper is very close to the paper \"On Least Square Estimation in Softmax Gating Mixture of Experts\" (Nguyen et al. 2024). The sections are so similar that may cause copyright issues. \n\nReferences:\n1. Krajewski, Jakub, et al. \"Scaling laws for fine-grained mixture of experts.\" arXiv preprint arXiv:2402.07871 (2024).\n\n2. Nguyen, Huy, Nhat Ho, and Alessandro Rinaldo. \"On least squares estimation in softmax gating mixture of experts.\" ICML (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: The authors should provide a more concise overview of the main contributions, and remove technical details that are repeated in later sections.**\n\nThanks for your suggestions. We will modify the contribution paragraph as below, and consider removing repeated details in Section 2 in the revision.\n\n**Contributions.** In this paper, we carry out a convergence analysis of the sigmoid gating MoE under two regimes of the gating parameters. The main objective is to compare the sample efficiency between the sigmoid gating and the softmax gating. Our contributions can be summarized as follows:\n\n**(C.1) Convergence rate for the regression function.** We demonstrate in Theorem 1 that the regression estimation $f_{\\widehat{G}_n}$ \n\nconverges to its true counterpart $f_{G_*}$ at the rate of order $\\mathcal{O}_P(n^{-1/2})$, which is parametric on the sample size $n$. This regression estimation rate is then utilized for determining the expert estimation rates.\n\n**(C.2) Expert estimation rates under the Regime 1.** Under the first regime, we first establish a condition called *strong identifiability* to characterize which types of experts would yield polynomial estimation rates. In particular, we find out that the rates for estimating experts formulated as feed-forward networks (FFN) with popular activation functions such as ReLU and GELU are of polynomial orders. By contrast, those for polynomial experts and input-indepedent experts could be of order $\\mathcal{O}_P(1/\\log(n)$. Such expert convergence behavior is similar to that when using the softmax gating.\n\n**(C.3) Expert estimation rates under the Regime 2.** Under the second regime, the regression estimation $f_{\\widehat{G}_n}$ \n\nconverge to a function taking the form of a sigmoid gating MoE which is different from $f_{G_*}$. From our derived weak identifiability condition, it follows that estimation rates for FFN experts with ReLU or GELU activation and polynomial experts are of orders $O_P(n^{-1/2})$, which are substantially faster than those when using the softmax gating (see Table 1). Therefore, the sigmoid gating is more sample efficient than the softmax gating.\n\n### **Q2: If in practice, when using top-k combination of small fine-grained experts, are we dealing in a regime where all true experts are over-specified in practice? If this is the case what are the implications for Regime 1? Does Regime 1 hold in practice?**\n\nThanks for your questions.\n\nFirstly, when using the regression estimation as a mixture of small fine-grained experts [34] (which could have up to millions of experts [35]), it is highly likely that we are dealing with the scenario when all the true experts are over-specified.\n\nSecondly, assume that we use a mixture of small fine-grained experts, and all the true experts are over-specified. Then, under the Regime 1, all the gating parameters $\\beta^*_{1j}$ are equal to zero, which means that the true mixture weights (gating values) are independent of the input $x$. Moreover, the fitted parameters $\\widehat{\\beta}^n_{1i}$ must also converge to zero. Consequently, all the fitted mixture weights also become independent of the input $x$. However, according to the definitions of two regimes, the Regime 1 is much less likely to occur than the Regime 2 where at least one among the over-specified parameters $\\beta^*_{1j}$ is different from zero.\n\n### **Q3: What are the practical implications of the convergence analysis?**\n\nThanks for your question. There are two important practical implications from our analysis:\n\n**(i) Expert specialization:** The convergence behavior of expert estimation allows us to capture how fast an expert learns a specific task, which is one of the most important problems in the MoE literature known as expert specialization (see [4]). As the sigmoid gating is more sample efficient than the softmax gating, our theories suggest that it would be better to use the sigmoid gating in this field. \n\n**(ii) Expert compatibility:** Compared to the softmax gating, the estimation rates for feed-forward expert networks with ReLU activation and polynomial experts when using the sigmoid gating are much faster. Thus, our theories indicate that the sigmoid gating is compatible with a broader class of experts than the softmax gating. This implication is particularly useful when people employ a mixture of fine-grained (shallow) expert networks [35]. \n\n### **Q4: The content and organization of the paper are very close to the paper [21].**\n\nThanks for your comment. However, we respectfully disagree that the content and organization of our paper are close to those of [21] for the following reasons:\n\n**1. Content:**\n\n*(1.1) Different objectives:* the objective of [21] is to figure out what types of experts are compatible with the softmax gating in terms of estimating experts. Meanwhile, our paper focuses on the sample efficiency comparison between the sigmoid gating and the softmax gating\n\n*(1.2) Analysis of similar expert types for comparison*: since our main goal is to demonstrate that the sigmoid gating is more sample efficient than the softmax gating, it is necessary to analyze experts considered in [21] for the sake of comparison. This probably makes our content look similar to that in [21]. However, the derived expert estimation rates are different, particularly under the Regime 2 where the sigmoid gating outperforms its softmax counterpart (see Table 1).\n\n**2. Organization:** In [21], sections for main results are divided based on the types of experts, namely Section 3 for strongly identifiable experts, Section 4.1 for ridge experts with strongly independent activation function, and Section 4.2 for polynomial experts. On the other hand, in our paper, sections for main results are organized based on two regimes of gating parameters. More specifically, Section 3.1 is for Regime 1, while Section 3.2 is for Regime 2.\n\n**References**\n\n*Due to the space limit, we leave the references to the General Response section.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ywvbhqRA34", "reviewer": "Reviewer_tkzC", "summary": "The paper argues that the sigmoid gating function is more sample efficient than the softmax gating function in mixture of experts (MoE) modeling. It removes competition and estimates the contribution each expert independently.\n\nEmpirical studies show that sigmoid gating achieves superior performance, but the paper aims to provide theoretical backing for this claim.\n\nThey consider a regression framework and analyze the rates of convergence of the least squares estimator in over-specified cases.\n\nThe convergence rates for expert estimation are derived under two regimes: Regime 1 (all over-specified parameters are 0) and Regime 2 (at least one over-specified parameter is not 0).", "questions": "the paper lacks justification for the choice of the regimes. We also need to deep dig to understand the reasons for strong identifiability conditions (for experts to be distinct enough, if i understand correctly), bracketing entropy (model is not overly complex) and concentration inequalities. it is quite difficult to read, so some more intuition and explanation would be helpful.\n\nMinor:\n124 in can seen", "limitations": "As mentioned by the authors, \n\nThe paper assumes that the ground-truth parameters are independent of the sample size, leading to point-wise rather than uniform estimation rates. \n\nThe assumption that the true regression function belongs to the parametric class of MoE models under sigmoid gating is restrictive. This assumption is likely to be violated in real-world settings.\n\nI still find these assumptions reasonable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper demonstrates that sigmoid gating is more sample efficient than softmax gating, requiring fewer samples to achieve the same level of accuracy in expert estimation.\n\nexperts with feed-forward networks and commonly used activations (ReLU, GELU) have faster convergence rates under sigmoid gating than softmax gating.\n\nThe sigmoid gating mechanism is compatible with a broader class of expert functions compared to softmax gating. (ReLU and GELU, as well as polynomial activations)", "weaknesses": "The results are heavily dependent on specific assumptions, such as the distinctness of expert parameters and the boundedness of the input space. If these assumptions are violated in practical scenarios, the theoretical guarantees may not hold\n\nWhile the derivations are very appropriate and i appreciate the citations, it would be interesting to see even toy experiments byt the authorsthat show similar conversions to the theoretical ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: The results are heavily dependent on specific assumptions, such as the distinctness of expert parameters and the boundedness of the input space. If these assumptions are violated in practical scenarios, the theoretical guarantees may not hold.**\n\nThanks for your comments. We will explain why the assumptions mentioned above are reasonable. \n\nFirstly, let us begin with the assumption of distinct expert parameters. This assumption is to ensure that all the experts are different from each other. In practice, it is memory-inefficient to have two identical experts as they have the same expertise but we have to store different weight parameters for them. If there are two identical experts, we can merge them by taking the summation of their weights.\n\nSecondly, regarding the assumption of the bounded input space, we would like to emphasize that this is a standard assumption in the literature of expert estimation in the MoE (see [12, 21]). Moreover, we can address the issue that the magnitude of the input goes to infinity by normalizing the input value, which has been recently applied in practice (see [1, 36]) and should not affect the current theory.\n\n### **Q2: It would be interesting to see even toy experiments by the authors that show similar conversions to the theoretical ones.**\n\nThanks for your suggestion. Please refer to our response to the Common Question 1 in the General Section for further details.\n\n### **Q3: The paper lacks justification for the choice of the regimes.**\n\nThanks for your comment. Actually, we have already included the justification for dividing the convergence analysis into two regimes in the \"Technical challenges\" paragraph (see lines 83-90). Let us summarize it here. \n\nFirstly, we would like to emphasize that the regimes are determined based on the gating convergence. Recall that the true number of experts $k_*$ is unknown, and we over-specify the true model by a mixture of $k$ experts where $k>k_*$. Thus, there must be at least one atom of $G_*$ fitted by two atoms of $\\widehat{G}_n$. \n\nWLOG, assume that $(\\hat{\\beta}^n_{1i},\\hat{\\eta}^n_i)\\to(\\beta^*_1,\\eta^*_1)$ for all $i\\in\\{1,2\\}$. \n\nThen, we have $h(x,\\hat{\\eta}^n_i)\\to h(x,\\eta^*_1)$ for all $i\\in\\{1,2\\}$. \n\nTherefore, the regression function $f_{\\widehat{G}_n}$ converges to \n\n$f_{G_*}$ only if \n$$\\sum_{i=1}^{2}\\sigma((\\hat{\\beta}^n_{1i})^{\\top}x+\\hat{\\beta}^n_{0i})\\to\\sigma((\\beta^*_{11})^{\\top}x+\\beta^*_{01}),$$\nfor almost every $x$, where $\\sigma$ denotes the sigmoid function. This above limit occurs iff $\\beta^*_{11}=0_d$. As a consequence, we propose conducting the analysis under two following complement regimes:\n\nRegime 1: All the overspecified parameters $\\beta^*_{1i}$ are equal to zero;\n\nRegime 2. At least one among the over-specified parameters $\\beta^*_{1i}$ is different from zero.\n\n### **Q4: What are the explanations for the strong identifiability condition?**\n\nThanks for your question. We will explain the strong identifiability condition both intuitively and technically as follows:\n\n**Intuitively**, the strong identifiability condition helps eliminate potential interactions among parameters expressed in the language of PDE (see Eq. (8) and Eq. (11) where gating parameters $\\beta_1$ interact with expert parameters $a$). Such interactions are demonstrated to result in significantly slow expert estimation rates (see Theorem 3 and Theorem 4).\n\n**Technically**, a key step in our proof techniques rely on the decomposition of the discrepancy between $f_{\\widehat{G}_n}(x)$ and \n\n$f_{G_*}(x)$ into a combination of linearly independent terms. This can be done by applying Taylor expansions to the function $F(x,\\beta_1,\\beta_0,\\eta):=\\sigma(\\beta_1^{\\top}x+\\beta_0)h(x,\\eta)$ defined as the product of the sigmoid gating and the expert function $h$. Thus, the condition is to ensure that terms in the decomposition are linearly independent.\n\n### **Q5: Typo issue.**\n\nThanks for pointing out. We will correct them in the revision of our paper.\n\n**References**\n\n[1] Z. Chi. On the representation collapse of sparse mixture of experts. Advances in NeurIPS, 2022.\n\n[12] N. Ho. Convergence rates for Gaussian mixtures of experts. In JMLR, 2022.\n\n[21] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024.\n\n[36] B. Li. Sparse mixture-of-experts are domain generalizable learners. In ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gK2hI6VTBi", "reviewer": "Reviewer_4kwq", "summary": "This paper provides a theoretical analysis of the convergence rate of the least squares estimator for learning MoE with sigmoid gating. Based on the results, the authors conclude that sigmoid gating enjoys a faster convergence rate and requires a smaller sample size to achieve the same error compared to softmax gating.", "questions": "See my questions raised in Weaknesses.", "limitations": "The authors have discussed certain limitations in Section 4.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The topic is very interesting. It is intriguing to see how the selection of gating functions can affect sample efficiency in MoE.\n2. This problem is particularly relevant to recent important topics regarding language models, given that MoE has been applied in popular models like Mixtral.", "weaknesses": "1. The biggest concern to me is that the results from this paper do not fully support the main claim that “sigmoid gating is more sample efficient than softmax gating”. If I understand correctly (correct me if I'm wrong), the comparison between sigmoid and softmax is not under the same scenario. The convergence rate for softmax in Table 1 is from the analysis in [21], where the ground truth is an MoE model with softmax gating, while this paper considers the setting where the ground truth is an MoE model with sigmoid gating. These are two different setups and do not necessarily imply that, given the same ground truth function, sigmoid is necessarily more sample efficient than softmax. \n\n2. The only existing empirical work discussing the potential superiority of sigmoid gating that the authors have mentioned is [3], where the provided intuition is that softmax introduces competition between experts, while sigmoid does not have this issue. This is very straightforward, although not yet formally and theoretically justified. On the other hand, this paper explains the advantage from the perspective of sample efficiency. I wonder how these two perspectives can be connected. Is there anything regarding the competition view that can be reflected in this paper’s analysis? I believe this is important for understanding the effect of sigmoid gating.\n\n3. Following point 2, it would be helpful to provide empirical results to further support the main conclusion that sigmoid gating is more sample efficient. Even experiments on toy data can help. Otherwise, the statement regarding sample efficiency lacks empirical evidence. Additionally, even \"sigmoid gating is better than softmax gating\" alone does not seem to be well-supported by empirical observations from the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: The results from this paper do not fully support the main claim that “sigmoid gating is more sample efficient than softmax gating” because the comparison between sigmoid and softmax is not under the same setup.**\n\nThanks for your comment. Let us explain why it is reasonable to compare the expert estimation rates under the MoE model when using the sigmoid gating versus when using the softmax gating.\n\nOur paper (resp. Nguyen et. al. [21]) consider a well-specified setting where the data are generated from a regression model where the regression function is a sigmoid gating (resp. softmax gating) MoE in order to lay the foundation for a more realistic yet challenging misspecified setting where the data are not necessarily generated from those models. \n\nUnder that misspecified setting, the regression function is an arbitrary function $g(x)$ which is not necessarily a mixture of experts. Then, the least square estimator $\\widehat{G}_n$ defined in equation (3) converges to the mixing measure $\\widetilde{G}\\in\\mathcal{G}_k$ that minimizes the distance \n\n$||f_{G}-g||_2$, \n\nwhere $f_{G}$ is the regression function taking the form of the sigmoid gating (resp. softmax gating) MoE.\n\nThe insights from our theories and from Nguyen et. al. [21] under the well-specified setting indicate that the Voronoi loss functions can be used to obtain the estimation rates of individual parameters of the least square estimator $\\widehat{G}_n$ to those of $\\widetilde{G}$, and therefore, the expert estimation rates. Let us recall from Table 1 that under the Regime 1 (all the over-specified gating parameters are zero), using the sigmoid gating leads to the same expert estimation rates as when using the softmax gating. However, under the Regime 2 (not all the over-specified gating parameters equal zero), which is more likely to occur, the sigmoid gating totally outperforms the softmax gating in terms of the rates for estimating feed-forward expert networks and polynomial experts ($n^{-1/2}$ compared to either $n^{-1/4}$ or $1/\\log(n)$).\n\nThus, under the misspecified setting, the expert estimation rates achieved when we use the sigmoid gating should be faster than those obtained when we use the softmax gating. This accounts for the validity of the expert estimation rate comparison between these two gating functions.\n\n### **Q2: Is there anything regarding the non-competition of the sigmoid gating that can be reflected in the analysis?**\n\nThanks for your question. Let us show the connection between the non-competition of the sigmoid gating and our analysis. In particular, when using the sigmoid gating, the experts do not share the gating parameters, that is, the mixture weights are independent of each other. Thus, the interaction between the gating and expert parameters which induces slow estimation rates when using the softmax gating (see Eq. (4) in [21]) does not hold if we use the sigmoid gating. As a result, the expert estimation rates when using the sigmoid gating are either comparable (under the Regime 1) or faster (under the Regime 2) than those when using the softmax gating. \n\n### **Q3: It would be helpful to provide empirical results to further support the main conclusion that sigmoid gating is more sample efficient. Even experiments on toy data can help.**\n\nThanks for your suggestion. Please refer to our response to the Common Question 1 in the General Section for further details.\n\n### **Q4: Even \"sigmoid gating is better than softmax gating\" alone does not seem to be well-supported by empirical observations from the literature.**\n\nThanks for your comment. However, we respectfully disagree that the claim \"sigmoid gating is better than softmax gating\" is not well-supported by empirical observations from the literature. In particular, there are two recent works [1, 3] on the applications of MoE in language modeling showing that the performances when using the sigmoid gating are comparable or even better than those when using the softmax gating.\n\nFinally, we would like to emphasize that our main goal is to demonstrate that the sigmoid gating is more sample efficient than the softmax gating from the perspective of the expert estimation problem. We have not attempted to show that the sigmoid gating is better than the softmax gating in general.\n\n**References**\n\n[1] Z. Chi. On the representation collapse of sparse mixture of experts. Advances in NeurIPS, 2022.\n\n[3] R. Csordás. Approximating two-layer feedforward networks for efficient transformers. Findings of the EMNLP 2023.\n\n[21] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General Response**

Dear AC and reviewers,

We would like to thank you for your value feedback and constructive comments, which have helped us improve the paper substantially. We are encouraged by the endorsement that:

- Sigmoid gating function has been **less explored** in the MoE field, and the topic is **very interesting** (Reviewer PmAq and Reviewer 4kwq).

- This problem is particularly **relevant to recent important topics** regarding language models (Reviewer 4kwq).

- The paper provides **rigorous proofs** and establishes convergence rates under different conditions (Reviewer PmAq). 

There is one common concerns from the reviewers regarding the synthetic experiments, which will be addressed in the sequel.

### **CQ1: Synthetic experiments for empirically justifying the theoretical results.**

Thanks for your suggestion. Actually, we already conducted a simulation study to empirically validate our theoretical results on the convergence rates of the least squares estimators under both the Regime 1 and the Regime 2 of the sigmoid gating MoE in Appendix C. 

Moreover, we just also carry out the following numerical experiments which empirically demonstrate that the sigmoid gating is more sample efficient than softmax gating in MoE. 

**Experimental setup.** From Table 1 in our paper, it can be seen that the sigmoid gating shares the same expert estimation rates as the softmax gating under the Regime 1. However, the former gating outperforms the latter in terms of expert estimation rates under the Regime 2, particularly for polynomial experts. Therefore, we will consider linear experts and the Regime 2 in our subsequent experiments. Due to the time limit, we will include additional experiments for other setups in the revision later.

In particular, we generate the data by first sampling $X_i \sim \mathrm{Uniform}([-1, 1]^d)$ for $i = 1, \ldots, n$. Then, we generate $Y_i$ according to the following model:

$$Y_{i} = g_{G_{*}}(X_{i}) + \epsilon_{i},$$

where the regression function $g_{G_{*}}(\cdot)$ take the form of a softmax gating MoE: 

$$\sum_{i=1}^{k_*} softmax((\beta^*_{1i})^{\top}x+\beta^*_{0i})\cdot \left((a_i^*)^\top x + b_{i}^{*}\right).$$

The input data dimension is $d = 32$. We employ $k_* = 8$ experts of the form $a^{\top}x+b$. The variance of Gaussian noise $\epsilon_i$ is $\nu = 0.01$. 

The ground-truth gating parameters $\beta^*_{0i}$ are drawn independently from an isotropic Gaussian distribution with zero mean and variance $\nu_g = 0.01/d$ for $1 \le i \le 6$, while $\beta^*_{1i}$ are drawn independently from an isotropic Gaussian distribution $\mathcal{N}(0_d, \nu_g I_{d})$ for $1\leq i\leq 7$, where $\nu_g=0.01/d$, and we set $\beta^*_{1i}=0_d$ for $i=8$. Similarly, the true expert parameters, $(a_i^*, b_i^*)$, are drawn independently of an isotropic Gaussian distribution with zero mean and variance $\nu_e = 1/d$ for all experts. 

**Training procedure.** For each sample size $n$, spanning from $10^3$ to $10^5$, we perform 20 experiments. In every experiment, we employ $k=k_*+1=9$ fitted experts, and the parameters initialization for the gating's and experts' parameters are adjusted to be near the true parameters, minimizing potential instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across $10$ epochs, employing a learning rate of $\eta = 0.1$ to fit a model to the synthetic data. 

**Results.** For each experiment, we calculate the Voronoi losses for every model and report the mean values for each sample size in Figure 1 in the attached PDF file. Error bars representing two standard deviations are also shown. 
In Figure 1, the Voronoi losses associated with the sigmoid gating vanish at the rate of $O(n^{-0.4})$, which nearly matches our theoretical results in Theorem 5 in our paper. Meanwhile, those associated with the softmax gating converge to zero at a very slow rate $O(n^{-0.11})$. This empirically shows that the sigmoid gating is more sample efficient than the softmax gating.

**References**

[4] D. Dai. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models.

[21] H. Nguyen. On least squares estimation in softmax gating mixture of experts. 

[34] Jakub Krajewski. Scaling laws for fine-grained mixture of experts.

[35] Xu Owen He. Mixture of A Million Experts.

Meta-review: This work presents a theoretical analysis to examine why sigmoid gating functions exhibit superior performance to softmax gating functions for mixture of experts models. Reviewers appreciated the novelty of the work, acknowledging that sigmoid gating functions are uncommon, and that the effect of selecting different gating functions is a curious topic worth investigating. The authors did an excellent job of responding to reviewer feedback, and while some concerns still remain with the presentation, many of the more technical issues have been resolved. I would strongly recommend that the authors consider making edits (perhaps by including passages from the discussion) to ensure future readers do not adopt the same misunderstandings, and aim to reduce technicality by moving less critical details to the supplementary material. Conditional on such edits, I recommend acceptance.
Decision: Accept (poster)