Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts

Authors: huy  nguyen
nhat  ho
alessandro  rinaldo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 13:00:43

Abstract: The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due to its inherent structure. In response, the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically to achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, in fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as a mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case in which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. In both cases, we find that experts formulated as feed-forward networks with commonly used activation such as $\mathrm{ReLU}$ and $\mathrm{GELU}$ enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.

Main paper: 
 introduction: 
 Mixture of experts (MoE) [15, 17] has recently emerged as a powerful machine learning model that helps scale up the model capacity while requiring a nearly constant computational overhead [34, 8] . In particular, it aggregates multiple sub-models called experts based on a gating network. Here, experts can be formulated as neural networks, and they specialize in different aspects of the data. For instance, in large language models [16, 31, 19, 40, 7, 5, 30] , one expert might focus on syntax, another on semantics, whereas the others concentrate on context. Meanwhile, the gating network takes the input data and calculates a set of weights that determine the contribution of each expert. This gating mechanism guarantees that the most relevant experts are assigned more weight for a given input. In addition to language modeling, MoE has also been leveraged in several other applications, including computer vision [32, 20, 33] , multi-task learning [12, 10] , multi-modal learning [11] , speech recognition [38, 9, 29] and reinforcement learning [3, 28] .In the above applications of MoE, practitioners frequently utilize the softmax gating function to compute the mixture weights. However, the use of softmax function might introduce an unexpected competition among experts, which leads to the representation collapse issue [2, 30, 18, 24] : when the weight of one expert increases, those of the others decrease accordingly. As a consequence, some experts will dominate the decision-making process, and overshadow the contributions of others. Therefore, the ability of MoE models to incorporate the diversity of expert specialization is partially limited. To this end, Csordás et al. [4] propose using the sigmoid function in place of the softmax to remove the unnecessary expert competition, and empirically demonstrate that it is indeed a better choice of gating function. On the other hand, the theoretical guarantee for that claim has remained missing in the literature. The main objective of this work is to provide a comprehensive analysis of the sigmoid gating function from the perspective of the expert estimation problem. In particular, we will show that sigmoid gating delivers superior sample efficiency for estimating the model parameters and allows for more general expert functions than softmax gating.Related works. A very recent line of research has focused on analyzing the convergence rates for expert estimation in Gaussian MoE models under a variety of assumptions and choices of gating functions. Assuming that data from an input-free gating Gaussian MoE, Ho et al. [14] demonstrated that the expert estimation rates for the maximum likelihood estimator vary with the algebraic independence among expert functions. Under the same setting but using instead softmax gating, Nguyen et al. [27] discovered that the expert estimation rates depend on the solvability of a system of polynomial equations resulting from an intrinsic interaction among gating and expert parameters. Subsequently, a Gaussian MoE with Top-K sparse softmax gate, designed to activate only a fraction of experts per input, was studied in [25] . The convergence analysis in that work revealed that activating exactly one expert for each input would eliminate the previous parameter interaction, thus boosting the expert estimation rates. Despite many important insights provided in this line of works, the mixture setting of Gaussian MoE is still far from practical. To that effect, Nguyen et al. [26] introduced a more realistic regression framework in which, conditionally on the features, the response variables are sampled from noisy realization of an unknown and deterministic softmax gating MoE-type regression function. They found that experts formulated as feed-forward networks with sigmoid or hyperbolic tangent activation functions enjoy faster estimation rates than those equipped with a polynomial activation. In this paper, we adopt the same regression setting and carry out an analogous analysis by considering instead sigmoid gating for the underlying regression function, as opposed to the more popular softmax gating considered in [26] . As we will see, the choice of gating function turns out to be critical, requiring a different type of analysis and leading to markedly different conclusions.Problem setup. We assume that the data (X 1 , Y 1 ), (X 2 , Y 2 ), . . . , (X n , Y n ) ∈ R d × R follow a standard regression modelEQUATIONwhere the features X 1 , X 2 , . . . , X n are i.i.d. samples from a probability distribution µ on R d and ϵ 1 , . . . , ϵ n are independent noise variables such that E[ϵ i |X i ] = 0 and Var(ϵ i |X i ) = ν for all 1 ≤ i ≤ n; for simplicity we further assume that they follow a Gaussian distribution. The unknown regression function f G * is formulated as a sigmoid-gated MoE with k * experts, i.e.,EQUATIONwhere the expert function x → h(x, η) is of parametric form, specified by parameter η ∈ R q . The regression function f G * is fully characterized by the unknown parameters(β * 0i , β * 1i , η * i ) k * i=1 in R × R d × R q ,which can be compactly encoded using the associated mixing measureG * := k * i=1 1 1+exp(-β * 0i ) δ (β * 1i ,η * i ), a weighted sum of Dirac measures δ. Least squares estimation. To estimate the unknown parameters (β * 0i , β * 1i , η * i ) k * i=1 (equivalently, the ground truth mixing measure G * ), we deploy the least squares method [35] and focus on the estimatorEQUATIONwhere G k (Θ) := {G = k ′ i=1 1 1+exp(-β0i) δ (β1i,ηi) : 1 ≤ k ′ ≤ k, (β 0i , β 1i , η i ) ∈ Θ} is the set of all mixing measures with at most k atoms. Since the true number of true experts k * is typically unknown in practice, we will assume that the number k of fitted experts is at least as large, i.e. that k > k * .Technical challenges. The main challenge in analyzing MoE models with sigmoid gating lies in the convergence behavior of the mixture weights. Specficially, since we fit the ground-truth MoE model (2) with a mixture of k > k * , there must be some true atoms (β * 1i , η * i ) fitted by more than one component; we will refer to the corresponding parameters β * 1i as over-specified parameters. To illustrate, suppose that ( βn 1i , ηn i ) → (β * 11 , η * 1 ) for i ∈ {1, 2}, in probability. Then, to ensure convergence of f Gn to f G * in the L 2 (µ)-norm we must have that, for i = 1, 2,2 i=1 1 1 + exp(-( βn 1i ) ⊤ x -βn 0i ) → 1 1 + exp(-(β * 11 ) ⊤ x -β * 01 ), as n → ∞, for µ-almost every x. Note that the above limit can be achieved only if β * 11 = 0 d . As a result, we will consider the two following regimes for the over-specified parameters β * 1i : Regime 1. All the over-specified parameters β * 1i are equal to 0 d ; Regime 2. At least one among the over-specified parameters β * 1i is different from 0 d . It is worth emphasizing that the second regime presents additional technical challenges, as the least squares estimator G n converges to a mixing measureG ∈ arg min G∈G k (Θ)\G k * (Θ) ∥f G -f G * ∥ L 2 (µ)that is in general different that the true mixing measure G * , as in the first regime.Contributions. In this paper, we carry out a convergence analysis of the sigmoid gating MoE under two regimes of the gating parameters. Our main objective is to compare the sample efficiency between the sigmoid gating and the softmax gating. The contributions of our paper are three-fold, and can be summarized as follows:(C.1) Convergence rate for the regression function. We demonstrate in Theorem 1 that the regression estimation f Gn converges to its true counterpart f G * at the rate of order O P (n -1/2 ), which is parametric on the sample size n. This regression estimation rate is then utilized for determining the expert estimation rates.(C.2) Expert estimation rates under the Regime 1. Under the first regime, we first establish a condition called strong identifiability to characterize which types of experts would yield polynomial estimation rates. In particular, we find out that the rates for estimating experts formulated as feedforward networks with popular activation functions such as ReLU and GELU are of polynomial orders. By contrast, those for polynomial experts and input-indepedent experts are slower than any polynomial rates and could be of order O P (1/ log(n). Such expert convergence behavior is similar to that when using the softmax gating.(C.3) Expert estimation rates under the Regime 2. Under the second regime, the regression estimation f Gn converge to a function taking the form of a sigmoid gating MoE which is different from f G * . From our derived weak identifiability condition, it follows that estimation rates for feed-forward expert networks with ReLU or GELU activation and polynomial experts are of orders O P (n -1/2 ), which are substantially faster than those when using the softmax gating (see also Table 1 ). Therefore, it follows that the sigmoid gating is more sample efficient than the softmax gating.Outline. The rest of the paper is organized as follows. In Section 2, we introduce some mild assumptions on the parameters, and then determine estimation rates for the regression function. Next, we establish convergence rates for expert estimation under both Regime 1 and the Regime 2 in Section 3. Then, we conduct some experiments to empirically justify our theory in Section 4. Finally, we provide a discussion on the practical implications, limitations and future directions of our work in Section 5. Full proofs of the results and additional experiments are deferred to the Appendices.Notation. For any n ∈ N, we denote [n] as the set = {1, 2, . . . , n}. Additionally, for any set S, we refer to |S| as its cardinality. Next, for any vectors v : Summary of expert estimation rates (up to a logarithmic factor) under the MoE models equipped with the sigmoid gating (ours) and the softmax gating [26] . In this work, we consider three types of expert functions including experts network with ReLU, GELU activations; polynomial experts; and input-independent experts.= (v 1 , v 2 , . . . , v d ) ∈ R d and α := (α 1 , α 2 , . . . , α d ) ∈ N d , we let v α = v α1 1 v α2 2 . . . v α d d , |v| := v 1 + v 2 + . . . + v d and α! := α 1 !α 2 ! . . . α d !,Polynomial Experts Inputindependent Experts Regime 1Regime 2 Regime 1 Regime 2Sigmoid O P (n -1/4 ) O P (n -1/2 ) O P (1/ log(n)) O P (n -1/2 ) O P (1/ log(n)) Softmax O P (n -1/4 ) O P (1/ log(n)) O P (1/ log(n)) 
preliminaries: 
 In this section, we demonstrate the parametric convergence rate for the least squares estimator of the regression function f Gn . We begin by describing the assumption that will be used throughout the paper, which are overall very mild.(A.1) Topological assumptions. To ensure the convergence of the least squares estimators, we assume the parameter space Θ ⊆ R × R d × R q is compact, and the input space X ⊆ R d is bounded.(A.2) Lipschitz experts. The expert function h(•, η) is assumed to be Lipschitz continuous with respect to η and bounded, µ-almost surely.(A.3) Distinct experts. The ground-truth expert parameters η * 1 , . . . , η * k * are pair-wise different so that the experts h(x, η * 1 ), . . . , h(x, η * k * ) are also distinct from each other. Given the above assumptions, we are now ready to present the main result of this section. Recall that we divide our analysis into two following regimes:• Regime 1. All the over-specified parameters β * 1i are equal to 0 d ; • Regime 2. At least one among the over-specified parameters β * 1i is different from 0 d .Let us begin with Regime 1. We show in Theorem 1 that the regression estimator f Gn converges to the true regression function f G * at the parametric rate.Theorem 1. Under the Regime 1 and with the least squares estimator G n defined in equation (3), the regression estimator f Gn admits the following rate of convergence to f G * :EQUATIONThe proof of Theorem 1 is in Appendix A. -f G * ∥ L 2 (µ) ≳ D( G n , G * ),then we obtain parameter estimation rates via the boundEQUATION). Consequently, those parameter estimation rates lead to our desired expert estimation rates.Next, under the Regime 2, the true model is misspecified, i.e., the regression estimator f Gn converges to the missepcified regression function f G rather than the true regression function f G * , whereG ∈ G k (Θ) := arg min G∈G k (Θ)\G k * (Θ) ∥f G -f G * ∥ L 2 (µ). By employing the same arguments as in Theorem 1, we also capture the parametric regression estimation rateO P ([log(n)/n] 1 2 ) for ∥f Gn -f G ∥ L 2 (µ) under this regime, which is stated in the following corollary.Corollary 1. Under the Regime 2, the regression estimator f Gn admits the following rate of conver-gence to f G : inf G∈G k (Θ) ∥f Gn -f G ∥ L 2 (µ) = O P ([log(n)/n] 1 2 ). 
convergence rates for expert estimation: 
 In this section we will establish the expert estimation rates under both Regime 1 and the Regime 2; see Section 3.1 and Section 3.2, respectively. Under each regime, we will formulate appropriate conditions, which we refer to as identifiability conditions, on the expert functions that will guarantee fast estimation rates. Furthermore, we will determine the (slow) estimation rates of some commonly used experts which fail to satisfy such identifiability conditions.Recall that under the Regime 1, all the over-specified parameters β * 1i , i.e., those fitted by at least two parameters, are equal to 0 d . Without loss of generality, we assume that β * 11 , . . . , β * 1( k+1) , . . . , β * 1k * are assumed to be fitted by exactly one estimator. Following a strategy that has been successfully used for analyzing MoE and mixture models (see, e.g. [21] and [26] ), in order to derive the expert estimation rates, it is sufficient to propose a Voronoi loss function D among parameters, and then prove the lower bound∥f Gn -f G * ∥ L 2 (µ) ≳ D( G n , G * ).Given the parametric regression estimation rate in Theorem 1, we then obtain thatD( G n , G * ) = O P ([log(n)/n] 1 2). Note that a key step in deriving the previous lower bound is to decompose the difference f Gn (x)f G * (x) into a combination of linearly independent terms.Towards that goal, we will Taylor expand to the product of mixture weight and expert functions F (x, β 1 , β 0 , η) := σ(x, β 1 , β 0 )h(x, η), where we let σ(x, β 1 , β 0 ) := 1 1+exp(-β ⊤ 1 x-β0) . To obtain the desired decomposition of f Gn (x)f G * (x), we provide below in Definition 1 a condition, which we refer to as a strong identifiability condition, that guarantees that the derivatives of the product F (x, β 1 , β 0 , η) w.r.t its parameters are linearly independent. Definition 1 (Strong identifiability). An expert function h(x, η) is called strongly identifiable if it is twice differentiable w.r.t its parameter η for µ-almost all x and, for any positive integer ℓ and any pair-wise distinct choices of parameters {(β 0i , β 1i , η i )} ℓ i=1 , the functions in the classesx → ∂ |γ1|+|γ2| F ∂β γ1 1 ∂η γ2 (x, 0 d , β 0i , η i ) : i ∈ [ℓ], (γ 1 , γ 2 ) ∈ N d × N q , 1 ≤ |γ 1 | + |γ 2 | ≤ 2 and x → ∂F ∂β α1 1 ∂β α2 0 ∂η α3 (x, β 1i , β 0i , η i ) : i ∈ [ℓ], (α 1 , α 2 , α 3 ) ∈ N d ×N×N q , |α 1 |+α 2 +|α 3 | = 1 are linearly independent, for µ-almost all x.Example. Consider experts formulated as two-layer neural networks, i.e., h(x, (a, b)) = φ(a ⊤ x + b), where φ is an activation function and (a, b) ∈ R d × R. If a = 0 d , then the expert h(•, (a, b)) is not strongly identifiable for any choices of the activation function φ. On the other hand, if a ̸ = 0 d and φ is one among popular activation functions such as ReLU and GELU, then h(•, (a, b)) is a strongly identifiable expert. By contrast, if φ is a polynomial, e.g. φ(z) = z p , then the expert h(•, (a, b)) does not meet the strong identifiability condition.Intuitively, the strong identifiability condition helps eliminate potential interactions among parameters expressed in the language of PDE (see equations (8) and (11) where gating parameters β 1 interact with expert parameters a). Such interactions are demonstrated to result in significantly slow expert estimation rates presented in Theorem 3 and 4). From the technical perspective, a key step in our proof techniques rely on the decomposition of the discrepancy f Gn (x)f G * (x) into a combination of linearly independent terms. This can be done by applying Taylor expansions to the function F (x, β 1 , β 0 , η) = σ(β ⊤ 1 x + β 0 )h(x, η) defined as the product of the sigmoid gating and the expert function h. Thus, the condition is to ensure that terms in the decomposition are linearly independent.Next, following the strategy first introduced by Manole et al. [21] and then adopted in several convergence analyses of MoE [22, 23] , we proceed to construct a loss function among parameters based on the notion of Voronoi cells.Voronoi loss. Given an arbitrary mixing measure G with k ′ ≤ k atoms, we distribute its atoms across the Voronoi cells {A j ≡ A j (G), j ∈ [k * ]} generated by the atoms of G * , whereEQUATIONwith ω i := (β 1i , η i ) and ω * j := (β * 1j , η * j ). In particular, the cardinality of the Voronoi cell A j corresponding to the least squares estimator G n in (3) is the number of fitted components assigned (and, likely, converging) to the true atoms ω * j . Then, we define the Voronoi loss functionEQUATIONwhere we let∆β 1ij := β 1i -β * 1j , ∆β 0ij := β 0i -β * 0j, and ∆η ij := η iη * j . Above, if the Voronoi cell A j is empty, then the summation term becomes zero by convention. Additionally, since D 1 (G, G * ) = 0 if and only if G ≡ G * , it follows that when D 1 (G, G * ) is sufficiently small, the differences ∆β 0ij , ∆β 1ij and ∆η ij are also small. This observation suggests that, though not symmetric in its arguments and therefore not a metric, D 1 (G, G * ) is nonetheless a suitable loss function for capturing the discrepancy between the least squares estimator G n and the true mixing measures G * . Furthermore, it is worth noting that the Voronoi loss function D 1 can be efficiently evaluated, as its computational complexity is of order O(k × k * ).In our next result, whose proof can be found in Appendix A.2, we show that the Voronoi loss between the least squares estimator and the true mixing measure is upper bounded, up to universal multiplicative constants, by the estimation error for the regression function. Theorem 2. Let x → h(x, η) be a strongly identifiable expert function. Then the lower bound∥f G -f G * ∥ L 2 (µ) ≳ D 1 (G, G * ),holds true for any G ∈ G k (Θ). As a consequence, this result together with the regression estimation rate in Theorem1 indicate that D 1 ( G n , G * ) = O P ([log(n)/n] 1 2 ).The following remarks regarding the results of Theorem 2 are in order. ). Note that as the expert h(•, η) is twice differentiable over a bounded domain, it is also a Lipschitz function. Thus,EQUATIONfor any i ∈ A j ( G n ), where L 1 ≥ 0 is a Lipschitz constant. The above bound indicates that if the strongly identifiable expert h(•, η * j ) is over-specified, i.e. fitted by at least two experts, then its estimation rate is of order O P ([log(n)/n] 1 4 ). (ii) Secondly, for exact-specified parameters β * 1j , η * j , where k + 1 ≤ j ≤ k * , the rates for estimating them are faster than those of their over-specified counterparts in Remark (i), standing at orderEQUATION2 ). By arguing as in equation (7) , we deduce that the expert h(•, η * j ) also enjoys the faster estimation rate of orderO P ([log(n)/n] 1 2), which is parametric on the sample size n. (iii) Putting the above two remarks together, we observe that when the expert functions are formulated as feed-forward networks with a widely used activation, namely ReLU and GELU, their estimation rates under the sigmoid gating MoE matches exactly those under the softmax gating MoE: see Theorem 3.2 in [26] .Non-strongly identifiable experts. The strong identifiability condition of Definition 1 is far from a technical requirement: it is in fact a crucial assumption. To illustrate its importance, we investigate the estimation rates of ridge experts of the form {h(x,(a * j , b * j )) = φ((a * j ) ⊤ x + b * j ), j ∈ [k * ]}, where φ is a scalar function, that fail to satisfy the strong identifiability condition. For ridge experts, the strong identifiability will not be met in two cases: when at least one among over-specified parameters a * j equals 0 d , for any arbitrary activation function φ (Scenario I) and when φ is a polynomial of the form φ(z) = z p , for p ∈ N (Scenario II). Scenario I. Without loss of generality, we may assume that a * 1 = 0 d , which means that the first expert φ((a * 1 ) ⊤ x + b * 1 ) becomes independent of the input x. This gives rise to an interaction among the gating parameter β 1 and the expert parameter a via the partial differential equation (PDE)EQUATIONwhere we recall that F (x;β 1 , β 0 , a, b) := σ(x; β 1 , β 0 )φ(a ⊤ x + b), and C b * 1 ,β *01 is some constant depending on b * 1 and β * 01 . The above PDE can be verified by taking the derivatives of F (x; β 1 , β 0 , a, b) w.r.t β 1 and a and using the fact that β * 11 = a * 1 = 0 d . Notably, the PDE (8) accounts for the violation of the strong identifiability of the expert φ((a * 1 ) ⊤ x + b * 1 ) under the Scenario I. To this end, we propose the following Voronoi loss to analyze the effects of such parameter interaction on the expert estimation rates in Theorem 3:EQUATIONThe following result establishes a minimax lower bound on the estimation error of G * in the D 2,r loss. Theorem 3. Suppose that the experts take the form φ(a ⊤ x + b). Then, under the Scenario I, we haveinf Gn∈G k (Θ) sup G∈G k (Θ)\G k * -1 (Θ) E f G [D 2,r ( G n , G)] ≳ n -1/2 ,for any r ≥ 1, where E f G indicates that the expectation taken w.r.t the product measure with f n G , and the infimum is over all estimators taking values in G k (Θ).See Appendix A.3 for the proof details. We highlight some important implications of Theorem 3.(i) The estimation rates for the parameters β * 1j , a * j and b * j are slower than O P (n -1/2r ), for any r ≥ 1. This means that they are slower than any polynomial rates, and could be of order O P (1/ log(n)).(ii) Using the same reasoning described after equation ( 7), we haveEQUATIONwhere L 2 > 0 is a Lipschitz constant. As a consequence, the rates for estimating experts φ((a * j ) ⊤ x + b * j ) are no better than those for estimating the parameters a * j and b * j , and could also be as slow as O P (1/ log(n)). This result suggests that all the expert parameters a * 1 , . . . , a * k * should be different from 0 d to hope for fast convergence rates. In other words, every expert of the form φ(a ⊤ x + b) in the MoE model should vary with the input value.Scenario II. Now, we consider the second scenario when φ is a polynomial of the form φ(z) = z p , for p ∈ N. For simplicity, we will only focus on the setting of p = 1, i.e., h(x, η * j ) = (a * j ) ⊤ x + b * j ; the case of p > 1 can be argued in a very similar fashion. The structure of the polynomial experts leads to a non-linear interactions among the parameters, expressed through the PDEEQUATIONwhereF (x; β 1 , β 0 , a, b) := σ(x; β 1 , β 0 )(a ⊤ x + b).This interaction is the reason why the polynomial experts are not strongly identifiable. Ultimately, this dependence yields slow rates of convergence for expert estimation, just like in Scenario I, as revealed by the next result, which deploys the Voronoi loss D 2,r of equation ( 9). Theorem 4. Suppose that the experts take the form (a ⊤ x + b) p , for some p ∈ N. Then, under the Scenario II and for any r ≥ 1, infGn∈G k (Θ) sup G∈G k (Θ)\G k * -1 (Θ) E f G [D 2,r ( G n , G)] ≳ n -1/2 ,where E f G indicates the expectation taken w.r.t the product measure with f n G .The proof of Theorem 4 is in Appendix A.4. To sum up, when either the experts are independent of the input or they take a polynomial form, their estimation rates could be as slow asEQUATION2 ).Recall that under the Regime 2, at least one among the over-specified parameters β * 1i is different from 0 d . As discussed in Section 2, the least squares regression estimator f Gn in this case converges toa regression function f G , where G ∈ G k (Θ) := arg min G∈G k (Θ)\G k * (Θ) ∥f G -f G * ∥ L 2 (µ). That is, the estimators of the parameters specifying f Gn converge to the parameters of f G . WLOG, we may assume that G :=EQUATION1+exp(-β0i) δ ( β1i,ηi) . Similarly to Regime 1, we also provide in Definition 2 a weak identifiability condition to characterize those expert functions that will have fast estimation rates under the Regime 2. Weakly identifiable experts are required to satisfy only a subset of the conditions imposed to strongly identifiable experts.Definition 2 (Weak identifiability). An expert function x → h(x, η) is called weakly identifiable if it is twice differentiable w.r.t its parameter η for µ-almost all x and, for any positive integer ℓ and any pair-wise distinct choices of parameters {(β 0i , β 1i , η i )} ℓ i=1 , the functions in the classx → ∂F ∂β α1 1 ∂β α2 0 ∂η α3 (x, β 1i , β 0i , η i ) : i ∈ [ℓ], (α 1 , α 2 , α 3 ) ∈ N d ×N×N q , |α 1 |+α 2 +|α 3 | = 1are linearly independent, for µ-almost all x.Example. Let us take an expert network h(x, (a, b)) = φ(a ⊤ x + b) as an example. It can be checked that if a ̸ = 0 d and the activation function φ is a ReLU or GELU or a polynomial, then the expert h(x, (a, b)) is weakly identifiable. Conversely, if a = 0 d , i.e. the expert does not depend on the input, then the weak identifiability condition is not met, regardless of the choice of the activation.We now describe the convergence rates for expert estimation under Regime 2. Since the analysis of input-independent experts can be done similarly to Theorem 3, we will focus only on weakly identifiable experts. As it turned out, the appropriate Voronoi loss function is given byEQUATION)Theorem 5. Let h(x, η) be a weakly identifiable expert. Then, for any mixing measureG ∈ G k (Θ), inf G∈G k (Θ) ∥f G -f G ∥ L 2 (µ) /D 3 (G, G) > 0.As a consequence, we obtain thatEQUATION).The proof of Theorem 5 is in Appendix A.5. As a direct consequence of the bound and the definition of D 3 , the convergence rates of the estimators βn 0i , βn 1i and ηn i are of the same of parametric orderO P ([log(n)/n] 1 2). Furthermore, by equation ( 7), we also establish that the convergence rate of the expert estimator h(x, ηn i ) is also of orderEQUATION). Comparing these estimation rates with those established by [26] assuming instead softmax gating, there are two main observations:(i) When the expert is a neural network with ReLU or GELU activation, the above parametric expert estimation rate obtained using the sigmoid gate is faster than the rate guranteed by the softmax gate, which could be of orderO P ([log(n)/n] 1 4 ) (see [Theorem 3.2, [26]]).(ii) When the activation is a polynomial, the gap between expert estimation rates when using the sigmoid gate versus when using the softmax gate becomes even more dramatic:O P ([log(n)/n] 1 2 ) compared to O P (1/ log(n)) (see Theorem 4.6 in [26]).The above remarks highlight the considerable benefits of deploying sigmoid gating over softmax gating in MoE models: provably faster estimation rates not only for expert networks with popular activations like ReLU and GELU, but also for those with polynomial activation. Expert parametersβ * 1i ∼ N (0 d , ν g I d ) 1 ≤ i ≤ 7 β * 1i = 0 d i = 8 β * 0i ∼ N (0, ν g ) a * i ∼ N (0 d , ν e I d ) b * i ∼ N (0, ν e ) 
numerical experiments: 
 In this section, we perform some numerical experiments to empirically demonstrate our claim that the sigmoid gating is more sample efficient than the softmax gating.From Table 1 , it can be seen that the sigmoid gating shares the same expert estimation rates as the softmax gating under the Regime 1. However, the former gating outperforms the latter under the Regime 2, particularly for ReLU experts and polynomial experts. Therefore, we will consider those experts under the Regime 2 in our subsequent experiments.Data generation. In particular, we generate the data by first samplingX i ∼ Uniform([-1, 1] d ) for i = 1, . . . , n.Then, we generate Y i according to the following model:EQUATIONwhere the regression function g G * (•) take the form of a softmax gating MoE:EQUATIONThe input data dimension is d = 32. We employ k * = 8 experts of the form ϕ(a ⊤ x + b), where ϕ is either the identity function or the ReLU function. The variance of Gaussian noise ϵ i is ν = 0.01.Experimental setup. We summarize the choices of the ground-truth parameters β * 0i , β * 1i , a * i and b * i for 1 ≤ i ≤ 8 in Table 2 , which satisfies the condition of the Regime 2.Training procedure. For each sample size n, spanning from 10 3 to 10 5 , we perform 20 experiments.In every experiment, we employ k = k * + 1 = 9 fitted experts, and the parameters initialization for the gating's and experts' parameters are adjusted to be near the true parameters, minimizing potential instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across 10 epochs, employing a learning rate of η = 0.1 to fit a model to the synthetic data.For each experiment, we calculate the Voronoi losses for every model and report the mean values for each sample size in Figure 1 . Error bars representing two standard deviations are also shown.In Figure 1 , when employing the ReLU experts, that is, ϕ is the ReLU function, the Voronoi loss for the sigmoid gating approaches zero at the rate of order O(n -0.51 ), which nearly matches our theoretical results in Theorem 5. Meanwhile, the loss for the softmax gating converges to zero at the slower rate O(n -0.24 ). On the other hand, when using the linear experts, that is, ϕ is the identity function, the vanishing rate of the Voronoi loss associated with the sigmoid gating is O(n -0.40 ), while that for the softmax gating is significantly slower, standing at O(n -0.11 ). These observations empirically shows that the sigmoid gating is more sample efficient than the softmax gating.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: All the experimental details are provided in Section 4 and Appendix C. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion: 
 In this paper, we carry out a convergence analysis of least squares expert estimation under MoE models with the sigmoid gating, which was found empirically to be robust to the representation collapse issue and to have a favorable performance in the MoE applications. We demonstrate that under both the gating regimes, the sigmoid gating requires a smaller sample size than the softmax gating to reach the same expert estimation error: that is, the sigmoid gating is more sample efficient than its softmax counterpart. Furthermore, we also verify that experts formulated as feed-forward networks with popular activation functions such as ReLU and GELU are more compatible with the sigmoid gating than other types of expert functions. Figure 1 : Log-log scaled plots displaying the empirical averages of the Voronoi losses when using the sigmoid gating (blue line) versus when using the sofmax gating (green line) under the same data.The red dash-dotted lines illustrate the fitted lines for determining the empirical convergence rates.Practical implications. Our theoretical findings support the following conclusions, which have considerable practical relevance.(I.1) The sigmoid gate is more sample efficient than the softmax gate for expert estimation. It can be seen from Table 1 that while the expert estimation rates obtained when using the sigmoid gate match those attained when using the softmax gate under the Regime 1, the former are totally faster than the latter under the Regime 2. Notably, Regime 2 is closer to practice since the gating values under this regime hinge upon the input while those under the Regime 1 are input-independent. As a consequence, we can claim that the sigmoid gate is more sample efficient than the softmax gate from the perspective of the expert estimation problem.(I.2) The sigmoid gate is compatible with a broader class of experts than the softmax gate.It follows from the expert characterization for Regime 1 (resp. Regime 2) in Definition 1 (resp. Definition 2) that formulating expert functions as feed-forward networks [36] with commonly used activation functions such as ReLU and GELU or even a polynomial activation will lead to faster expert estimation rates, and thus, require a smaller sample size to reach the same tolerance of estimating experts compared to the softmax gate. Thus, our theories indicate that the sigmoid gating is compatible with a broader class of experts than the softmax gating. This implication is particularly useful when people employ a mixture of fine-grained (shallow) expert networks [13] .Limitations and future directions. There are two main limitations of our current analysis:1. We assume implicitly that the ground-truth parameters are independent of the sample size. Therefore, the expert estimation rates established in the paper are point-wise rather than the desirable uniform rates. A potential approach to cope with this problem is using the techniques for deriving the uniform parameter estimation rates in mixture models [6] and in MoE models [37] . However, those techniques are only valid for input-independent gating MoE, and we believe that further technical tools should be developed to adapt such framework to the sigmoid gating MoE. Thus, we leave it for future development.The assumption that the true regression function belongs to the parametric class of MoE models under sigmoid gating is, of course, quite restrictive, and likely to be violated in real-world settings. This issue can be alleviated by assuming the data are sampled from a regression framework with an arbitrary regression function g, which is not necessarily formulated as a mixture of experts. Under that setting, the least squares estimatorG n converges to G ∈ arg min G∈G k (Θ) ∥f G -g∥ L 2 (µ) .Nevertheless, it requires the comprehensive knowledge of the universal approximation power of the sigmoid function, which has been slightly explored in [1] , to determine the expert estimation rate.Therefore, we leave this potential direction for future work.Supplement to "Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts"In this supplementary material, we present proofs for the main results of the paper in Appendix A, while we study the identifiability of the sigmoid gating mixture of experts (MoE) in Appendix B. Lastly, we provide additional numerical experiments to verify our theory in Appendix C.In this appendix, we provide proofs for the theoretical results introduced in Section 2 and Section 3 of the paper.To streamline the arguments for this proof, we need to define some necessary notations. First of all, we letF k (Θ) := {f G (x) : G ∈ G k (Θ)} stand for the set of all regression functions in G k (Θ).Next, for each δ > 0, we define the L 2 (µ)-ball centered around the regression function f G * (x) and intersected with the set F k (Θ) asF k (Θ, δ) := f ∈ F k (Θ) : ∥f -f G * ∥ L 2 (µ) ≤ δ .Furthermore, the size of the above ball is captured by the following integral as suggested in [35] .EQUATIONin whichH B (t, F k (Θ, t), ∥ • ∥ L 2 (µ)) denotes the bracketing entropy [35] of F k (Θ, t) under the L 2 (µ)-norm, and t ∨ δ := max{t, δ}. By arguing in a similar fashion to Theorem 7.4 and Theorem 9.2 in [35] with adapted notations to this work, we achieve the following lemma: Lemma 1. Let Ψ(δ) ≥ J B (δ, F k (Θ, δ)) such that Ψ(δ)/δ 2 is a non-increasing function of δ. Then, for some universal constant c and for some sequence (δ n ) that satisfies √ nδ 2 n ≥ cΨ(δ n ), the following holds for any δ ≥ δ n :P ∥f Gn -f G * ∥ L 2 (µ) > δ ≤ c exp - nδ 2 c 2 .General Picture. If we are able to demonstrate that the following bound holds for any ε ∈ (0, 1/2]:EQUATIONthen it follows thatEQUATIONBy choosingΨ(δ) = δ • [log(1/δ)] 1/2, then Ψ(δ)/δ 2 is a non-increasing function of δ. Moreover, equation (17) suggests that Ψ(δ) ≥ J B (δ, F k (Θ, δ)). Additionally, we set δ n = log(n)/n, and then get that √ nδ 2 n ≥ cΨ(δ n ) for some universal constant c. According to Lemma 1, we achieve the desired conclusion of the theorem. As a consequence, it suffices to establish the bound (16) .Proof of inequality (16) . Note that as the expert functions are bounded, we have that |f G (x)| ≤ M for almost every x, where M > 0 is a bounded constant.Next, let τ ≤ ε and {ζ 1 , . . . , ζ N } be the τ -cover under the L 2 (µ)-norm of the set F k (Θ), where N := N (τ, F k (Θ), ∥ • ∥ L ∞ ) stands for the η-covering number of the metric space (F k (Θ), ∥ • ∥ L ∞ ). Now, we consider brackets of the form [L i (x), U i (x)], where we defineL i (x) := max{ζ i (x) -τ, 0}, U i (x) := max{ζ i (x) + τ, M }, for all i ∈ [N ]. Then, it can be verified that F k (Θ) ⊂ ∪ N i=1 [L i (x), U i (x)] and U i (x) -L i (x) ≤ min{2τ, M }. Thus, we obtain that ∥U i -L i ∥ 2 L 2 (µ) = (U i -L i ) 2 dµ(x) ≤ 4τ 2 dµ(x) = 4τ 2 , which indicates that ∥U i -L i ∥ L 2 (µ) ≤ 2τ .From the definition of the bracketing entropy, we arrive atEQUATIONConsequently, it is sufficient to provide an upper bound for the covering number N . For that purpose, let us denote∆ := {(β 0 , β 1 ) ∈ R × R d : (β 0 , β 1 , η) ∈ Θ} and Ω := {η ∈ R q : (β 0 , β 1 , η) ∈ Θ}.Recall that Θ is a compact set, therefore, ∆ and Ω are also compact. Then, there exist τ -covers ∆ τ and Ω τ for ∆ and Ω, respectively. Furthermore, it can be validated that|∆ τ | ≤ O P (τ -(d+1)k ), |Ω τ | ≲ O P (τ -qk ).Given a mixing measureG = k i=1 exp(β 0i )δ (β1i,ηi) ∈ G k (Θ), we take into account two other mixing measures defined as:G := k i=1 1 1 + exp(-β 0i ) δ (β1i,η i ) , Ǧ := k i=1 1 1 + exp(-β0i ) δ ( β1i,ηi) .Above, ηi ∈ Ω τ such that ηi is the closest to η i in that set, while ( β0i , β1i ) ∈ ∆ τ is the closest to (β 0i , β 1i ) in that set. It follows from the above formulations that∥f G -f G ∥ L ∞ = sup x∈X k i=1 1 1 + exp(-(β 1i ) ⊤ x -β 0i ) • [h(x, η i ) -h(x, ηi )] ≤ k i=1 sup x∈X 1 1 + exp(-(β 1i ) ⊤ x -β 0i ) • |h(x, η i ) -h(x, ηi )| ≤ k i=1 sup x∈X |h(x, η i ) -h(x, ηi )| ≤ k i=1 sup x∈X L 1 • ∥η i -ηi ∥ ≤ kL 1 τ ≲ τ.Here, the second inequality occurs as the sigmoid weight is not larger than one and the third inequality follows from the fact that the expert function h(x, •) is a Lipschitz function with some Lipschitz constant L 1 > 0.Subsequently, we have∥f G -f Ǧ∥ L ∞ = sup x∈X k i=1 1 1 + exp(-(β 1i ) ⊤ x -β 0i ) - 1 1 + exp(-( β1i ) ⊤ x -β0i ) • h(x, ηi ) ≤ k i=1 sup x∈X 1 1 + exp(-(β 1i ) ⊤ x -β 0i ) - 1 1 + exp(-( β1i ) ⊤ x -β0i ) • |h(x, ηi )| ≤ k i=1 sup x∈X M ′ 1 1 + exp(-(β 1i ) ⊤ x -β 0i ) - 1 1 + exp(-( β1i ) ⊤ x -β0i ) ≤ k i=1 sup x∈X M ′ L 2 (∥β 1i -β1i ∥ • ∥x∥ + |β 0i -β0i |) ≤ kM ′ (τ • B + τ ) ≲ τ.Above, the second inequality is due to the fact that the expert function is bounded, that is, |h(x, ηi )| ≤ M ′ . The third one happens as the sigmoid function is a Lipschitz function with some Lipschitz constant L 2 > 0, while the fourth inequality occurs since the input space is bounded.∥f G -f Ǧ∥ L ∞ ≤ ∥f G -f G ∥ L ∞ + ∥f G -f G ∥ L ∞ ≲ τ.From the definition of the covering number, it yields thatEQUATIONPutting equations ( 18) and ( 19) together, we get thatH B (2τ, F k (Θ), ∥ • ∥ L 2 (µ) ) ≲ log(1/τ ).Let τ = ε/2, then we reach the desired boundH B (ε, F k (Θ), ∥.∥ L 2 (µ) ) ≲ log(1/ε).Hence, the proof is totally completed.In this proof, we aim to establish the following inequality:EQUATIONFor that purpose, we divide the proof of the above inequality into local and global parts as below.Local part: In this part, we focus only on demonstrating the following local inequality:EQUATIONAssume by contrary that the above claim does not hold true, then there exists a sequence of mixing measuresEQUATIONas n → ∞. Let us recall thatD 1n := k j=1 i∈Aj 1 1 + exp(-β n 0i ) - 1 1 + exp(-β * 0j ) + k j=1 i∈Aj ∥∆β n 1ij ∥ 2 + ∥∆η n ij ∥ 2 + k * j= k+1 i∈Aj |∆β n 0ij | + ∥∆β n 1ij ∥ + ∥∆η n ij ∥ ,where∆β n 0ij := β n 0i -β * 0j , ∆β n 1ij := β n 1i -β * 1j and ∆η n ij := η n i -η * j . Since D 1n → 0 as n → ∞, we deduce that • For 1 ≤ j ≤ k: i∈Aj 1 1+exp(-β n 0i ) → 1 1+exp(-β * 0j ) and (β n 1i , η n i ) → (β * 1j , η * j ), for any i ∈ A j ; • For k + 1 ≤ j ≤ k * : (β n 0i , β n 1i , η n i ) → (β * 0j , β * 1j , η * j ) for any i ∈ A j .Step 1 -Taylor expansion: In this step, we decompose the term f Gn (x)f G * (x) using Taylor expansion. Firstly, let us denoteEQUATIONLet us denote σ(x, β 1 , β 0 ) := 1 1+exp(-β ⊤ 1 x-β0) . Then, by means of the second-order Taylor expansion, we can decompose the term A n (x) defined above asEQUATIONwhere R 1 (x) is a Taylor remainder such that R 1 (x)/D 1n → 0 as n → ∞, and we defineT n j,i,γ1,γ2 := 1 γ! (∆β n 1ij ) γ1 (∆η n ij ) γ2 , for any j ∈ [ k], i ∈ A j , γ 1 ∈ N d and γ 2 ∈ N q such that 1 ≤ |γ 1 | + |γ 2 | ≤ 2.Next, let us denoteW n j := i∈Aj 1 1 + exp(-β n 0i ) - 1 1 + exp(-β * 0j ), for any j ∈ [ k], then the term B n (x) can be represented asEQUATIONFinally, recall that |A j | = 1 for any k + 1 ≤ j ≤ k * , we can decompose the term C n (x) by applying the first-order Taylor expansion as follows:EQUATIONwhere R 2 (x) is a Taylor remainder such that R 2 (x)/D 1n → 0 as n → ∞, and we defineS n j,α1,α2,α3 := i∈Aj 1 α! (∆β n 1ij ) α1 (∆β n 0ij ) α2 (∆η n ij ) α3 , for any k + 1 ≤ j ≤ k * , α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that |α 1 | + α 2 + |α 3 | = 1.Combine the decomposition of the terms A n (x), B n (x) and C n (x) in equations ( 24), ( 25) and ( 26), we rewrite the difference f Gn (x)f G * (x) asEQUATION)Step 2 -Non-vanishing coefficients: In this step, we show that at least one among the ratios T n j,i,γ1,γ2 /D 1n , W n j /D 1n and S n j,α1,α2,α3 /D 1n does not go to zero as n tends to infinity. Indeed, assume by contrary that all of them converge to zero, i.e.T n j,i,γ1,γ2D 1n → 0, W n j D 1n → 0, S n j,α1,α2,α3 D 1n → 0as n → ∞. Then, it follows that• 1 D1n k j=1 i∈Aj 1 1+exp(-β n 0i ) - 1 1+exp(-β * 0j ) = 1 D1n • k j=1 |W n j | → 0; • 1 D1n k j=1 i∈Aj ∥∆β n 1ij ∥ 2 = 1 D1n • k j=1 i∈Aj d u=1 |T n j,i,2e d,u ,0q | → 0, where e d,u is a vector in R d whose u-th entry is one while other entries are zero;• 1 D1n k j=1 i∈Aj ∥∆η n ij ∥ 2 = 1 D1n • k j=1 i∈Aj q v=1 |T n j,i,0 d ,2eq,v | → 0; • 1 D1n k * j= k+1 i∈Aj ∥∆β n 1ij ∥ 1 = 1 D1n • k * j= k+1 d u=1 |S n j,e d,u ,0,0q | → 0; • 1 D1n k * j= k+1 i∈Aj |∆β n 0ij | = 1 D1n • k * j= k+1 |S n j,0 d ,1,0q | → 0; • 1 D1n k * j= k+1 i∈Aj ∥∆η n ij ∥ 1 = 1 D1n • k * j= k+1 q v=1 |S n j,0 d ,0,eq,v | → 0.By taking the summation of the above limits, we obtain that1 D 1n k j=1 i∈Aj 1 1 + exp(-β n 0i ) - 1 1 + exp(-β * 0j ) + k j=1 i∈Aj ∥∆β n 1ij ∥ 2 + ∥∆η n ij ∥ 2 + k * j= k+1 i∈Aj |∆β n 0ij | + ∥∆β n 1ij ∥ 1 + ∥∆η n ij ∥ 1 → 0.Due to the topological equivalence between the 1-norm and the 2-norm, we deduce that1 = 1 D 1n k j=1 i∈Aj 1 1 + exp(-β n 0i ) - 1 1 + exp(-β * 0j ) + k j=1 i∈Aj ∥∆β n 1ij ∥ 2 + ∥∆η n ij ∥ 2 + k * j= k+1 i∈Aj |∆β n 0ij | + ∥∆β n 1ij ∥ + ∥∆η n ij ∥ → 0,which is a contradiction. As a consequence, at least one among the ratios T n j,i,γ1,γ2 /D 1n , W n j /D 1n and S n j,α1,α2,α3 /D 1n must not approach zero as n → ∞.Step 3 -Application of Fatou's lemma: In this step, we use the Fatou's lemma to argue against the result in Step 2, and then, achieve the desired inequality in equation (21) .In particular, let us denote by m n the maximum of the absolute values of T n j,i,γ1,γ2 /D 1n , W n j /D 1n and S n j,α1,α2,α3 /D 1n . Since at least one among those ratios must not approach zero as n → ∞, we get that 1/m n ̸ → ∞ as n → ∞.Recall from the hypothesis in equation ( 22) that ∥f Gnf G * ∥ L 2 (µ) /D 1n → 0 as n → ∞, which indicates that ∥f Gnf G * ∥ L 1 (µ) /D 1n → 0 due to the equivalence between L 1 (µ)-norm and L 2 (µ)norm. By means of the Fatou's lemma, we have0 = lim n→∞ ∥f Gn -f G * ∥ L 1 (µ) m n D 1n ≥ lim inf n→∞ |f Gn (x) -f G * (x)| m n D 1n dµ(x) ≥ 0. This result implies that [f Gn (x) -f G * (x)]/[m n D 1n ] → 0 for almost every x.Let us denoteT n j,i,γ1,γ2 /m n D 1n → t j,i,γ1,γ2 , W n j /m n D 1n → w j , S n j,α1,α2,α3 /m n D 1n → s j,α1,α2,α3, with a note that at least one among the limits t j,i,γ1,γ2 , w j and s j,α1,α2,α3 is non-zero. Then, from the decomposition in equation ( 27), we deduce thatk j=1 i∈Aj 2 |γ|=1 t j,i,γ1,γ2 • ∂ |γ1| σ ∂β γ1 1 (x, 0 d , β0i ) ∂ |γ2| h ∂η γ2 (x, η * j ) + k j=1 w j • h(x, η * j ) + k * j= k+1 |α|=1 s j,α1,α2,α3 • ∂ |α1|+α2 σ ∂β α1 1 ∂β α2 0 (x, β * 1j , β * 0j ) ∂ |α3| h ∂η α3 (x, η * j ) = 0,for almost every x. Note that the expert function h(•, η) is strongly identifiable, then the above equation implies that t j,γ1,γ2 = w j = s j,α1,α2,α3 = 0, for any j∈ [k * ], α 1 , γ 1 ∈ N d , α 2 ∈ N and γ 2 , α 3 ∈ N q such that 1 ≤ |γ 1 | + |γ 2 | ≤ 2 and |α 1 | + α 2 + |α 3 | = 1.This contradicts the fact that at least one among the limits s i,α1,α2,α3 is different from zero.Hence, we obtain the local inequality in equation (21) . Consequently, there exists some ε ′ > 0 such thatinf G∈G k (Θ):D1(G,G * )≤ε ′ ∥f G -f G * ∥ L 2 (µ) /D 1 (G, G * ) > 0.Global part: Given the above result, it suffices to demonstrate thatEQUATION)Assume by contrary that the inequality (28) does not hold true, then we can find a sequence of mixing measuresG ′ n ∈ G k (Θ) such that D 1 (G ′ n , G * ) > ε ′ and lim n→∞ ∥f G ′ n -f G * ∥ L 2 (µ) D 1 (G ′ n , G * ) = 0, which indicates that ∥f G ′ n -f G * ∥ L 2 (µ) → 0 as n → ∞.Recall that Θ is a compact set, therefore, we can replace the sequence G ′ n by one of its subsequences that converges to a mixing measureG ′ ∈ G k (Ω). Since D 1 (G ′ n , G * ) > ε ′ , we deduce that D 1 (G ′ , G * ) > ε ′ .Next, by invoking the Fatou's lemma, we have that0 = lim n→∞ ∥f G ′ n -f G * ∥ 2 L 2 (µ) ≥ lim inf n→∞ f G ′ n (x) -f G * (x) 2 dµ(x).Thus, we get that f G ′ (x) = f G * (x) for almost every x. From Proposition 1, we deduce thatG ′ ≡ G * . Consequently, it follows that D 1 (G ′ , G * ) = 0, contradicting the fact that D 1 (G ′ , G * ) > ε ′ > 0.Hence, the proof is completed.Lemma 2. If the following holds for any r ≥ 1:EQUATIONthen we obtain thatEQUATION)Proof of Lemma 2. Indeed, from the Gaussian assumption on the noise variables ϵ i , we obtain thatY i |X i ∼ N (f G * (X i ), σ 2 ) for all i ∈ [n].Next, the assumption in equation ( 29) indicates for sufficiently small ε > 0 and a fixed constant C 1 > 0 which we will choose later, we can find a mixing measureG ′ * ∈ G k (Θ) such that D 2,r (G ′ * , G * ) = 2ε and ∥f G ′ * -f G * ∥ L 2 (µ) ≤ C 1 ε.From Le Cam's lemma [39] , as the Voronoi loss function D 2,r satisfies the weak triangle inequality, we obtain thatEQUATIONwhere the second inequality is due to the fact thatKL(N (f G ′ * (X), σ 2 ), N (f G * (X), σ 2 )) = (f G ′ * (X) -f G * (X)) 2 2σ 2 .By choosing ε = n -1/2 , we obtain that ε • exp(-C 1 nε 2 ) = n -1/2 exp(-C 1 ). As a consequence, we achieve the desired minimax lower bound in equation (30) .Main proof. Following from the result of Lemma 2, it is sufficient to show that the following limit holds true for any r ≥ 1:EQUATION)To this end, we need to construct a sequence of mixing measures(G n ) that satisfies D 2,r (G n , G * ) → 0 and ∥f Gn -f G * ∥ L 2 (µ) D 2,r (G n , G * ) → 0,as n → ∞. Recall that under the Case 2, at least one among parameters a * 1 , . . . , a * k * is equal to 0 d . Without loss of generality, we may assume that a * 1 = 0 d . Next, let us take into account the sequence(G n ) with k * + 1 atoms in which • exp(-β n 01 ) = exp(-β n 02 ) = 1 + 2 exp(-β * 01 ) and β n 0i = β * 0(i-1) for any 3 ≤ i ≤ k * + 1; • β n 11 = β n 12 = β * 11 = 0 d and β n 1i = β * 1(i-1) for any 3 ≤ i ≤ k * + 1; • a n 1 = a n 2 = a * 1 = 0 d and a n i = a * i-1 for any 3 ≤ i ≤ k * + 1; • b n 1 = b * 1 + c n , b n 2 = b * 1 + 2c n and b n i = b * i-1 for any 3 ≤ i ≤ k * + 1,where c ∈ R will be chosen later. Consequently, we get thatD 2,r (G n , G * ) = c r n r + (2c) r n r = O(n -r ).Gn -f G * ∥ L 2 (µ) /D 2,r (G n , G * ) → 0.In particular, following from the decomposition of f Gn (x)f G * (x) in equation ( 23) and the above parameter setting, we have thatf Gn (x) -f G * (x) = 2 i=1 φ((a n i ) ⊤ x + b n i ) 1 + exp(-(β n 1i ) ⊤ x -β n 0i ) - φ((a * 1 ) ⊤ x + b * 1 ) 1 + exp(-β n 0i ) + 2 i=1 1 1 + exp(-β n 0i ) - 1 1 + exp(-β * 01 ) • φ((a * 1 ) ⊤ x + b * 1 ) + k * +1 i=3 φ((a n i ) ⊤ x + b n i ) 1 + exp(-(β n 1i ) ⊤ x -β n 0i ) - φ((a * i-1 ) ⊤ x + b * i-1 ) 1 + exp(-(β * 1(i-1) ) ⊤ x -β * 0(i-1) ) = 2 i=1 φ(b n i ) 1 + exp(-β n 0i ) - φ(b * 1 ) 1 + exp(-β n 0i ) = 1 2 + 2 exp(-β * 01 ) • 2 i=1 [φ(b n i ) -φ(b * 1 )].Thus, we deduce that[2 + 2 exp(-β * 01 )] • [f Gn (x) -f G * (x)] = 2 i=1 [φ(b n i ) -φ(b * 1 )].When r is an odd natural number: By applying the Taylor expansion up to order r-th, we get that[2 + 2 exp(-β * 01 )] • [f Gn (x) -f G * (x)] = 2 i=1 r α=1 (b n i -b * 1 ) α α! φ (α) (b * 1 ) + R 1 (x) = r α=1 (1 + 2 α )c α α!n α • φ (α) (b * 1 ) + R 1 (x),where R 1 (x) is the Taylor remainder such that R 1 (x)/D 2,r (G n , G * ) → 0 as n → ∞.Note that r α=1(1+2 α )φ (α) (b * 1 ) α!n α• c α is an odd-order polynomial of c. Thus, we can choose c as a root of this polynomial, which leads to the fact thatf Gn (x) -f G * (x) = R 1 (x) 2 + 2 exp(-β * 01 ).From the above results, we deduce that[f Gn (x) -f G * (x)]/D 2,r (G n , G * ) → 0 for almost every x.As a consequence, we achieve that ∥f Gn -f G * ∥ L 2 (µ) /D 2,r (G n , G * ) → 0 as n → ∞.When r is an even natural number: By means of the Taylor expansion of order (r + 1)-th, we have[2 + 2 exp(-β * 01 )] • [f Gn (x) -f G * (x)] = r+1 α=1 (1 + 2 α )c α α!n α • φ (α) (b * 1 ) + R 2 (x), where R 2 (x) is a Taylor remainder such that R 2 (x)/D 2,r (G n , G * ) → 0. Since r+1 α=1 (1+2 α )σ (α) (b * 1 ) α!n α• c α is an odd-degree polynomial of variable c, we can argue in a similar fashion to the scenario when r is odd to obtain that ∥fGn -f G * ∥ L 2 (µ) /D 2,r (G n , G * ) → 0.Combine results from the above two scenarios of r, we obtain the claim in equation (32) .Following from the result of Lemma 2, it is sufficient to show that the following limit holds true for any r ≥ 1:EQUATIONTherefore, given an arbitrary mixing measure G :=k i=1 1 1 + exp(-β0i ) δ ( β1i,ηi) ∈ G k (Θ), wefocus only on demonstrating thatEQUATIONAssume by contrary that the above inequality does not hold true, then there exists a sequence ofEQUATIONas n → ∞. Let us denote byA n i := A i (G n ) aVoronoi cell of G n generated by the j-th components of G. Since our arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size, i.e. A j = A n i . Additionally, since G n and G share the same number of atoms k under the Regime 2 and D 3n → 0, the Voronoi cell A i has only one element for any i ∈ [k]. WLOG, we assume thatA i = {i} for all i ∈ [k], which follows that (β n 0i , β 1i , η n i ) → ( β0i , β1i , ηi ) as n → ∞ for any i ∈ [k]. Thus, the Voronoi loss D 3n can be represented asEQUATIONwhere we denote ∆ βn 0i := β n 0i -β0i , ∆ βn 1i := β n 1i -β1i and ∆η n i := η n iηi . Now, we divide the proof of local part into three steps as follows:Step 1 -Taylor expansion: In this step, we decompose the term f Gn (x)f G (x) using Taylor expansion. Let us denote σ(x, β 1 , β 0 ) := 1 1+exp(-β ⊤ 1 x-β0) , then we haveEQUATIONwhere R 1 (x) is a Taylor remainder such that R 1 (x)/D 3n → 0 as n → ∞, andS n i,α1,α2,α3 := |α|=1 1 α! (∆ βn 1i ) α1 (∆ βn 0i ) α2 (∆η n i ) α3 , for any i ∈ [k], α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that |α 1 | + α 2 + |α 3 | = 1.Step 2 -Non-vanishing coefficients: In this step, we show that at least one among the ratios S n i,α1,α2,α3 /D 3n does not go to zero as n tends to infinity. Assume by contrary that all of them converge to zero, i.e. S n i,α1,α2,α3 /D 3n → 0 as n → ∞, for any i∈ [k], α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that |α 1 | + α 2 + |α 3 | = 1. Then, it follows that • 1 D3n • k i=1 ∥∆ βn 1i ∥ 1 = 1 D3n • k i=1 d u=1 |S n i,e d,u ,0,0q | → 0; • 1 D3n • k i=1 |∆ βn 0i | = 1 D3n • k i=1 |S n i,0 d ,1,0q | → 0; • 1 D3n • k i=1 ∥∆η n i ∥ 1 = 1 D3n • k i=1 q v=1 |S n i,0 d ,0,eq,v | → 0.By taking the summation of the above three limits, we deduce that1 D 3n • k i=1 |∆ βn 0i | + ∥∆ βn 1i ∥ 1 + ∥∆η n i ∥ 1 → 0.Due to the topological equivalence between the 1-norm and 2-norm, we achieve that1 = 1 D 3n • k i=1 |∆ βn 0i | + ∥∆ βn 1i ∥ + ∥∆η n i ∥ → 0,as n → ∞, which is a contradiction. Thus, at least one among the ratios S n i,α1,α2,α3 /D 3n must not approach zero as n → ∞.Step 3 -Application of Fatou's lemma: In this step, we use the Fatou's lemma to argue against the result in Step 2, and then, achieve the desired inequality in equation (35) .In particular, let us by m n the maximum of the absolute values of S n i,α1,α2,α3 /D 3n . Since at least one among those ratios must not approach zero as n → ∞, we get that 1/m n ̸ → ∞ as n → ∞.Recall from the hypothesis in equation (36) that ∥f Gn -f G ∥ L 2 (µ) /D 3n → 0 as n → ∞, which indicates that ∥f Gn -f G ∥ L 1 (µ) /D 3n→ 0 due to the equivalence between L 1 (µ)-norm and L 2 (µ)norm. By means of the Fatou's lemma, we have0 = lim n→∞ ∥f Gn -f G ∥ L 1 (µ) m n D 3n ≥ lim inf n→∞ |f Gn (x) -f G (x)| m n D 3n dµ(x) ≥ 0. This result implies that [f Gn (x) -f G (x)]/[m n D 3n ] → 0 for almost every x.Let us denote S n i,α1,α2,α3 /m n D 3n → s i,α1,α2,α3 as n → ∞ with a note that at least one among the limits s i,α1,α2,α3 is non-zero. Then, it follows from the decomposition in equation (38) that k i=1 |α|=1 s i,α1,α2,α3 • ∂ |α1|+α2 σ ∂β α1 1 ∂β α2 0 (x, β1i , β0i ) ∂ |α3| h ∂η α3 (x, ηi ) = 0,for almost every x. Note that the expert function h(•, η) is ..., then the above equation implies that s i,α1,α2,α3 = 0, for any i∈ [k], α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that |α 1 | + α 2 + |α 3 | = 1.This contradicts the fact that at least one among the limits s i,α1,α2,α3 is different from zero.Hence, we obtain the local inequality in equation (35) .In this appendix, we explore the identifiability of the sigmoid gating MoE model in Proposition 1.Proposition 1. If the equation f G (x) = f G * (x) holds true for almost every x, then it follows that G ≡ G ′ .Proof of Proposition 1. From the assumption of this proposition, we haveEQUATIONSince the expert function h(•, η) is identifiable, the set {h(x,η ′ i ) : i ∈ [k ′ ]}, where η ′ 1 , . . . , η ′ k ′ are distinct vectors for some k ′ ∈ N, is linearly independent. If k ̸ = k * , then there exists some i ∈ [k] such that η i ̸ = η * j for any j ∈ [k * ]. This implies that 1 1+exp(-β1i) ⊤ x-β0i) = 0, which is a contradiction. Thus, we must have that k = k * . As a result, 1 1 + exp(-β 1i ) ⊤ x -β 0i ) : i ∈ [k] = 1 1 + exp(-β * 1i ) ⊤ x -β * 0i ) : i ∈ [k * ] ,for almost every x. WLOG, we may assume thatEQUATIONfor almost every x for any i ∈ [k * ]. It is worth noting that the sigmoid function is invariant to translations, then equation (40) indicates that β 1i = β * 1i + v 1 and β 0i = β * 0i + v 0 for some v 1 ∈ R d and v 0 ∈ R. Nevertheless, due to the assumptions β 1k = β * 1k = 0 d and β 0k = β * 0k = 0, we have that v 1 = 0 d and v 0 = 0. Consequently, it follows that β 1i = β * 1i and β 0i = β * 0i for any i ∈ [k * ]. Then, equation ( 39) can be represented asEQUATIONfor almost every x. Next, let us denote J 1 , J 2 , . . . , J m as a partition of the index set[k * ], where m ≤ k, such that (β 0i , β 1i ) = (β * 0i ′ , β * 1i ′ ) for any i, i ′ ∈ J j and j ∈ [k * ].On the other hand, when i and i ′ do not belong to the same set J j , we let (β 0i , β 1i ) ̸ = (β 0i ′ , β 1i ′ ). Thus, we can reformulate equation (41) asm j=1 i∈Jj 1 1 + exp(-β 1i ) ⊤ x -β 0i ) • h(x, η i ) = m j=1 i∈Jj 1 1 + exp(-β * 1i ) ⊤ x -β * 0i ) • h(x, η * i ),for almost every x. Recall that β 1i = β * 1i and β 0i = β * 0i for any i ∈ [k * ], then the above leads to {η i : i ∈ J j } ≡ {η * i : i ∈ J j }, for any j ∈ [m]. As a result, we achieve thatG = m j=1 i∈Jj 1 1 + exp(-β 0i ) • δ (β1i,ηi) = m j=1 i∈Jj 1 1 + exp(-β * 0i ) • δ (β * 1i ,η * i ) = G * .Hence, the proof is totally completed.In this appendix, we conduct a simulation study to empirically validate our theoretical results on the convergence rates of the least squares estimators under both the Regime 1 and the Regime 2 of the sigmoid gating MoE. All the subsequent experiments are conducted on a MacBook Air equipped with an M1 chip CPU.Regime 1. We begin with numerical experiments for the Regime 1 in which we sample synthetic data based on the model described in equation (1) . In particular, we generate {(X i , Y i )} n i=1 ⊂ R d × R by first sampling X i ∼ Uniform([-1, 1] d ) for i = 1, . . . , n. Then, we generate Y i according to the following model:EQUATIONwhere the regression function f G * (•) is defined as:EQUATIONThe input data dimension is set at d = 32. We employ k * = 8 experts of the form φ (a * i ) ⊤ x + b * i , where the activation function is either the ReLU function or the identity function. The variance of Gaussian noise ϵ i is specified as ν = 0.01.The ground-truth gating parameters β * 0i ∈ R are drawn independently from an isotropic Gaussian distribution with zero mean and variance ν g = 0.01/d for 1 ≤ i ≤ 6, while we set β * 11 = . . . = β * 1k * = 0 d . Similarly, the true expert parameters, (a * i , b * i ) ∈ R d × R, are drawn independently of an isotropic Gaussian distribution with zero mean and variance ν e = 1/d for all experts. These parameters remain unchanged for all the experiments (see also Table 3) .Training procedure. For each sample size n, spanning from 10 3 to 10 5 , we perform 20 experiments. In every experiment, we employ k = k * + 1 = 9 fitted experts, and the parameters initialization for the gating's and experts' parameters are adjusted to be near the true parameters, minimizing potential Table 3 : Ground-truth Parameters for Gating and Experts. The variance for the gating parameters is ν g = 0.01/d, and for the expert parameters is ν e = 1/d.Expert parametersβ * 1i = 0 d β * 0i ∼ N (0, ν g ) a * i ∼ N (0 d , ν e I d ) b * i ∼ N (0, ν e )Table 4 : True Parameters for Gating and Experts. The variance for the gating parameters is ν g = 0.01/d and for the expert parameters is ν e = 1/d.Expert parameters The blue lines depict the Voronoi loss associated with the ReLU experts, while the green lines correspond to that of the linear experts. The red dash-dotted lines are used to illustrate the fitted lines for determining the empirical convergence rates. See Appendix C for the experimental details.β * 1i ∼ N (0 d , ν g I d ) 1 ≤ i ≤ 7 β * 1i = 0 d i = 8 β * 0i ∼ N (0, ν g ) a i ∼ N (0 d , ν e I d ) b * i ∼ N (0, ν e )instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across 10 epochs, employing a learning rate of η = 0.1 to fit a model to the synthetic data.Results. For each experiment, we calculate the Voronoi losses for every model and report the mean values for each sample size in Figure 2a . Error bars representing two standard deviations are also shown. In Figure 2a , both the empirical convergence rates corresponding to the ReLU experts and the linear experts are analyzed under the over-specified setting. It can be seen that the use of ReLU experts induces a rapid convergence rate of order O(n -0.46 ), while the linear experts lead to a considerably slower rate of order O(n -0.02 ). Those empirical rates match our theoretical results in Theorem 2 and Theorem 4, respectively.Regime 2. In the experiments for the Regime 2, we apply the same experimental setup as in those for the Regime 1 except for the generation of true parameters β * 1i . More specifically, β * 1i are drawn independently from an isotropic Gaussian distribution N (0 d , ν g I d ) for 1 ≤ i ≤ 8, where ν g = 0.01/d, while we set β * 1i = 0 d for i = 8 (see also Table 4 ). Results. It can be observed from Figure 2b that the uses of ReLU experts and linear experts both lead to fast empirical convergence rates of orders O(n -0.48 ) and O(n -0.42 ), respectively. Those empirical rates align with our theoretical findings in Theorem 5.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The error bars are reported in Figure 1 and Figure 2 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: All the information on the computer resources are provided in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments. Justification: We have read and followed all the NeurIPS code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Given the theoretical nature of the paper, we do not think there are any positive or negative societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This is a theoretical work, and we do not release any data or models that have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: We do not use any existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We will release the code for our simulations after the review process. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This work presents a theoretical analysis to examine why sigmoid gating functions exhibit superior performance to softmax gating functions for mixture of experts models. Reviewers appreciated the novelty of the work, acknowledging that sigmoid gating functions are uncommon, and that the effect of selecting different gating functions is a curious topic worth investigating. The authors did an excellent job of responding to reviewer feedback, and while some concerns still remain with the presentation, many of the more technical issues have been resolved. I would strongly recommend that the authors consider making edits (perhaps by including passages from the discussion) to ensure future readers do not adopt the same misunderstandings, and aim to reduce technicality by moving less critical details to the supplementary material. Conditional on such edits, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents a theoretical analysis of expert estimation in MoE models using sigmoid gating, in contrast to the more widely used softmax gating. The authors show that sigmoid gating leads to better sample efficiency compared to softmax gating for estimating expert parameters. In particular, the paper analyzes convergence rates for least squares estimation under two regimes: 
* Regime 1) when all over-specified gating parameters are zero
* Regime 2) when at least one of the over-specified gating parameters is non-zero

One of the key findings is that experts formulated as NNs with common activations like ReLU and GELU obtain faster convergence rates under sigmoid gating compared to softmax gating. This work provides theoretical justification for the potential empirical benefits of sigmoid gating, showing it requires smaller sample sizes to achieve the same estimation error as softmax gating.
strength_weakness: strength_weakness: 1) The structure of the paper is very confusing and could be much improved. There is way too much repetition in the paper. For example, the section Technical challenges in the introduction which defines Regime 1 and 2 (Lines 83-92) is largely copy pasted to lines 155-165 and is redundant. The various sections in the introductions particularly the contributions should be kept at a more high level and not just copy paste the method section. The authors should provide a more concise overview of the problem and main contributions. Remove technical details that are repeated in later sections.

2) The recent insights into training Sparse MoEs demonstrate that "the common practice of setting the size of experts in MoE to mirror the feed-forward layer (of a base dense network) is not optimal at almost any computational budget" (Jakub Krajewski et al. 2024). If in practice, top-k combination of small fine-grained experts is proven to be more effective than a single large expert, are we dealing in a regime where all true experts are *over-specified* in practice? If this is the case what are the implications for Regime 1? Does it mean that none of the experts are input-dependent as all the over-specified parameters $β^*_{1i}$ are equal to $0_d$? Does Regime 1 hold in practice?

3) It would be best if the authors could find some practical implications of their findings. The theoretical work presented in the paper does not establish any relationship to any of the SoTA MoE models in the literature. It is unclear to me how impactful the convergence analysis with the considered regimes is in practical MoE research works.

4) At present, the content and organization of the paper is very close to the paper "On Least Square Estimation in Softmax Gating Mixture of Experts" (Nguyen et al. 2024). The sections are so similar that may cause copyright issues. 

References:
1. Krajewski, Jakub, et al. "Scaling laws for fine-grained mixture of experts." arXiv preprint arXiv:2402.07871 (2024).

2. Nguyen, Huy, Nhat Ho, and Alessandro Rinaldo. "On least squares estimation in softmax gating mixture of experts." ICML (2024).
questions: questions: Minor point:
- misspecified is misspelled in line 175.dq
limitations: limitations: Two limitations concerning the assumptions used in the analyses are discussed in the limitations section.
paper_summary: paper_summary: The paper argues that the sigmoid gating function is more sample efficient than the softmax gating function in mixture of experts (MoE) modeling. It removes competition and estimates the contribution each expert independently.

Empirical studies show that sigmoid gating achieves superior performance, but the paper aims to provide theoretical backing for this claim.

They consider a regression framework and analyze the rates of convergence of the least squares estimator in over-specified cases.

The convergence rates for expert estimation are derived under two regimes: Regime 1 (all over-specified parameters are 0) and Regime 2 (at least one over-specified parameter is not 0).
strength_weakness: strength_weakness: The results are heavily dependent on specific assumptions, such as the distinctness of expert parameters and the boundedness of the input space. If these assumptions are violated in practical scenarios, the theoretical guarantees may not hold

While the derivations are very appropriate and i appreciate the citations, it would be interesting to see even toy experiments byt the authorsthat show similar conversions to the theoretical ones.
questions: questions: the paper lacks justification for the choice of the regimes. We also need to deep dig to understand the reasons for strong identifiability conditions (for experts to be distinct enough, if i understand correctly), bracketing entropy (model is not overly complex) and concentration inequalities. it is quite difficult to read, so some more intuition and explanation would be helpful.

Minor:
124 in can seen
limitations: limitations: As mentioned by the authors, 

The paper assumes that the ground-truth parameters are independent of the sample size, leading to point-wise rather than uniform estimation rates. 

The assumption that the true regression function belongs to the parametric class of MoE models under sigmoid gating is restrictive. This assumption is likely to be violated in real-world settings.

I still find these assumptions reasonable.
paper_summary: paper_summary: This paper provides a theoretical analysis of the convergence rate of the least squares estimator for learning MoE with sigmoid gating. Based on the results, the authors conclude that sigmoid gating enjoys a faster convergence rate and requires a smaller sample size to achieve the same error compared to softmax gating.
strength_weakness: strength_weakness: 1. The biggest concern to me is that the results from this paper do not fully support the main claim that “sigmoid gating is more sample efficient than softmax gating”. If I understand correctly (correct me if I'm wrong), the comparison between sigmoid and softmax is not under the same scenario. The convergence rate for softmax in Table 1 is from the analysis in [21], where the ground truth is an MoE model with softmax gating, while this paper considers the setting where the ground truth is an MoE model with sigmoid gating. These are two different setups and do not necessarily imply that, given the same ground truth function, sigmoid is necessarily more sample efficient than softmax. 

2. The only existing empirical work discussing the potential superiority of sigmoid gating that the authors have mentioned is [3], where the provided intuition is that softmax introduces competition between experts, while sigmoid does not have this issue. This is very straightforward, although not yet formally and theoretically justified. On the other hand, this paper explains the advantage from the perspective of sample efficiency. I wonder how these two perspectives can be connected. Is there anything regarding the competition view that can be reflected in this paper’s analysis? I believe this is important for understanding the effect of sigmoid gating.

3. Following point 2, it would be helpful to provide empirical results to further support the main conclusion that sigmoid gating is more sample efficient. Even experiments on toy data can help. Otherwise, the statement regarding sample efficiency lacks empirical evidence. Additionally, even "sigmoid gating is better than softmax gating" alone does not seem to be well-supported by empirical observations from the literature.
questions: questions: See my questions raised in Weaknesses.
limitations: limitations: The authors have discussed certain limitations in Section 4.

Comments: title: Thank You!
Response to Reviewer 4kwq
Thank you!
Thank You
comment: Dear Reviewer 4kwq,

We would like to thank you for increasing your rating to 6, we really appreciate that. We will include the connection of the universal approximation properties of the sigmoid and softmax functions and the derivation of expert estimation rates under the same data generation setting in the revision of our paper.

Best,

The Authors
Thank the authors for addressing my concerns. I believe the step of connecting the two data generation processes with the universal approximation properties of the sigmoid and softmax functions is essential in clarifying and legitimizing the comparison. It would be beneficial to include this in the revised version.

I have adjusted my score accordingly.
Dear Reviewer 4kwq,

Thanks for your question. We would like to confirm that the analysis of the sigmoid gating (resp. softmax gating) MoE under the same data generation setting is implied by the analysis in our paper (resp. in [a]). 

**Assumption on the regression function $g$.** Firstly, let us recall a result on the universal approximation of the sigmoid function in [b]. In particular, let $g:\mathcal{X}\to\mathbb{R}$ be a function such that there is a Fourier representation of the form

$$g(x)=\int_{\mathcal{X}}e^{i\omega\cdot x}\tilde{g}(\omega)d\omega,$$

for some complex-valued function $\tilde{g}(x)$ for which $\omega\tilde{g}(\omega)$ is integrable and the term

$$C_g=\int_{\mathcal{X}}||\omega||_2|\tilde{g}(\omega)|d\omega$$

is finite. Then, there exists a linear combination of $\tilde{k}$ sigmoidal functions $f_{\widetilde{G}}(x)$ such that

$$\int_{\mathcal{X}}[f_{\widetilde{G}}(x)-g(x)]^2d\mu(x)\leq\frac{(2C_g)^2}{\tilde{k}}.$$

**Analysis for the sigmoid gating MoE.** Subsequently, we combine the above result with our current analysis in the paper. More specifically, by treating the mixing measure $\widetilde{G}$ as the mixing measure $G_*$ in the paper, we are able to design a Voronoi loss 

$$\mathcal{D}(G,\widetilde{G})=\sum_{j=1}^{k}\sum_{i\in\mathcal{A}_j}$$

$$\Big[|\beta_{0i}-\tilde{\beta}_{0j}|$$

$$+||\beta_{1i}-\tilde{\beta}_{1j}||$$

$$+||\eta_{i}-\tilde{\eta}_{j}||\Big],$$ 

and show that 

$$\mathcal{D}(\widehat{G}_n,\widetilde{G})=O_P(\sqrt{\log(n)/n}).$$

From this bound, we deduce that the expert estimation rates are of order $O_P(n^{-1/2})$.

**Analysis for the softmax gating MoE.** Similarly, we also leverage the universal approximation of the softmax function in [c], and the analysis in [a] to derive the expert estimation rates under the same data generation setting. In particular, we find out that the rates for estimating feed-forward expert networks with ReLU activation are of order $n^{-1/4}$, while those for polynomial experts are slower than any polynomial rates, and could be as slow as $1/\log(n)$.

**As the discussion period deadline is approaching, please let us know if you have any further concerns. We are more than happy to answer your questions. Additionally, if you find that our response sufficiently addresses your concerns, we hope that you will re-evaluate the paper and increase the rating. Thank you again!**

**References**

[a] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024.

[b] Andrew R. Barron. Universal Approximation Bounds for Superpositions of a Sigmoidal Function. IEEE Transactions on Information Theory, 1993.

[c] Assaf J. Zeevi. Error Bounds for Functional Approximation and Estimation Using Mixtures of Experts. IEEE Transactions on Information Theory, 1998.
Thank the authors for their response. I’m curious about how the new theoretical results presented relate to those in the original paper. Are they completely independent, an extension of the original findings, or already implied by them? If they are a more generalized version of the original results, do we need to introduce any new assumptions here, particularly any assumptions on $g(x)$?
Dear Reviewer 4kwq,

We would like to thank you for raising your concerns. **We hope that our following response will address those concerns, and eventually convince you to increase your rating**.

Per your suggestion, we conduct both theoretical and empirical sample efficiency comparisons between the sigmoid gating and the softmax gating under the setting where the data are generated from the same source. Below are the expert estimation rates when using the softmax gating MoE and the sigmoid gating MoE to fit the data, respectively.

**Same data generation setting:** The data $(X_1,Y_1),\ldots,(X_n,Y_n)$ are generated from a regression framework 
$$Y_i=g(X_i)+\epsilon_i,$$
where the features $X_1,\ldots,X_n$ are sampled from a probability distribution $\mu$, and $\epsilon_1,\ldots,\epsilon_n$ are independent Gaussian noise variables such that $\mathbb{E}[\epsilon_i|X_i]=0$ and $Var(\epsilon_i|X_i)=\nu$ for all $1\leq i\leq n$. The unknown regression function $g(x)$ is not necessarily a mixture of experts (MoE). Then, we use either the sigmoid gating MoE or the softmax gating MoE to fit the data. 

In particular, the least square estimator $\widehat{G}_n$ in Eq.(3) now converges to a mixing measure $\widetilde{G}\in\mathcal{G}_k(\Theta)$ where

$$\widetilde{G}\in\arg\min_{G}||f_{G}-g||_2,$$

in which $f_{G}$ is the regression taking the form of the MoE associated with either the sigmoid gating or the softmax gating. Below are the expert estimation rates resulted from our convergence analysis:

**When $f_{G}$ is the softmax gating MoE:** the rates for estimating feed-forward expert networks with ReLU activation are of order $n^{-1/4}$, while those for polynomial experts are slower than any polynomial rates, and could be as slow as $1/\log(n)$;

**When $f_{G}$ is the sigmoid gating MoE:** the estimation rates for feed-forward expert networks with ReLU activation and polynomial experts share the same order of $n^{-1/2}$, which are significantly faster than those when using the softmax gating MoE.

**Empirical validation:** To justify the above theoretical results, we conduct a simulation study in the General Response section where we take polynomial experts into account. The empirical result totally matches its theoretical rates. 

As a consequence, **we can conclude that the sigmoid gating is more sample efficient than the softmax gating even when fitting the same data**.

Please feel free to let us know if you have any further concerns regarding the paper. We are more than happy to address all of them.

Best,

The Authors
Thank you to the authors for their responses. However, I am not entirely sure how the authors' response addresses my first question. My question was: this paper and Nguyen et al. [21] consider different settings. This paper considers the case where the data are generated with sigmoid gating, and a model with sigmoid gating is trained to fit the data; Nguyen et al. [21] consider the case where the data are generated with softmax gating, and a model with softmax gating is trained to fit the data. Since the data in these two cases are not generated in the same way, we cannot claim that sigmoid gating is more sample efficient than softmax if we use them to fit the same data. In other words, if the paper had shown that, given the same data generated in a certain way, using sigmoid to fit the data is more sample efficient than using softmax, then I would find the conclusion well justified.

The new empirical results seem to be done in the desired setting, where the data are always generated with softmax gating, whether using the sigmoid gating model or the softmax gating model to fit the data.
Dear Reviewer PmAq,

We would like to thank you for increasing the rating to 5. We will definitely incorporate the modifications into the revision of our paper as discussed. Please feel free to let us know if you have any further concerns.

Best,

The Authors
Thanks for your detailed response. Most of my concerns are addressed. I would encourage the authors to improve the conciseness of the presentation in the revised version as discussed in the comments. I raise my rating to 5
Dear Reviewer tkzC,

We want to thank you for increasing your score to 6. 

Please let us know if you still have any concerns about the paper.

Best,

The Authors
Thank you for addressing the questions. As I am leaning towards acceptance, I raised my score to 6.