Title: Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Venue: NeurIPS 2024
Accepted: yes
Keywords: mixture of experts

sigmoid gating function

softmax gating function

sample efficiency
Abstract: The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due to its inherent structure. In response, the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically to achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, in fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as a mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case in which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. In both cases, we find that experts formulated as feed-forward networks with commonly used activation such as $\mathrm{ReLU}$ and $\mathrm{GELU}$ enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts

Abstract

The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due to its inherent structure. In response, the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically to achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, in fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as a mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case in which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. In both cases, we find that experts formulated as feed-forward networks with commonly used activation such as ReLU and GELU enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.

1 Introduction

Mixture of experts (MoE) [15, 17] has recently emerged as a powerful machine learning model that helps scale up the model capacity while requiring a nearly constant computational overhead [34, 8]. In particular, it aggregates multiple sub-models called experts based on a gating network. Here, experts can be formulated as neural networks, and they specialize in different aspects of the data. For instance, in large language models [16, 31, 19, 40, 7, 5, 30], one expert might focus on syntax, another on semantics, whereas the others concentrate on context. Meanwhile, the gating network takes the input data and calculates a set of weights that determine the contribution of each expert. This gating mechanism guarantees that the most relevant experts are assigned more weight for a given input. In addition to language modeling, MoE has also been leveraged in several other applications, including computer vision [32, 20, 33], multi-task learning [12, 10], multi-modal learning [11], speech recognition [38, 9, 29] and reinforcement learning [3, 28].

In the above applications of MoE, practitioners frequently utilize the softmax gating function to compute the mixture weights. However, the use of softmax function might introduce an unexpected competition among experts, which leads to the representation collapse issue [2, 30, 18, 24]: when the weight of one expert increases, those of the others decrease accordingly. As a consequence, some experts will dominate the decision-making process, and overshadow the contributions of others. Therefore, the ability of MoE models to incorporate the diversity of expert specialization is partially limited. To this end, Csordás et al. [4] propose using the sigmoid function in place of the softmax to remove the unnecessary expert competition, and empirically demonstrate that it is indeed a better choice of gating function. On the other hand, the theoretical guarantee for that claim has remained missing in the literature. The main objective of this work is to provide a comprehensive analysis of the sigmoid gating function from the perspective of the expert estimation problem. In particular, we will show that sigmoid gating delivers superior sample efficiency for estimating the model parameters and allows for more general expert functions than softmax gating.

Related works. A very recent line of research has focused on analyzing the convergence rates for expert estimation in Gaussian MoE models under a variety of assumptions and choices of gating functions. Assuming that data from an input-free gating Gaussian MoE, Ho et al. [14] demonstrated that the expert estimation rates for the maximum likelihood estimator vary with the algebraic independence among expert functions. Under the same setting but using instead softmax gating, Nguyen et al. [27] discovered that the expert estimation rates depend on the solvability of a system of polynomial equations resulting from an intrinsic interaction among gating and expert parameters. Subsequently, a Gaussian MoE with Top-K sparse softmax gate, designed to activate only a fraction of experts per input, was studied in [25]. The convergence analysis in that work revealed that activating exactly one expert for each input would eliminate the previous parameter interaction, thus boosting the expert estimation rates. Despite many important insights provided in this line of works, the mixture setting of Gaussian MoE is still far from practical. To that effect, Nguyen et al. [26] introduced a more realistic regression framework in which, conditionally on the features, the response variables are sampled from noisy realization of an unknown and deterministic softmax gating MoE-type regression function. They found that experts formulated as feed-forward networks with sigmoid or hyperbolic tangent activation functions enjoy faster estimation rates than those equipped with a polynomial activation. In this paper, we adopt the same regression setting and carry out an analogous analysis by considering instead sigmoid gating for the underlying regression function, as opposed to the more popular softmax gating considered in [26]. As we will see, the choice of gating function turns out to be critical, requiring a different type of analysis and leading to markedly different conclusions.

Problem setup. We assume that the data ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . . , ( X n , Y n ) ∈ R d × R follow a standard regression model

where the features X 1 , X 2 , . . . , X n are i.i.d. samples from a probability distribution µ on R d and ϵ 1 , . . . , ϵ n are independent noise variables such that E [ ϵ i | X i ] = 0 and Var( ϵ i | X i ) = ν for all 1 ≤ i ≤ n ; for simplicity we further assume that they follow a Gaussian distribution. The unknown regression function f G ∗ is formulated as a sigmoid-gated MoE with k ∗ experts, i.e.,

where the expert function x ↦→ h ( x, η ) is of parametric form, specified by parameter η ∈ R q . The regression function f G ∗ is fully characterized by the unknown parameters ( β ∗ 0 i , β ∗ 1 i , η ∗ i ) k ∗ i =1 in R × R d × R q , which can be compactly encoded using the associated mixing measure G ∗ := ∑ k ∗ i =1 1 1+exp( -β ∗ 0 i ) δ ( β ∗ 1 i ,η ∗ i ) , a weighted sum of Dirac measures δ .

Least squares estimation. To estimate the unknown parameters ( β ∗ 0 i , β ∗ 1 i , η ∗ i ) k ∗ i =1 (equivalently, the ground truth mixing measure G ∗ ), we deploy the least squares method [35] and focus on the estimator

where G k (Θ) := { G = ∑ k ′ i =1 1 1+exp( -β 0 i ) δ ( β 1 i ,η i ) : 1 ≤ k ′ ≤ k, ( β 0 i , β 1 i , η i ) ∈ Θ } is the set of all mixing measures with at most k atoms. Since the true number of true experts k ∗ is typically unknown in practice, we will assume that the number k of fitted experts is at least as large, i.e. that k > k ∗ .

Technical challenges. The main challenge in analyzing MoE models with sigmoid gating lies in the convergence behavior of the mixture weights. Specficially, since we fit the ground-truth MoE model (2) with a mixture of k > k ∗ , there must be some true atoms ( β ∗ 1 i , η ∗ i ) fitted by more than one component; we will refer to the corresponding parameters β ∗ 1 i as over-specified parameters . To illustrate, suppose that ( ˆ β n 1 i , ˆ η n i ) → ( β ∗ 11 , η ∗ 1 ) for i ∈ { 1 , 2 } , in probability. Then, to ensure convergence of f ̂ G n to f G ∗ in the L 2 ( µ ) -norm we must have that, for i = 1 , 2 ,

as n →∞ , for µ -almost every x . Note that the above limit can be achieved only if β ∗ 11 = 0 d . As a result, we will consider the two following regimes for the over-specified parameters β ∗ 1 i :

Regime 1. All the over-specified parameters β ∗ 1 i are equal to 0 d ;

Regime 2. At least one among the over-specified parameters β ∗ 1 i is different from 0 d .

It is worth emphasizing that the second regime presents additional technical challenges, as the least squares estimator ̂ G n converges to a mixing measure G ∈ arg min G ∈G k (Θ) \G k ∗ (Θ) ∥ f G -f G ∗ ∥ L 2 ( µ ) that is in general different that the true mixing measure G ∗ , as in the first regime.

Contributions. In this paper, we carry out a convergence analysis of the sigmoid gating MoE under two regimes of the gating parameters. Our main objective is to compare the sample efficiency between the sigmoid gating and the softmax gating. The contributions of our paper are three-fold, and can be summarized as follows:

(C.1) Convergence rate for the regression function. We demonstrate in Theorem 1 that the regression estimation f ̂ G n converges to its true counterpart f G ∗ at the rate of order O P ( n -1 / 2 ) , which is parametric on the sample size n . This regression estimation rate is then utilized for determining the expert estimation rates.

(C.2) Expert estimation rates under the Regime 1. Under the first regime, we first establish a condition called strong identifiability to characterize which types of experts would yield polynomial estimation rates. In particular, we find out that the rates for estimating experts formulated as feedforward networks with popular activation functions such as ReLU and GELU are of polynomial orders. By contrast, those for polynomial experts and input-indepedent experts are slower than any polynomial rates and could be of order O P (1 / log( n ) . Such expert convergence behavior is similar to that when using the softmax gating.

(C.3) Expert estimation rates under the Regime 2. Under the second regime, the regression estimation f ̂ G n converge to a function taking the form of a sigmoid gating MoE which is different from f G ∗ . From our derived weak identifiability condition, it follows that estimation rates for feed-forward expert networks with ReLU or GELU activation and polynomial experts are of orders O P ( n -1 / 2 ) , which are substantially faster than those when using the softmax gating (see also Table 1). Therefore, it follows that the sigmoid gating is more sample efficient than the softmax gating.

Outline. The rest of the paper is organized as follows. In Section 2, we introduce some mild assumptions on the parameters, and then determine estimation rates for the regression function. Next, we establish convergence rates for expert estimation under both Regime 1 and the Regime 2 in Section 3. Then, we conduct some experiments to empirically justify our theory in Section 4. Finally, we provide a discussion on the practical implications, limitations and future directions of our work in Section 5. Full proofs of the results and additional experiments are deferred to the Appendices.

Notation. For any n ∈ N , we denote [ n ] as the set = { 1 , 2 , . . . , n } . Additionally, for any set S , we refer to | S | as its cardinality. Next, for any vectors v := ( v 1 , v 2 , . . . , v d ) ∈ R d and α := ( α 1 , α 2 , . . . , α d ) ∈ N d , we let v α = v α 1 1 v α 2 2 . . . v α d d , | v | := v 1 + v 2 + . . . + v d and α ! := α 1 ! α 2 ! . . . α d ! , while ∥ v ∥ stands for its 2 -norm value. Lastly, for any two positive sequences ( a n ) n ≥ 1 and ( b n ) n ≥ 1 , we write a n = O ( b n ) or a n ≲ b n if a n ≤ Cb n for all n ∈ N , where C > 0 is some universal constant. The notation a n = O P ( b n ) indicates that a n /b n is stochastically bounded.

2 Preliminaries

In this section, we demonstrate the parametric convergence rate for the least squares estimator of the regression function f ̂ G n . We begin by describing the assumption that will be used throughout the paper, which are overall very mild.

(A.1) Topological assumptions. To ensure the convergence of the least squares estimators, we assume the parameter space Θ ⊆ R × R d × R q is compact, and the input space X ⊆ R d is bounded.

(A.2) Lipschitz experts. The expert function h ( · , η ) is assumed to be Lipschitz continuous with respect to η and bounded, µ -almost surely.

(A.3) Distinct experts. The ground-truth expert parameters η ∗ 1 , . . . , η ∗ k ∗ are pair-wise different so that the experts h ( x, η ∗ 1 ) , . . . , h ( x, η ∗ k ∗ ) are also distinct from each other.

Given the above assumptions, we are now ready to present the main result of this section. Recall that we divide our analysis into two following regimes:

Regime 1. All the over-specified parameters β ∗ 1 i are equal to 0 d ;

Regime 2. At least one among the over-specified parameters β ∗ 1 i is different from 0 d .

Let us begin with Regime 1. We show in Theorem 1 that the regression estimator f ̂ G n converges to the true regression function f G ∗ at the parametric rate.

Theorem 1. Under the Regime 1 and with the least squares estimator ̂ G n defined in equation (3) , the regression estimator f ̂ G n admits the following rate of convergence to f G ∗ :

The proof of Theorem 1 is in Appendix A.1. The above result indicates that if we are able to design a loss function among parameters D that satisfies the lower bound ∥ f ̂ G n -f G ∗ ∥ L 2 ( µ ) ≳ D ( ̂ G n , G ∗ ) , then we obtain parameter estimation rates via the bound D ( ̂ G n , G ∗ ) = O P ([log( n ) /n ] 1 2 ) . Consequently, those parameter estimation rates lead to our desired expert estimation rates.

Corollary 1. Under the Regime 2, the regression estimator f ̂ G n admits the following rate of convergence to f G : inf G ∈G k (Θ) ∥ f ̂ G n -f G ∥ L 2 ( µ ) = O P ([log( n ) /n ] 1 2 ) .

Next, under the Regime 2, the true model is misspecified, i.e., the regression estimator f ̂ G n converges to the missepcified regression function f G rather than the true regression function f G ∗ , where G ∈ G k (Θ) := arg min G ∈G k (Θ) \G k ∗ (Θ) ∥ f G -f G ∗ ∥ L 2 ( µ ) . By employing the same arguments as in Theorem 1, we also capture the parametric regression estimation rate O P ([log( n ) /n ] 1 2 ) for ∥ f ̂ G n -f G ∥ L 2 ( µ ) under this regime, which is stated in the following corollary.

3 Convergence Rates for Expert Estimation

In this section we will establish the expert estimation rates under both Regime 1 and the Regime 2; see Section 3.1 and Section 3.2, respectively. Under each regime, we will formulate appropriate

conditions, which we refer to as identifiability conditions , on the expert functions that will guarantee fast estimation rates. Furthermore, we will determine the (slow) estimation rates of some commonly used experts which fail to satisfy such identifiability conditions.

3.1 Regime 1 of Gating Parameters

Recall that under the Regime 1, all the over-specified parameters β ∗ 1 i , i.e., those fitted by at least two parameters, are equal to 0 d . Without loss of generality, we assume that β ∗ 11 , . . . , β ∗ 1 ¯ k are over-specified parameters, where 1 ≤ ¯ k ≤ k ∗ . The remaining parameters β ∗ 11 = . . . = β ∗ 1 ¯ k = 0 d . Meanwhile, β ∗ 1( ¯ k +1) , . . . , β ∗ 1 k ∗ are assumed to be fitted by exactly one estimator.

Following a strategy that has been successfully used for analyzing MoE and mixture models (see, e.g. .

̂ G n -∗

[21] and [26]), in order to derive the expert estimation rates, it is sufficient to propose a V oronoi loss function D among parameters, and then prove the lower bound ∥ f ̂ G n -f G ∗ ∥ L 2 ( µ ) ≳ D ( ̂ G n , G ∗ ) Given the parametric regression estimation rate in Theorem 1, we then obtain that D ( ̂ G n , G ∗ ) = O P ([log( n ) /n ] 1 2 ) . Note that a key step in deriving the previous lower bound is to decompose the difference f ( x ) f G ( x ) into a combination of linearly independent terms.

Towards that goal, we will Taylor expand to the product of mixture weight and expert functions F ( x, β 1 , β 0 , η ) := σ ( x, β 1 , β 0 ) h ( x, η ) , where we let σ ( x, β 1 , β 0 ) := 1 1+exp( -β ⊤ 1 x -β 0 ) . To obtain the desired decomposition of f ̂ G n ( x ) -f G ∗ ( x ) , we provide below in Definition 1 a condition, which we refer to as a strong identifiability condition, that guarantees that the derivatives of the product F ( x, β 1 , β 0 , η ) w.r.t its parameters are linearly independent.

Definition 1 (Strong identifiability) . An expert function h ( x, η ) is called strongly identifiable if it is twice differentiable w.r.t its parameter η for µ -almost all x and, for any positive integer ℓ and any pair-wise distinct choices of parameters { ( β 0 i , β 1 i , η i ) } ℓ i =1 , the functions in the classes

are linearly independent, for µ -almost all x .

Example. Consider experts formulated as two-layer neural networks, i.e., h ( x, ( a, b )) = φ ( a ⊤ x + b ) , where φ is an activation function and ( a, b ) ∈ R d × R . If a = 0 d , then the expert h ( · , ( a, b )) is not strongly identifiable for any choices of the activation function φ . On the other hand, if a = 0 d and φ is one among popular activation functions such as ReLU and GELU , then h ( · , ( a, b )) is a strongly identifiable expert. By contrast, if φ is a polynomial, e.g. φ ( z ) = z p , then the expert h ( · , ( a, b )) does not meet the strong identifiability condition.

̸

Intuitively, the strong identifiability condition helps eliminate potential interactions among parameters expressed in the language of PDE (see equations (8) and (11) where gating parameters β 1 interact with expert parameters a ). Such interactions are demonstrated to result in significantly slow expert estimation rates presented in Theorem 3 and 4). From the technical perspective, a key step in our proof techniques rely on the decomposition of the discrepancy f ̂ G n ( x ) -f G ∗ ( x ) into a combination of linearly independent terms. This can be done by applying Taylor expansions to the function F ( x, β 1 , β 0 , η ) = σ ( β ⊤ 1 x + β 0 ) h ( x, η ) defined as the product of the sigmoid gating and the expert function h . Thus, the condition is to ensure that terms in the decomposition are linearly independent.

Next, following the strategy first introduced by Manole et al. [21] and then adopted in several convergence analyses of MoE [22, 23], we proceed to construct a loss function among parameters based on the notion of Voronoi cells.

Voronoi loss. Given an arbitrary mixing measure G with k ′ ≤ k atoms, we distribute its atoms across the Voronoi cells {A j ≡ A j ( G ) , j ∈ [ k ∗ ] } generated by the atoms of G ∗ , where

̸

with ω i := ( β 1 i , η i ) and ω ∗ j := ( β ∗ 1 j , η ∗ j ) . In particular, the cardinality of the Voronoi cell A j corresponding to the least squares estimator ̂ G n in (3) is the number of fitted components assigned (and, likely, converging) to the true atoms ω ∗ j . Then, we define the Voronoi loss function where we let ∆ β 1 ij := β 1 i -β ∗ 1 j , ∆ β 0 ij := β 0 i -β ∗ 0 j , and ∆ η ij := η i -η ∗ j . Above, if the Voronoi cell A j is empty, then the summation term becomes zero by convention. Additionally, since D 1 ( G,G ∗ ) = 0 if and only if G ≡ G ∗ , it follows that when D 1 ( G,G ∗ ) is sufficiently small, the differences ∆ β 0 ij , ∆ β 1 ij and ∆ η ij are also small. This observation suggests that, though not symmetric in its arguments and therefore not a metric, D 1 ( G,G ∗ ) is nonetheless a suitable loss function for capturing the discrepancy between the least squares estimator ̂ G n and the true mixing measures G ∗ . Furthermore, it is worth noting that the Voronoi loss function D 1 can be efficiently evaluated, as its computational complexity is of order O ( k × k ∗ ) .

In our next result, whose proof can be found in Appendix A.2, we show that the Voronoi loss between the least squares estimator and the true mixing measure is upper bounded, up to universal multiplicative constants, by the estimation error for the regression function.

Theorem 2. Let x ↦→ h ( x, η ) be a strongly identifiable expert function. Then the lower bound

holds true for any G ∈ G k (Θ) . As a consequence, this result together with the regression estimation rate in Theorem 1 indicate that D 1 ( ̂ G n , G ∗ ) = O P ([log( n ) /n ] 1 2 ) . The following remarks regarding the results of Theorem 2 are in order.

(i) It follows from the formulation of the loss function D 1 that the estimation rates for the overspecified parameters β ∗ 1 j , η ∗ 1 j , where j ∈ [ ¯ k ] , are all of order O P ([log( n ) /n ] 1 4 ) . Note that as the expert h ( · , η ) is twice differentiable over a bounded domain, it is also a Lipschitz function. Thus, letting ̂ G n := ∑ ̂ k n i =1 1 1+exp( -̂ β 0 i ) δ ( ̂ β n 1 i , ̂ η n i ) , we obtain that for any i ∈ A j ( ̂ G n ) , where L 1 ≥ 0 is a Lipschitz constant. The above bound indicates that if the strongly identifiable expert h ( · , η ∗ j ) is over-specified, i.e. fitted by at least two experts, then its estimation rate is of order O P ([log( n ) /n ] 1 4 ) .

(ii) Secondly, for exact-specified parameters β ∗ 1 j , η ∗ j , where ¯ k +1 ≤ j ≤ k ∗ , the rates for estimating them are faster than those of their over-specified counterparts in Remark (i), standing at order O P ([log( n ) /n ] 1 2 ) . By arguing as in equation (7), we deduce that the expert h ( · , η ∗ j ) also enjoys the faster estimation rate of order O P ([log( n ) /n ] 1 2 ) , which is parametric on the sample size n .

(iii) Putting the above two remarks together, we observe that when the expert functions are formulated as feed-forward networks with a widely used activation, namely ReLU and GELU , their estimation rates under the sigmoid gating MoE matches exactly those under the softmax gating MoE: see Theorem 3.2 in [26].

Non-strongly identifiable experts. The strong identifiability condition of Definition 1 is far from a technical requirement: it is in fact a crucial assumption. To illustrate its importance, we investigate the estimation rates of ridge experts of the form { h ( x, ( a ∗ j , b ∗ j )) = φ (( a ∗ j ) ⊤ x + b ∗ j ) , j ∈ [ k ∗ ] } , where φ is a scalar function, that fail to satisfy the strong identifiability condition. For ridge experts, the strong identifiability will not be met in two cases: when at least one among over-specified parameters a ∗ j equals 0 d , for any arbitrary activation function φ ( Scenario I ) and when φ is a polynomial of the form φ ( z ) = z p , for p ∈ N ( Scenario II ).

Scenario I. Without loss of generality, we may assume that a ∗ 1 = 0 d , which means that the first expert φ (( a ∗ 1 ) ⊤ x + b ∗ 1 ) becomes independent of the input x . This gives rise to an interaction among the gating parameter β 1 and the expert parameter a via the partial differential equation (PDE)

where we recall that F ( x ; β 1 , β 0 , a, b ) := σ ( x ; β 1 , β 0 ) φ ( a ⊤ x + b ) , and C b ∗ 1 ,β ∗ 01 is some constant depending on b ∗ 1 and β ∗ 01 . The above PDE can be verified by taking the derivatives of F ( x ; β 1 , β 0 , a, b ) w.r.t β 1 and a and using the fact that β ∗ 11 = a ∗ 1 = 0 d . Notably, the PDE (8) accounts for the violation of the strong identifiability of the expert φ (( a ∗ 1 ) ⊤ x + b ∗ 1 ) under the Scenario I. To this end, we propose the following Voronoi loss to analyze the effects of such parameter interaction on the expert estimation rates in Theorem 3:

The following result establishes a minimax lower bound on the estimation error of G ∗ in the D 2 ,r loss.

Theorem 3. Suppose that the experts take the form φ ( a ⊤ x + b ) . Then, under the Scenario I, we have

for any r ≥ 1 , where E f G indicates that the expectation taken w.r.t the product measure with f n G , and the infimum is over all estimators taking values in G k (Θ) .

See Appendix A.3 for the proof details. We highlight some important implications of Theorem 3.

(i) The estimation rates for the parameters β ∗ 1 j , a ∗ j and b ∗ j are slower than O P ( n -1 / 2 r ) , for any r ≥ 1 . This means that they are slower than any polynomial rates, and could be of order O P (1 / log( n )) .

(ii) Using the same reasoning described after equation (7), we have

where L 2 > 0 is a Lipschitz constant. As a consequence, the rates for estimating experts φ (( a ∗ j ) ⊤ x + b ∗ j ) are no better than those for estimating the parameters a ∗ j and b ∗ j , and could also be as slow as O P (1 / log( n )) . This result suggests that all the expert parameters a ∗ 1 , . . . , a ∗ k ∗ should be different from 0 d to hope for fast convergence rates. In other words, every expert of the form φ ( a ⊤ x + b ) in the MoE model should vary with the input value.

Scenario II. Now, we consider the second scenario when φ is a polynomial of the form φ ( z ) = z p , for p ∈ N . For simplicity, we will only focus on the setting of p = 1 , i.e., h ( x, η ∗ j ) = ( a ∗ j ) ⊤ x + b ∗ j ; the case of p > 1 can be argued in a very similar fashion. The structure of the polynomial experts leads to a non-linear interactions among the parameters, expressed through the PDE

where F ( x ; β 1 , β 0 , a, b ) := σ ( x ; β 1 , β 0 )( a ⊤ x + b ) . This interaction is the reason why the polynomial experts are not strongly identifiable. Ultimately, this dependence yields slow rates of convergence for expert estimation, just like in Scenario I, as revealed by the next result, which deploys the Voronoi loss D 2 ,r of equation (9).

Theorem 4. Suppose that the experts take the form ( a ⊤ x + b ) p , for some p ∈ N . Then, under the Scenario II and for any r ≥ 1 , where E f G indicates the expectation taken w.r.t the product measure with f n G .

The proof of Theorem 4 is in Appendix A.4. To sum up, when either the experts are independent of the input or they take a polynomial form, their estimation rates could be as slow as O P ([log( n ) /n ] 1 2 ) .

3.2 Regime 2 of Gating Parameters

Recall that under the Regime 2, at least one among the over-specified parameters β ∗ 1 i is different from 0 d . As discussed in Section 2, the least squares regression estimator f ̂ G n in this case converges to a regression function f G , where G ∈ G k (Θ) := arg min G ∈G k (Θ) \G k ∗ (Θ) ∥ f G -f G ∗ ∥ L 2 ( µ ) . That is, the estimators of the parameters specifying f ̂ G n converge to the parameters of f G . WLOG, we may assume that G := ∑ k i =1 1 1+exp( -¯ β 0 i ) δ ( ¯ β 1 i , ¯ η i ) .

Similarly to Regime 1, we also provide in Definition 2 a weak identifiability condition to characterize those expert functions that will have fast estimation rates under the Regime 2. Weakly identifiable experts are required to satisfy only a subset of the conditions imposed to strongly identifiable experts.

Definition 2 (Weak identifiability) . An expert function x ↦→ h ( x, η ) is called weakly identifiable if it is twice differentiable w.r.t its parameter η for µ -almost all x and, for any positive integer ℓ and any pair-wise distinct choices of parameters { ( β 0 i , β 1 i , η i ) } ℓ i =1 , the functions in the class

are linearly independent, for µ -almost all x .

̸

Example. Let us take an expert network h ( x, ( a, b )) = φ ( a ⊤ x + b ) as an example. It can be checked that if a = 0 d and the activation function φ is a ReLU or GELU or a polynomial, then the expert h ( x, ( a, b )) is weakly identifiable. Conversely, if a = 0 d , i.e. the expert does not depend on the input, then the weak identifiability condition is not met, regardless of the choice of the activation.

We now describe the convergence rates for expert estimation under Regime 2. Since the analysis of input-independent experts can be done similarly to Theorem 3, we will focus only on weakly identifiable experts. As it turned out, the appropriate V oronoi loss function is given by

Theorem 5. Let h ( x, η ) be a weakly identifiable expert. Then, for any mixing measure G ∈ G k (Θ) ,

As a consequence, we obtain that inf G ∈G k (Θ) D 3 ( ̂ G n , G ) = O P ([log( n ) /n ] 1 2 ) .

The proof of Theorem 5 is in Appendix A.5. As a direct consequence of the bound and the definition of D 3 , the convergence rates of the estimators ˆ β n 0 i , ˆ β n 1 i and ˆ η n i are of the same of parametric order O P ([log( n ) /n ] 1 2 ) . Furthermore, by equation (7), we also establish that the convergence rate of the expert estimator h ( x, ˆ η n i ) is also of order O P ([log( n ) /n ] 1 2 ) . Comparing these estimation rates with those established by [26] assuming instead softmax gating, there are two main observations:

(i) When the expert is a neural network with ReLU or GELU activation, the above parametric expert estimation rate obtained using the sigmoid gate is faster than the rate guranteed by the softmax gate, which could be of order O P ([log( n ) /n ] 1 4 ) (see [Theorem 3.2, [26]]).

(ii) When the activation is a polynomial, the gap between expert estimation rates when using the sigmoid gate versus when using the softmax gate becomes even more dramatic: O P ([log( n ) /n ] 1 2 ) compared to O P (1 / log( n )) (see Theorem 4.6 in [26]).

The above remarks highlight the considerable benefits of deploying sigmoid gating over softmax gating in MoE models: provably faster estimation rates not only for expert networks with popular activations like ReLU and GELU , but also for those with polynomial activation.

4 Numerical Experiments

In this section, we perform some numerical experiments to empirically demonstrate our claim that the sigmoid gating is more sample efficient than the softmax gating.

From Table 1, it can be seen that the sigmoid gating shares the same expert estimation rates as the softmax gating under the Regime 1. However, the former gating outperforms the latter under the Regime 2, particularly for ReLU experts and polynomial experts. Therefore, we will consider those experts under the Regime 2 in our subsequent experiments.

Data generation. In particular, we generate the data by first sampling X i ∼ Uniform([ -1 , 1] d ) for i = 1 , . . . , n . Then, we generate Y i according to the following model:

where the regression function g G ∗ ( · ) take the form of a softmax gating MoE:

The input data dimension is d = 32 . We employ k ∗ = 8 experts of the form ϕ ( a ⊤ x + b ) , where ϕ is either the identity function or the ReLU function. The variance of Gaussian noise ϵ i is ν = 0 . 01 .

Experimental setup. We summarize the choices of the ground-truth parameters β ∗ 0 i , β ∗ 1 i , a ∗ i and b ∗ i for 1 ≤ i ≤ 8 in Table 2, which satisfies the condition of the Regime 2.

Training procedure. For each sample size n , spanning from 10 3 to 10 5 , we perform 20 experiments. In every experiment, we employ k = k ∗ +1 = 9 fitted experts, and the parameters initialization for the gating's and experts' parameters are adjusted to be near the true parameters, minimizing potential instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across 10 epochs, employing a learning rate of η = 0 . 1 to fit a model to the synthetic data. For each experiment, we calculate the Voronoi losses for every model and report the mean values for each sample size in Figure 1. Error bars representing two standard deviations are also shown.

Results. In Figure 1, when employing the ReLU experts, that is, ϕ is the ReLU function, the Voronoi loss for the sigmoid gating approaches zero at the rate of order O ( n -0 . 51 ) , which nearly matches our theoretical results in Theorem 5. Meanwhile, the loss for the softmax gating converges to zero at the slower rate O ( n -0 . 24 ) . On the other hand, when using the linear experts, that is, ϕ is the identity function, the vanishing rate of the Voronoi loss associated with the sigmoid gating is O ( n -0 . 40 ) , while that for the softmax gating is significantly slower, standing at O ( n -0 . 11 ) . These observations empirically shows that the sigmoid gating is more sample efficient than the softmax gating.

5 Discussion

In this paper, we carry out a convergence analysis of least squares expert estimation under MoE models with the sigmoid gating, which was found empirically to be robust to the representation collapse issue and to have a favorable performance in the MoE applications. We demonstrate that under both the gating regimes, the sigmoid gating requires a smaller sample size than the softmax gating to reach the same expert estimation error: that is, the sigmoid gating is more sample efficient than its softmax counterpart. Furthermore, we also verify that experts formulated as feed-forward networks with popular activation functions such as ReLU and GELU are more compatible with the sigmoid gating than other types of expert functions.

3

10

3

10

4

10

5

sample size

n

10

-

3

10

-

2

Loss

Rate =

O

(

n

-

0

.

51

)

Rate =

O

(

n

-

0

.

24

)

Sigmoid MoE

D

3

(

̂

G

n

, G

∗

)

0

.

1

n

-

0

.

51

D

3

(

̂

G

n

, G

∗

)

0

.

1

n

-

0

.

24

D

3

(

̂

G

n

, G

∗

)

0

.

1

n

-

0

.

24

(a)

ReLU

Expert

10

3

10

4

10

5

sample size

n

10

-

4

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

10

Loss

Rate =

O

(

n

-

0

.

40

)

Rate =

O

(

n

-

0

.

11

)

Sigmoid MoE

D

3

(

̂

G

n

, G

∗

)

0

.

6

n

-

0

.

40

D

3

(

̂

G

n

, G

∗

)

12

.

9

n

-

0

.

11

D

3

(

̂

G

n

, G

∗

)

12

.

9

n

-

0

.

11

(b) Linear Expert

Practical implications. Our theoretical findings support the following conclusions, which have considerable practical relevance.

(I.1) The sigmoid gate is more sample efficient than the softmax gate for expert estimation. It can be seen from Table 1 that while the expert estimation rates obtained when using the sigmoid gate match those attained when using the softmax gate under the Regime 1, the former are totally faster than the latter under the Regime 2. Notably, Regime 2 is closer to practice since the gating values under this regime hinge upon the input while those under the Regime 1 are input-independent. As a consequence, we can claim that the sigmoid gate is more sample efficient than the softmax gate from the perspective of the expert estimation problem.

(I.2) The sigmoid gate is compatible with a broader class of experts than the softmax gate. It follows from the expert characterization for Regime 1 (resp. Regime 2) in Definition 1 (resp. Definition 2) that formulating expert functions as feed-forward networks [36] with commonly used activation functions such as ReLU and GELU or even a polynomial activation will lead to faster expert estimation rates, and thus, require a smaller sample size to reach the same tolerance of estimating experts compared to the softmax gate. Thus, our theories indicate that the sigmoid gating is compatible with a broader class of experts than the softmax gating. This implication is particularly useful when people employ a mixture of fine-grained (shallow) expert networks [13].

Limitations and future directions. There are two main limitations of our current analysis:

We assume implicitly that the ground-truth parameters are independent of the sample size. Therefore, the expert estimation rates established in the paper are point-wise rather than the desirable uniform rates. A potential approach to cope with this problem is using the techniques for deriving the uniform parameter estimation rates in mixture models [6] and in MoE models [37]. However, those techniques are only valid for input-independent gating MoE, and we believe that further technical tools should be developed to adapt such framework to the sigmoid gating MoE. Thus, we leave it for future development.

The assumption that the true regression function belongs to the parametric class of MoE models under sigmoid gating is, of course, quite restrictive, and likely to be violated in real-world settings. This issue can be alleviated by assuming the data are sampled from a regression framework with an arbitrary regression function g , which is not necessarily formulated as a mixture of experts. Under that setting, the least squares estimator ̂ G n converges to ˜ G ∈ arg min G ∈G k (Θ) ∥ f G -g ∥ L 2 ( µ ) . Nevertheless, it requires the comprehensive knowledge of the universal approximation power of the sigmoid function, which has been slightly explored in [1], to determine the expert estimation rate. Therefore, we leave this potential direction for future work.

References

A. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory , 39(3):930-945, 1993.

Z. Chi, L. Dong, S. Huang, D. Dai, S. Ma, B. Patra, S. Singhal, P. Bajaj, X. Song, X.-L. Mao, H. Huang, and F. Wei. On the representation collapse of sparse mixture of experts. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Y. Chow, A. Tulepbergenov, O. Nachum, D. Gupta, M. Ryu, M. Ghavamzadeh, and C. Boutilier. A Mixture-of-Expert Approach to RL-based Dialogue Management. In The Eleventh International Conference on Learning Representations , 2023.

R. Csordás, K. Irie, and J. Schmidhuber. Approximating two-layer feedforward networks for efficient transformers. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 674-692, Singapore, Dec. 2023. Association for Computational Linguistics.

D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.04088 , 2024.

D. Do, H. Nguyen, K. Nguyen, and N. Ho. Minimax optimal rate for parameter estimation in multivariate deviated models. In Advances in Neural Information Processing Systems , volume 36, pages 30096-30133. Curran Associates, Inc., 2023.

N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In ICML , 2022.

W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research , 23:1-39, 2022.

A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented Transformer for Speech Recognition. In Proc. Interspeech 2020 , pages 5036-5040, 2020.

S. Gupta, S. Mukherjee, K. Subudhi, E. Gonzalez, D. Jose, A. Awadallah, and J. Gao. Sparsely activated mixture-of-experts are robust multi-task learners. arXiv preprint arxiv 2204.0768 , 2022.

X. Han, H. Nguyen, C. Harris, N. Ho, and S. Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. In Advances in Neural Information Processing Systems , 2024.

H. Hazimeh, Z. Zhao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, R. Mazumder, L. Hong, and E. Chi. DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 29335-29347. Curran Associates, Inc., 2021.

X. O. He. Mixture of a million experts. arXiv preprint arXiv:2407.04153 , 2024.

N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence rates for Gaussian mixtures of experts. Journal of Machine Learning Research , 23(323):1-81, 2022.

R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation , 3, 1991.

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts. arxiv preprint arxiv 2401.04088 , 2024.

M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation , 6:181-214, 1994.

J. Lee-Thorp and J. Ainslie. Sparse mixers: Combining moe and mixing to build a more efficient BERT. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 58-75, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.

D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations , 2021.

H. Liang, Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, and Z. Wang. M 3 ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with ModelAccelerator Co-design. In NeurIPS , 2022.

T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14979-15006. PMLR, 17-23 Jul 2022.

H. Nguyen, P. Akbarian, and N. Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML , 2024.

H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the ICML , 2024.

H. Nguyen, P. Akbarian, T. Pham, T. Nguyen, S. Zhang, and N. Ho. Statistical advantages of perturbing cosine router in mixture of experts. arXiv preprint arXiv:2405.14131 , 2024.

H. Nguyen, P. Akbarian, F. Yan, and N. Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations , 2024.

H. Nguyen, N. Ho, and A. Rinaldo. On least square estimation in softmax gating mixture of experts. In Proceedings of the ICML , 2024.

H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems , 2023.

J. S. Obando Ceron, G. Sokar, T. Willi, C. Lyle, J. Farebrother, J. N. Foerster, G. K. Dziugaite, D. Precup, and P. S. Castro. Mixtures of experts unlock parameter scaling for deep RL. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 38520-38540. PMLR, 21-27 Jul 2024.

F. Peng, R. A. Jacobs, and M. A. Tanner. Bayesian Inference in Mixtures-of-Experts and Hierarchical Mixtures-of-Experts Models With an Application to Speech Recognition. Journal of the American Statistical Association , 91(435):953-960, 1996.

Q. Pham, G. Do, H. Nguyen, T. Nguyen, C. Liu, M. Sartipi, B. T. Nguyen, S. Ramasamy, X. Li, S. Hoi, and N. Ho. Competesmoe - effective training of sparse mixture of experts via competition. arXiv preprint arXiv:2402.02526 , 2024.

J. Puigcerver, C. Riquelme, B. Mustafa, and N. Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations , 2024.

C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pint, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems , volume 34, pages 8583-8595. Curran Associates, Inc., 2021.

C. Ruiz, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In NeurIPS , 2021.

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In In International Conference on Learning Representations , 2017.

S. van de Geer. Empirical processes in M-estimation . Cambridge University Press, 2000.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

F. Yan, H. Nguyen, D. Le, P. Akbarian, and N. Ho. Understanding expert structures on minimax parameter estimation in contaminated mixture of experts. arXiv preprint arXiv:2410.12258 , 2024.

Z. You, S. Feng, D. Su, and D. Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech , 2021.

B. Yu. Assouad, Fano, and Le Cam. Festschrift for Lucien Le Cam , pages 423-435, 1997.

Y. Zhou, N. Du, Y. Huang, D. Peng, C. Lan, D. Huang, S. Shakeri, D. So, A. Dai, Y. Lu, Z. Chen, Q. Le, C. Cui, J. Laudon, and J. Dean. Brainformers: Trading simplicity for efficiency. In International Conference on Machine Learning , pages 42531-42542. PMLR, 2023.

Supplement to 'Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts'

In this supplementary material, we present proofs for the main results of the paper in Appendix A, while we study the identifiability of the sigmoid gating mixture of experts (MoE) in Appendix B. Lastly, we provide additional numerical experiments to verify our theory in Appendix C.

A Proof of Main Results

In this appendix, we provide proofs for the theoretical results introduced in Section 2 and Section 3 of the paper.

A.1 Proof of Theorem 1

To streamline the arguments for this proof, we need to define some necessary notations. First of all, we let F k (Θ) := { f G ( x ) : G ∈ G k (Θ) } stand for the set of all regression functions in G k (Θ) . Next, for each δ > 0 , we define the L 2 ( µ ) -ball centered around the regression function f G ∗ ( x ) and intersected with the set F k (Θ) as

Furthermore, the size of the above ball is captured by the following integral as suggested in [35].

in which H B ( t, F k (Θ , t ) , ∥ · ∥ L 2 ( µ ) ) denotes the bracketing entropy [35] of F k (Θ , t ) under the L 2 ( µ ) -norm, and t ∨ δ := max { t, δ } . By arguing in a similar fashion to Theorem 7.4 and Theorem 9.2 in [35] with adapted notations to this work, we achieve the following lemma:

Lemma 1. Let Ψ( δ ) ≥ J B ( δ, F k (Θ , δ )) such that Ψ( δ ) /δ 2 is a non-increasing function of δ . Then, for some universal constant c and for some sequence ( δ n ) that satisfies √ nδ 2 n ≥ c Ψ( δ n ) , the following holds for any δ ≥ δ n :

General Picture. If we are able to demonstrate that the following bound holds for any ε ∈ (0 , 1 / 2] :

then it follows that

By choosing Ψ( δ ) = δ · [log(1 /δ )] 1 / 2 , then Ψ( δ ) /δ 2 is a non-increasing function of δ . Moreover, equation (17) suggests that Ψ( δ ) ≥ J B ( δ, F k (Θ , δ )) . Additionally, we set δ n = √ log( n ) /n , and then get that √ nδ 2 n ≥ c Ψ( δ n ) for some universal constant c . According to Lemma 1, we achieve the desired conclusion of the theorem. As a consequence, it suffices to establish the bound (16).

Proof of inequality (16) . Note that as the expert functions are bounded, we have that | f G ( x ) | ≤ M for almost every x , where M > 0 is a bounded constant.

Next, let τ ≤ ε and { ζ 1 , . . . , ζ N } be the τ -cover under the L 2 ( µ ) -norm of the set F k (Θ) , where N := N ( τ, F k (Θ) , ∥ · ∥ L ∞ ) stands for the η -covering number of the metric space ( F k (Θ) , ∥ · ∥ L ∞ ) . Now, we consider brackets of the form [ L i ( x ) , U i ( x )] , where we define

for all i ∈ [ N ] . Then, it can be verified that F k (Θ) ⊂ ∪ N i =1 [ L i ( x ) , U i ( x )] and U i ( x ) -L i ( x ) ≤ min { 2 τ, M } . Thus, we obtain that which indicates that ∥ U i -L i ∥ L 2 ( µ ) ≤ 2 τ . From the definition of the bracketing entropy, we arrive at

Consequently, it is sufficient to provide an upper bound for the covering number N . For that purpose, let us denote ∆ := { ( β 0 , β 1 ) ∈ R × R d : ( β 0 , β 1 , η ) ∈ Θ } and Ω := { η ∈ R q : ( β 0 , β 1 , η ) ∈ Θ } . Recall that Θ is a compact set, therefore, ∆ and Ω are also compact. Then, there exist τ -covers ∆ τ and Ω τ for ∆ and Ω , respectively. Furthermore, it can be validated that

Given a mixing measure G = ∑ k i =1 exp( β 0 i ) δ ( β 1 i ,η i ) ∈ G k (Θ) , we take into account two other mixing measures defined as:

Above, ˇ η i ∈ Ω τ such that ˇ η i is the closest to η i in that set, while ( ˇ β 0 i , ˇ β 1 i ) ∈ ∆ τ is the closest to ( β 0 i , β 1 i ) in that set. It follows from the above formulations that

Here, the second inequality occurs as the sigmoid weight is not larger than one and the third inequality follows from the fact that the expert function h ( x, · ) is a Lipschitz function with some Lipschitz constant L 1 > 0 .

Subsequently, we have

Above, the second inequality is due to the fact that the expert function is bounded, that is, | h ( x, ˇ η i ) | ≤ M ′ . The third one happens as the sigmoid function is a Lipschitz function with some Lipschitz constant L 2 > 0 , while the fourth inequality occurs since the input space is bounded.

By the triangle inequality, we obtain

From the definition of the covering number, it yields that

Putting equations (18) and (19) together, we get that

Let τ = ε/ 2 , then we reach the desired bound

Hence, the proof is totally completed.

A.2 Proof of Theorem 2

In this proof, we aim to establish the following inequality:

For that purpose, we divide the proof of the above inequality into local and global parts as below. Local part: In this part, we focus only on demonstrating the following local inequality:

Assume by contrary that the above claim does not hold true, then there exists a sequence of mixing measures G n = ∑ k ∗ i =1 δ ( β n 0 i ,β n 1 i ,η n i ) in G k (Θ) such that D 1 n := D 1 ( G n , G ∗ ) → 0 and

as n →∞ . Let us recall that

where ∆ β n 0 ij := β n 0 i -β ∗ 0 j , ∆ β n 1 ij := β n 1 i -β ∗ 1 j and ∆ η n ij := η n i -η ∗ j . Since D 1 n → 0 as n →∞ , we deduce that

For ¯ k +1 ≤ j ≤ k ∗ : ( β n 0 i , β n 1 i , η n i ) → ( β ∗ 0 j , β ∗ 1 j , η ∗ j ) for any i ∈ A j .

Step 1 - Taylor expansion: In this step, we decompose the term f G n ( x ) -f G ∗ ( x ) using Taylor expansion. Firstly, let us denote

Let us denote σ ( x, β 1 , β 0 ) := 1 1+exp( -β ⊤ 1 x -β 0 ) . Then, by means of the second-order Taylor expansion, we can decompose the term A n ( x ) defined above as

where R 1 ( x ) is a Taylor remainder such that R 1 ( x ) / D 1 n → 0 as n →∞ , and we define

for any j ∈ [ ¯ k ] , i ∈ A j , γ 1 ∈ N d and γ 2 ∈ N q such that 1 ≤ | γ 1 | + | γ 2 | ≤ 2 . Next, let us denote

for any j ∈ [ ¯ k ] , then the term B n ( x ) can be represented as

Finally, recall that |A j | = 1 for any ¯ k +1 ≤ j ≤ k ∗ , we can decompose the term C n ( x ) by applying the first-order Taylor expansion as follows:

where R 2 ( x ) is a Taylor remainder such that R 2 ( x ) / D 1 n → 0 as n →∞ , and we define

for any ¯ k +1 ≤ j ≤ k ∗ , α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that | α 1 | + α 2 + | α 3 | = 1 .

Combine the decomposition of the terms A n ( x ) , B n ( x ) and C n ( x ) in equations (24), (25) and (26), we rewrite the difference f G n ( x ) -f G ∗ ( x ) as

Step 2 - Non-vanishing coefficients: In this step, we show that at least one among the ratios T n j,i,γ 1 ,γ 2 / D 1 n , W n j / D 1 n and S n j,α 1 ,α 2 ,α 3 / D 1 n does not go to zero as n tends to infinity. Indeed, assume by contrary that all of them converge to zero, i.e.

1 D 1 n ∑ ¯ k j =1 ∣ ∣ ∣ ∑ i ∈A j 1 1+exp( -β n 0 i ) -1 1+exp( -β ∗ 0 j ) ∣ ∣ ∣ = 1 D 1 n · ∑ ¯ k j =1 | W n j |→ 0 ; · 1 D 1 n ∑ ¯ k j =1 ∑ i ∈A j ∥ ∆ β n 1 ij ∥ 2 = 1 D 1 n · ∑ ¯ k j =1 ∑ i ∈A j ∑ d u =1 | T n j,i, 2 e d,u , 0 q |→ 0 , where e d,u is a vector in R d whose u -th entry is one while other entries are zero; · 1 D 1 n ∑ ¯ k j =1 ∑ i ∈A j ∥ ∆ η n ij ∥ 2 = 1 D 1 n · ∑ ¯ k j =1 ∑ i ∈A j ∑ q v =1 | T n j,i, 0 d , 2 e q,v |→ 0 ; · 1 D 1 n ∑ k ∗ j =¯ k +1 ∑ i ∈A j ∥ ∆ β n 1 ij ∥ 1 = 1 D 1 n · ∑ k ∗ j =¯ k +1 ∑ d u =1 | S n j,e d,u , 0 , 0 q |→ 0 ; · 1 D 1 n ∑ k ∗ j =¯ k +1 ∑ i ∈A j | ∆ β n 0 ij | = 1 D 1 n · ∑ k ∗ j =¯ k +1 | S n j, 0 d , 1 , 0 q |→ 0 ; · 1 D 1 n ∑ k ∗ j =¯ k +1 ∑ i ∈A j ∥ ∆ η n ij ∥ 1 = 1 D 1 n · ∑ k ∗ j =¯ k +1 ∑ q v =1 | S n j, 0 d , 0 ,e q,v |→ 0 .

as n →∞ . Then, it follows that

By taking the summation of the above limits, we obtain that

Due to the topological equivalence between the 1-norm and the 2-norm, we deduce that

which is a contradiction. As a consequence, at least one among the ratios T n j,i,γ 1 ,γ 2 / D 1 n , W n j / D 1 n and S n j,α 1 ,α 2 ,α 3 / D 1 n must not approach zero as n →∞ .

Step 3 - Application of Fatou's lemma: In this step, we use the Fatou's lemma to argue against the result in Step 2, and then, achieve the desired inequality in equation (21).

In particular, let us denote by m n the maximum of the absolute values of T n j,i,γ 1 ,γ 2 / D 1 n , W n j / D 1 n and S n j,α 1 ,α 2 ,α 3 / D 1 n . Since at least one among those ratios must not approach zero as n →∞ , we get that 1 /m n ̸→∞ as n →∞ .

Recall from the hypothesis in equation (22) that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / D 1 n → 0 as n → ∞ , which indicates that ∥ f G n -f G ∗ ∥ L 1 ( µ ) / D 1 n → 0 due to the equivalence between L 1 ( µ ) -norm and L 2 ( µ ) -norm. By means of the Fatou's lemma, we have

This result implies that [ f G n ( x ) -f G ∗ ( x )] / [ m n D 1 n ] → 0 for almost every x .

Let us denote

with a note that at least one among the limits t j,i,γ 1 ,γ 2 , w j and s j,α 1 ,α 2 ,α 3 is non-zero. Then, from the decomposition in equation (27), we deduce that

for almost every x . Note that the expert function h ( · , η ) is strongly identifiable, then the above equation implies that t j,γ 1 ,γ 2 = w j = s j,α 1 ,α 2 ,α 3 = 0 , for any j ∈ [ k ∗ ] , α 1 , γ 1 ∈ N d , α 2 ∈ N and γ 2 , α 3 ∈ N q such that 1 ≤ | γ 1 | + | γ 2 | ≤ 2 and | α 1 | + α 2 + | α 3 | = 1 . This contradicts the fact that at least one among the limits s i,α 1 ,α 2 ,α 3 is different from zero.

Hence, we obtain the local inequality in equation (21). Consequently, there exists some ε ′ > 0 such that

Global part: Given the above result, it suffices to demonstrate that

Assume by contrary that the inequality (28) does not hold true, then we can find a sequence of mixing measures G ′ n ∈ G k (Θ) such that D 1 ( G ′ n , G ∗ ) > ε ′ and

which indicates that ∥ f G ′ n -f G ∗ ∥ L 2 ( µ ) → 0 as n →∞ . Recall that Θ is a compact set, therefore, we can replace the sequence G ′ n by one of its subsequences that converges to a mixing measure G ′ ∈ G k (Ω) . Since D 1 ( G ′ n , G ∗ ) > ε ′ , we deduce that D 1 ( G ′ , G ∗ ) > ε ′ .

Next, by invoking the Fatou's lemma, we have that

Thus, we get that f G ′ ( x ) = f G ∗ ( x ) for almost every x . From Proposition 1, we deduce that G ′ ≡ G ∗ . Consequently, it follows that D 1 ( G ′ , G ∗ ) = 0 , contradicting the fact that D 1 ( G ′ , G ∗ ) > ε ′ > 0 .

Hence, the proof is completed.

A.3 Proof of Theorem 3

Lemma 2. If the following holds for any r ≥ 1 :

then we obtain that

Proof of Lemma 2. Indeed, from the Gaussian assumption on the noise variables ϵ i , we obtain that Y i | X i ∼ N ( f G ∗ ( X i ) , σ 2 ) for all i ∈ [ n ] . Next, the assumption in equation (29) indicates for sufficiently small ε > 0 and a fixed constant C 1 > 0 which we will choose later, we can find a mixing measure G ′ ∗ ∈ G k (Θ) such that D 2 ,r ( G ′ ∗ , G ∗ ) = 2 ε and ∥ f G ′ ∗ -f G ∗ ∥ L 2 ( µ ) ≤ C 1 ε . From Le Cam's lemma [39], as the Voronoi loss function D 2 ,r satisfies the weak triangle inequality, we obtain that

where the second inequality is due to the fact that

By choosing ε = n -1 / 2 , we obtain that ε · exp ( -C 1 nε 2 ) = n -1 / 2 exp( -C 1 ) . As a consequence, we achieve the desired minimax lower bound in equation (30).

Main proof. Following from the result of Lemma 2, it is sufficient to show that the following limit holds true for any r ≥ 1 :

To this end, we need to construct a sequence of mixing measures ( G n ) that satisfies D 2 ,r ( G n , G ∗ ) → 0 and

as n →∞ . Recall that under the Case 2, at least one among parameters a ∗ 1 , . . . , a ∗ k ∗ is equal to 0 d . Without loss of generality, we may assume that a ∗ 1 = 0 d . Next, let us take into account the sequence ( G n ) with k ∗ +1 atoms in which

a n 1 = a n 2 = a ∗ 1 = 0 d and a n i = a ∗ i -1 for any 3 ≤ i ≤ k ∗ +1 ;

β n 11 = β n 12 = β ∗ 11 = 0 d and β n 1 i = β ∗ 1( i -1) for any 3 ≤ i ≤ k ∗ +1 ;

b n 1 = b ∗ 1 + c n , b n 2 = b ∗ 1 + 2 c n and b n i = b ∗ i -1 for any 3 ≤ i ≤ k ∗ +1 ,

where c ∈ R will be chosen later. Consequently, we get that

Next, we demonstrate that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / D 2 ,r ( G n , G ∗ ) → 0 . In particular, following from the decomposition of f G n ( x ) -f G ∗ ( x ) in equation (23) and the above parameter setting, we have that

Thus, we deduce that

When r is an odd natural number: By applying the Taylor expansion up to order r -th, we get that

Note that [ ∑ r α =1 (1+2 α ) φ ( α ) ( b ∗ 1 ) α ! n α · c α ] is an odd-order polynomial of c . Thus, we can choose c as a root of this polynomial, which leads to the fact that where R 1 ( x ) is the Taylor remainder such that R 1 ( x ) / D 2 ,r ( G n , G ∗ ) → 0 as n →∞ .

From the above results, we deduce that [ f G n ( x ) -f G ∗ ( x )] / D 2 ,r ( G n , G ∗ ) → 0 for almost every x . As a consequence, we achieve that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / D 2 ,r ( G n , G ∗ ) → 0 as n →∞ .

When r is an even natural number: By means of the Taylor expansion of order ( r +1) -th, we have

Since [ ∑ r +1 α =1 (1+2 α ) σ ( α ) ( b ∗ 1 ) α ! n α · c α ] is an odd-degree polynomial of variable c , we can argue in a similar fashion to the scenario when r is odd to obtain that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / D 2 ,r ( G n , G ∗ ) → 0 . Combine results from the above two scenarios of r , we obtain the claim in equation (32).

where R 2 ( x ) is a Taylor remainder such that R 2 ( x ) / D 2 ,r ( G n , G ∗ ) → 0 .

A.4 Proof of Theorem 4

Following from the result of Lemma 2, it is sufficient to show that the following limit holds true for any r ≥ 1 :

To this end, we need to construct a sequence of mixing measures ( G n ) that satisfies D 2 ,r ( G n , G ∗ ) → 0 and

as n →∞ . Recall that under the Case 2, at least one among parameters a ∗ 1 , . . . , a ∗ k ∗ is equal to 0 d . Without loss of generality, we may assume that a ∗ 1 = 0 d . Next, let us take into account the sequence ( G n ) with k ∗ +1 atoms in which

β n 01 = β n 02 such that

β n 11 = β n 12 = β ∗ 11 = 0 d and β n 1 i = β ∗ 1( i -1) for any 3 ≤ i ≤ k ∗ +1 ;

and β n 0 i = β ∗ 0( i -1) for any 3 ≤ i ≤ k ∗ +1 ;

a n 1 = a n 2 = a ∗ 1 and a n i = a ∗ i -1 for any 3 ≤ i ≤ k ∗ +1 ;

Consequently, we get that

Next, we demonstrate that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / D 2 ,r ( G n , G ∗ ) → 0 . In particular, following from the decomposition of f G n ( x ) -f G ∗ ( x ) in equation (23) and the above parameter setting, we have that

From the above result, we deduce that [ f G n ( x ) -f G ∗ ( x )] / D 2 ,r ( G n , G ∗ ) → 0 for almost every x . As a consequence, we achieve that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / D 2 ,r ( G n , G ∗ ) → 0 as n →∞ . Hence, we obtain the claim in equation (33).

A.5 Proof of Theorem 5

Following from the result of Corollary 1, it suffices to establish the following inequality:

for any mixing measure G ∈ G k (Θ) . For that purpose, we divide the proof of the above inequality into local and global parts as in Appendix A.2. However, since the arguments for the global part remain the same (up to some changes of notations) for the over-specified setting, they will be omitted.

Therefore, given an arbitrary mixing measure G := ∑ k i =1 1 1 + exp( -¯ β 0 i ) δ ( ¯ β 1 i , ¯ η i ) ∈ G k (Θ) , we focus only on demonstrating that

Assume by contrary that the above inequality does not hold true, then there exists a sequence of mixing measures G n = ∑ k i =1 1 1 + exp( -β n 0 i ) δ ( β n 1 i ,η n i ) in G k (Θ) such that D 3 n := D 3 ( G n , G ) → 0 and

as n →∞ . Let us denote by A n i := A i ( G n ) a Voronoi cell of G n generated by the j -th components of G . Since our arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size, i.e. A j = A n i .

Additionally, since G n and G share the same number of atoms k under the Regime 2 and D 3 n → 0 , the Voronoi cell A i has only one element for any i ∈ [ k ] . WLOG, we assume that A i = { i } for all i ∈ [ k ] , which follows that ( β n 0 i , β n 1 i , η n i ) → ( ¯ β 0 i , ¯ β 1 i , ¯ η i ) as n →∞ for any i ∈ [ k ] .

Thus, the Voronoi loss D 3 n can be represented as

where we denote ∆ ¯ β n 0 i := β n 0 i -¯ β 0 i , ∆ ¯ β n 1 i := β n 1 i -¯ β 1 i and ∆¯ η n i := η n i -¯ η i .

Now, we divide the proof of local part into three steps as follows:

Step 1 - Taylor expansion: In this step, we decompose the term f G n ( x ) -f G ( x ) using Taylor expansion. Let us denote σ ( x, β 1 , β 0 ) := 1 1+exp( -β ⊤ 1 x -β 0 ) , then we have

where R 1 ( x ) is a Taylor remainder such that R 1 ( x ) / D 3 n → 0 as n →∞ , and

for any i ∈ [ k ] , α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that | α 1 | + α 2 + | α 3 | = 1 .

Step 2 - Non-vanishing coefficients: In this step, we show that at least one among the ratios S n i,α 1 ,α 2 ,α 3 / D 3 n does not go to zero as n tends to infinity. Assume by contrary that all of them converge to zero, i.e. S n i,α 1 ,α 2 ,α 3 / D 3 n → 0 as n → ∞ , for any i ∈ [ k ] , α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that | α 1 | + α 2 + | α 3 | = 1 . Then, it follows that

By taking the summation of the above three limits, we deduce that

Due to the topological equivalence between the 1-norm and 2-norm, we achieve that

Step 3 - Application of Fatou's lemma: In this step, we use the Fatou's lemma to argue against the result in Step 2, and then, achieve the desired inequality in equation (35).

as n →∞ , which is a contradiction. Thus, at least one among the ratios S n i,α 1 ,α 2 ,α 3 / D 3 n must not approach zero as n →∞ .

In particular, let us denote by m n the maximum of the absolute values of S n i,α 1 ,α 2 ,α 3 / D 3 n . Since at least one among those ratios must not approach zero as n →∞ , we get that 1 /m n ̸→∞ as n →∞ . Recall from the hypothesis in equation (36) that ∥ f G n -f G ∥ L 2 ( µ ) / D 3 n → 0 as n → ∞ , which indicates that ∥ f G n -f G ∥ L 1 ( µ ) / D 3 n → 0 due to the equivalence between L 1 ( µ ) -norm and L 2 ( µ ) -norm. By means of the Fatou's lemma, we have

This result implies that [ f G n ( x ) -f G ( x )] / [ m n D 3 n ] → 0 for almost every x .

Let us denote S n i,α 1 ,α 2 ,α 3 /m n D 3 n → s i,α 1 ,α 2 ,α 3 as n →∞ with a note that at least one among the limits s i,α 1 ,α 2 ,α 3 is non-zero. Then, it follows from the decomposition in equation (38) that

for almost every x . Note that the expert function h ( · , η ) is ..., then the above equation implies that s i,α 1 ,α 2 ,α 3 = 0 , for any i ∈ [ k ] , α 1 ∈ N d , α 2 ∈ N and α 3 ∈ N q such that | α 1 | + α 2 + | α 3 | = 1 . This contradicts the fact that at least one among the limits s i,α 1 ,α 2 ,α 3 is different from zero.

Hence, we obtain the local inequality in equation (35).

B Identifiability of the Sigmoid Gating Mixture of Experts

In this appendix, we explore the identifiability of the sigmoid gating MoE model in Proposition 1. Proposition 1. If the equation f G ( x ) = f G ∗ ( x ) holds true for almost every x , then it follows that G ≡ G ′ .

Proof of Proposition 1. From the assumption of this proposition, we have

̸

̸

Since the expert function h ( · , η ) is identifiable, the set { h ( x, η ′ i ) : i ∈ [ k ′ ] } , where η ′ 1 , . . . , η ′ k ′ are distinct vectors for some k ′ ∈ N , is linearly independent. If k = k ∗ , then there exists some i ∈ [ k ] such that η i = η ∗ j for any j ∈ [ k ∗ ] . This implies that 1 1+exp( -β 1 i ) ⊤ x -β 0 i ) = 0 , which is a contradiction. Thus, we must have that k = k ∗ . As a result, for almost every x . WLOG, we may assume that

for almost every x for any i ∈ [ k ∗ ] . It is worth noting that the sigmoid function is invariant to translations, then equation (40) indicates that β 1 i = β ∗ 1 i + v 1 and β 0 i = β ∗ 0 i + v 0 for some v 1 ∈ R d and v 0 ∈ R . Nevertheless, due to the assumptions β 1 k = β ∗ 1 k = 0 d and β 0 k = β ∗ 0 k = 0 , we have that v 1 = 0 d and v 0 = 0 . Consequently, it follows that β 1 i = β ∗ 1 i and β 0 i = β ∗ 0 i for any i ∈ [ k ∗ ] . Then, equation (39) can be represented as

̸

for almost every x . Next, let us denote J 1 , J 2 , . . . , J m as a partition of the index set [ k ∗ ] , where m ≤ k , such that ( β 0 i , β 1 i ) = ( β ∗ 0 i ′ , β ∗ 1 i ′ ) for any i, i ′ ∈ J j and j ∈ [ k ∗ ] . On the other hand, when i and i ′ do not belong to the same set J j , we let ( β 0 i , β 1 i ) = ( β 0 i ′ , β 1 i ′ ) . Thus, we can reformulate equation (41) as

for almost every x . Recall that β 1 i = β and β 0 i = β for any i [ k ] , then the above leads to

∗ 1 i ∗ 0 i ∈ ∗ ∗ , for any j ∈ [ m ] . As a result, we achieve that

Hence, the proof is totally completed.

C Additional Experiments

In this appendix, we conduct a simulation study to empirically validate our theoretical results on the convergence rates of the least squares estimators under both the Regime 1 and the Regime 2 of the sigmoid gating MoE. All the subsequent experiments are conducted on a MacBook Air equipped with an M1 chip CPU.

Regime 1. We begin with numerical experiments for the Regime 1 in which we sample synthetic data based on the model described in equation (1). In particular, we generate { ( X i , Y i ) } n i =1 ⊂ R d × R by first sampling X i ∼ Uniform([ -1 , 1] d ) for i = 1 , . . . , n . Then, we generate Y i according to the following model:

where the regression function f G ∗ ( · ) is defined as:

The input data dimension is set at d = 32 . We employ k ∗ = 8 experts of the form φ ( ( a ∗ i ) ⊤ x + b ∗ i ) , where the activation function is either the ReLU function or the identity function. The variance of Gaussian noise ϵ i is specified as ν = 0 . 01 .

The ground-truth gating parameters β ∗ 0 i ∈ R are drawn independently from an isotropic Gaussian distribution with zero mean and variance ν g = 0 . 01 /d for 1 ≤ i ≤ 6 , while we set β ∗ 11 = . . . = β ∗ 1 k ∗ = 0 d . Similarly, the true expert parameters, ( a ∗ i , b ∗ i ) ∈ R d × R , are drawn independently of an isotropic Gaussian distribution with zero mean and variance ν e = 1 /d for all experts. These parameters remain unchanged for all the experiments (see also Table 3).

Training procedure. For each sample size n , spanning from 10 3 to 10 5 , we perform 20 experiments. In every experiment, we employ k = k ∗ +1 = 9 fitted experts, and the parameters initialization for the gating's and experts' parameters are adjusted to be near the true parameters, minimizing potential

Gating parameters

Expert parameters

β

∗

1

i

= 0

d

β

∗

0

i

∼ N

(0

, ν

g

)

a

∗

i

∼ N

(0

d

, ν

e

I

d

)

b

∗

i

∼ N

(0

, ν

e

)

Gating parameters

Expert parameters

10

3

10

4

10

5

sample size

n

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

Loss

Rate =

O

(

n

-

0

.

46

)

Rate =

O

(

n

-

0

.

02

)

ReLU experts

D

1

(

̂

G

n

, G

∗

)

1

.

4

n

-

0

.

46

Linear experts

D

2

(

̂

G

n

, G

∗

)

1

.

6

n

-

0

.

02

Linear experts

D

2

(

̂

G

n

, G

∗

)

1

.

6

n

-

0

.

02

10

3

10

4

10

5

sample size

n

10

-

3

10

-

2

Loss

Rate =

O

(

n

-

0

.

48

)

Rate =

O

(

n

-

0

.

42

)

ReLU experts

D

3

(

̂

G

n

, G

)

0

.

2

n

-

0

.

48

Linear experts

D

3

(

̂

G

n

, G

)

0

.

2

n

-

0

.

42

Linear experts

D

3

(

̂

G

n

, G

)

0

.

2

n

-

0

.

42

(a) Regime 1

(b) Regime 2

instabilities from the optimization process. Subsequently, we execute the stochastic gradient descent algorithm across 10 epochs, employing a learning rate of η = 0 . 1 to fit a model to the synthetic data.

Results. For each experiment, we calculate the Voronoi losses for every model and report the mean values for each sample size in Figure 2a. Error bars representing two standard deviations are also shown. In Figure 2a, both the empirical convergence rates corresponding to the ReLU experts and the linear experts are analyzed under the over-specified setting. It can be seen that the use of ReLU experts induces a rapid convergence rate of order O ( n -0 . 46 ) , while the linear experts lead to a considerably slower rate of order O ( n -0 . 02 ) . Those empirical rates match our theoretical results in Theorem 2 and Theorem 4, respectively.

Regime 2. In the experiments for the Regime 2, we apply the same experimental setup as in those for the Regime 1 except for the generation of true parameters β ∗ 1 i . More specifically, β ∗ 1 i are drawn independently from an isotropic Gaussian distribution N (0 d , ν g I d ) for 1 ≤ i ≤ 8 , where ν g = 0 . 01 /d , while we set β ∗ 1 i = 0 d for i = 8 (see also Table 4).

Results. It can be observed from Figure 2b that the uses of ReLU experts and linear experts both lead to fast empirical convergence rates of orders O ( n -0 . 48 ) and O ( n -0 . 42 ) , respectively. Those empirical rates align with our theoretical findings in Theorem 5.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's contributions and scope are reflected accurately in the abstract and introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the Discussion Section we describe possible extensions and generalizations of our analysis.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Each theoretical result contains the full set of assumptions. All the proofs are in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the experimental details are provided in Section 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No] .

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experimental details are provided in Section 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bars are reported in Figure 1 and Figure 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the information on the computer resources are provided in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and followed all the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Given the theoretical nature of the paper, we do not think there are any positive or negative societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical work, and we do not release any data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We will release the code for our simulations after the review process.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of the convergence rate of the least squares estimator for learning MoE with sigmoid gating. Based on the results, the authors conclude that sigmoid gating enjoys a faster convergence rate and requires a smaller sample size to achieve the same error compared to softmax gating.\n\nSTRENGTHS:\n1. The topic is very interesting. It is intriguing to see how the selection of gating functions can affect sample efficiency in MoE.\n2. This problem is particularly relevant to recent important topics regarding language models, given that MoE has been applied in popular models like Mixtral.\n\nWEAKNESSES:\n1. The biggest concern to me is that the results from this paper do not fully support the main claim that “sigmoid gating is more sample efficient than softmax gating”. If I understand correctly (correct me if I'm wrong), the comparison between sigmoid and softmax is not under the same scenario. The convergence rate for softmax in Table 1 is from the analysis in [21], where the ground truth is an MoE model with softmax gating, while this paper considers the setting where the ground truth is an MoE model with sigmoid gating. These are two different setups and do not necessarily imply that, given the same ground truth function, sigmoid is necessarily more sample efficient than softmax. \n\n2. The only existing empirical work discussing the potential superiority of sigmoid gating that the authors have mentioned is [3], where the provided intuition is that softmax introduces competition between experts, while sigmoid does not have this issue. This is very straightforward, although not yet formally and theoretically justified. On the other hand, this paper explains the advantage from the perspective of sample efficiency. I wonder how these two perspectives can be connected. Is there anything regarding the competition view that can be reflected in this paper’s analysis? I believe this is important for understanding the effect of sigmoid gating.\n\n3. Following point 2, it would be helpful to provide empirical results to further support the main conclusion that sigmoid gating is more sample efficient. Even experiments on toy data can help. Otherwise, the statement regarding sample efficiency lacks empirical evidence. Additionally, even \"sigmoid gating is better than softmax gating\" alone does not seem to be well-supported by empirical observations from the literature.\n\nQUESTIONS:\nSee my questions raised in Weaknesses.\n\nLIMITATIONS:\nThe authors have discussed certain limitations in Section 4.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper argues that the sigmoid gating function is more sample efficient than the softmax gating function in mixture of experts (MoE) modeling. It removes competition and estimates the contribution each expert independently.\n\nEmpirical studies show that sigmoid gating achieves superior performance, but the paper aims to provide theoretical backing for this claim.\n\nThey consider a regression framework and analyze the rates of convergence of the least squares estimator in over-specified cases.\n\nThe convergence rates for expert estimation are derived under two regimes: Regime 1 (all over-specified parameters are 0) and Regime 2 (at least one over-specified parameter is not 0).\n\nSTRENGTHS:\nThe paper demonstrates that sigmoid gating is more sample efficient than softmax gating, requiring fewer samples to achieve the same level of accuracy in expert estimation.\n\nexperts with feed-forward networks and commonly used activations (ReLU, GELU) have faster convergence rates under sigmoid gating than softmax gating.\n\nThe sigmoid gating mechanism is compatible with a broader class of expert functions compared to softmax gating. (ReLU and GELU, as well as polynomial activations)\n\nWEAKNESSES:\nThe results are heavily dependent on specific assumptions, such as the distinctness of expert parameters and the boundedness of the input space. If these assumptions are violated in practical scenarios, the theoretical guarantees may not hold\n\nWhile the derivations are very appropriate and i appreciate the citations, it would be interesting to see even toy experiments byt the authorsthat show similar conversions to the theoretical ones.\n\nQUESTIONS:\nthe paper lacks justification for the choice of the regimes. We also need to deep dig to understand the reasons for strong identifiability conditions (for experts to be distinct enough, if i understand correctly), bracketing entropy (model is not overly complex) and concentration inequalities. it is quite difficult to read, so some more intuition and explanation would be helpful.\n\nMinor:\n124 in can seen\n\nLIMITATIONS:\nAs mentioned by the authors, \n\nThe paper assumes that the ground-truth parameters are independent of the sample size, leading to point-wise rather than uniform estimation rates. \n\nThe assumption that the true regression function belongs to the parametric class of MoE models under sigmoid gating is restrictive. This assumption is likely to be violated in real-world settings.\n\nI still find these assumptions reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of expert estimation in MoE models using sigmoid gating, in contrast to the more widely used softmax gating. The authors show that sigmoid gating leads to better sample efficiency compared to softmax gating for estimating expert parameters. In particular, the paper analyzes convergence rates for least squares estimation under two regimes: \n* Regime 1) when all over-specified gating parameters are zero\n* Regime 2) when at least one of the over-specified gating parameters is non-zero\n\nOne of the key findings is that experts formulated as NNs with common activations like ReLU and GELU obtain faster convergence rates under sigmoid gating compared to softmax gating. This work provides theoretical justification for the potential empirical benefits of sigmoid gating, showing it requires smaller sample sizes to achieve the same estimation error as softmax gating.\n\nSTRENGTHS:\n- The paper provides rigorous proofs and establishes convergence rates under different conditions.\n- Sigmoid gating function has been less explored in the MoE field and analyzing its behaviour for expert estimation problem is interesting.\n\nWEAKNESSES:\n1) The structure of the paper is very confusing and could be much improved. There is way too much repetition in the paper. For example, the section Technical challenges in the introduction which defines Regime 1 and 2 (Lines 83-92) is largely copy pasted to lines 155-165 and is redundant. The various sections in the introductions particularly the contributions should be kept at a more high level and not just copy paste the method section. The authors should provide a more concise overview of the problem and main contributions. Remove technical details that are repeated in later sections.\n\n2) The recent insights into training Sparse MoEs demonstrate that \"the common practice of setting the size of experts in MoE to mirror the feed-forward layer (of a base dense network) is not optimal at almost any computational budget\" (Jakub Krajewski et al. 2024). If in practice, top-k combination of small fine-grained experts is proven to be more effective than a single large expert, are we dealing in a regime where all true experts are *over-specified* in practice? If this is the case what are the implications for Regime 1? Does it mean that none of the experts are input-dependent as all the over-specified parameters $β^*_{1i}$ are equal to $0_d$? Does Regime 1 hold in practice?\n\n3) It would be best if the authors could find some practical implications of their findings. The theoretical work presented in the paper does not establish any relationship to any of the SoTA MoE models in the literature. It is unclear to me how impactful the convergence analysis with the considered regimes is in practical MoE research works.\n\n4) At present, the content and organization of the paper is very close to the paper \"On Least Square Estimation in Softmax Gating Mixture of Experts\" (Nguyen et al. 2024). The sections are so similar that may cause copyright issues. \n\nReferences:\n1. Krajewski, Jakub, et al. \"Scaling laws for fine-grained mixture of experts.\" arXiv preprint arXiv:2402.07871 (2024).\n\n2. Nguyen, Huy, Nhat Ho, and Alessandro Rinaldo. \"On least squares estimation in softmax gating mixture of experts.\" ICML (2024).\n\nQUESTIONS:\nMinor point:\n- misspecified is misspelled in line 175.dq\n\nLIMITATIONS:\nTwo limitations concerning the assumptions used in the analyses are discussed in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}