Title: Demystifying Softmax Gating Function in Gaussian Mixture of Experts
Venue: NeurIPS 2023
Accepted: yes
Keywords: Mixture of Experts

Maximum Likelihood Estimation

Voronoi Loss Function

Algebraic Geometry.
Abstract: Understanding the parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamental theoretical challenges associated with the softmax gating function: (i) the identifiability only up to the translation of parameters; (ii) the intrinsic interaction via partial differential equations between the softmax gating and the expert functions in the Gaussian density; (iii) the complex dependence between the numerator and denominator of the conditional density of softmax gating Gaussian mixture of experts. We resolve these challenges by proposing novel Voronoi loss functions among parameters and establishing the convergence rates of maximum likelihood estimator (MLE) for solving parameter estimation in these models. When the true number of experts is unknown and over-specified, our findings show a connection between the convergence rate of the MLE and a solvability problem of a system of polynomial equations.

Demystifying Softmax Gating Function in Gaussian Mixture of Experts

Abstract

Understanding the parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamental theoretical challenges associated with the softmax gating function: (i) the identifiability only up to the translation of parameters; (ii) the intrinsic interaction via partial differential equations between the softmax gating and the expert functions in the Gaussian density; (iii) the complex dependence between the numerator and denominator of the conditional density of softmax gating Gaussian mixture of experts. We resolve these challenges by proposing novel Voronoi loss functions among parameters and establishing the convergence rates of maximum likelihood estimator (MLE) for solving parameter estimation in these models. When the true number of experts is unknown and over-specified, our findings show a connection between the convergence rate of the MLE and a solvability problem of a system of polynomial equations.

1 Introduction

Softmax gating Gaussian mixture of experts [32, 37], a class of statistical machine learning models that combine multiple simpler models, known as expert functions of the covariates, via softmax gating networks to form more complex and accurate models, has found widespread use in various applications, including speech recognition [51, 64, 65], natural language processing [14, 20, 17, 54, 21], computer vision [52, 3, 15, 44], and other applications [26, 49, 7, 8, 48, 5, 6]. Regarding the applications of the softmax gating Gaussian mixture of experts in medicine [43] and physical sciences [39], the parameters of each expert function play an important role in capturing the heterogeneity of data. Thus, the main objective of these works is to conduct statistical inference for those parameters, which leads to a need for convergence rates of parameter estimation in the softmax gating Gaussian mixture of experts. However, a comprehensive theoretical understanding of parameter estimation in that model has still remained a long-standing open problem in the literature.

Parameter estimation has been studied quite extensively in standard mixture models. In his seminal work, Chen et al. [9] established the convergence rate O ( n -1 / 4 ) of parameter estimation in over-fitted univariate mixture models, namely, the settings when the number of true components is unknown and over-specified, and the family of distributions is strongly identifiable in the second order, e.g., location Gaussian distributions. That slow and non-standard rate is due to the collapse of some parameters into single pararameter or the convergence of weights to zero, which leads to the singularity of Fisher information matrix around the true parameters. Then, Nguyen et al. [50] and Ho et al. [29] utilized Wasserstein metrics to achieve this rate under the multivariate settings of second-order strongly identifiable mixture models. Recently, Ho et al. [28] demonstrated that rates of the MLE can strictly depend on the number of over-specified components when the mixture models are not strongly

identifiable, such as location-scale Gaussian mixtures. The minimax optimal behaviors of parameter estimation were studied in [27, 46]. From the computational side, the statistical guarantee of the expectation-maximization (EM), e.g., [12], and moment methods had also been studied under both exact-fitted [2, 1, 25] and over-fitted settings [19, 18, 61, 16, 62] of mixture models.

Compared to mixture models, there has been less research on parameter estimation of mixture of experts. When the gating networks are independent of the covariates, Ho et al. [30] employed the generalized Wasserstein loss function [59] to study the convergence rates of parameter estimation in Gaussian mixture of experts. They proved that these rates are determined by the algebraic independence of the expert functions and the partial differential equations with respect to the parameters. Later, Do et al. [13] extended these results to general mixture of experts with covariate-free gating network. Statistical guarantees of optimization methods for solving parameter estimation in Gaussian mixture of experts with covariate-free gating functions were studied in [11, 67, 41, 63]. When the gating networks are softmax functions, parameter estimation becomes more challenging to understand due to the complex structures of the softmax gating function in the Gaussian mixture of experts. Before describing these phenomena in further details, we begin by formally introducing the softmax gating Gaussian mixture of experts and related notions.

Problem setting: Assume that ( X 1 , Y 1 ) , . . . , ( X n , Y n ) ∈ R d × R are i.i.d. samples drawn from the softmax gating Gaussian mixture of experts of order k ∗ whose conditional density function g G ∗ ( Y | X ) is given by:

where f ( . | µ, σ ) is a Gaussian density function with mean µ and variance σ . Here, we define G ∗ := ∑ k ∗ i =1 exp( β ∗ 0 i ) δ ( β ∗ 1 i ,a ∗ i ,b ∗ i ,σ ∗ i ) as a true but unknown mixing measure , that is, a combination of Dirac measures δ associated with true parameters θ ∗ i := ( β ∗ 0 i , β ∗ 1 i , a ∗ i , b ∗ i , σ ∗ i ) . Notably, G ∗ is not necessarily a probability measure as the summation of its weights can be different from one. For the purpose of the theory, we assume that θ ∗ i ∈ Θ ⊂ R × R d × R d × R × R + where Θ is a compact set, and X ∈ X ⊂ R d where X is a bounded set. Furthermore, we let ( a ∗ 1 , b ∗ 1 , σ ∗ 1 ) , . . . , ( a ∗ k ∗ , b ∗ k ∗ , σ ∗ k ∗ ) be pairwise distinct and at least one among β ∗ 11 , . . . , β ∗ 1 k ∗ be non-zero to guarantee the dependence of softmax gating function on the covariate X . Finally, we assume that the covariate X follows a continuous distribution to ensure that the softmax gating Gaussian mixture of experts is at least identifiable up to translations (see Proposition 1).

Maximum likelihood estimation. Since the value of true order k ∗ is unknown in practice, to estimate the unknown parameters in the softmax gating Gaussian mixture of experts (1), we consider using maximum likelihood estimation (MLE) within a class of all mixing measures with at most k components, which is defined as follows:

where O k (Θ) := { G = ∑ k ′ i =1 exp( β 0 i ) δ ( β 1 i ,a i ,b i ,σ i ) : 1 ≤ k ′ ≤ k and ( β 0 i , β 1 i , a i , b i , σ i ) ∈ Θ } . To guarantee that the MLE ̂ G n is a consistent estimator of G ∗ , we need k ≥ k ∗ . In this paper, we study the convergence rate of the MLE ̂ G n to the true mixing measure G ∗ under both the exact-fitted settings, namely when k = k ∗ , and the over-fitted settings, namely when k > k ∗ , of the softmax gating Gaussian mixture of experts.

Fundamental challenges from the softmax gating function: There are three fundamental challenges arising from the softmax gating function that create various obstacles in our convergence analysis:

(i) Firstly, parameters β ∗ 1 i , β ∗ 0 i of the softmax gating function are not identifiable as those of the covariate-independent gating function in previous work. Instead, they are identifiable up to translations, that is, the softmax gating value does not change when we translate β ∗ 0 i to β ∗ 0 i + t 1 and β ∗ 1 i to β ∗ 1 i + t 2 for any t 1 ∈ R and t 2 ∈ R d . As a consequence, we need to introduce an infimum operator in the Voronoi loss functions (see equations (4) and (6)) to deal with this issue.

(ii) Secondly, a key step in our proof techniques is to decompose the density discrepancy g ̂ G n ( Y | X ) -g G ∗ ( Y | X ) into a linear combination of linearly independent elements using Taylor expansions.

However, since the numerators and denominators of softmax gating functions are dependent, we cannot apply the Taylor expansions directly to that density discrepancy as in previous work [30, 13]. Moreover, there are two intrinsic interactions between parameters of the softmax gating's numerators and the Gaussian density function via the following partial differential equations (PDEs), which induce a lot of linearly dependent derivative terms in the Taylor expansions:

where u ( Y | X ; β 1 , a, b, σ ) := exp( β ⊤ 1 X ) · f ( Y | a ⊤ X + b, σ ) . Therefore, it takes us great effort to group those linearly dependent terms together to obtain the desired linear combination of linearly independent terms.

(iii) Lastly, given the above linear combination of linearly independent elements, when the density estimation g ̂ G n ( Y | X ) converges to the true density g G ∗ ( Y | X ) , the associated coefficients in that combination also go to zero. Then, via some transformations, those limits lead to a system of polynomial equations introduced in equation (9). This system admits a much more complex structure than what considered in previous work [30, 13].

These fundamental challenges from the softmax gating function suggest that the previous loss functions, such as Wasserstein distance [50, 28, 30], being employed to study parameter estimation in standard mixture models or mixture of experts with covariate-free gating functions are no longer sufficient as they heavily rely on the assumptions that the weights of these models are independent of the covariates.

Main contributions: To tackle these challenges of the softmax gating function, we propose two novel Voronoi losses among parameters and establish the lower bounds of the Hellinger distance, denoted as h ( · , · ) , of the mixing densities of softmax gating Gaussian mixture of experts in terms of these Voronoi losses to capture the behaviors of the MLE. Our results can be summarized as follows (see also Table 1):

Exact-fitted settings : When k = k ∗ , we demonstrate that the Hellinger lower bound E X [ h ( g G ( ·| X ) , g G ∗ ( ·| X ))] ≥ C · D 1 ( G,G ∗ ) holds for any mixing measure G ∈ O k (Θ) , where C is some universal constant and the Voronoi metric D 1 ( G,G ∗ ) is defined as:

where ∆ t 2 β 1 ij := β 1 i -β ∗ 1 j -t 2 , ∆ a ij := a i -a ∗ j , ∆ b ij := b i -b ∗ j , ∆ σ ij := σ i -σ ∗ j . The infimum over t 1 ∈ R and t 2 ∈ R d is to account for the identifiability up to the translation of ( β ∗ 0 j , β ∗ 1 j ) k ∗ j =1 . Furthermore, A j is a Voronoi cell of mixing measure G generated by the true component ω ∗ j := ( β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j ) for all 1 ≤ j ≤ k ∗ [47], which is defined as follows:

̸

where we denote ω i := ( β 1 i , a i , b i , σ i ) . It is worth noting that the cardinality of each V oronoi cell A j indicates the number of components of G approximating the true component ω ∗ j of G ∗ . As E X [ h ( g ̂ G n ( ·| X ) , g G ∗ ( ·| X ))] = O ( n -1 / 2 ) , that lower bound of Hellinger distance indicates that D 1 ( ̂ G n , G ∗ ) = O ( n -1 / 2 ) . Therefore, the rates of estimating exp( β ∗ 0 j ) , β ∗ 1 j (up to translations) and a ∗ j , β ∗ j , σ ∗ j are of optimal order O ( n -1 / 2 ) .

Over-fitted settings : When k > k ∗ , the lower bound of Hellinger distance in terms of the Voronoi metric D 1 in the exact-fitted settings is no longer sufficient due to the collapse of softmax of vectors in possibly k dimensions to softmax of vectors in k ∗ dimensions. Our approach is to define more

fine-grained Voronoi metric D 2 ( G,G ∗ ) to capture such collapse, which is given by:

for any mixing measure G ∈ O k (Θ) . Here, the values of function ¯ r ( · ) are determined by the solvability of a system of polynomial equations defined in equation (9). We then show in Lemma 1 that ¯ r (2) = 4 , ¯ r (3) = 6 , and we conjecture that ¯ r ( m ) = 2 m for any m ≥ 2 .

In high level, the aforementioned system of polynomial equations arises from the PDEs in equation (3) when we establish the lower bound E X [ h ( g G ( ·| X ) , g G ∗ ( ·| X ))] ≥ C ′ D 2 ( G,G ∗ ) for any G ∈ O k (Θ) for some universal constant C ′ . Since E X [ h ( g ̂ G n ( ·| X ) , g G ∗ ( ·| X ))] = O ( n -1 / 2 ) , we also have D 2 ( ̂ G n , G ∗ ) = O ( n -1 / 2 ) under the over-fitted settings of the softmax gating Gaussian mixture of experts. As a consequence, the rates for estimating true parameters whose Voronoi cells have only one component of the MLE are of order O ( n -1 / 2 ) . On the other hand, for true parameters exp( β ∗ 0 j ) , β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j whose Voronoi cells have more than one component of the MLE, the estimation rates are respectively O ( n -1 / 2¯ r ( |A j | ) ) for β ∗ 1 j , b ∗ j , O ( n -1 / ¯ r ( |A j | ) ) for a ∗ j , σ ∗ j , and O ( n -1 / 2 ) for exp( β ∗ 0 j ) . This rich spectrum of parameter estimation rates is due to the complex interaction between the softmax gating and the expert functions.

Table 1: Summary of density estimation and parameter estimation rates in the softmax gating Gaussian mixture of experts under both the exact-fitted and over-fitted settings. Recall that the cardinality of each Voronoi cell A j gives the number of fitted components approximating true component ω ∗ j = ( β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j ) (see equation (5)). Furthermore, the notation ¯ r ( |A j | ) stands for the solvability of the system of polynomial equations (9). For instance, if ω ∗ j is fitted by two components, then we have |A j | = 2 and ¯ r ( |A j | ) = 4 . Please refer to Lemma 1 for more details of the values of function ¯ r .

D

O

O

O

O

Practical implications: Although the slow rates of the MLE under the over-fitted settings of the softmax gating Gaussian mixture of experts may seem discouraging, a practical implication of these results is that we should not choose the number of experts k to be very large compared to the true number of experts k ∗ . Furthermore, the slow rates can also be useful for post-processing procedures, such as merge-truncate-merge procedure [24], with the MLE to reduce the number of experts so as to consistently estimate k ∗ when the number of data is sufficiently large. In particular, an important insight from the theoretical results is that we can merge the MLE parameters that are close and within the range of their rates of convergence or truncate the parameters that lead to small weights of the experts. As the sample size becomes sufficiently large, the reduced number of experts may converge to the true number of experts. We leave an investigation of such model selection with the Gaussian mixture of experts via the rates of MLE for future work.

Organization: The paper is organized as follows. In Section 2, we first provide background on the identifiability and rate of conditional density estimation in the softmax gating Gaussian mixture of experts. Next, we proceed to establish the convergence rate of the MLE under both the exact-fitted and over-fitted settings of these models in Section 3. Then, we conclude the paper with a few discussions in Section 4. Finally, full proofs of the results and a simulation study are provided in the Appendices.

Notation: Firstly, we denote [ n ] := { 1 , 2 , . . . , n } for any positive integer n . Next, for any vector u ∈ R d and z := ( z 1 , z 2 , . . . , z d ) ∈ N d , we denote u z = u z 1 1 u z 2 2 . . . u z d d , | u | := u 1 + u 2 + . . . + u d and z ! := z 1 ! z 2 ! . . . z d ! , while ∥ u ∥ represents for its 2 -norm value. Additionally, the notation | A | indicates the cardinality of any set A . Given any two positive sequences { a n } n ≥ 1 and { b n } n ≥ 1 , we write a n = O ( b n ) or a n ≲ b n if a n ≤ Cb n for all n ∈ N , where C > 0 is some universal constant. Lastly, for any two probability density functions p, q dominated by the Lebesgue measure µ , we denote

h 2 ( p, q ) = 1 2 ∫ ( √ p - √ q ) 2 dµ as the their squared Hellinger distance and V ( p, q ) = 1 2 ∫ | p -q | dµ as their Total Variation distance.

2 Background

In this section, we begin with the following result on the identifiability of the softmax gating Gaussian mixture of experts, which was previously studied in [36].

Proposition 1 (Identifiability of the softmax gating Gaussian mixture of experts) . For any mixing measures G = ∑ k i =1 exp( β 0 i ) δ ( β 1 i ,a i ,b i ,σ i ) and G ′ = ∑ k ′ i =1 exp( β ′ 0 i ) δ ( β ′ 1 i ,a ′ i ,b ′ i ,σ ′ i ) , if we have g G ( Y | X ) = g G ′ ( Y | X ) for almost surely ( X,Y ) , then it follows that k = k ′ and G ≡ G ′ t 1 ,t 2 where G ′ t 1 ,t 2 := ∑ k ′ i =1 exp( β ′ 0 i + t 1 ) δ ( β ′ 1 i + t 2 ,a ′ i ,b ′ i ,σ ′ i ) for some t 1 ∈ R and t 2 ∈ R d .

Given the consistency of the MLE, it is natural to ask about its convergence rate to the true parameters. Our next result establishes the convergence rate of conditional density estimation g ̂ G n ( Y | X ) to the true conditional density g G ∗ ( Y | X ) , which lays an important foundation for the study of MLE's convergence rate.

Proof of Proposition 1 is in Appendix B.1. The identifiability of the softmax gating Gaussian mixture of experts guarantees that the MLE ̂ G n (2) converges to the true mixing measure G ∗ (up to the translation of the parameters in the softmax gating).

Proposition 2 (Density estimation rate) . Given the MLE in equation (2) , the conditional density estimation g ̂ G n ( Y | X ) has the following convergence rate:

where c and C are universal constants.

Proof of Proposition 2 is in Appendix B.2. The result of Proposition 2 indicates that under either the exact-fitted or over-fitted settings of the softmax gating Gaussian mixture of experts, the rate of the conditional density function g ̂ G n ( Y | X ) to the true one g G ∗ ( Y | X ) under Hellinger distance is of order O ( n -1 / 2 ) (up to some logarithmic factors), which is parametric on the sample size.

From density estimation to parameter estimation: The parametric rate of the conditional density estimation in Proposition 2 suggests that as long as we can establish the Hellinger lower bound E X [ h ( g G ( ·| X ) , g G ∗ ( ·| X ))] ≳ D ( G,G ∗ ) for any mixing measure G ∈ O k (Θ) for some metric D among the parameters, then we obtain directly the parametric convergence rate of the MLE under the metric D . Therefore, the main focus of the next section is to determine such metric D and to establish that lower bound under either exact-fitted or over-fitted settings of the Gaussian mixture of experts.

3 Convergence Rate of the Maximum Likelihood Estimation

In this section, we first study the convergence rate of the MLE under the exact-fitted settings of the softmax gating Gaussian mixture of experts in Section 3.1. Then, we move to the over-fitted settings in Section 3.2. Finally, we provide a proof sketch of the theories in Section 3.3.

3.1 Exact-fitted Settings

For the exact-fitted settings, namely, when the chosen number of experts k is equal to the true number of experts k ∗ , as we mentioned in the introduction, the proper metric between the MLE and the true mixing measure is the metric D 1 defined in equation (4), which is given by:

where ∆ t 2 β 1 ij := β 1 i -β ∗ 1 j -t 2 , ∆ a ij := a i -a ∗ j , ∆ b ij := b i -b ∗ j , ∆ σ ij := σ i -σ ∗ j . Here, A j is a Voronoi cell of G generated by ( β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j ) for all 1 ≤ j ≤ k ∗ . Furthermore, the infimum is taken with respect to ( t 1 , t 2 ) ∈ R × R d such that β ∗ 0 j + t 1 and β ∗ 1 j + t 2 still lie inside the domain of the parameter space Θ .

It is clear that D 1 ( G,G ∗ ) = 0 if and only if G ≡ G ∗ (up to translation). When D 1 ( G,G ∗ ) is sufficiently small, there exist t 1 , t 2 such that all of ∆ t 2 β 1 ij , ∆ a ij , ∆ b ij , ∆ σ ij , and ∑ i ∈A j exp( β 0 i ) -exp( β ∗ 0 j + t 1 ) are sufficiently small as well. Therefore, the loss function D 1 provides a useful metric to measure the difference between the MLE and the true mixing measure. For any fixed t 1 , t 2 , the computation of the summations in D 1 only has the complexity of the order O ( k 2 ∗ ) . To solve the optimization with respect to t 1 , t 2 in the metric D 1 , we can utilize the projected subgradient method with fixed step size [4], which has the complexity of the order O ( ε -2 ) as the functions of t 1 and t 2 are convex where ε is a desired tolerance. Therefore, the total computational complexity of approximating the value of the Voronoi loss function D 1 is at the order of O ( k 2 ∗ /ε 2 ) .

The following result establishes the lower bound of the Hellinger distance between the conditional densities in terms of the loss function D 1 between corresponding mixing measures, which in turn leads to the convergence rate of the MLE.

Theorem 1. Given the exact-fitted settings of the softmax gating Gaussian mixture of experts (1) , i.e., k = k ∗ , we find that

for any G ∈ E k ∗ (Θ) := O k ∗ (Θ) \ O k ∗ -1 (Θ) where C 1 is some universal constant depending only on G ∗ and Θ . As a consequence, there exist universal constants C ′ 1 and c 1 such that the convergence rate of the MLE ̂ G n under the exact-fitted settings satisfies:

Proof of Theorem 1 is in Appendix A.1. The parametric convergence rate of the MLE to G ∗ under the metric D 1 suggests that the rates of estimating the true parameters exp( β ∗ 0 j ) , β ∗ 1 j (up to translation), a ∗ j , b ∗ j , σ ∗ j for j ∈ [ k ∗ ] are of order O ( n -1 / 2 ) , which are optimal up to logarithmic factors.

3.2 Over-fitted Settings

We now consider the over-fitted settings of the softmax gating Gaussian mixture of experts. Different from the exact-fitted settings, the softmax weights associated with the MLE collapse to the softmax weights of the mixture of true experts as long as the MLE approaches the true mixing measure G ∗ . More concretely, we can relabel the supports of the MLE ̂ G n with ˆ k n components ( ˆ k n ≤ k ) based on the Voronoi cells A n j := A j ( ̂ G n ) such that we can rewrite it as ̂ G n = ∑ k ∗ j =1 ∑ i ∈A n j exp( ̂ β n 0 i ) δ ( ̂ β n 1 i , ̂ a n i , ̂ b n i , ̂ σ n i ) where ∑ k ∗ j =1 |A n j | = ˆ k n , ( a n i , ̂ b n i , σ n i ) → ( a ∗ j , b ∗ j , σ ∗ j ) , as n approaches infinity for all 1 ≤ i ≤ A n j and j ∈ [ k ∗ ] .

The collapse of the softmax weights along with the PDEs (3) between the softmax gating and the expert functions in the Gaussian density create a complex interaction among the estimated parameters. To disentangle such interaction, we rely on the solvability of a novel system of polynomial equations defined in equation (9). In particular, for any m ≥ 2 , we define ¯ r ( m ) as the smallest natural number r such that the following system of polynomial equations:

for any ( ℓ 1 , ℓ 2 ) ∈ N d × N such that 0 ≤ | ℓ 1 | ≤ r , 0 ≤ ℓ 2 ≤ r -| ℓ 1 | and | ℓ 1 | + ℓ 2 ≥ 1 , does not have any non-trivial solution for the unknown variables { p 1 j , p 2 j , p 3 j , p 4 j , p 5 j } j m =1 , namely, all of p 5 j are non-zero and at least one among p 3 j is different from zero. The ranges of α 1 , α 2 , α 3 , α 4 in the above

sum satisfy I ℓ 1 ,ℓ 2 = { α = ( α 1 , α 2 , α 3 , α 4 ) ∈ N d × N d × N × N : α 1 + α 2 = ℓ 1 , | α 2 | + α 3 +2 α 4 = ℓ 2 } . When d = 1 and r = 2 , that system of equations becomes

It is clear that we have non-trivial solutions p 5 j = 1 , p 1 j = 0 for all j ∈ [ m ] , | p 21 | = p 31 = 1 , | p 22 | = p 32 = -1 , p 41 = p 42 = -1 / 2 , p 2 j = p 3 j = p 4 j = 0 for 3 ≤ j ≤ m .

When d = 1 and r = 3 , the system of equations can be written as follows:

It can be seen that the following is a non-trivial solution of the above system: p 5 j = 1 , p 1 j = p 2 j = 0 for all j ∈ [ m ] , p 31 = √ 3 3 , p 32 = -√ 3 3 , p 41 = p 42 = -1 6 , p 3 j = p 4 j = 0 for 3 ≤ j ≤ m . Therefore, we obtain that ¯ r ( m ) ≥ 4 when m ≥ 2 and d = 1 .

In general, when d = 1 , the system of equations has ( r 2 +3 r ) / 2 equations. Intuitively, when m is sufficiently larger than ( r 2 +3 r ) / 2 , the system may not have a non-trivial solution. For general dimension d and parameter m ≥ 2 , finding the exact value of ¯ r ( m ) is a non-trivial central problem in algebraic geometry [55]. When m is small, the following lemma provides specific values for ¯ r ( m ) .

Proof of Lemma 1 is in Appendix B.3. As m increases, so does the value of ¯ r ( m ) . We conjecture that ¯ r ( m ) = 2 m and leave the proof of that conjecture to future work.

By constructing the Voronoi loss function:

Theorem 2. Under the over-fitted settings of the softmax gating Gaussian mixture of experts (1) , namely, when k > k ∗ , we obtain that the following result demonstrates that the convergence rates of the MLE under the over-fitted settings of the softmax gating Gaussian mixture of experts are determined by ¯ r ( · ) .

for any G ∈ O k (Θ) where C 2 is some universal constant depending only on G ∗ and Θ . Therefore, that lower bound leads to the following convergence rate of the MLE:

where C ′ 2 and c 2 are some universal constants.

Proof of Theorem 2 is in Appendix A.2. A few comments with the result of Theorem 2 are in order.

(i) Rates of individual parameters: The convergence rate O ( n -1 / 2 ) (up to some logarithmic term) of the MLE under the loss function D 2 implies that for the true parameters exp( β ∗ 0 j ) , β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j whose Voronoi cells have only one component of the MLE, the rates for estimating them are O ( n -1 / 2 ) up to some logarithmic factor. On the other hand, for true parameters with greater than one component in their Voronoi cells, the rates for estimating β ∗ 1 j , b ∗ j are O ( n -1 / 2¯ r ( |A n j | ) ) while those for a ∗ j , σ ∗ j are O ( n -1 / ¯ r ( |A n j | ) ) (up to logarithmic factors). As the maximum value of |A n j | is ˆ k n -k ∗ + 1 , it indicates that these rates (up to logarithmic factors) can be as worse as O ( n -1 / ¯ r ( ˆ k n -k ∗ +1) ) for estimating a ∗ j , σ ∗ j and O ( n -1 / 2¯ r ( ˆ k n -k ∗ +1) ) for estimating β ∗ 1 j , b ∗ j .

(iii) Comparison with covariate-free gating network: We would like to remark that the results being established for parameter estimation under the softmax gating network settings of over-fitted Gaussian mixture of experts are in stark difference from those under the covariate-free gating network settings of these models [30], namely, when the gating function is independent of the covariates X . In particular, Theorem 2 in [30] shows that when the gating networks are independent of the covariates, the convergence rates of estimating a ∗ j are at the order of O ( n -1 / 4 ) (up to some logarithmic factor), which are independent of the number of over-fitted components. It is different from the rates of a ∗ j whose Voronoi cells have more than one component in the softmax gating settings, which depends on the number of components that we over-fit the Gaussian mixture of experts (see discussion (i) after Theorem 2). Furthermore, the rates of estimating b ∗ j , σ ∗ j when the gating networks are independent of covariates are determined by a system of polynomial equations that is much simpler than the system of equations (9) when the gating networks are softmax function. These differences are mainly due to the intrinsic interaction characterized by partial differential equations with respect to the parameters between the softmax gating networks and the expert functions in Gaussian distribution.

(ii) Computation of Voronoi loss function D 2 : Similar to the Voronoi loss function D 1 in the exact-fitted setting, the loss function D 2 is also computationally efficient. In particular, for any fixed t 1 , t 2 , the computation of the summations in the formulation of D 2 is at the order O ( k × k ∗ ) , which is linear on k when k ∗ is fixed. Furthermore, we can solve the convex optimization problem with respect to t 1 , t 2 with computational complexity at the order of O ( ε -2 ) via the projected gradient descent method with fixed step size where ε is the error. Therefore, the total computational complexity of approximating the Voronoi loss function D 2 is at the order of O ( k × k ∗ /ε 2 ) .

3.3 Proof Sketch

In this section, we provide a proof sketch for Theorems 1 and 2. To simplify the ensuing discussion, the loss function D in the proof sketch is implicitly understood as either D 1 or D 2 depending on the settings of the softmax gating Gaussian mixture of experts. Since the Hellinger distance h is lower bounded by the Total Variation distance V , to obtain the bounds in equations (7) and (10), it is sufficient to show that E X [ V ( g G ( ·| X ) , g G ∗ ( ·| X ))] ≳ D ( G,G ∗ ) . To establish this bound, we respectively prove its local and global versions by contradiction as follows:

Local version : In this part, we aim to show the following local inequality:

Assume that this claim does not hold true, that is, there exists a sequence G n = ∑ k n i =1 exp( β n 0 i ) δ ( β n 1 i ,a n i ,b n i ,σ n i ) ∈ O k (Θ) such that both E X [ V ( g G n ( ·| X ) , g G ∗ ( ·| X ))] / D ( G n , G ∗ ) and D ( G n , G ∗ ) approach zero as n tends to infinity. This implies that for any j ∈ [ k ∗ ] , we have ∑ i ∈A j exp( β n 0 i ) → exp( β ∗ 0 j ) and ( β n 1 i , a n i , b n i , σ n i ) → ( β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j ) and for all i ∈ A j . For the sake of presentation, we simplify the loss function D by assuming that it is minimized when t 1 = 0 and t 2 = 0 d . Now, we decompose the quantity

as follows:

where we define u ( Y | X ; β 1 , a, b, σ ) := exp( β ⊤ 1 X ) f ( Y | a ⊤ X + b, σ ) and v ( Y | X ; β 1 ) := exp( β ⊤ 1 X ) g G n ( Y | X ) . Next, for each j ∈ [ k ∗ ] and i ∈ A j , we denote h 1 ( X,a ∗ j , b ∗ j ) := ( a ∗ j ) ⊤ X + b ∗ j and then apply the Taylor expansions to the functions u ( Y | X ; β n 1 i , a n i , b n i , σ n i ) and v ( Y | X ; β n 1 i ) up to orders r 1 j and r 2 j (which we will choose later), respectively, as follows:

where R 1 ij ( X,Y ) and R 2 ij ( X,Y ) are Taylor remainders such that R ρij ( X,Y ) / D ( G n , G ∗ ) vanishes as n →∞ for ρ ∈ { 1 , 2 } . As a result, the limit of Q n / D ( G n , G ∗ ) when n goes to infinity can be seen as a linear combination of elements of the following set:

which is shown to be linearly independent. By the Fatou's lemma, we demonstrate that Q n / D ( G n , G ∗ ) goes to zero as n → ∞ , implying that all the coefficients in the representation of Q n / D ( G n , G ∗ ) , denoted by T n ℓ 1 ,ℓ 2 ( j ) / D ( G n , G ∗ ) and S n γ ( j ) / D ( G n , G ∗ ) , vanish when n →∞ . Given that result, we aim to select the Taylor orders r 1 j and r 2 j such that at least one among the limits of T n ℓ 1 ,ℓ 2 ( j ) / D ( G n , G ∗ ) and S n γ ( j ) / D ( G n , G ∗ ) is different from zero, which leads to a contradiction. Hence, we obtain the local version of the desired inequality.

Below are the details of choosing appropriate Taylor orders in each setting.

Exact-fitted settings: Under this setting, since k ∗ is known, each of the Voronoi cells A j for j ∈ [ k ∗ ] has only one element. Thus, for any i ∈ A j , we have exp( β n 0 i ) → exp( β ∗ 0 j ) and ( β n 1 i , a n i , b n i , σ n i ) → ( β ∗ 1 j , a ∗ j , b ∗ j , σ ∗ j ) . Given that result, we will select r 1 j = r 2 j = 1 for all j ∈ [ k ∗ ] as it suffices to show that at least one among the limits of T n ℓ 1 ,ℓ 2 ( j ) / D ( G n , G ∗ ) and S n γ ( j ) / D ( G n , G ∗ ) is different from zero. In particular, if all of them vanished, we would take the sum of all the limits of T n ℓ 1 ,ℓ 2 ( j ) / D ( G n , G ∗ ) for ( ℓ 1 , ℓ 2 ) such that 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 , which leads to a contradiction that 1 = D ( G n , G ∗ ) / D ( G n , G ∗ ) → 0 .

Over-fitted settings: As k ∗ becomes unknown in this scenario, we need higher Taylor orders to obtain the same result as in the exact-fitted setting. We will reuse the proof by contradiction method to find out those orders. More specifically, assume that all the limits of T n ℓ 1 ,ℓ 2 ( j ) / D ( G n , G ∗ ) and S n γ ( j ) / D ( G n , G ∗ ) equal zero. After some steps of considering typical limits as in the previous setting which requires r 2 j = 2 for all j ∈ [ k ∗ ] , we encounter the following system of polynomial equations:

for all ( ℓ 1 , ℓ 2 ) ∈ N d × N such that 0 ≤ | ℓ 1 | ≤ r 1 j , 0 ≤ ℓ 2 ≤ r 1 j - | ℓ 1 | and | ℓ 1 | + ℓ 2 ≥ 1 for some j ∈ [ k ∗ ] . Due to the construction of this system, it must have at least one non-trivial

solution. Therefore, if we choose r 1 j = ¯ r ( |A j | ) for all j ∈ [ k ∗ ] , then the above system does not admit any non-trivial solutions, which leads to a contradiction. Hence, we obtain the local inequality in equation (12), which suggests that we can find a positive constant ε ′ such that inf G ∈O k (Θ): D ( G,G ∗ ) ≤ ε ′ E X [ V ( g G ( ·| X ) , g G ∗ ( ·| X ))] / D ( G,G ∗ ) > 0 .

Global version: Therefore, it is sufficient to demonstrate the following global inequality:

Assume that this claim is not true, then we can find a mixing measure G ′ ∈ O k (Θ) such that g G ′ ( Y | X ) = g G ∗ ( Y | X ) for almost surely ( X,Y ) . According to Proposition 1, we get that D ( G ′ , G ∗ ) = 0 , which contradicts the hypothesis D ( G ′ , G ∗ ) > ε ′ . These arguments hold for both exact-fitted and over-fitted settings up to some changes of notations.

Hence, the proof sketch is completed.

4 Discussion

In the paper, we study the convergence rates of parameter estimation under both the exact-fitted and over-fitted settings of the softmax gating Gaussian mixture of experts. We introduce novel Voronoi loss functions among parameters to resolve fundamental theoretical challenges posed by the softmax gating function, including identifiability up to the translation of parameters, the interaction between softmax weights and expert functions, and the dependence between the numerator and denominator of the conditional density function. When the true number of experts is known, we demonstrate that the rates for estimating true parameters are parametric on the sample size. On the other hand, when the true number of experts is unknown and over-specified, these estimation rates turn out to be determined by the solvability of a system of polynomial equations.

There are a few natural directions arising from the paper that we leave for furture work:

First, our work does not consider the top-K sparse softmax gating function, which has been widely used to scale up massive deep learning architectures [68, 54, 21]. It is practically important to extend the current theories to establish the convergence rates of parameter estimation in the Gaussian mixture of experts with that gating function.

Second, the paper only takes into account the regression settings, namely when the distribution of Y is assumed to be continuous. Given that mixture of experts has also been used in classification settings [22, 31, 53, 34, 35, 60], namely when Y is a discrete response variable, it is desirable to establish a comprehensive theory for parameter estimation under these settings of mixtures of experts.

Third, the theories developed in the paper lay an important foundation for understanding parameter estimation in more complex models, including hierarchical mixture of experts [33, 51, 37, 66] and multigate mixture of experts [44, 26, 45].

Finally, the convergence rates of the MLE in this work are established under the wellspecified settings, namely when the data are drawn from the softmax gating Gaussian mixture of experts. Nevertheless, the convergence analysis of the MLE under the misspecified settings, namely when the data are not necessarily generated from that model, has remained poorly understood. Under those settings, the MLE ̂ G n converges to the mixing measures G ∈ arg min G ∈O k (Θ) KL ( g G ( Y | X ) , p ( Y | X )) where p ( Y | X ) is the true conditional density function of Y given X , and it is not a softmax gating Gaussian mixture of experts. Additionally, the notation KL stands for the Kullback-Leibler divergence. The insights from our theories under the well-specified setting indicate that the Voronoi loss functions can be used to obtain the precise rates of individual parameters of the MLE ̂ G n to those of the mixing measure G .

References

A. Anandkumar, D. Hsu, and S. M. Kakade. A method of moments for mixture models and hidden markov models. In COLT , 2012.

S. Balakrishnan, M. J. Wainwright, and B. Yu. Statistical guarantees for the EM algorithm: From population to sample-based analysis. Annals of Statistics , 45:77-120, 2017.

H. Bao, W. Wang, L. Dong, Q. Liu, O.-K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei. VLMo: Unified vision-language pre-training with mixture-of-modality-experts. In Advances in Neural Information Processing Systems , 2022.

S. Boyd and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

F. Chamroukhi and B. T. Huynh. Regularized Maximum-Likelihood Estimation of Mixtureof-Experts for Regression and Clustering. In 2018 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2018.

F. Chamroukhi and B.-T. Huynh. Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models. Journal de la Société Française de Statistique , 160(1):57-85, 2019.

F. Chamroukhi, A. Samé, G. Govaert, and P. Aknin. Time series modeling by a regression approach based on a latent process. Neural Networks , 22(5-6):593-602, 2009. Publisher: Elsevier.

F. Chamroukhi, A. Samé, G. Govaert, and P. Aknin. A hidden process regression model for functional data description. Application to curve discrimination. Neurocomputing , 73(7-9):12101221, 2010.

J. H. Chen. Optimal rate of convergence for finite mixture models. Annals of Statistics , 23(1):221-233, 1995.

K. Chen, L. Xu, and H. Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural Networks , 12(9):1229-1252, 1999.

Y. Chen, X. Yi, and C. Caramanis. A convex formulation for mixed regression with two components: Minimax optimal rates. In COLT , 2014.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, Sept. 1977.

D. Do, L. Do, and X. Nguyen. Strong identifiability and parameter learning in regression with heterogeneous response. arXiv preprint arXiv:2212.04091 , 2022.

T. G. Do, H. K. Le, T. Nguyen, Q. Pham, B. T. Nguyen, T.-N. Doan, C. Liu, S. Ramasamy, X. Li, and S. HOI. HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Singapore, Dec. 2023. Association for Computational Linguistics.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

N. Doss, Y. Wu, P. Yang, and H. H. Zhou. Optimal estimation of high-dimensional Gaussian location mixtures. The Annals of Statistics , 51(1):62 - 95, 2023. Publisher: Institute of Mathematical Statistics.

N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In ICML , 2022.

R. Dwivedi, N. Ho, K. Khamaru, M. J. Wainwright, M. I. Jordan, and B. Yu. Sharp analysis of expectation-maximization for weakly identifiable models. AISTATS , 2020.

R. Dwivedi, N. Ho, K. Khamaru, M. J. Wainwright, M. I. Jordan, and B. Yu. Singularity, misspecification, and the convergence rate of EM. Annals of Statistics , 44:2726-2755, 2020.

D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in a deep mixture of experts. In ICLR Workshops , 2014.

W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research , 23:1-39, 2022.

I. C. Gormley and T. B. Murphy. A mixture of experts model for rank data with applications in election studies. The Annals of Applied Statistics , 2(4):1452 - 1477, 2008. Publisher: Institute of Mathematical Statistics.

P. J. Green. Iteratively Reweighted Least Squares for Maximum Likelihood Estimation, and some Robust and Resistant Alternatives. Journal of the Royal Statistical Society. Series B (Methodological) , 46(2):149-192, 1984. Publisher: [Royal Statistical Society, Wiley].

A. Guha, N. Ho, and X. Nguyen. On posterior contraction of parameters and interpretability in Bayesian mixture modeling. Bernoulli , 27(4):2159-2188, 2021.

M. Hardt and E. Price. Tight bounds for learning a mixture of two gaussians. In STOC , 2015.

H. Hazimeh, Z. Zhao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, R. Mazumder, L. Hong, and E. Chi. DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 29335-29347. Curran Associates, Inc., 2021.

P. Heinrich and J. Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. The Annals of Statistics , 46(6):2844-2870, 2018.

N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistics , 44:2726-2755, 2016.

N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics , 10:271-307, 2016.

N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence rates for Gaussian mixtures of experts. Journal of Machine Learning Research , 23(323):1-81, 2022.

B. T. Huynh and F. Chamroukhi. Estimation and feature selection in mixtures of generalized linear experts models. arXiv preprint arXiv:1907.06994 , 2019.

R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation , 3, 1991.

R. A. Jacobs, F. Peng, and M. A. Tanner. A Bayesian Approach to Model Selection in Hierarchical Mixtures-of-Experts Architectures. Neural Networks , 10(2):231-241, 1997.

W. Jiang and M. A. Tanner. Hierarchical Mixtures-of-Experts for Generalized Linear Models: Some Results on Denseness and Consistency. In D. Heckerman and J. Whittaker, editors, Proceedings of the Seventh International Workshop on Artificial Intelligence and Statistics , volume R2 of Proceedings of Machine Learning Research . PMLR, Jan. 1999.

W. Jiang and M. A. Tanner. On the Approximation Rate of Hierarchical Mixtures-of-Experts for Generalized Linear Models. Neural Computation , 11(5):1183-1198, July 1999.

W. Jiang and M. A. Tanner. On the identifiability of mixtures-of-experts. Neural Networks , 9:1253-1258, 1999.

M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation , 6:181-214, 1994.

B. Krishnapuram, L. Carin, M. Figueiredo, and A. Hartemink. Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE Transactions on Pattern Analysis and Machine Intelligence , 27(6):957-968, 2005.

M. Kuusela, T. Vatanen, E. Malmi, T. Raiko, T. Aaltonen, and Y. Nagai. Semi-supervised anomaly detection - towards model-independent searches of new physics. Journal of Physics: Conference Series , 368:012032, 2011.

J. Kwon and C. Caramanis. EM Converges for a Mixture of Many Linear Regressions. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1727-1736. PMLR, Aug. 2020.

J. Kwon, N. Ho, and C. Caramanis. On the minimax optimality of the EM algorithm for learning two-component mixed linear regression. In AISTATS , 2021.

J. Kwon, W. Qian, C. Caramanis, Y. Chen, and D. Davis. Global Convergence of the EM Algorithm for Mixtures of Two Component Linear Regression. In A. Beygelzimer and D. Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2055-2110. PMLR, June 2019.

Q. Li, R. Shi, , and F. Liang. Drug sensitivity prediction with high-dimensional mixture regression. PLoS ONE , 2019.

H. Liang, Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, and Z. Wang. M 3 ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with ModelAccelerator Co-design. In NeurIPS , 2022.

J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi. Modeling Task Relationships in Multi-Task Learning with Multi-Gate Mixture-of-Experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '18, pages 1930-1939, New York, NY, USA, 2018. Association for Computing Machinery. event-place: London, United Kingdom.

T. Manole and N. Ho. Uniform convergence rates for maximum likelihood estimation under two-component gaussian mixture models. arXiv preprint arXiv:2006.00704 , 2020.

T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14979-15006. PMLR, 17-23 Jul 2022.

L. Montuelle and E. Le Pennec. Mixture of Gaussian regressions model with logistic weights, a penalized maximum likelihood approach. Electronic Journal of Statistics , 8(1):1661-1695, 2014.

B. Mustafa, C. Ruiz, J. Puigcerver, R. Jenatton, and N. Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. In NeurIPS , 2022.

X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. Annals of Statistics , 4(1):370-400, 2013.

F. Peng, R. A. Jacobs, and M. A. Tanner. Bayesian Inference in Mixtures-of-Experts and Hierarchical Mixtures-of-Experts Models With an Application to Speech Recognition. Journal of the American Statistical Association , 91(435):953-960, 1996.

C. Ruiz, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In NeurIPS , 2021.

M. E. Ruiz and P. Srinivasan. Hierarchical Text Categorization Using Neural Networks. Information Retrieval , 5(1):87-118, Jan. 2002.

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR) , 2017.

B. Sturmfels. Solving Systems of Polynomial Equations . Providence, R.I, 2002.

H. Teicher. On the mixture of distributions. Annals of Statistics , 31:55-73, 1960.

H. Teicher. Identifiability of mixtures. Annals of Statistics , 32:244-248, 1961.

S. van de Geer. Empirical processes in M-estimation . Cambridge University Press, 2000.

C. Villani. Optimal transport: Old and New . Springer, 2008.

S. Waterhouse and A. Robinson. Classification using hierarchical mixtures of experts. In Proceedings of IEEE Workshop on Neural Networks for Signal Processing , pages 177-186, 1994.

Y. Wu and P. Yang. Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics , 48:1987-2007, 2020.

Y. Wu and H. H. Zhou. Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in o ( √ n ) iterations. Mathematical Statistics and Learning , 4:143-220, 2021.

X. Yi, C. Caramanis, and S. Sanghavi. Alternating minimization for mixed linear regression. In ICML , 2014.

Z. You, S. Feng, D. Su, and D. Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech , 2021.

Z. You, S. Feng, D. Su, and D. Yu. Speechmoe2: Mixture-of-experts model with improved routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 7217-7221, 2022.

Y. Zhao, R. Schwartz, J. Sroka, and J. Makhoul. Hierarchical mixtures of experts methodology applied to continuous speech recognition. In Advances in Neural Information Processing Systems , volume 7, 1994.

K. Zhong, P. Jain, and I. S. Dhillon. Mixed linear regression with multiple components. In NeurIPS , 2016.

Y. Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Y . Zhao, A. M. Dai, Z. Chen, Q. V . Le, and J. Laudon. Mixture-of-Experts with Expert Choice Routing. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Supplement to 'Demystifying Softmax Gating Function in Gaussian Mixture of Experts'

In this supplementary material, we present the proofs of Theorems 1 and 2 in Appendix A, and then provide proofs for the remaining results in Appendix B. Finally, we carry out a simulation study to illustrate the various convergence rates that were derived in Theorems 1 and 2 in Appendix C.

A Proofs of Main Results

In this appendix, we provide proof for Theorem 1 in Appendix A.1, while leave that for Theorem 2 in Appendix A.2. Prior to discussing in more detail, let us recall some notations for high-dimensional settings that we will use in our proofs. Firstly, for any vector u := ( u 1 , u 2 , . . . , u d ) ∈ R d and z := ( z 1 , z 2 , . . . , z d ) ∈ N d , we denote u z = u z 1 1 u z 2 2 . . . u z d d , | u | := u 1 + u 2 + . . . + u d and z ! := z 1 ! z 2 ! . . . z d ! . Additionally, 0 d denotes the vector zero in R d , whereas the notation 1 stands for the indicator function. Finally, we denote h 1 ( X,a,b ) = a ⊤ X + b and h 2 ( X,σ ) = σ as mean and variance expert functions in this work for any ( a, b, σ ) ∈ R d × R × R + and X ∈ X .

A.1 Proof of Theorem 1

General Picture : In this proof, we focus mainly on establishing the following bound:

Then, the above bound together with the result of Proposition 2 lead to the conclusion of Theorem 1. Local version : Firstly, we prove the local version of the inequality (14):

Suppose that the inequality in equation (15) does not hold true, then we can find a sequence G n := ∑ k ∗ i =1 exp( β n 0 i ) δ ( β n 1 i ,a n i ,b n i ,σ n i ) ∈ E k ∗ (Θ) such that as n →∞ . Next, we consider the Voronoi cells A n j := A j ( G n ) , for j ∈ [ k ∗ ] , of the mixing measure G n generated by the true components of G ∗ . Since the argument in this proof is asymptotic, we assume without loss of generality (WLOG) that those Voronoi cells are independent of n for all n ∈ N , i.e. A j = A n j . Additionally, since k ∗ is known under the exact-fitted settings and D 1 ( G n , G ∗ ) → 0 , the Voronoi cell A j has only one element for any j ∈ [ k ∗ ] . WLOG, we assume that A j = { j } for all j ∈ [ k ∗ ] , which follows that ( a n j , b n j , σ n j ) → ( a ∗ j , b ∗ j , σ ∗ j ) as n → ∞ . Furthermore, there exist t 1 ∈ R and t 2 ∈ R d independent of n such that exp( β n 0 j ) → exp( β ∗ 0 j + t 1 ) and β n 1 j → β ∗ 1 j + t 2 as n approaches infinity for all j ∈ [ k ∗ ] . It indicates that we can upper bound the Voronoi loss function D 1 as D 1 ( G n , G ∗ ) ≤ D ′ 1 ( G n , G ∗ ) , where

in which

As E X [ V ( g G n ( ·| X ) , g G ∗ ( ·| X ))] / D 1 ( G n , G ∗ ) → 0 when n →∞ , we also obtain that

Step 1: Density Decomposition

Subsequently, we consider Q n := [ ∑ k ∗ j =1 exp(( β ∗ 1 j + t 2 ) ⊤ X + β ∗ 0 j + t 1 )] · [ g G n ( Y | X ) -g G ∗ ( Y | X )] , which can decomposed as

where we denote u ( Y | X ; β 1 , a, b, σ ) := exp( β ⊤ 1 X ) f ( Y | a ⊤ X + b, σ ) and v ( Y | X ; β 1 ) := exp( β ⊤ 1 X ) g G n ( Y | X ) . Next, by means of the first-order Taylor expansion, we rewrite A n as

where R 1 ( X,Y ) is a Taylor remainder such that R 1 ( X,Y ) / D ′ 1 ( G n , G ∗ ) → 0 as n →∞ . Here, the first equality is due to the following partial differential equation for the univariate Gaussian density:

while the second equality is obtained by defining ℓ 1 = α 1 + α 2 , ℓ 2 = | α 2 | + α 3 +2 α 4 and I ℓ 1 ,ℓ 2 := { α = ( α i ) 4 i =1 ∈ N d × N d × N × N : α 1 + α 2 = ℓ 1 , α 3 +2 α 4 = ℓ 2 -| α 2 | } , (18) for all ( ℓ 1 , ℓ 2 ) ∈ N d × N such that 1 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 . Analogously, B n can be rewritten as where R 2 ( X,Y ) is a Taylor remainder such that R 2 ( X,Y ) / D ′ 1 ( G n , G ∗ ) → 0 as n →∞ . From the formulations of A n , B n and E n , we can represent Q n as the following linear combination

with coefficients being denoted by T n ℓ 1 ,ℓ 2 ( j ) and S n γ ( j ) for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 and 0 ≤ | γ | ≤ 1 where

̸

and

Step 2: Non-vanishing coefficients

Now, we will demonstrate by contradiction that at least one among terms of the forms T n ℓ 1 ,ℓ 2 ( j ) / D ′ 1 ( G n , G ∗ ) and S n γ ( j ) / D ′ 1 ( G n , G ∗ ) does not approach zero. Indeed, assume that all of them vanish when n →∞ , then we get

Similarly, by considering the limits of T n ℓ 1 ,ℓ 2 ( j ) / D ′ 1 ( G n , G ∗ ) for all j ∈ [ k ∗ ] and 1 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 , we obtain that

Combine the results in equations (22) and 23, we have 1 = D ′ 1 ( G n , G ∗ ) / D ′ 1 ( G n , G ∗ ) → 0 , which is a contradiction. As a result, not all the limits of T n ℓ 1 ,ℓ 2 ( j ) / D ′ 1 ( G n , G ∗ ) and S n γ ( j ) / D ′ 1 ( G n , G ∗ ) equal to zero.

Step 3: Fatou's lemma involvement

Thus, let m n be the maximum of the absolute values of those terms, we have that 1 /m n ̸→∞ . Then, the Fatou's lemma says that

By assumption, the left-hand side of the above equation equals to zero, therefore, the integrand in the right-hand side also equals to zero for almost surely ( X,Y ) , which leads to the following limit: Q n / [ m n D ′ 1 ( G n , G ∗ )] → 0 as n →∞ for almost surely ( X,Y ) . More specifically, we have

for almost surely ( X,Y ) , where η ℓ 1 ,ℓ 2 ( j ) and ω γ ( j ) are the limits of T n ℓ 1 ,ℓ 2 ( j ) / [ m n D ′ 1 ( G n , G ∗ )] and S n γ ( j ) / [ m n D ′ 1 ( G n , G ∗ )] , respectively, for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 and 0 ≤ | γ | ≤ 1 . Here, at least one among η ℓ 1 ,ℓ 2 ( j ) and ω γ ( j ) is different from zero. On the other hand, since the set

Thus, we reach the local inequality in 15, that is, there exists ε ′ > 0 that satisfies is linearly independent (see Lemma 2 at the end of this proof), we obtain that η ℓ 1 ,ℓ 2 ( j ) = ω γ ( j ) = 0 for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 and 0 ≤ | γ | ≤ 1 , which is a contradiction.

̸

Global version : Thus, it suffices to prove its following global inequality:

Assume by contrary that there exists a sequence G ′ n ∈ E k ∗ (Θ) that satisfies

Therefore, we obtain that E X [ V ( g G ′ n ( ·| X ) , g G ∗ ( ·| X ))] → 0 as n →∞ . Since the set Θ is compact, we are able to replace the sequence G ′ n by its subsequence which converges to some mixing measure G ′ ∈ E k ∗ (Θ) such that D ( G ′ , G ∗ ) > ε ′ . Then, by the Fatou's lemma, we get

which implies that

Thus, we obtain that g G ′ ( Y | X ) = g G ∗ ( Y | X ) for almost surely ( X,Y ) . Now that the softmax gating Gaussian mixture of experts is identifiable up to a translation (see Proposition 1), the mixing measure G ′ admits the form G ′ = ∑ k ∗ i =1 exp( β ∗ 0 τ ( i ) + t 1 ) δ ( β ∗ 1 τ ( i ) + t 2 ,a ∗ τ ( i ) ,b ∗ τ ( i ) ,σ ∗ τ ( i ) ) for some t 1 ∈ R and t 2 ∈ R d , where τ is some permutation of the set { 1 , 2 , . . . , k } . This leads to the fact that D 1 ( G ′ , G ∗ ) = 0 , which contradicts the hypothesis D 1 ( G ′ , G ∗ ) > ε ′ > 0 . Hence, we obtain the inequality in equation (14).

To complete the proof, we will show the previous claim regarding the independence of elements in W 1 in the following lemma:

Lemma 2. The set W 1 defined in equation (25) is linearly independent w.r.t X and Y .

Proof of Lemma 2. Assume that the following holds for almost surely ( X,Y ) :

where η ℓ 1 ,ℓ 2 ( j ) ∈ R and ω γ ( j ) ∈ R . Then, we need to show that η ℓ 1 ,ℓ 2 ( j ) = ω γ ( j ) = 0 , for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 and 0 ≤ | γ | ≤ 1 . The above equation is equivalent to

for almost surely ( X,Y ) . Since β ∗ 11 , . . . , β ∗ 1 k ∗ are k ∗ distinct values, we get that the set { exp(( β ∗ 1 j + t 2 ) ⊤ X ) : j ∈ [ k ∗ ] } is linearly independent, which implies that

for all j ∈ [ k ∗ ] for almost surely ( X,Y ) . Obviously, the above equation is a polynomial of X ∈ X , where X is a compact subset of R d . Then, we achieve that

for all j ∈ [ k ∗ ] and ζ ∈ { 0 , 1 } , for almost surely ( X,Y ) . Again, as ( a ∗ j , b ∗ j , σ ∗ j ) for j ∈ [ k ∗ ] are k ∗ distinct tuples, we have that (( a ∗ j ) ⊤ X + b ∗ j , σ ∗ j ) for j ∈ [ k ∗ ] are also k ∗ distinct tuples for almost surely X . Therefore, { ∂ ℓ 2 f ∂h ℓ 2 1 ( Y | ( a ∗ j ) ⊤ X + b ∗ j , σ ∗ j ) , g G ∗ ( Y | X ) } is a linearly independent set. As a result, η ℓ 1 ,ℓ 2 ( j ) = ω γ ( j ) = 0 for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 and 0 ≤ | γ | ≤ 1 . Hence, the proof is completed.

A.2 Proof of Theorem 2

In this proof, we adapt the framework in Appendix A.1 to the setting of Theorem 2. However, since the arguments utilized for the global version part remain the same (up to some changes of notations) for the over-fitted settings, they will not be presented here again. Thus, we focus only on proving the following local inequality:

Assume that the above claim is not true, then there exists a sequence of mixing measures G n := ∑ k n i =1 exp( β n 0 i ) δ ( β n 1 i ,a n i ,b n i ,σ n i ) ∈ O k (Θ) such that when n tends to infinity. Since the proof argument is asymptotic, we also assume that k n = k ′ for all n ≥ 1 . Following the proof argument of Theorem 1 in Appendix A.1, we also assume that the Voronoi cells A j = A n j does not change with n for all j ∈ [ k ∗ ] . Additionally, since D 2 ( G n , G ∗ ) → 0 , we have ( a n i , b n i , σ n i ) → ( a ∗ j , b ∗ j , σ ∗ j ) for any i ∈ A j as n approaches infinity. Furthermore, there exist t 1 ∈ R and t 2 ∈ R d such that ∑ i ∈A j exp( β n 0 i ) → exp( β ∗ 0 j + t 1 ) and β n 1 i → β ∗ 1 j + t 2 for any i ∈ A j and j ∈ [ k ∗ ] . Then, we can upper bound the Voronoi loss function D 2 as D 2 ( G n , G ∗ ) ≤ D ′ 2 ( G n , G ∗ ) , where

D ′ 2 ( G n , G ∗ ) := ∑ j : |A j | > 1 ∑ i ∈A j exp( β n 0 i ) ( ∥ (∆ t 2 β n 1 ij , ∆ b n ij ) ∥ ¯ r ( |A j | ) + ∥ (∆ a n ij , ∆ σ n ij ) ∥ ¯ r ( |A j | ) / 2 ) + ∑ j : |A j | =1 ∑ i ∈A j exp( β n 0 i ) ∥ (∆ t 2 β 1 ij , ∆ a n ij , ∆ b n ij , ∆ σ n ij ) ∥ + k ∗ ∑ j =1 ∣ ∣ ∣ ∑ i ∈A j exp( β n 0 i ) -exp( β ∗ 0 j + t 1 ) ∣ ∣ ∣ , in which

Recall that E X [ V ( g G n ( ·| X ) , g G ∗ ( ·| X ))] / D 2 ( G n , G ∗ ) → 0 as n →∞ , which leads to E X [ V ( g G n ( ·| X ) , g G ∗ ( ·| X ))] / D ′ 2 ( G n , G ∗ ) → 0 .

Step 1: Density Decomposition

In this step, we decompose the quantity Q n = [ ∑ k ∗ j =1 exp(( β ∗ 1 j + t 2 ) ⊤ X + β ∗ 0 j + t 1 )] · [ g G n ( Y | X ) -g G ∗ ( Y | X )] with abuse of notations in Appendix A.1 as follows:

where we denote u ( Y | X ; β 1 , a, b, σ ) := exp( β ⊤ 1 X ) f ( Y | a ⊤ X + b, σ ) and v ( Y | X ; β 1 ) := exp( β ⊤ 1 X ) g G n ( Y | X ) . Since each Voronoi cell A j possibly has more than one element, we continue to decompose A n and B n as follows:

and

Now, we apply the first-order Taylor expansions to two terms A n, 1 and B n, 1 as in equations (17) and (19), while for A n, 2 and B n, 2 , we use the Taylor expansions of orders ¯ r ( |A j | ) and 2 , respectively, for each j : |A j | > 1 as follows:

As a result, Q n can be represented as

Here, I ℓ 1 ,ℓ 2 := { α = ( α i ) 4 i =1 ∈ N d × N d × N × N : α 1 + α 2 = ℓ 1 , α 3 +2 α 4 = ℓ 2 -| α 2 | } for any ( ℓ 1 , ℓ 2 ) ∈ N 2 such that 1 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2¯ r ( |A j | ) , while R 3 ( X,Y ) and R 4 ( X,Y ) are Taylor remainders such that R p ( X,Y ) / D ′ 2 ( G n , G ∗ ) → 0 when n →∞ for p ∈ { 3 , 4 } .

with coefficients T n ℓ 1 ,ℓ 2 ( j ) and S n γ ( j ) being defined for any j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2 and 0 ≤ | γ | ≤ 1 as

̸

̸

Step 2: Non-vanishing coefficients

Next, we will show that not all the quantities T n ℓ 1 ,ℓ 2 ( j ) / D ′ 2 ( G n , G ∗ ) and S n γ ( j ) / D ′ 2 ( G n , G ∗ ) go to 0 as n →∞ . Assume that all of them vanish when n tends to infinity. Then, by arguing similarly as in equations (22) and (23), we obtain that

Putting the above limit and the formulation of D 2 ( G n , G ∗ ) together, we deduce that

which indicates that there exists some index j ∗ ∈ [ k ∗ ] such that |A j ∗ | > 1 and

for all t 2 ∈ R d . Without loss of generality, we may assume that j ∗ = 1 . Recall that for ( ℓ 1 , ℓ 2 ) ∈ N d × N such that 1 ≤ | ℓ 1 | + ℓ 2 ≤ ¯ r ( |A 1 | ) , we have T n ℓ 1 ,ℓ 2 (1) / D ′ 2 ( G n , G ∗ ) → 0 as n →∞ . Thus, by dividing this ratio and the left hand side of the above equation and let t 2 = 0 , we obtain that

Let us define M n := max {∥ ∆ t 2 β n 1 i 1 ∥ , ∥ ∆ a n i 1 ∥ 1 / 2 , | ∆ b n i 1 | , | ∆ σ n i 1 | 1 / 2 : i ∈ A 1 } and β n := max i ∈A 1 exp( β n 0 i ) . Note that the sequence exp( β n 0 i ) /β n is bounded, therefore, we can replace it by its subsequence that has a positive limit p 2 5 i := lim n →∞ exp( β n 0 i ) /β n . Thus, at least one among p 2 5 i , for i ∈ A 1 , equals 1.

for all ( ℓ 1 , ℓ 2 ) such that 1 ≤ | ℓ 1 | + ℓ 2 ≤ ¯ r ( |A 1 | ) .

In addition, we also define

Here, at least one of p 1 i , p 2 i , p 3 i and p 4 i for i ∈ A 1 equals either 1 or -1 . Next, we divide both the numerator and the denominator of the ratio in equation (29) by β n M ℓ 1 + ℓ 2 n , and then achieve the following system of polynomial equations:

for all ( ℓ 1 , ℓ 2 ) ∈ N d × N such that 1 ≤ | ℓ 1 | + ℓ 2 ≤ ¯ r ( |A 1 | ) . However, based on the definition of ¯ r ( |A 1 | ) , the above system has no non-trivial solutions, which is a contradiction. Thus, not all the quantities T n ℓ 1 ,ℓ 2 ( j ) / D ′ 2 ( G n , G ∗ ) and S n γ ( j ) / D ′ 2 ( G n , G ∗ ) go to 0 as n →∞ .

Step 3: Fatou's lemma involvement

Subsequently, we denote by m n be the maximum of the absolute values of those quantities. Based on the result in Step 2, we know that 1 /m n ̸→ ∞ . Then, by applying the Fatou's lemma as in equation (24), we get that Q n / [ m n D ′ 2 ( G n , G ∗ )] → 0 as n →∞ for almost surely ( X,Y ) . It follows

from the decomposition of Q n in equation (28) that

for almost surely ( X,Y ) , where η ℓ 1 ,ℓ 2 ( j ) and ω γ ( j ) denote the limits of T n ℓ 1 ,ℓ 2 ( j ) / [ m n D ′ 2 ( G n , G ∗ )] and S n γ ( j ) / [ m n D ′ 2 ( G n , G ∗ )] as n →∞ , respectively, for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2¯ r ( |A j | ) and 0 ≤ | γ | ≤ 1 + 1 {|A j | > 1 } . By definition, at least one among η ℓ 1 ,ℓ 2 ( j ) and ω γ ( j ) is different from zero. Nevertheless, as the set

for all j ∈ [ k ∗ ] , 0 ≤ 2 | ℓ 1 | + ℓ 2 ≤ 2¯ r ( |A j | ) and 0 ≤ | γ | ≤ 1 + 1 {|A j | > 1 } , which is a contradiction. Hence, we achieve the inequality in equation (27), and complete the proof.

B Proofs of Auxiliary Results

In this appendix, we provide proofs for the results of Proposition 1 and Proposition 2 in Appendix B.1 and Appendix B.2, respectively, while we leave that for Lemma 1 in Appendix B.3.

B.1 Proof of Proposition 1

Given the notations in Proposition 1, assume that the equation g G ( Y | X ) = g G ′ ( Y | X ) holds true, that is,

{ exp(( β 1 i ) ⊤ X + β 0 i ) ∑ k j =1 exp(( β 1 j ) ⊤ X + β 0 j ) : i ∈ [ k ] } ≡ { exp(( β ′ 1 i ) ⊤ X + β ′ 0 i ) ∑ k j =1 exp(( β ′ 1 j ) ⊤ X + β ′ 0 j ) : i ∈ [ k ] } , for almost surely X . For simplicity, we may assume that for almost surely ( X,Y ) . Then, it follows from the identifiability of the location-scale Gaussian mixtures [56, 57] that the number of components and the weight set of the mixing measure G equal to those of its counterpart G ′ , i.e. k = k ′ and

for all i ∈ [ k ] . Since the softmax function is invariant to translation, we get that β 0 i = β ′ 0 i + t 1 and β 1 i = β ′ 1 i + t 2 for some t 1 ∈ R and t 2 ∈ R d . Therefore, equation (31) reduces to

for almost surely ( X,Y ) , where u ( Y | X ; β 1 , a, b, σ ) := exp( β ⊤ 1 X ) f ( Y | a ⊤ X + b, σ ) for all i ∈ [ k ] . Next, we will partition the index set [ k ] into q subsets U 1 , U 2 , . . . , U q such that for each ℓ ∈ [ q ] , we have exp( β 0 i ) = exp( β 0 i ′ ) for any i, i ′ ∈ U ℓ . As a result, equation (32) can be rewritten as

for almost surely ( X,Y ) . Given the above equation, for each ℓ ∈ [ q ] , we obtain that for almost surely X , which directly leads to

WLOG, we assume that ( a i , b i , σ i ) = ( a ′ i , b ′ i , σ ′ i ) for all i ∈ U ℓ . Consequently,

or equivalently, G ≡ G ′ t 1 ,t 2 . Hence, the proof is completed.

B.2 Proof of Proposition 2

Our proof will be based on the convergence rates of density estimation from MLE in Theorem 7.4 in [58]. Before stating this result here, let us introduce some necessary notations. Firstly, let P k (Θ) be the set of conditional densities of all mixing measures in O k (Θ) , i.e., P k (Θ) := { g G ( Y | X ) : G ∈ O k (Θ) } . Additionally, we define

Next, for each δ > 0 , the Hellinger ball centered around the conditional density g G ∗ ( Y | X ) and intersected with the set P 1 / 2 k (Θ) is denoted by

Finally, in order to measure the size of the above set, [58] proposes using the following quantity:

Theorem 3 (Theorem 7.4, [58]) . Take Ψ( δ ) ≥ J B ( δ, ˜ P 1 / 2 k (Θ , δ )) that satisfies Ψ( δ ) /δ 2 is a nonincreasing function of δ . Then, for some universal constant c and for some sequence ( δ n ) such that √ nδ 2 n ≥ c Ψ( δ n ) , we achieve that where H B ( t, ˜ P 1 / 2 k (Θ , t ) , ∥ · ∥ ) denotes the bracketing entropy [58] of ˜ P 1 / 2 k (Θ , u ) under the 2 -norm, and t ∨ δ := max { t, δ } . Now, we are ready to recall the statement of Theorem 7.4 in [58]:

for all δ ≥ δ n .

The proof of this theorem can be seen in [58].

Proof of Proposition 2. Back to our main proof, since

for any t > 0 , it follows from equation (33) that

where we apply the upper bound of a bracketing entropy in Lemma 3 (cf. the end of this proof) in the second inequality. Let Ψ( δ ) = δ · [log(1 /δ )] 1 / 2 , we have Ψ( δ ) /δ 2 is a non-increasing function of θ . Moreover, the above equation deduces that Ψ( δ ) ≥ J B ( δ, ˜ P 1 / 2 k (Θ , δ )) . Additionally, let δ n = √ log( n ) /n , we have that √ nδ 2 n ≥ c Ψ( δ n ) for some universal constant c . As all the assumptions are met, Theorem 3 gives us for some universal constant C that depends only on Θ .

For completion, we will provide the result regarding the upper bound of a bracketing entropy in the following lemma:

Lemma 3. Assume that Θ is a bounded set, then the following inequality holds true for any 0 ≤ ε ≤ 1 / 2 :

Proof of Lemma 3. Firstly, we will establish an upper bound for the univariate Gaussian density f ( Y | a ⊤ X + b, σ ) . Since both X and Θ are bounded sets, there exist positive constants κ, u, ℓ such that -κ ≤ a ⊤ X + b ≤ κ and ℓ ≤ σ ≤ u . As a result,

For any | Y | ≥ 2 κ , we have that ( Y -h 1 ) 2 2 h 2 ≥ Y 2 8 u , which leads to

Subsequently, let η ≤ ε , we assume that the set P k (Θ) has an η -cover (under ℓ 1 -norm) denoted by { π 1 , . . . , π N } , where N := N ( η, P k (Θ) , ∥· ∥ 1 ) is known as the η -covering number of P k (Θ) . Then, we will build up the brackets of the form [ ν i ( Y | X ) , µ i ( Y | X )] for all i ∈ [ N ] as follows:

Putting the above results together, we obtain that f ( Y | a ⊤ X + b, σ ) ≤ K ( Y | X ) , where we define K ( Y | X ) := 1 √ 2 πℓ exp ( -Y 2 8 u ) if | Y | ≥ 2 κ , and K ( Y | X ) := 1 √ 2 πℓ otherwise.

Consequently, it can be checked that P k (Θ) ⊂ ⋃ N i =1 [ ν i ( Y | X ) , µ i ( Y | X )] with a note that µ i ( Y | X ) -ν i ( Y | X ) ≤ min { 2 η, K ( Y | X ) } . Next, for each i ∈ [ N ] , we attempt to give an upper bound for where R := max { 2 κ, √ 8 u } log(1 /η ) and R ′ is some positive constant. By definition of the bracketing entropy, since H B ( R ′ η, P k (Θ) , ∥ · ∥ 1 ) is the logarithm of the smallest number of brackets of size R ′ η necessary to cover P k (Θ) , we achieve that

Assume that the following upper bound for the covering number log N ( η, P k (Θ) , ∥ · ∥ 1 ) ≲ log(1 /η ) holds true (proof is provided below), then the above result leads to

By selecting η = ε/R ′ , we receive that H B ( ε, P k (Θ) , ∥ · ∥ 1 ) ≲ log(1 /ε ) . Furthermore, since the Hellinger distance is upper bounded by the L 1 -norm, we reach the desired conclusion:

Upper bound of the covering number. For completion, we will establish the following upper bound for the covering number, i.e.,

For each mixing measure G = ∑ k i =1 exp( β 0 i ) δ ( β 1 i ,a i ,b i ,σ i ) ∈ O k (Θ) , we consider another one denoted by ˜ G := ∑ k i =1 exp( β 0 i ) δ ( β 1 i ,a i ,b i ,σ i ) , where ( a i , b i , σ i ) ∈ Ω η such that ( a i , b i , σ i ) are the closest to ( a i , b i , σ i ) in that set for all i ∈ [ k ] . In addition, we also take into account the following mixing measure G := ∑ k i =1 exp( β 0 i ) δ ( β 1 i ,a i ,b i ,σ i ) , where ( β 0 i , β 1 i ) ∈ ∆ η are the closest to ( β 0 i , β 1 i ) in that set. We can verify that the conditional density g G belongs to the following set:

Let us denote ∆ := { ( β 0 , β 1 ) ∈ R × R d : ( β 0 , β 1 , a, b, σ ) ∈ Θ } and Ω := { ( a, b, σ ) ∈ R d × R × R + : ( β 0 , β 1 , a, b, σ ) ∈ Θ } . Since Θ is a compact set, ∆ and Ω are also compact. Thus, we can find η -covers ∆ η and Ω η for ∆ and Ω , respectively. It can be verified that | ∆ η | ≤ O ( η -( d +1) k ) and | Ω η | ≲ O ( η -( d +2) k ) .

Let us denote Softmax( β ⊤ 1 i X + β 0 i ) := exp( β ⊤ 1 i X + β 0 i ) ∑ k j =1 exp( β ⊤ 1 j X + β 0 j ) . From the formulation of ˜ G , we get the following bounds:

where the second inequality follows from the facts that X is a bounded set. Note that Softmax is a Lipschitz function with Lipschitz constant L ≥ 0 . Additionally, since X is a bounded set, there exists a constant B > 0 such that ∥ X ∥ ≤ B for any X ∈ X . As a result, we get where the second inequality follows from the fact that Softmax is a Lipschitz function and the Gaussian density f ( Y | ( a i ) ⊤ X + b i , σ i ) is bounded. Putting the bounds in equations (34) and (35) together with the triangle inequality, we receive that

which means that R is an η -cover (not necessarily smallest) of the metric space ( P k (Θ) , ∥ · ∥ 1 ) . By definition of the covering number, we know that

which implies that,

Hence, the proof is completed.

B.3 Proof of Lemma 1

First of all, let us recall the system of polynomial equations of interest here:

where I ℓ 1 ,ℓ 2 = { α = ( α 1 , α 2 , α 3 , α 4 ) ∈ N d × N d × N × N : α 1 + α 2 = ℓ 1 , α 3 +2 α 4 = ℓ 2 -| α 2 |} for any ( ℓ 1 , ℓ 2 ) ∈ N d × N such that 0 ≤ | ℓ 1 | ≤ r , 0 ≤ ℓ 2 ≤ r -| ℓ 1 | and | ℓ 1 | + ℓ 2 ≥ 1 .

In this proof, we denote p 1 j = ( p 1 j 1 , p 1 j 2 , . . . , p 1 jd ) and p 2 j = ( p 2 j 1 , p 2 j 2 , . . . , p 2 jd ) .

When m = 2 : By observing a portion of the above system when ℓ 1 = 0 d , which is given by

It follows from Proposition 2.1 in [28] that the smallest natural number r such that the system (37) does not have any non-trivial solutions when m = 2 is r = 4 . It is worth noting that a solution of the system 37 is considered non-trivial in [28] if all the values of p 5 j are different from zero, whereas at least one among p 3 j is non-zero, which aligns with our definition of non-trivial solutions for the system (36). Thus, we get ¯ r ( m ) ≤ 4 , and it suffices to demonstrate that ¯ r ( m ) > 3 . Indeed, when r = 3 , the system in equation (36) can be written as follows:

It can be seen that the following is a non-trivial solution of the above system: p 5 j = 1 , p 1 j = p 2 j = 0 d for all j ∈ [ m ] , p 31 = 1 , p 32 = -1 , p 41 = p 42 = -1 2 . Therefore, we obtain that ¯ r ( m ) > 3 , which leads to ¯ r ( m ) = 4 .

When m = 3 : Note that ¯ r ( m ) is a monotonically increasing function of m . Therefore, it follows from the previous result that ¯ r ( m ) > ¯ r (2) = 4 , or equivalently, ¯ r ( m ) ≥ 5 . Additionally, according to Proposition 2.1 in [28], we deduce that ¯ r ( m ) ≤ 6 based on the reduced system in equation (37). Thus, we only need to show that ¯ r ( m ) > 5 . Indeed, the system (36) when r = 5 is a combination of

the system in equation (38) and the following system:

We can verify that the following is a non-trivial solution of this system:

Hence, we conclude that ¯ r ( m ) = 6 .

C Experiments

In this appendix, we conduct a simulation study to empirically validate our theoretical results on the convergence rates of maximum likelihood estimation (MLE) in the softmax gating Gaussian mixture of experts established in Theorem 1 and Theorem 2.

C.1 Numerical Schemes

We illustrate the heterogeneity convergence rates of the MLE under the softmax gating Gaussian mixture of experts via exact-fitted and over-fitted models, which correspond to the exact-fitted

and over-fitted settings described in Sections 3.1 and 3.2, respectively. For each case, we let X be uniformly distributed over [0 , 1] and we generate observations Y from the conditional density g G ∗ ( Y | X ) of softmax gating Gaussian mixture of experts model in equation (1). Here, the true mixing measure G ∗ = ∑ k ∗ i =1 exp( β ∗ 0 i ) δ ( β ∗ 1 i ,a ∗ i ,b ∗ i ,σ ∗ i ) , where k ∗ = 2 , is specified as follows:

Then, we compute the MLE ̂ G n w.r.t. a number of components k for each sample. For both of these settings, we choose k ∈ { k ∗ , k ∗ +1 , k ∗ +2 } . In order to perform the MLE, we use a numerical scheme based on the EM algorithm [12] similar to the one used by Chamroukhi et al. [7, 8]. Note that the main difference with a classical EM is in the maximization step, as there are no closed formulas for updating the softmax gating parameters ( β 0 i , β 1 i ) , ∀ i ∈ [ k ] . For this purpose, following the results of Chamroukhi et al. [7, 8], see also [38, 10, 23], we use a multi-class iterative reweighted least-squares algorithm. All code for our simulation study below was written in Python 3.9.13 on a standard Unix machine.

We choose the convergence criteria ϵ = 10 -6 and 2000 maximum EM iterations. Our goal is to illustrate the theoretical properties of the estimator ̂ G n . Therefore, we have initialized the EM algorithm in a favourable way. More specifically, we first randomly partitioned the set { 1 , . . . , k } into k ∗ index sets J 1 , . . . , J k ∗ , each containing at least one point, for any given k and k ∗ and for each replication. Finally, we sampled β ∗ 1 j (resp. a ∗ j , b ∗ j , σ ∗ j ) from a unique Gaussian distribution centered on β ∗ 1 t (resp. a ∗ t , b ∗ t , σ ∗ t ), with vanishing covariance so that j ∈ J t .

C.2 Empirical Convergence Rates

The empirical mean of discrepancies D 1 and D 2 between ̂ G n and G ∗ , and the choice of k for exactfitted and over-fitted models are reported in Figures 1 and 2, respectively. It can be observed that the average discrepancies from ̂ G n to G 0 vanish at a rate of O ( n -1 / 2 ) up to a logarithmic factor, as envisaged by Theorems 1 and 2. Although these empirical rates of convergence are similar for the two models, they imply that the convergence behaviour of the individual fitted parameters is very different in an over-fitted setting, which was already discussed in more detail in the Section 3.2.

C.2.1 Exact-fitted Model

We generate 40 samples of size n for each setting, given 200 different choices of sample size n between 10 2 and 10 5 . The empirical parametric convergence rate of the MLE to G ∗ under the metric D 1 in Figure 1 is consistent with the theoretical rates of estimating the true parameters exp( β ∗ 0 i ) , β ∗ 1 i (up to translation), a ∗ i , b ∗ i , σ ∗ i for i ∈ [ k ∗ ] , which are of order O ( n -1 / 2 ) up to logarithmic factors.

C.2.2 Over-fitted Model

In the over-fitted setting, we generate 40 samples of size n for each setting, given 200 different choices of sample size n between n min ≈ 14 ∗ 10 3 for k = k ∗ +1 , n min ≈ 27 ∗ 10 3 for k = k ∗ +2 and n max = 10 5 . To the best of our knowledge, there is still a lack of theoretical understanding of EM performance, in particular an established algorithm that enjoys global convergence for the parameter estimation of the over-fitted softmax gating Gaussian mixture of experts. The most related theoretical results are only for the mixture of expert with covariate-free gating networks in [41, 40, 42]. This explains why in Figure 2 for over-fitted setting, we have not plotted the error bar due to the instability of the EM algorithm for finding the global solution. Moreover, the sample size must be large enough so that the empirical behaviour of the MLE from the EM algorithm matches the theoretical rate of order O ( n -1 / 2 ) up to a logarithmic term.

6

8

10

log(sample size)

-

4

-

3

-

2

-

1

0

1

log(loss)

1.9

n

-

0

.

50049

D

1

(

̂

G

n

, G

∗

)

9

.

5

10

.

0

10

.

5

11

.

0

11

.

5

log(sample size)

-

1

0

1

2

3

log(loss)

4.9

n

-

0

.

4926

D

2

(

̂

G

n

, G

∗

)

(a) Over-fitted,

k

= 3

,

n

min

= 14070

10

.

25

10

.

50

10

.

75

11

.

00

11

.

25

11

.

50

log(sample size)

0

1

2

3

log(loss)

5.7

n

-

0

.

49297

D

2

(

̂

G

n

, G

∗

)

(b) Over-fitted,

k

= 4

,

n

min

= 26733

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the long-standing challenge of parameter estimation in softmax gating Gaussian mixture of experts. The authors identify three theoretical challenges related to softmax gating: (i) the non-uniqueness of parameter estimation, (ii) the interaction between softmax gating and expert functions, and (iii) the complex dependence in the conditional density. They propose Vononoi loss functions to overcome these challenges and establish convergence rates for the maximum likelihood estimator (MLE). Additionally, they discover a connection between the MLE rate and a polynomial equation solvability problem when the number of experts is unknown or over-specified.\n\nSTRENGTHS:\nThe paper proposes a novel solution to a long-standing problem in the literature, addressing three fundamental challenges associated with softmax gating. By introducing Vononoi loss functions and establishing convergence rates for the maximum likelihood estimator, the authors offer a fresh perspective on parameter estimation in these models. The paper contributes to the theoretical understanding of softmax gating Gaussian mixture of experts. It tackles the issues of non-uniqueness, interaction between gating and expert functions, and complex dependence in the conditional density. By resolving these challenges, the authors enhance our knowledge of the underlying principles and dynamics of these models. The paper establishes a connection between the MLE rate and a solvability problem of a system of polynomial equations. This connection adds an interesting aspect to the study, highlighting the potential relationships between statistical estimation and mathematical problem-solving.\n\nWEAKNESSES:\nFirstly, the absence of a comparison or discussion of popular techniques such as Gumbel-softmax [1] and max-propagate [2] raises concerns. These techniques are widely used in probabilistic MOE methods, and their exclusion from the paper may limit its completeness and applicability.\n\nSecondly, the lack of experimental validation is another potential issue. The practical relevance of the proposed methods is claimed by the authors, but without empirical evidence or experiments, it is challenging to assess the true impact and effectiveness of the proposed approach in real-world scenarios.\n\n[1] E. Jang, et al. \"Categorical Reparameterization with Gumbel-Softmax.\" International Conference on Learning Representations (2017).\n\n[2] J. Ren, et al. \"Probabilistic mixture-of-experts for efficient deep reinforcement learning.\" arXiv preprint arXiv:2104.09122 (2021).\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
The authors explore the theoretical complexity of standard Gaussian Mixture of Experts and propose a Vononoi loss which estimates different components of the MLE using different Voronoi cells. 

Update: I have read the rebuttal and the other reviewers' responses, and will keep the score at 7 (accept).

STRENGTHS:
The authors establish versions of the Voronoi loss in multiple scenarios, both in fixed settings and over-fitted settings. 

The Voronoi loss function in both settings is computationally efficient up to a linear order of the number of experts, assuming that the true number of experts is fixed.

WEAKNESSES:
Unfortunately, there does not seem to be any practical exploration of the Voronoi loss in the real world setting to verify the theoretical findings, that should be fine due to the posit of the paper as a more theoretical work.

QUESTIONS:
Minor notes: there seems to be a switching between Vononoi vs Voronoi in the paper. Make sure to standardize them!

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes convergence rates for parameters (Gaussian mean / variance and expert weight) for softmax Gaussian mixture of experts. The rates depend on the setting: exact-fitted vs. over-fitted. The solutions are in terms of Voronoi losses and Hellinger distances.\n\nSTRENGTHS:\n**Originality**\n\nTo the best of my knowledge, this paper is the first to provide convergence rates for softmax Gaussian mixture of experts, making it an original paper. Likewise, the methodology for obtaining this result involves a rather involved setup, which struck me as highly non-trivial, again, supporting claims of originality. Of course, this is not my area of expertise, so it is difficult for me to accurately assess these points.\n\n**Quality**\n\nI believe the quality of the paper is strong. The authors provide a thorough description of the setup for softmax Gaussian mixture of experts, with rigorous proofs for the cases of exact-fitted and over-fitted number of experts. I have not assessed the quality of the proofs in the appendix.\n\n\n**Clarity**\n\nDespite tackling a complex theoretical problem and proposing a fairly involved solution, I found the paper was presented well. Terms are defined clearly, and the authors walk the reader through the main results before diving in to more detail.\n\n**Significance**\n\nThe results seem marginally significant, as the authors claim that they have some practical implications for setting the number of experts and post-processing techniques, like merge-truncate-merge. However, fully demonstrating these practical insights, as well as empirical verification of the convergence rates, are not included in the paper, somewhat limiting their significance.\n\nWEAKNESSES:\nI have several concerns regarding the **significance** of the results.\n\nFirst, these results target the convergence of the softmax mixture of expert parameters, however, in practical settings, I would imagine that these parameters are the outputs of deep networks, which may complicate the convergence picture.\n\nSecond, the convergence results are derived by considering the number of “true” experts, however, it’s unclear to me whether there is a notion of a “true” expert, particularly if the parameterization is used inside of a neural network / model to model a latent variable.\n\nFinally, while I understand that this is an entirely theoretical paper, connecting it more closely to the practical settings in which softmax mixture of experts are used would help to more clearly demonstrate the significance of the paper. For instance, even if it’s in a toy setting, is there any way to empirically verify the convergence rates for exact-fitted vs. over-fitted setups?\n\nRegarding **presentation**, I found the paper overall quite clear given the complexity of the result. However, some diagrams (there are none in the main paper) could possibly fill in any gaps in the reader’s understanding. For instance, diagrams showing the softmax Gaussian mixture of experts setting, as well as other concepts, like Voronoi cells, Hellinger distance, etc., would be useful additions to the paper.\n\nQUESTIONS:\nMy main question is regarding the significance of the result and connecting the result to practical empirical settings — what can be done to make this connection clearer?\n\nLIMITATIONS:\nThe authors have adequately addressed the main limitations of their work in the Discussion section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies convergence rate of Maximum Likelihood Estimate (MLE) of parameters of Gaussian mixture of experts with softmax gating. Unlike some of the previous papers, the authors allow for the gating probabilities to be input dependent. Convergence rate (upper bound) is provided for both the setting where the number of components is known, and when only a lower bound is known. For the latter case, the authors show connection between the MLE convergence rate and solvability of a central system of polynomial equations.\n\nSTRENGTHS:\n* Makes progress on a difficult problem complicated by lack of identifiability and complex interactions between parameters of the gating mechanism and of the individual components. \n\n* Proposes a new \"Voronoi loss\", which lower bound the Hellinger distance, and is crafted to capture properties of the mixture of experts model (the involved Voronoi cells roughly correspond to the subsets of the input space dominated by a particular expert under the ground truth). \n\n* AFAICT provides the first MLE rates for models where the gating mixture is input dependent.\n\nWEAKNESSES:\n* As someone with only cursory understanding of the area (emergency reviewer), the paper was hard to follow and full of technical detail that to me felt like obfuscating rather than facilitating understanding.\n\n* Unclear what can a practitioner take away, except that they should set the number of components as low as possible. (This is a very minor weakness, as I don't think every paper needs to have direct application.)\n\nQUESTIONS:\n* With the problem of identifiability, what happens if we introduce some penalty ($\\ell^1$, $\\ell^2$)? I was surprised that this (perhaps naive) solution was not discussed in the background section or anywhere else.\n\n* Could you make more explicit how the level of separation between individual components of the ground truth impacts the convergence rate? Intuitively, the more separated, the easier the estimation problem should be.\n\n* Potentially related question: What do you mean when you say that a particular Voronoi cell contains only one component of the MLE? Isn't the MLE a random variable? (Wouldn't that imply that the identity of cells which have only one component changes? Also what about the cells which have zero MLE components in them?)\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the problem of parameter estimation for the **softmax-gated mixture of Gaussian experts**. For simplicity, first consider the case in which the mixture weights do not depend on the covariates X. Then, this model can be seen as a mixture of linear regressions where each “regression component” is corrupted by Gaussian noise. The paper considers the much more challenging case where the mixture weights depend on the covariates. The mixing weights are modeled as linear functions which are normalized by applying softmax to the vector of mixing weights.\n\nThe paper proposes a novel \"Voronoi distance\" for the parameter space, which takes into account the invariances of the parametric model, and is used to determine convergence rates for the maximum likelihood estimator (MLE). In the well-specified case where $k$, the number of components of the model used, is equal to $k^*$, the true number of components, the MLE parameters converge at the parametric rate $n^{-1/2}$. In the overparametrized case where $k > k^*$, the rates of the component parameters are slower and depend on how many components of the overparametrized model fall inside the Voronoi cell of the true model’s components.\n\nSTRENGTHS:\n- The paper overcomes significant technical challenges that come from using softmax gates for the mixing weights. The main contribution of the paper is showing that the expected total variation distance of the MLE’s conditional distribution $\\hat{p}(Y|X)$ from the true conditional distribution $p(Y|X)$ is *lower bounded* by the proposed Voronoi distance. The Voronoi distance takes care of the inherent symmetries of the parameter space. For example, the mixing weight parameters can only be identified up to global translations due to the softmax operation.\n- Since one can show that the expected TV distance of the MLE converges at the parametric rate $n^{-1/2}$ by computing the bracketing number of the parametric family [van de Geer, 2000; Theorem 7.2], this leads to an upper bound on the Voronoi distance and thus the $\\ell_2$ distance of the MLE parameters. \n- To be more precise, the paper proposes a family of Voronoi distances, for both the well-specified case and overparameterized case. The convergence rates for overparametrized cases are determined implicitly by a system of polynomial equations which in turn come from analyzing the Taylor expansion of the model’s conditional density w.r.t. the parameters.\n\nWEAKNESSES:\n**Exposition.** Though the technical contributions of this paper are solid, its exposition could be improved to make it more accessible to non-expert readers.\n\n- **System of polynomial equations (Section 1).** The full details of the system of polynomial equations in the introduction is difficult to follow and seems unnecessary since it is reproduced in Section 3. It would be easier and less intimidating for the readers if the authors omitted the full details and simply noted that $r(m)$ is related to whether some system of polynomial equations admits no non-trivial solutions. Then, one could end with “we conjecture r(m) = 2m, and establish this for the first few cases m=1,2,3, in Lemma 1.” It would also be great if the authors could provide more explanation for how one arrives at these polynomial equations.\n- **Computation of the Voronoi loss function (Section 3.2).** This would fit better in the experiment section. In typical estimation settings, we do not have access to the true parameters anyway. This quantity is only computable in synthetic settings where the experimenter knows the true parameters.\n- **Practical implication paragraphs.** These paragraphs in Section 1 and Section 3.2 are redundant.\n\nQUESTIONS:\n- The authors mention in Section 1 that understanding the parameter estimation problem for mixture of experts models has been a long-standing open problem. Could the authors provide explanation for why this was the case? What were the technical barriers and what recent technical advances or breakthroughs have paved the way for the results established here?\n- Which aspects of the Voronoi distances are novel? Assigning the model’s components according to the Voronoi cells of true components seems like a natural thing to do for any mixture model. Is it the \\inf over translations of the mixing weights? Or the rate functions r(m) related to the system of polynomial equations?\n\n**Editorial comment**\n\n- In the introduction, it would motivate the paper’s result more if the authors provided simple and practical examples showing the importance of parameter estimation in this particular mixture of experts model rather than just estimating the density. Perhaps there are examples in which the parameters in the mixture of experts model are interpretable and have practical value?\n- Typo. First paragraph of Section 3.3. Total variation distance should be V not h.\n- It would be helpful to explain what mixing measures when they are first introduced in Section 1. Perhaps after Eq.(1) in the **Problem setting** paragraph.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}