SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huy Nguyen† TrungTin Nguyen\* Nhat Ho†
Department of Statistics and Data Sciences, The University of Texas at Austin†
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
{huynm, minhnhat}@utexas.edu, trung-tin.nguyen@inria.fr
Understanding the parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamental theoretical challenges associated with the softmax gating function: (i) the identifiability only up to the translation of parameters; (ii) the intrinsic interaction via partial differential equations between the softmax gating and the expert functions in the Gaussian density; (iii) the complex dependence between the numerator and denominator of the conditional density of softmax gating Gaussian mixture of experts. We resolve these challenges by proposing novel Voronoi loss functions among parameters and establishing the convergence rates of maximum likelihood estimator (MLE) for solving parameter estimation in these models. When the true number of experts is unknown and over-specified, our findings show a connection between the convergence rate of the MLE and a solvability problem of a system of polynomial equations.
Softmax gating Gaussian mixture of experts [32, 37], a class of statistical machine learning models that combine multiple simpler models, known as expert functions of the covariates, via softmax gating networks to form more complex and accurate models, has found widespread use in various applications, including speech recognition [51, 64, 65], natural language processing [14, 20, 17, 54, 21], computer vision [52, 3, 15, 44], and other applications [26, 49, 7, 8, 48, 5, 6]. Regarding the applications of the softmax gating Gaussian mixture of experts in medicine [43] and physical sciences [39], the parameters of each expert function play an important role in capturing the heterogeneity of data. Thus, the main objective of these works is to conduct statistical inference for those parameters, which leads to a need for convergence rates of parameter estimation in the softmax gating Gaussian mixture of experts. However, a comprehensive theoretical understanding of parameter estimation in that model has still remained a long-standing open problem in the literature.
Parameter estimation has been studied quite extensively in standard mixture models. In his seminal work, Chen et al. [9] established the convergence rate  $\mathcal{O}(n^{-1/4})$  of parameter estimation in over-fitted univariate mixture models, namely, the settings when the number of true components is unknown and over-specified, and the family of distributions is strongly identifiable in the second order, e.g., location Gaussian distributions. That slow and non-standard rate is due to the collapse of some parameters into single parameter or the convergence of weights to zero, which leads to the singularity of Fisher information matrix around the true parameters. Then, Nguyen et al. [50] and Ho et al. [29] utilized Wasserstein metrics to achieve this rate under the multivariate settings of second-order strongly identifiable mixture models. Recently, Ho et al. [28] demonstrated that rates of the MLE can strictly depend on the number of over-specified components when the mixture models are not strongly
identifiable, such as location-scale Gaussian mixtures. The minimax optimal behaviors of parameter estimation were studied in [27, 46]. From the computational side, the statistical guarantee of the expectation-maximization (EM), e.g., [12], and moment methods had also been studied under both exact-fitted [2, 1, 25] and over-fitted settings [19, 18, 61, 16, 62] of mixture models.
Compared to mixture models, there has been less research on parameter estimation of mixture of experts. When the gating networks are independent of the covariates, Ho et al. [30] employed the generalized Wasserstein loss function [59] to study the convergence rates of parameter estimation in Gaussian mixture of experts. They proved that these rates are determined by the algebraic independence of the expert functions and the partial differential equations with respect to the parameters. Later, Do et al. [13] extended these results to general mixture of experts with covariate-free gating network. Statistical guarantees of optimization methods for solving parameter estimation in Gaussian mixture of experts with covariate-free gating functions were studied in [11, 67, 41, 63]. When the gating networks are softmax functions, parameter estimation becomes more challenging to understand due to the complex structures of the softmax gating function in the Gaussian mixture of experts. Before describing these phenomena in further details, we begin by formally introducing the softmax gating Gaussian mixture of experts and related notions.
Problem setting: Assume that  $(X_{1},Y_{1}),\ldots ,(X_{n},Y_{n})\in \mathbb{R}^{d}\times \mathbb{R}$  are i.i.d. samples drawn from the softmax gating Gaussian mixture of experts of order  $k_{*}$  whose conditional density function  $g_{G_{*}}(Y|X)$  is given by:
$$
g _ {G _ {*}} (Y | X) := \sum_ {i = 1} ^ {k _ {*}} \frac {\exp \left(\left(\beta_ {1 i} ^ {*}\right) ^ {\top} X + \beta_ {0 i} ^ {*}\right)}{\sum_ {j = 1} ^ {k _ {*}} \exp \left(\left(\beta_ {1 j} ^ {*}\right) ^ {\top} X + \beta_ {0 j} ^ {*}\right)} \cdot f (Y | \left(a _ {i} ^ {*}\right) ^ {\top} X + b _ {i} ^ {*}, \sigma_ {i} ^ {*}), \tag {1}
$$
where  $f(\cdot |\mu ,\sigma)$  is a Gaussian density function with mean  $\mu$  and variance  $\sigma$ . Here, we define  $G_{*}:= \sum_{i = 1}^{k_{*}}\exp (\beta_{0i}^{*})\delta_{(\beta_{1i}^{*},a_{i}^{*},b_{i}^{*},\sigma_{i}^{*})}$  as a true but unknown mixing measure, that is, a combination of Dirac measures  $\delta$  associated with true parameters  $\theta_{i}^{*}\coloneqq (\beta_{0i}^{*},\beta_{1i}^{*},a_{i}^{*},b_{i}^{*},\sigma_{i}^{*})$ . Notably,  $G_{*}$  is not necessarily a probability measure as the summation of its weights can be different from one. For the purpose of the theory, we assume that  $\theta_{i}^{*}\in \Theta \subset \mathbb{R}\times \mathbb{R}^{d}\times \mathbb{R}^{d}\times \mathbb{R}\times \mathbb{R}_{+}$  where  $\Theta$  is a compact set, and  $X\in \mathcal{X}\subset \mathbb{R}^d$  where  $\mathcal{X}$  is a bounded set. Furthermore, we let  $(a_1^*,b_1^*,\sigma_1^*),\ldots ,(a_{k_*}^*,b_{k_*}^*,\sigma_{k_*}^*)$  be pairwise distinct and at least one among  $\beta_{11}^{*},\dots,\beta_{1k_{*}}^{*}$  be non-zero to guarantee the dependence of softmax gating function on the covariate  $X$ . Finally, we assume that the covariate  $X$  follows a continuous distribution to ensure that the softmax gating Gaussian mixture of experts is at least identifiable up to translations (see Proposition 1).
Maximum likelihood estimation. Since the value of true order  $k_{*}$  is unknown in practice, to estimate the unknown parameters in the softmax gating Gaussian mixture of experts (1), we consider using maximum likelihood estimation (MLE) within a class of all mixing measures with at most  $k$  components, which is defined as follows:
$$
\widehat {G} _ {n} \in \underset {G \in \mathcal {O} _ {k} (\Theta)} {\arg \max } \frac {1}{n} \sum_ {i = 1} ^ {n} \log \left(g _ {G} \left(Y _ {i} \mid X _ {i}\right)\right), \tag {2}
$$
where  $\mathcal{O}_k(\Theta) := \{G = \sum_{i=1}^{k'} \exp(\beta_{0i}) \delta_{(\beta_{1i}, a_i, b_i, \sigma_i)} : 1 \leq k' \leq k$  and  $(\beta_{0i}, \beta_{1i}, a_i, b_i, \sigma_i) \in \Theta\}$ . To guarantee that the MLE  $\widehat{G}_n$  is a consistent estimator of  $G_*$ , we need  $k \geq k_*$ . In this paper, we study the convergence rate of the MLE  $\widehat{G}_n$  to the true mixing measure  $G_*$  under both the exact-fitted settings, namely when  $k = k_*$ , and the over-fitted settings, namely when  $k > k_*$ , of the softmax gating Gaussian mixture of experts.
Fundamental challenges from the softmax gating function: There are three fundamental challenges arising from the softmax gating function that create various obstacles in our convergence analysis:
(i) Firstly, parameters  $\beta_{1i}^{*},\beta_{0i}^{*}$  of the softmax gating function are not identifiable as those of the covariate-independent gating function in previous work. Instead, they are identifiable up to translations, that is, the softmax gating value does not change when we translate  $\beta_{0i}^{*}$  to  $\beta_{0i}^{*} + t_{1}$  and  $\beta_{1i}^{*}$  to  $\beta_{1i}^{*} + t_{2}$  for any  $t_1\in \mathbb{R}$  and  $t_2\in \mathbb{R}^d$  . As a consequence, we need to introduce an infimum operator in the Voronoi loss functions (see equations (4) and (6)) to deal with this issue.
(ii) Secondly, a key step in our proof techniques is to decompose the density discrepancy  $g_{\widehat{G}_n}(Y|X) - g_{G_*(Y|X)}$  into a linear combination of linearly independent elements using Taylor expansions.
However, since the numerators and denominators of softmax gating functions are dependent, we cannot apply the Taylor expansions directly to that density discrepancy as in previous work [30, 13]. Moreover, there are two intrinsic interactions between parameters of the softmax gating's numerators and the Gaussian density function via the following partial differential equations (PDEs), which induce a lot of linearly dependent derivative terms in the Taylor expansions:
$$
\frac {\partial^ {2} u}{\partial \beta_ {1} \partial b} = \frac {\partial u}{\partial a}; \quad \frac {\partial^ {2} u}{\partial b ^ {2}} = 2 \frac {\partial u}{\partial \sigma}, \tag {3}
$$
where  $u(Y|X; \beta_1, a, b, \sigma) \coloneqq \exp(\beta_1^\top X) \cdot f(Y|a^\top X + b, \sigma)$ . Therefore, it takes us great effort to group those linearly dependent terms together to obtain the desired linear combination of linearly independent terms.
(iii) Lastly, given the above linear combination of linearly independent elements, when the density estimation  $g_{\widehat{G}_n}(Y|X)$  converges to the true density  $g_{G_*(} (Y|X)$ , the associated coefficients in that combination also go to zero. Then, via some transformations, those limits lead to a system of polynomial equations introduced in equation (9). This system admits a much more complex structure than what considered in previous work [30, 13].
These fundamental challenges from the softmax gating function suggest that the previous loss functions, such as Wasserstein distance [50, 28, 30], being employed to study parameter estimation in standard mixture models or mixture of experts with covariate-free gating functions are no longer sufficient as they heavily rely on the assumptions that the weights of these models are independent of the covariates.
Main contributions: To tackle these challenges of the softmax gating function, we propose two novel Voronoi losses among parameters and establish the lower bounds of the Hellinger distance, denoted as  $h(\cdot ,\cdot)$ , of the mixing densities of softmax gating Gaussian mixture of experts in terms of these Voronoi losses to capture the behaviors of the MLE. Our results can be summarized as follows (see also Table 1):
1. Exact-fitted settings: When  $k = k_*$ , we demonstrate that the Hellinger lower bound  $\mathbb{E}_X[h(g_G(\cdot | X), g_{G_*)}(\cdot | X))] \geq C \cdot \mathcal{D}_1(G, G_*)$  holds for any mixing measure  $G \in \mathcal{O}_k(\Theta)$ , where  $C$  is some universal constant and the Voronoi metric  $\mathcal{D}_1(G, G_*)$  is defined as:
$$
\begin{array}{l} \mathcal {D} _ {1} (G, G _ {*}) := \inf _ {t _ {1}, t _ {2}} \sum_ {j = 1} ^ {k _ {*}} \left[ \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) \| (\Delta_ {t _ {2}} \beta_ {1 i j}, \Delta a _ {i j}, \Delta b _ {i j}, \Delta \sigma_ {i j}) \| \right. \\ \left. + \left| \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i}\right) - \exp \left(\beta_ {0 j} ^ {*} + t _ {1}\right) \right| \right], \tag {4} \\ \end{array}
$$
where  $\Delta_{t_2}\beta_{1ij} := \beta_{1i} - \beta_{1j}^* - t_2$ ,  $\Delta a_{ij} := a_i - a_j^*$ ,  $\Delta b_{ij} := b_i - b_j^*$ ,  $\Delta \sigma_{ij} := \sigma_i - \sigma_j^*$ . The infimum over  $t_1 \in \mathbb{R}$  and  $t_2 \in \mathbb{R}^d$  is to account for the identifiability up to the translation of  $(\beta_{0j}^*, \beta_{1j}^*)_{j=1}^{k_*}$ . Furthermore,  $\mathcal{A}_j$  is a Voronoi cell of mixing measure  $G$  generated by the true component  $\omega_j^* := (\beta_{1j}^*, a_j^*, b_j^*, \sigma_j^*)$  for all  $1 \leq j \leq k_*$  [47], which is defined as follows:
$$
\mathcal {A} _ {j} \equiv \mathcal {A} _ {j} (G) := \{i \in \{1, 2, \dots , k \}: \| \omega_ {i} - \omega_ {j} ^ {*} \| \leq \| \omega_ {i} - \omega_ {\ell} ^ {*} \|, \forall \ell \neq j \}, \tag {5}
$$
where we denote  $\omega_{i} := (\beta_{1i}, a_{i}, b_{i}, \sigma_{i})$ . It is worth noting that the cardinality of each Voronoi cell  $\mathcal{A}_{j}$  indicates the number of components of  $G$  approximating the true component  $\omega_{j}^{*}$  of  $G_{*}$ . As  $\mathbb{E}_X[h(g_{\widehat{G}_n}(\cdot | X), g_{G_*(\cdot | X))}] = \mathcal{O}(n^{-1/2})$ , that lower bound of Hellinger distance indicates that  $\mathcal{D}_1(\widehat{G}_n, G_*) = \mathcal{O}(n^{-1/2})$ . Therefore, the rates of estimating  $\exp(\beta_{0j}^*), \beta_{1j}^*$  (up to translations) and  $a_{j}^*, \beta_{j}^*, \sigma_{j}^*$  are of optimal order  $\mathcal{O}(n^{-1/2})$ .
2. Over-fitted settings: When  $k > k_{*}$ , the lower bound of Hellinger distance in terms of the Voronoi metric  $\mathcal{D}_1$  in the exact-fitted settings is no longer sufficient due to the collapse of softmax of vectors in possibly  $k$  dimensions to softmax of vectors in  $k_{*}$  dimensions. Our approach is to define more
fine-grained Voronoi metric  $\mathcal{D}_2(G,G_*)$  to capture such collapse, which is given by:
$$
\begin{array}{l} \mathcal{D}_{2}(G,G_{*}):= \inf_{t_{1},t_{2}}\sum_{j:|\mathcal{A}_{j}| > 1}\sum_{i\in \mathcal{A}_{j}}\exp (\beta_{0i})\Big(\| (\Delta_{t_{2}}\beta_{1ij},\Delta b_{ij})\|^{\bar{r} (|\mathcal{A}_{j}|)} + \| (\Delta a_{ij},\Delta \sigma_{ij})\|^{\bar{r} (|\mathcal{A}_{j}|) / 2}\Big) \\ + \sum_ {j: | \mathcal {A} _ {j} | = 1} \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i}\right) \| \left(\Delta_ {t _ {2}} \beta_ {1 i j}, \Delta a _ {i j}, \Delta b _ {i j}, \Delta \sigma_ {i j}\right) \| + \sum_ {j = 1} ^ {k _ {*}} \left| \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i}\right) - \exp \left(\beta_ {0 j} ^ {*} + t _ {1}\right) \right|, \tag {6} \\ \end{array}
$$
for any mixing measure  $G \in \mathcal{O}_k(\Theta)$ . Here, the values of function  $\bar{r}(\cdot)$  are determined by the solvability of a system of polynomial equations defined in equation (9). We then show in Lemma 1 that  $\bar{r}(2) = 4$ ,  $\bar{r}(3) = 6$ , and we conjecture that  $\bar{r}(m) = 2m$  for any  $m \geq 2$ .
In high level, the aforementioned system of polynomial equations arises from the PDEs in equation (3) when we establish the lower bound  $\mathbb{E}_X[h(g_G(\cdot |X),g_{G_*(\cdot |X))]\geq C'\mathcal{D}_2(G,G_*)$  for any  $G\in \mathcal{O}_k(\Theta)$  for some universal constant  $C^\prime$ . Since  $\mathbb{E}_X[h(g_{\widehat{G}_n}(\cdot |X),g_{G_*}(\cdot |X))] = \mathcal{O}(n^{-1 / 2})$ , we also have  $\mathcal{D}_2(\widehat{G}_n,G_*) = \mathcal{O}(n^{-1 / 2})$  under the over-fitted settings of the softmax gating Gaussian mixture of experts. As a consequence, the rates for estimating true parameters whose Voronoi cells have only one component of the MLE are of order  $\mathcal{O}(n^{-1 / 2})$ . On the other hand, for true parameters  $\exp (\beta_{0j}^{*}),\beta_{1j}^{*},a_{j}^{*},b_{j}^{*},\sigma_{j}^{*}$  whose Voronoi cells have more than one component of the MLE, the estimation rates are respectively  $\mathcal{O}(n^{-1 / 2\bar{r} (|\mathcal{A}_j|)})$  for  $\beta_{1j}^{*},b_{j}^{*}$ ,  $\mathcal{O}(n^{-1 / \bar{r} (|\mathcal{A}_j|)})$  for  $a_j^*,\sigma_j^*$ , and  $\mathcal{O}(n^{-1 / 2})$  for  $\exp (\beta_{0j}^{*})$ . This rich spectrum of parameter estimation rates is due to the complex interaction between the softmax gating and the expert functions.
Table 1: Summary of density estimation and parameter estimation rates in the softmax gating Gaussian mixture of experts under both the exact-fitted and over-fitted settings. Recall that the cardinality of each Voronoi cell  $\mathcal{A}_j$  gives the number of fitted components approximating true component  $\omega_j^* = (\beta_{1j}^*, a_j^*, b_j^*, \sigma_j^*)$  (see equation (5)). Furthermore, the notation  $\bar{r}(|\mathcal{A}_j|)$  stands for the solvability of the system of polynomial equations (9). For instance, if  $\omega_j^*$  is fitted by two components, then we have  $|\mathcal{A}_j| = 2$  and  $\bar{r}(|\mathcal{A}_j|) = 4$ . Please refer to Lemma 1 for more details of the values of function  $\bar{r}$ .
Setting	Loss Function	gG*(Y|X)	exp(β0j*)	β1j*, bj*	aj*,σj*
Exact-fitted	D1	O(n-1/2)	O(n-1/2)	O(n-1/2)	O(n-1/2)
Over-fitted	D2	O(n-1/2)	O(n-1/2)	O(n-1/2r(|Aj|))	O(n-1/rr(|Aj|))
Practical implications: Although the slow rates of the MLE under the over-fitted settings of the softmax gating Gaussian mixture of experts may seem discouraging, a practical implication of these results is that we should not choose the number of experts  $k$  to be very large compared to the true number of experts  $k_{*}$ . Furthermore, the slow rates can also be useful for post-processing procedures, such as merge-truncate-merge procedure [24], with the MLE to reduce the number of experts so as to consistently estimate  $k_{*}$  when the number of data is sufficiently large. In particular, an important insight from the theoretical results is that we can merge the MLE parameters that are close and within the range of their rates of convergence or truncate the parameters that lead to small weights of the experts. As the sample size becomes sufficiently large, the reduced number of experts may converge to the true number of experts. We leave an investigation of such model selection with the Gaussian mixture of experts via the rates of MLE for future work.
Organization: The paper is organized as follows. In Section 2, we first provide background on the identifiability and rate of conditional density estimation in the softmax gating Gaussian mixture of experts. Next, we proceed to establish the convergence rate of the MLE under both the exact-fitted and over-fitted settings of these models in Section 3. Then, we conclude the paper with a few discussions in Section 4. Finally, full proofs of the results and a simulation study are provided in the Appendices.
Notation: Firstly, we denote  $[n] := \{1, 2, \dots, n\}$  for any positive integer  $n$ . Next, for any vector  $u \in \mathbb{R}^d$  and  $z := (z_1, z_2, \ldots, z_d) \in \mathbb{N}^d$ , we denote  $u^z = u_1^{z_1} u_2^{z_2} \ldots u_d^{z_d}$ ,  $|u| := u_1 + u_2 + \ldots + u_d$  and  $z! := z_1! z_2! \ldots z_d!$ , while  $\|u\|$  represents for its 2-norm value. Additionally, the notation  $|A|$  indicates the cardinality of any set  $A$ . Given any two positive sequences  $\{a_n\}_{n \geq 1}$  and  $\{b_n\}_{n \geq 1}$ , we write  $a_n = \mathcal{O}(b_n)$  or  $a_n \lesssim b_n$  if  $a_n \leq C b_n$  for all  $n \in \mathbb{N}$ , where  $C > 0$  is some universal constant. Lastly, for any two probability density functions  $p, q$  dominated by the Lebesgue measure  $\mu$ , we denote
$h^2 (p,q) = \frac{1}{2}\int (\sqrt{p} -\sqrt{q})^2 d\mu$  as the their squared Hellinger distance and  $V(p,q) = \frac{1}{2}\int |p - q|d\mu$  as their Total Variation distance.
In this section, we begin with the following result on the identifiability of the softmax gating Gaussian mixture of experts, which was previously studied in [36].
Proposition 1 (Identifiability of the softmax gating Gaussian mixture of experts). For any mixing measures  $G = \sum_{i=1}^{k} \exp(\beta_{0i}) \delta_{(\beta_{1i}, a_i, b_i, \sigma_i)}$  and  $G' = \sum_{i=1}^{k'} \exp(\beta_{0i}') \delta_{(\beta_{1i}', a_i', b_i', \sigma_i')}$ , if we have  $g_G(Y|X) = g_{G'}(Y|X)$  for almost surely  $(X,Y)$ , then it follows that  $k = k'$  and  $G \equiv G_{t_1,t_2}'$  where  $G_{t_1,t_2}' := \sum_{i=1}^{k'} \exp(\beta_{0i}' + t_1) \delta_{(\beta_{1i}' + t_2,a_i',b_i',\sigma_i')}$  for some  $t_1 \in \mathbb{R}$  and  $t_2 \in \mathbb{R}^d$ .
Proof of Proposition 1 is in Appendix B.1. The identifiability of the softmax gating Gaussian mixture of experts guarantees that the MLE  $\widehat{G}_n$  (2) converges to the true mixing measure  $G_{*}$  (up to the translation of the parameters in the softmax gating).
Given the consistency of the MLE, it is natural to ask about its convergence rate to the true parameters. Our next result establishes the convergence rate of conditional density estimation  $g_{\widehat{G}_n}(Y|X)$  to the true conditional density  $g_{G_*(}(\mathcal{Y}|X)$ , which lays an important foundation for the study of MLE's convergence rate.
Proposition 2 (Density estimation rate). Given the MLE in equation (2), the conditional density estimation  $g_{\widehat{G}_n}(Y|X)$  has the following convergence rate:
$$
\mathbb {P} (\mathbb {E} _ {X} [ h (g _ {\widehat {G} _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)) ] > C (\log (n) / n) ^ {1 / 2}) \lesssim \exp (- c \log n),
$$
where  $c$  and  $C$  are universal constants.
Proof of Proposition 2 is in Appendix B.2. The result of Proposition 2 indicates that under either the exact-fitted or over-fitted settings of the softmax gating Gaussian mixture of experts, the rate of the conditional density function  $g_{\widehat{G}_n}(Y|X)$  to the true one  $g_{G_*(}Y|X)$  under Hellinger distance is of order  $\mathcal{O}(n^{-1/2})$  (up to some logarithmic factors), which is parametric on the sample size.
From density estimation to parameter estimation: The parametric rate of the conditional density estimation in Proposition 2 suggests that as long as we can establish the Hellinger lower bound  $\mathbb{E}_X[h(g_G(\cdot |X),g_{G_*)}(\cdot |X))]\gtrsim \mathcal{D}(G,G_*)$  for any mixing measure  $G\in \mathcal{O}_k(\Theta)$  for some metric  $\mathcal{D}$  among the parameters, then we obtain directly the parametric convergence rate of the MLE under the metric  $\mathcal{D}$ . Therefore, the main focus of the next section is to determine such metric  $\mathcal{D}$  and to establish that lower bound under either exact-fitted or over-fitted settings of the Gaussian mixture of experts.
3 Convergence Rate of the Maximum Likelihood Estimation
In this section, we first study the convergence rate of the MLE under the exact-fitted settings of the softmax gating Gaussian mixture of experts in Section 3.1. Then, we move to the over-fitted settings in Section 3.2. Finally, we provide a proof sketch of the theories in Section 3.3.
3.1 Exact-fitted Settings
For the exact-fitted settings, namely, when the chosen number of experts  $k$  is equal to the true number of experts  $k_{*}$ , as we mentioned in the introduction, the proper metric between the MLE and the true mixing measure is the metric  $\mathcal{D}_1$  defined in equation (4), which is given by:
$$
\begin{array}{l} \mathcal {D} _ {1} (G, G _ {*}) := \inf _ {t _ {1}, t _ {2}} \sum_ {j = 1} ^ {k _ {*}} \left[ \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) \| (\Delta_ {t _ {2}} \beta_ {1 i j}, \Delta a _ {i j}, \Delta b _ {i j}, \Delta \sigma_ {i j}) \| \right. \\ \left. + \Big | \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}) \Big | \right], \\ \end{array}
$$
where  $\Delta_{t_2}\beta_{1ij} \coloneqq \beta_{1i} - \beta_{1j}^* - t_2$ ,  $\Delta a_{ij} \coloneqq a_i - a_j^*$ ,  $\Delta b_{ij} \coloneqq b_i - b_j^*$ ,  $\Delta \sigma_{ij} \coloneqq \sigma_i - \sigma_j^*$ . Here,  $\mathcal{A}_j$  is a Voronoi cell of  $G$  generated by  $(\beta_{1j}^*, a_j^*, b_j^*, \sigma_j^*)$  for all  $1 \leq j \leq k_*$ . Furthermore, the infimum is taken with respect to  $(t_1, t_2) \in \mathbb{R} \times \mathbb{R}^d$  such that  $\beta_{0j}^* + t_1$  and  $\beta_{1j}^* + t_2$  still lie inside the domain of the parameter space  $\Theta$ .