SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is clear that  $\mathcal{D}_1(G,G_*) = 0$  if and only if  $G\equiv G_{*}$  (up to translation). When  $\mathcal{D}_1(G,G_*)$  is sufficiently small, there exist  $t_1,t_2$  such that all of  $\Delta_{t_2}\beta_{1ij},\Delta a_{ij},\Delta b_{ij},\Delta \sigma_{ij}$ , and  $\sum_{i\in \mathcal{A}_j}\exp (\beta_{0i})-\exp (\beta_{0j}^* +t_1)$  are sufficiently small as well. Therefore, the loss function  $\mathcal{D}_1$  provides a useful metric to measure the difference between the MLE and the true mixing measure. For any fixed  $t_1,t_2$ , the computation of the summations in  $\mathcal{D}_1$  only has the complexity of the order  $\mathcal{O}(k_*^2)$ . To solve the optimization with respect to  $t_1,t_2$  in the metric  $\mathcal{D}_1$ , we can utilize the projected subgradient method with fixed step size [4], which has the complexity of the order  $\mathcal{O}(\varepsilon^{-2})$  as the functions of  $t_1$  and  $t_2$  are convex where  $\varepsilon$  is a desired tolerance. Therefore, the total computational complexity of approximating the value of the Voronoi loss function  $\mathcal{D}_1$  is at the order of  $\mathcal{O}(k_*^2 /\varepsilon^2)$ .
The following result establishes the lower bound of the Hellinger distance between the conditional densities in terms of the loss function  $\mathcal{D}_1$  between corresponding mixing measures, which in turn leads to the convergence rate of the MLE.
Theorem 1. Given the exact-fitted settings of the softmax gating Gaussian mixture of experts (1), i.e.,  $k = k_{*}$ , we find that
$$
\mathbb {E} _ {X} \left[ h \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] \geq C _ {1} \cdot \mathcal {D} _ {1} (G, G _ {*}), \tag {7}
$$
for any  $G \in \mathcal{E}_{k_{*}}(\Theta) \coloneqq \mathcal{O}_{k_{*}}(\Theta) \setminus \mathcal{O}_{k_{*} - 1}(\Theta)$  where  $C_1$  is some universal constant depending only on  $G_{*}$  and  $\Theta$ . As a consequence, there exist universal constants  $C_1'$  and  $c_{1}$  such that the convergence rate of the MLE  $\widehat{G}_n$  under the exact-fitted settings satisfies:
$$
\mathbb {P} \left(\mathcal {D} _ {1} \left(\widehat {G} _ {n}, G _ {*}\right) > C _ {1} ^ {\prime} (\log (n) / n) ^ {1 / 2}\right) \lesssim \exp \left(- c _ {1} \log n\right). \tag {8}
$$
Proof of Theorem 1 is in Appendix A.1. The parametric convergence rate of the MLE to  $G_{*}$  under the metric  $\mathcal{D}_1$  suggests that the rates of estimating the true parameters  $\exp(\beta_{0j}^{*}), \beta_{1j}^{*}$  (up to translation),  $a_j^*, b_j^*, \sigma_j^*$  for  $j \in [k_*]$  are of order  $\mathcal{O}(n^{-1/2})$ , which are optimal up to logarithmic factors.
3.2 Over-fitted Settings
We now consider the over-fitted settings of the softmax gating Gaussian mixture of experts. Different from the exact-fitted settings, the softmax weights associated with the MLE collapse to the softmax weights of the mixture of true experts as long as the MLE approaches the true mixing measure  $G_{*}$ . More concretely, we can relabel the supports of the MLE  $\widehat{G}_n$  with  $\hat{k}_n$  components  $(\hat{k}_n \leq k)$  based on the Voronoi cells  $\mathcal{A}_j^n \coloneqq \mathcal{A}_j(\widehat{G}_n)$  such that we can rewrite it as  $\widehat{G}_n = \sum_{j=1}^{k_*} \sum_{i \in \mathcal{A}_j^n} \exp(\widehat{\beta}_{0i}^n) \delta_{(\widehat{\beta}_{1i}^n, \widehat{a}_i^n, \widehat{b}_i^n, \widehat{\sigma}_i^n)}$  where  $\sum_{j=1}^{k_*} |\mathcal{A}_j^n| = \hat{k}_n$ ,  $(\widehat{a}_i^n, \widehat{b}_i^n, \widehat{\sigma}_i^n) \to (a_j^*, b_j^*, \sigma_j^*)$ ,
$$
\sum_ {i \in \mathcal {A} _ {j} ^ {n}} \frac {\exp \left(\left(\widehat {\beta} _ {1 i} ^ {n}\right) ^ {\top} X + \widehat {\beta} _ {0 i} ^ {n}\right)}{\sum_ {j ^ {\prime} = 1} ^ {k _ {*}} \sum_ {i ^ {\prime} \in \mathcal {A} _ {j ^ {\prime}} ^ {n}} \exp \left(\left(\widehat {\beta} _ {1 i ^ {\prime}} ^ {n}\right) ^ {\top} X + \widehat {\beta} _ {0 i ^ {\prime}} ^ {n}\right)} \rightarrow \frac {\exp \left(\left(\beta_ {1 j} ^ {*}\right) ^ {\top} X + \beta_ {0 j} ^ {*}\right)}{\sum_ {j ^ {\prime} = 1} ^ {k _ {*}} \exp \left(\left(\beta_ {1 j ^ {\prime}} ^ {*}\right) ^ {\top} X + \beta_ {0 j ^ {\prime}} ^ {*}\right)}
$$
as  $n$  approaches infinity for all  $1 \leq i \leq \mathcal{A}_j^n$  and  $j \in [k_*]$ .
The collapse of the softmax weights along with the PDEs (3) between the softmax gating and the expert functions in the Gaussian density create a complex interaction among the estimated parameters. To disentangle such interaction, we rely on the solvability of a novel system of polynomial equations defined in equation (9). In particular, for any  $m \geq 2$ , we define  $\bar{r}(m)$  as the smallest natural number  $r$  such that the following system of polynomial equations:
$$
\sum_ {j = 1} ^ {m} \sum_ {\left(\alpha_ {1}, \alpha_ {2}, \alpha_ {3}, \alpha_ {4}\right) \in \mathcal {I} _ {\ell_ {1}, \varepsilon_ {2}}} \frac {p _ {5 j} ^ {2} p _ {1 j} ^ {\alpha_ {1}} p _ {2 j} ^ {\alpha_ {2}} p _ {3 j} ^ {\alpha_ {3}} p _ {4 j} ^ {\alpha_ {4}}}{\alpha_ {1} ! \alpha_ {2} ! \alpha_ {3} ! \alpha_ {4} !} = 0, \tag {9}
$$
for any  $(\ell_1,\ell_2)\in \mathbb{N}^d\times \mathbb{N}$  such that  $0\leq |\ell_1|\leq r,0\leq \ell_2\leq r - |\ell_1|$  and  $|\ell_1| + \ell_2\geq 1$  , does not have any non-trivial solution for the unknown variables  $\{p_{1j},p_{2j},p_{3j},p_{4j},p_{5j}\}_{j = 1}^{m}$  , namely, all of  $p_{5j}$  are non-zero and at least one among  $p_{3j}$  is different from zero. The ranges of  $\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4}$  in the above
sum satisfy  $\mathcal{I}_{\ell_1,\ell_2} = \{\alpha = (\alpha_1,\alpha_2,\alpha_3,\alpha_4)\in \mathbb{N}^d\times \mathbb{N}^d\times \mathbb{N}\times \mathbb{N}: \alpha_1 + \alpha_2 = \ell_1, |\alpha_2| + \alpha_3 + 2\alpha_4 = \ell_2\}$ . When  $d = 1$  and  $r = 2$ , that system of equations becomes
$$
\begin{array}{l} \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j} = 0, \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j} ^ {2} = 0, \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} (p _ {1 j} p _ {3 j} + p _ {2 j}) = 0, \\ \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {3 j} = 0, \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \Big (\frac {1}{2} p _ {3 j} ^ {2} + p _ {4 j} \Big) = 0. \\ \end{array}
$$
It is clear that we have non-trivial solutions  $p_{5j} = 1$ ,  $p_{1j} = 0$  for all  $j \in [m]$ ,  $|p_{21}| = p_{31} = 1$ ,  $|p_{22}| = p_{32} = -1$ ,  $p_{41} = p_{42} = -1/2$ ,  $p_{2j} = p_{3j} = p_{4j} = 0$  for  $3 \leq j \leq m$ .
When  $d = 1$  and  $r = 3$ , the system of equations can be written as follows:
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j} = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {3 j} = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} (p _ {2 j} + p _ {1 j} p _ {3 j}) = 0,
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j} ^ {2} = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2} p _ {3 j} ^ {2} + p _ {4 j}\right) = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{3 !} p _ {3 j} ^ {3} + p _ {3 j} p _ {4 j}\right) = 0, \\ \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j} ^ {3} = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \Big (\frac {1}{2} p _ {1 j} ^ {2} p _ {3 j} + p _ {1 j} p _ {2 j} \Big) = 0, \\ \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2} p _ {1 j} \cdot p _ {3 j} ^ {2} + p _ {1 j} p _ {4 j} + p _ {2 j} p _ {3 j}\right) = 0. \\ \end{array}
$$
It can be seen that the following is a non-trivial solution of the above system:  $p_{5j} = 1$ ,  $p_{1j} = p_{2j} = 0$  for all  $j \in [m]$ ,  $p_{31} = \frac{\sqrt{3}}{3}$ ,  $p_{32} = -\frac{\sqrt{3}}{3}$ ,  $p_{41} = p_{42} = -\frac{1}{6}$ ,  $p_{3j} = p_{4j} = 0$  for  $3 \leq j \leq m$ . Therefore, we obtain that  $\bar{r}(m) \geq 4$  when  $m \geq 2$  and  $d = 1$ .
In general, when  $d = 1$ , the system of equations has  $(r^2 + 3r) / 2$  equations. Intuitively, when  $m$  is sufficiently larger than  $(r^2 + 3r) / 2$ , the system may not have a non-trivial solution. For general dimension  $d$  and parameter  $m \geq 2$ , finding the exact value of  $\bar{r}(m)$  is a non-trivial central problem in algebraic geometry [55]. When  $m$  is small, the following lemma provides specific values for  $\bar{r}(m)$ .
Lemma 1. For any  $d \geq 1$ , when  $m = 2$ ,  $\bar{r}(m) = 4$ . When  $m = 3$ ,  $\bar{r}(m) = 6$ .
Proof of Lemma 1 is in Appendix B.3. As  $m$  increases, so does the value of  $\bar{r}(m)$ . We conjecture that  $\bar{r}(m) = 2m$  and leave the proof of that conjecture to future work.
By constructing the Voronoi loss function:
$$
\begin{array}{l} \mathcal {D} _ {2} (G, G _ {*}) := \inf  _ {t _ {1}, t _ {2}} \sum_ {j: | \mathcal {A} _ {j} | > 1} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) \Big (\| (\Delta_ {t _ {2}} \beta_ {1 i j}, \Delta b _ {i j}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |)} + \| (\Delta a _ {i j}, \Delta \sigma_ {i j}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |) / 2} \Big) \\ + \sum_ {j: | \mathcal {A} _ {j} | = 1} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) \| (\Delta_ {t _ {2}} \beta_ {1 i j}, \Delta a _ {i j}, \Delta b _ {i j}, \Delta \sigma_ {i j}) \| + \sum_ {j = 1} ^ {k _ {*}} \Big | \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}) \Big |, \\ \end{array}
$$
the following result demonstrates that the convergence rates of the MLE under the over-fitted settings of the softmax gating Gaussian mixture of experts are determined by  $\bar{r} (\cdot)$ .
Theorem 2. Under the over-fitted settings of the softmax gating Gaussian mixture of experts (1), namely, when  $k > k_{*}$ , we obtain that
$$
\mathbb {E} _ {X} \left[ h \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] \geq C _ {2} \cdot \mathcal {D} _ {2} (G, G _ {*}), \tag {10}
$$
for any  $G \in \mathcal{O}_k(\Theta)$  where  $C_2$  is some universal constant depending only on  $G_*$  and  $\Theta$ . Therefore, that lower bound leads to the following convergence rate of the MLE:
$$
\mathbb {P} \left(\mathcal {D} _ {2} \left(\widehat {G} _ {n}, G _ {*}\right) > C _ {2} ^ {\prime} \left(\log (n) / n\right) ^ {1 / 2}\right) \lesssim \exp \left(- c _ {2} \log n\right), \tag {11}
$$
where  $C_2^\prime$  and  $c_{2}$  are some universal constants.
Proof of Theorem 2 is in Appendix A.2. A few comments with the result of Theorem 2 are in order.
(i) Rates of individual parameters: The convergence rate  $\mathcal{O}(n^{-1/2})$  (up to some logarithmic term) of the MLE under the loss function  $\mathcal{D}_2$  implies that for the true parameters  $\exp(\beta_{0j}^*), \beta_{1j}^*, a_j^*, b_j^*, \sigma_j^*$  whose Voronoi cells have only one component of the MLE, the rates for estimating them are  $\mathcal{O}(n^{-1/2})$  up to some logarithmic factor. On the other hand, for true parameters with greater than one component in their Voronoi cells, the rates for estimating  $\beta_{1j}^*$ ,  $b_j^*$  are  $\mathcal{O}(n^{-1/2\bar{r}(|\mathcal{A}_j^n|)})$  while those for  $a_j^*$ ,  $\sigma_j^*$  are  $\mathcal{O}(n^{-1/\bar{r}(|\mathcal{A}_j^n|)})$  (up to logarithmic factors). As the maximum value of  $|\mathcal{A}_j^n|$  is  $\hat{k}_n - k_* + 1$ , it indicates that these rates (up to logarithmic factors) can be as worse as  $\mathcal{O}(n^{-1/\bar{r}(\hat{k}_n - k_* + 1)})$  for estimating  $a_j^*$ ,  $\sigma_j^*$  and  $\mathcal{O}(n^{-1/2\bar{r}(\hat{k}_n - k_* + 1)})$  for estimating  $\beta_{1j}^*$ ,  $b_j^*$ .
(ii) Computation of Voronoi loss function  $\mathcal{D}_2$ : Similar to the Voronoi loss function  $\mathcal{D}_1$  in the exact-fitted setting, the loss function  $\mathcal{D}_2$  is also computationally efficient. In particular, for any fixed  $t_1, t_2$ , the computation of the summations in the formulation of  $\mathcal{D}_2$  is at the order  $\mathcal{O}(k \times k_*)$ , which is linear on  $k$  when  $k_*$  is fixed. Furthermore, we can solve the convex optimization problem with respect to  $t_1, t_2$  with computational complexity at the order of  $\mathcal{O}(\varepsilon^{-2})$  via the projected gradient descent method with fixed step size where  $\varepsilon$  is the error. Therefore, the total computational complexity of approximating the Voronoi loss function  $\mathcal{D}_2$  is at the order of  $\mathcal{O}(k \times k_* / \varepsilon^2)$ .
(iii) Comparison with covariate-free gating network: We would like to remark that the results being established for parameter estimation under the softmax gating network settings of over-fitted Gaussian mixture of experts are in stark difference from those under the covariate-free gating network settings of these models [30], namely, when the gating function is independent of the covariates  $X$ . In particular, Theorem 2 in [30] shows that when the gating networks are independent of the covariates, the convergence rates of estimating  $a_{j}^{*}$  are at the order of  $\mathcal{O}(n^{-1/4})$  (up to some logarithmic factor), which are independent of the number of over-fitted components. It is different from the rates of  $a_{j}^{*}$  whose Voronoi cells have more than one component in the softmax gating settings, which depends on the number of components that we over-fit the Gaussian mixture of experts (see discussion (i) after Theorem 2). Furthermore, the rates of estimating  $b_{j}^{*}$ ,  $\sigma_{j}^{*}$  when the gating networks are independent of covariates are determined by a system of polynomial equations that is much simpler than the system of equations (9) when the gating networks are softmax function. These differences are mainly due to the intrinsic interaction characterized by partial differential equations with respect to the parameters between the softmax gating networks and the expert functions in Gaussian distribution.
3.3 Proof Sketch
In this section, we provide a proof sketch for Theorems 1 and 2. To simplify the ensuing discussion, the loss function  $\mathcal{D}$  in the proof sketch is implicitly understood as either  $\mathcal{D}_1$  or  $\mathcal{D}_2$  depending on the settings of the softmax gating Gaussian mixture of experts. Since the Hellinger distance  $h$  is lower bounded by the Total Variation distance  $V$ , to obtain the bounds in equations (7) and (10), it is sufficient to show that  $\mathbb{E}_X[V(g_G(\cdot |X),g_{G_*(\cdot |X))]\gtrsim \mathcal{D}(G,G_*)$ . To establish this bound, we respectively prove its local and global versions by contradiction as follows:
Local version: In this part, we aim to show the following local inequality:
$$
\lim  _ {\varepsilon \rightarrow 0} \inf  _ {G \in \mathcal {O} _ {k} (\Theta): \mathcal {D} (G, G _ {*}) \leq \varepsilon} \mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} (G, G _ {*}) > 0. \tag {12}
$$
Assume that this claim does not hold true, that is, there exists a sequence  $G_{n} = \sum_{i=1}^{k_{n}} \exp(\beta_{0i}^{n}) \delta_{(\beta_{1i}^{n}, a_{i}^{n}, b_{i}^{n}, \sigma_{i}^{n})} \in \mathcal{O}_{k}(\Theta)$  such that both  $\mathbb{E}_{X}[V(g_{G_{n}}(\cdot|X), g_{G_{*}}(\cdot|X))] / \mathcal{D}(G_{n}, G_{*})$  and  $\mathcal{D}(G_{n}, G_{*})$  approach zero as  $n$  tends to infinity. This implies that for any  $j \in [k_{*}]$ , we have  $\sum_{i \in \mathcal{A}_{j}} \exp(\beta_{0i}^{n}) \to \exp(\beta_{0j}^{*})$  and  $(\beta_{1i}^{n}, a_{i}^{n}, b_{i}^{n}, \sigma_{i}^{n}) \to (\beta_{1j}^{*}, a_{j}^{*}, b_{j}^{*}, \sigma_{j}^{*})$  and for all  $i \in \mathcal{A}_{j}$ . For the sake of presentation, we simplify the loss function  $\mathcal{D}$  by assuming that it is minimized when  $t_{1} = 0$  and  $t_{2} = 0_{d}$ . Now, we decompose the quantity
$$
Q _ {n} = \left[ \sum_ {j ^ {\prime} = 1} ^ {k _ {*}} \exp \left(\left(\beta_ {1 j ^ {\prime}} ^ {*}\right) ^ {\top} X + \beta_ {0 j ^ {\prime}} ^ {*}\right) \right] \cdot \left[ g _ {G _ {n}} (Y | X) - g _ {G _ {*}} (Y | X) \right]
$$
as follows:
$$
\begin{array}{l} Q _ {n} = \sum_ {j = 1} ^ {k _ {*}} \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i} ^ {n}\right) \left[ u \left(Y | X; \beta_ {1 i} ^ {n}, a _ {i} ^ {n}, b _ {i} ^ {n}, \sigma_ {i} ^ {n}\right) - u \left(Y | X; \beta_ {1 j} ^ {*}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right) - v \left(Y | X; \beta_ {1 i} ^ {n}\right) \right. \\ + v (Y | X; \beta_ {1 j} ^ {*}) \biggr ] + \sum_ {j = 1} ^ {k _ {*}} \Big (\sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) - \exp (\beta_ {0 j} ^ {*}) \Big) \Big [ u (Y | X; \beta_ {0 j} ^ {*}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}) - v (Y | X; \beta_ {1 j} ^ {*}) \Big ], \\ \end{array}
$$
where we define  $u(Y|X; \beta_1, a, b, \sigma) \coloneqq \exp(\beta_1^\top X)f(Y|a^\top X + b, \sigma)$  and  $v(Y|X; \beta_1) \coloneqq \exp(\beta_1^\top X)g_{G_n}(Y|X)$ . Next, for each  $j \in [k_*]$  and  $i \in \mathcal{A}_j$ , we denote  $h_1(X, a_j^*, b_j^*) \coloneqq (a_j^*)^\top X + b_j^*$  and then apply the Taylor expansions to the functions  $u(Y|X; \beta_{1i}^n, a_i^n, b_i^n, \sigma_i^n)$  and  $v(Y|X; \beta_{1i}^n)$  up to orders  $r_{1j}$  and  $r_{2j}$  (which we will choose later), respectively, as follows:
$$
\begin{array}{l} u \left(Y | X; \beta_ {1 i} ^ {n}, a _ {i} ^ {n}, b _ {i} ^ {n}, \sigma_ {i} ^ {n}\right) - u \left(Y | X; \beta_ {1 j} ^ {*}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right) \\ = \sum_ {| \ell_ {1} | + \ell_ {2} = 1} ^ {2 r _ {1 j}} T _ {\ell_ {1}, \ell_ {2}} ^ {n} (j) X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*}\right) ^ {\top} X\right) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) + R _ {1 i j} (X, Y), \\ \end{array}
$$
$$
v (Y | X; \beta_ {1 i} ^ {n}) - v (Y | X; \beta_ {1 j} ^ {*}) = \sum_ {| \gamma | = 1} ^ {r _ {2 j}} S _ {\gamma} ^ {n} (j) X ^ {\gamma} \exp ((\beta_ {1 j} ^ {*}) ^ {\top} X) g _ {G _ {n}} (Y | X) + R _ {2 i j} (X, Y),
$$
where  $R_{1ij}(X,Y)$  and  $R_{2ij}(X,Y)$  are Taylor remainders such that  $R_{\rho ij}(X,Y) / \mathcal{D}(G_n,G_*)$  vanishes as  $n\to \infty$  for  $\rho \in \{1,2\}$ . As a result, the limit of  $Q_{n} / \mathcal{D}(G_{n},G_{*})$  when  $n$  goes to infinity can be seen as a linear combination of elements of the following set:
$$
\begin{array}{l} \mathcal {W}: = \left\{X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*}\right) ^ {\top} X\right) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} \left(Y \mid \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right): j \in [ k _ {*} ], 0 \leq 2 | \ell_ {1} | + \ell_ {2} \leq 2 r _ {1 j} \right\} \\ \cup \left\{X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*}\right) ^ {\top} X\right) g _ {G _ {*}} (Y | X): j \in \left[ k _ {*} \right], 0 \leq | \gamma | \leq r _ {2 j} \right\}, \\ \end{array}
$$
which is shown to be linearly independent. By the Fatou's lemma, we demonstrate that  $Q_{n} / \mathcal{D}(G_{n}, G_{*})$  goes to zero as  $n \to \infty$ , implying that all the coefficients in the representation of  $Q_{n} / \mathcal{D}(G_{n}, G_{*})$ , denoted by  $T_{\ell_1, \ell_2}^n(j) / \mathcal{D}(G_n, G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}(G_{n}, G_{*})$ , vanish when  $n \to \infty$ . Given that result, we aim to select the Taylor orders  $r_{1j}$  and  $r_{2j}$  such that at least one among the limits of  $T_{\ell_1, \ell_2}^n(j) / \mathcal{D}(G_n, G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}(G_n, G_*)$  is different from zero, which leads to a contradiction. Hence, we obtain the local version of the desired inequality.
Below are the details of choosing appropriate Taylor orders in each setting.
Exact-fitted settings: Under this setting, since  $k_{*}$  is known, each of the Voronoi cells  $\mathcal{A}_j$  for  $j \in [k_{*}]$  has only one element. Thus, for any  $i \in \mathcal{A}_j$ , we have  $\exp(\beta_{0i}^n) \to \exp(\beta_{0j}^*)$  and  $(\beta_{1i}^n, a_i^n, b_i^n, \sigma_i^n) \to (\beta_{1j}^*, a_j^*, b_j^*, \sigma_j^*)$ . Given that result, we will select  $r_{1j} = r_{2j} = 1$  for all  $j \in [k_{*}]$  as it suffices to show that at least one among the limits of  $T_{\ell_1, \ell_2}^n(j) / \mathcal{D}(G_n, G_*)$  and  $S_\gamma^n(j) / \mathcal{D}(G_n, G_*)$  is different from zero. In particular, if all of them vanished, we would take the sum of all the limits of  $T_{\ell_1, \ell_2}^n(j) / \mathcal{D}(G_n, G_*)$  for  $(\ell_1, \ell_2)$  such that  $0 \leq 2|\ell_1| + \ell_2 \leq 2$ , which leads to a contradiction that  $1 = \mathcal{D}(G_n, G_*) / \mathcal{D}(G_n, G_*) \to 0$ .
Over-fitted settings: As  $k_*$  becomes unknown in this scenario, we need higher Taylor orders to obtain the same result as in the exact-fitted setting. We will reuse the proof by contradiction method to find out those orders. More specifically, assume that all the limits of  $T_{\ell_1,\ell_2}^n (j) / \mathcal{D}(G_n,G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}(G_{n},G_{*})$  equal zero. After some steps of considering typical limits as in the previous setting which requires  $r_{2j} = 2$  for all  $j\in [k_{*}]$ , we encounter the following system of polynomial equations:
$$
\sum_ {i \in \mathcal {A} _ {j}} \sum_ {(\alpha_ {1}, \alpha_ {2}, \alpha_ {3}, \alpha_ {4}) \in \mathcal {I} _ {\ell_ {1}, \ell_ {2}}} \frac {p _ {5 i} ^ {2} p _ {1 i} ^ {\alpha_ {1}} p _ {2 i} ^ {\alpha_ {2}} p _ {3 i} ^ {\alpha_ {3}} p _ {4 i} ^ {\alpha_ {4}}}{\alpha_ {1} ! \alpha_ {2} ! \alpha_ {3} ! \alpha_ {4} !} = 0,
$$
for all  $(\ell_1,\ell_2)\in \mathbb{N}^d\times \mathbb{N}$  such that  $0\leq |\ell_1|\leq r_{1j}$ ,  $0\leq \ell_{2}\leq r_{1j} - |\ell_{1}|$  and  $|\ell_1| + \ell_2\geq 1$  for some  $j\in [k_{*}]$ . Due to the construction of this system, it must have at least one non-trivial
solution. Therefore, if we choose  $r_{1j} = \bar{r} (|\mathcal{A}_j|)$  for all  $j\in [k_{*}]$ , then the above system does not admit any non-trivial solutions, which leads to a contradiction. Hence, we obtain the local inequality in equation (12), which suggests that we can find a positive constant  $\varepsilon^{\prime}$  such that  $\inf_{G\in \mathcal{O}_k(\Theta):\mathcal{D}(G,G_*)\leq \varepsilon '}\mathbb{E}_X[V(g_G(\cdot |X),g_{G_*)}(\cdot |X))] / \mathcal{D}(G,G_*) > 0$
Global version: Therefore, it is sufficient to demonstrate the following global inequality:
$$
\inf  _ {G \in \mathcal {O} _ {k} (\Theta), \mathcal {D} (G, G _ {*}) > \varepsilon^ {\prime}} \mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] / \mathcal {D} (G, G _ {*}) > 0. \tag {13}
$$
Assume that this claim is not true, then we can find a mixing measure  $G' \in \mathcal{O}_k(\Theta)$  such that  $g_{G'}(Y|X) = g_{G_*(Y|X)}$  for almost surely  $(X,Y)$ . According to Proposition 1, we get that  $\mathcal{D}(G',G_*) = 0$ , which contradicts the hypothesis  $\mathcal{D}(G',G_*) > \varepsilon'$ . These arguments hold for both exact-fitted and over-fitted settings up to some changes of notations.
Hence, the proof sketch is completed.
In the paper, we study the convergence rates of parameter estimation under both the exact-fitted and over-fitted settings of the softmax gating Gaussian mixture of experts. We introduce novel Voronoi loss functions among parameters to resolve fundamental theoretical challenges posed by the softmax gating function, including identifiability up to the translation of parameters, the interaction between softmax weights and expert functions, and the dependence between the numerator and denominator of the conditional density function. When the true number of experts is known, we demonstrate that the rates for estimating true parameters are parametric on the sample size. On the other hand, when the true number of experts is unknown and over-specified, these estimation rates turn out to be determined by the solvability of a system of polynomial equations.
There are a few natural directions arising from the paper that we leave for furture work: