SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- First, our work does not consider the top-K sparse softmax gating function, which has been widely used to scale up massive deep learning architectures [68, 54, 21]. It is practically important to extend the current theories to establish the convergence rates of parameter estimation in the Gaussian mixture of experts with that gating function.
- Second, the paper only takes into account the regression settings, namely when the distribution of  $Y$  is assumed to be continuous. Given that mixture of experts has also been used in classification settings [22, 31, 53, 34, 35, 60], namely when  $Y$  is a discrete response variable, it is desirable to establish a comprehensive theory for parameter estimation under these settings of mixtures of experts.
- Third, the theories developed in the paper lay an important foundation for understanding parameter estimation in more complex models, including hierarchical mixture of experts [33, 51, 37, 66] and multigate mixture of experts [44, 26, 45].
- Finally, the convergence rates of the MLE in this work are established under the well-specified settings, namely when the data are drawn from the softmax gating Gaussian mixture of experts. Nevertheless, the convergence analysis of the MLE under the misspecified settings, namely when the data are not necessarily generated from that model, has remained poorly understood. Under those settings, the MLE  $\widehat{G}_n$  converges to the mixing measures  $\overline{G} \in \arg \min_{G \in \mathcal{O}_k(\Theta)} \mathrm{KL}(g_G(Y|X), p(Y|X))$  where  $p(Y|X)$  is the true conditional density function of  $Y$  given  $X$ , and it is not a softmax gating Gaussian mixture of experts. Additionally, the notation KL stands for the Kullback-Leibler divergence. The insights from our theories under the well-specified setting indicate that the Voronoi loss functions can be used to obtain the precise rates of individual parameters of the MLE  $\widehat{G}_n$  to those of the mixing measure  $\overline{G}$ .
NH acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
[63] X. Yi, C. Caramanis, and S. Sanghavi. Alternating minimization for mixed linear regression. In ICML, 2014.
[64] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech, 2021.
[65] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe2: Mixture-of-experts model with improved routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7217-7221, 2022.
[66] Y. Zhao, R. Schwartz, J. Sroke, and J. Makhoul. Hierarchical mixtures of experts methodology applied to continuous speech recognition. In Advances in Neural Information Processing Systems, volume 7, 1994.
[67] K. Zhong, P. Jain, and I. S. Dhillon. Mixed linear regression with multiple components. In NeurIPS, 2016.
[68] Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Y. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon. Mixture-of-Experts with Expert Choice Routing. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
Supplement to “Demystifying Softmax Gating Function in Gaussian Mixture of Experts”
In this supplementary material, we present the proofs of Theorems 1 and 2 in Appendix A, and then provide proofs for the remaining results in Appendix B. Finally, we carry out a simulation study to illustrate the various convergence rates that were derived in Theorems 1 and 2 in Appendix C.
A Proofs of Main Results
In this appendix, we provide proof for Theorem 1 in Appendix A.1, while leave that for Theorem 2 in Appendix A.2. Prior to discussing in more detail, let us recall some notations for high-dimensional settings that we will use in our proofs. Firstly, for any vector  $u \coloneqq (u_{1}, u_{2}, \ldots, u_{d}) \in \mathbb{R}^{d}$  and  $z \coloneqq (z_{1}, z_{2}, \ldots, z_{d}) \in \mathbb{N}^{d}$ , we denote  $u^{z} = u_{1}^{z_{1}} u_{2}^{z_{2}} \ldots u_{d}^{z_{d}}$ ,  $|u| \coloneqq u_{1} + u_{2} + \ldots + u_{d}$  and  $z! \coloneqq z_{1}! z_{2}! \ldots z_{d}!$ . Additionally,  $\mathbf{0}_{d}$  denotes the vector zero in  $\mathbb{R}^{d}$ , whereas the notation  $\mathbf{1}$  stands for the indicator function. Finally, we denote  $h_{1}(X, a, b) = a^{\top} X + b$  and  $h_{2}(X, \sigma) = \sigma$  as mean and variance expert functions in this work for any  $(a, b, \sigma) \in \mathbb{R}^{d} \times \mathbb{R} \times \mathbb{R}_{+}$  and  $X \in \mathcal{X}$ .
General Picture: In this proof, we focus mainly on establishing the following bound:
$$
\mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] \geq C _ {1} \cdot \mathcal {D} _ {1} (G, G _ {*}). \tag {14}
$$
Then, the above bound together with the result of Proposition 2 lead to the conclusion of Theorem 1.
Local version: Firstly, we prove the local version of the inequality (14):
$$
\lim  _ {\varepsilon \rightarrow 0} \inf  _ {G \in \mathcal {E} _ {k _ {*}} (\Theta): \mathcal {D} _ {1} (G, G _ {*}) \leq \varepsilon} \mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} _ {1} (G, G _ {*}) > 0. \tag {15}
$$
Suppose that the inequality in equation (15) does not hold true, then we can find a sequence  $G_{n} \coloneqq \sum_{i=1}^{k_{*}} \exp(\beta_{0i}^{n}) \delta_{(\beta_{1i}^{n}, a_{i}^{n}, b_{i}^{n}, \sigma_{i}^{n})} \in \mathcal{E}_{k_{*}}(\Theta)$  such that
$$
\mathbb {E} _ {X} \left[ V \left(g _ {G _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} _ {1} \left(G _ {n}, G _ {*}\right)\rightarrow 0,
$$
$$
\mathcal {D} _ {1} (G _ {n}, G _ {*}) \to 0,
$$
as  $n\to \infty$ . Next, we consider the Voronoi cells  $\mathcal{A}_j^n\coloneqq \mathcal{A}_j(G_n)$ , for  $j\in [k_{*}]$ , of the mixing measure  $G_{n}$  generated by the true components of  $G_{*}$ . Since the argument in this proof is asymptotic, we assume without loss of generality (WLOG) that those Voronoi cells are independent of  $n$  for all  $n\in \mathbb{N}$ , i.e.  $\mathcal{A}_j = \mathcal{A}_j^n$ . Additionally, since  $k_{*}$  is known under the exact-fitted settings and  $\mathcal{D}_1(G_n,G_*)\to 0$ , the Voronoi cell  $\mathcal{A}_j$  has only one element for any  $j\in [k_{*}]$ . WLOG, we assume that  $\mathcal{A}_j = \{j\}$  for all  $j\in [k_{*}]$ , which follows that  $(a_j^n,b_j^n,\sigma_j^n)\rightarrow (a_j^*,b_j^*,\sigma_j^*)$  as  $n\to \infty$ . Furthermore, there exist  $t_1\in \mathbb{R}$  and  $t_2\in \mathbb{R}^d$  independent of  $n$  such that  $\exp (\beta_{0j}^{n})\to \exp (\beta_{0j}^{*} + t_{1})$  and  $\beta_{1j}^{n}\rightarrow \beta_{1j}^{*} + t_{2}$  as  $n$  approaches infinity for all  $j\in [k_{*}]$ . It indicates that we can upper bound the Voronoi loss function  $\mathcal{D}_1$  as  $\mathcal{D}_1(G_n,G_*)\leq \mathcal{D}_1'(G_n,G_*)$ , where
$$
\mathcal {D} _ {1} ^ {\prime} (G _ {n}, G _ {*}) := \sum_ {j = 1} ^ {k _ {*}} \left[ \exp (\beta_ {0 j} ^ {n}) \| (\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}, \Delta a _ {j} ^ {n}, \Delta b _ {j} ^ {n}, \Delta \sigma_ {j} ^ {n}) \| + \left| \exp (\beta_ {0 j} ^ {n}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}) \right| \right],
$$
$$
\Delta_ {t _ {2}} \beta_ {1 j} ^ {n} := \beta_ {1 j} ^ {n} - \beta_ {1 j} ^ {*} - t _ {2}, \quad \Delta a _ {j} ^ {n} := a _ {j} ^ {n} - a _ {j} ^ {*},
$$
$$
\Delta b _ {j} ^ {n} := b _ {j} ^ {n} - b _ {j} ^ {*}, \quad \Delta \sigma_ {j} ^ {n} := \sigma_ {j} ^ {n} - \sigma_ {j} ^ {*}.
$$
As  $\mathbb{E}_X[V(g_{G_n}(\cdot |X),g_{G_*(\cdot |X))] / \mathcal{D}_1(G_n,G_*)\to 0$  when  $n\to \infty$  , we also obtain that
$$
\mathbb {E} _ {X} \left[ V \left(g _ {G _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} _ {1} ^ {\prime} \left(G _ {n}, G _ {*}\right)\rightarrow 0.
$$
Step 1: Density Decomposition
Subsequently, we consider  $Q_{n} \coloneqq [\sum_{j=1}^{k_{*}} \exp((\beta_{1j}^{*} + t_{2})^{\top} X + \beta_{0j}^{*} + t_{1})] \cdot [g_{G_{n}}(Y|X) - g_{G_{*}}(Y|X)]$ , which can be decomposed as
$$
\begin{array}{l} Q _ {n} = \sum_ {j = 1} ^ {k _ {*}} \exp \left(\beta_ {0 j} ^ {n}\right) \left[ u \left(Y | X; \beta_ {1 j} ^ {n}, a _ {j} ^ {n}, b _ {j} ^ {n}, \sigma_ {j} ^ {n}\right) - u \left(Y | X; \beta_ {1 j} ^ {*} + t _ {2}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right) \right] \\ - \sum_ {j = 1} ^ {k _ {*}} \exp \left(\beta_ {0 j} ^ {n}\right) \left[ v (Y | X; \beta_ {1 j} ^ {n}) - v (Y | X; \beta_ {1 j} ^ {*} + t _ {2}) \right] \\ + \sum_ {j = 1} ^ {k _ {*}} \Big (\exp (\beta_ {0 j} ^ {n}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}) \Big) \Big [ u (Y | X; \beta_ {1 j} ^ {*} + t _ {2}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}) - v (Y | X; \beta_ {1 j} ^ {*} + t _ {2}) \Big ], \\ := A _ {n} + B _ {n} + E _ {n}, \tag {16} \\ \end{array}
$$
where we denote  $u(Y|X; \beta_1, a, b, \sigma) \coloneqq \exp(\beta_1^\top X) f(Y|a^\top X + b, \sigma)$  and  $v(Y|X; \beta_1) \coloneqq \exp(\beta_1^\top X) g_{G_n}(Y|X)$ . Next, by means of the first-order Taylor expansion, we rewrite  $A_n$  as
$$
\begin{array}{l} A _ {n} = \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \alpha | = 1} \frac {\exp \left(\beta_ {0 j} ^ {n}\right)}{2 ^ {\alpha_ {4}} \alpha !} \left(\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}\right) ^ {\alpha_ {1}} \left(\Delta a _ {i j} ^ {n}\right) ^ {\alpha_ {2}} \left(\Delta b _ {j} ^ {n}\right) ^ {\alpha_ {3}} \left(\Delta \sigma_ {j} ^ {n}\right) ^ {\alpha_ {4}} \\ \times X ^ {\alpha_ {1} + \alpha_ {2}} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) \cdot \frac {\partial^ {| \alpha_ {2} | + \alpha_ {3} + 2 \alpha_ {4}} f}{\partial h _ {1} ^ {| \alpha_ {2} | + \alpha_ {3} + 2 \alpha_ {4}}} (Y | \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) + R _ {1} (X, Y) \\ = \sum_ {j = 1} ^ {k _ {*}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 1} ^ {2} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1}}, \ell_ {2}} \frac {\exp \left(\beta_ {0 j} ^ {n}\right)}{2 ^ {\alpha_ {4}} \alpha !} \left(\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}\right) ^ {\alpha_ {1}} \left(\Delta a _ {j} ^ {n}\right) ^ {\alpha_ {2}} \left(\Delta b _ {j} ^ {n}\right) ^ {\alpha_ {3}} \left(\Delta \sigma_ {j} ^ {n}\right) ^ {\alpha_ {4}} \\ \times X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) \cdot \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} \left(Y \mid \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right) + R _ {1} (X, Y), \tag {17} \\ \end{array}
$$
where  $R_{1}(X,Y)$  is a Taylor remainder such that  $R_{1}(X,Y) / \mathcal{D}_{1}^{\prime}(G_{n},G_{*})\to 0$  as  $n\to \infty$ . Here, the first equality is due to the following partial differential equation for the univariate Gaussian density:
$$
\frac {\partial^ {\alpha_ {4}} f}{\partial h _ {2} ^ {\alpha_ {4}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) = \frac {1}{2 ^ {\alpha_ {4}}} \cdot \frac {\partial^ {2 \alpha_ {4}} f}{\partial h _ {1} ^ {2 \alpha_ {4}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}),
$$
while the second equality is obtained by defining  $\ell_1 = \alpha_1 + \alpha_2$ ,  $\ell_2 = |\alpha_2| + \alpha_3 + 2\alpha_4$  and
$$
\mathcal {I} _ {\ell_ {1}, \ell_ {2}} := \left\{\alpha = \left(\alpha_ {i}\right) _ {i = 1} ^ {4} \in \mathbb {N} ^ {d} \times \mathbb {N} ^ {d} \times \mathbb {N} \times \mathbb {N}: \alpha_ {1} + \alpha_ {2} = \ell_ {1}, \alpha_ {3} + 2 \alpha_ {4} = \ell_ {2} - | \alpha_ {2} | \right\}, \tag {18}
$$
for all  $(\ell_1,\ell_2)\in \mathbb{N}^d\times \mathbb{N}$  such that  $1\leq 2|\ell_1| + \ell_2\leq 2$  . Analogously,  $B_{n}$  can be rewritten as
$$
B _ {n} = - \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \gamma | = 1} \frac {\exp \left(\beta_ {0 j} ^ {n}\right)}{\gamma !} \left(\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}\right) ^ {\gamma} X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) g _ {G _ {n}} (Y | X) + R _ {2} (X, Y), \tag {19}
$$
where  $R_{2}(X,Y)$  is a Taylor remainder such that  $R_{2}(X,Y) / \mathcal{D}_{1}^{\prime}(G_{n},G_{*})\to 0$  as  $n\to \infty$ . From the formulations of  $A_{n}$ ,  $B_{n}$  and  $E_{n}$ , we can represent  $Q_{n}$  as the following linear combination
$$
\begin{array}{l} Q _ {n} = \sum_ {j = 1} ^ {k _ {*}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 0} ^ {2} T _ {\ell_ {1}, \ell_ {2}} ^ {n} (j) \cdot X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} \left(Y \mid \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right) \\ + \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \gamma | = 0} ^ {1} S _ {\gamma} ^ {n} (j) \cdot X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) g _ {G _ {n}} (Y | X) + R _ {1} (X, Y) + R _ {2} (X, Y), \\ \end{array}
$$
with coefficients being denoted by  $T_{\ell_1,\ell_2}^n (j)$  and  $S_{\gamma}^{n}(j)$  for all  $j\in [k_{*}]$ ,  $0\leq 2|\ell_1| + \ell_2\leq 2$  and  $0\leq |\gamma |\leq 1$  where
$$
T _ {\ell_ {1}, \ell_ {2}} ^ {n} (j) = \left\{ \begin{array}{l l} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1}, \ell_ {2}}} \frac {\exp \left(\beta_ {0 j} ^ {n}\right)}{2 ^ {\alpha_ {4}} \alpha !} \left(\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}\right) ^ {\alpha_ {1}} \left(\Delta a _ {j} ^ {n}\right) ^ {\alpha_ {2}} \left(\Delta b _ {j} ^ {n}\right) ^ {\alpha_ {3}} \left(\Delta \sigma_ {j} ^ {n}\right) ^ {\alpha_ {4}}, & (\ell_ {1}, \ell_ {2}) \neq (\mathbf {0} _ {d}, 0), \\ \exp \left(\beta_ {0 j} ^ {n}\right) - \exp \left(\beta_ {0 j} ^ {*} + t _ {1}\right), & (\ell_ {1}, \ell_ {2}) = (\mathbf {0} _ {d}, 0); \end{array} \right. \tag {20}
$$
$$
S _ {\gamma} ^ {n} (j) = \left\{ \begin{array}{l l} - \frac {\exp \left(\beta_ {0 j} ^ {n}\right)}{\gamma !} \left(\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}\right) ^ {\gamma}, & | \gamma | \neq 0, \\ - \exp \left(\beta_ {0 j} ^ {n}\right) + \exp \left(\beta_ {0 j} ^ {*} + t _ {1}\right), & | \gamma | = 0. \end{array} \right. \tag {21}
$$
Step 2: Non-vanishing coefficients
Now, we will demonstrate by contradiction that at least one among terms of the forms  $T_{\ell_1,\ell_2}^n (j) / \mathcal{D}_1'(G_n,G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}_{1}^{\prime}(G_{n},G_{*})$  does not approach zero. Indeed, assume that all of them vanish when  $n\to \infty$ , then we get
$$
\frac {1}{\mathcal {D} _ {1} ^ {\prime} \left(G _ {n} , G _ {*}\right)} \cdot \sum_ {j = 1} ^ {k _ {*}} \left| \exp \left(\beta_ {0 j} ^ {n}\right) - \exp \left(\beta_ {0 j} ^ {*} + t _ {1}\right)\right| = \sum_ {j = 1} ^ {k _ {*}} \frac {\left| T _ {\mathbf {0} _ {d} , 0} ^ {n} (j) \right|}{\mathcal {D} _ {1} ^ {\prime} \left(G _ {n} , G _ {*}\right)} \rightarrow 0. \tag {22}
$$
Similarly, by considering the limits of  $T_{\ell_1,\ell_2}^n (j) / \mathcal{D}_1'(G_n,G_*)$  for all  $j\in [k_{*}]$  and  $1\leq 2|\ell_1| + \ell_2\leq 2$ , we obtain that
$$
\frac {1}{\mathcal {D} _ {1} ^ {\prime} \left(G _ {n} , G _ {*}\right)} \cdot \sum_ {j = 1} ^ {k _ {*}} \exp \left(\beta_ {0 j} ^ {n}\right) \| \left(\Delta_ {t _ {2}} \beta_ {1 j} ^ {n}, \Delta a _ {j} ^ {n}, \Delta b _ {j} ^ {n}, \Delta \sigma_ {j} ^ {n}\right) \| \rightarrow 0. \tag {23}
$$
Combine the results in equations (22) and 23, we have  $1 = \mathcal{D}_1'(G_n,G_*) / \mathcal{D}_1'(G_n,G_*)\to 0$  , which is a contradiction. As a result, not all the limits of  $T_{\ell_1,\ell_2}^n (j) / \mathcal{D}_1'(G_n,G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}_{1}^{\prime}(G_{n},G_{*})$  equal to zero.
Step 3: Fatou's lemma involvement
Thus, let  $m_{n}$  be the maximum of the absolute values of those terms, we have that  $1 / m_{n} \neq \infty$ . Then, the Fatou's lemma says that
$$
\lim  _ {n \rightarrow \infty} \frac {\mathbb {E} _ {X} \left[ V \left(g _ {G _ {n}} (\cdot | X) , g _ {G _ {*}} (\cdot | X)\right)\right]}{m _ {n} \cdot \mathcal {D} _ {1} ^ {\prime} \left(G _ {n} , G _ {*}\right)} \geq \int \lim  _ {n \rightarrow \infty} \inf  \frac {\left| g _ {G _ {n}} (Y | X) - g _ {G _ {*}} (Y | X) \right|}{2 m _ {n} \cdot \mathcal {D} _ {1} ^ {\prime} \left(G _ {n} , G _ {*}\right)} d (X, Y). \tag {24}
$$
By assumption, the left-hand side of the above equation equals to zero, therefore, the integrand in the right-hand side also equals to zero for almost surely  $(X,Y)$ , which leads to the following limit:  $Q_{n} / [m_{n}\mathcal{D}_{1}^{\prime}(G_{n},G_{*})]\to 0$  as  $n\to \infty$  for almost surely  $(X,Y)$ . More specifically, we have
$$
\begin{array}{l} \sum_ {j = 1} ^ {k _ {*}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 0} ^ {2} \eta_ {\ell_ {1}, \ell_ {2}} (j) \cdot X ^ {\ell_ {1}} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) \\ + \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \gamma | = 0} ^ {1} \omega_ {\gamma} (j) \cdot X ^ {\gamma} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) g _ {G _ {*}} (Y | X) = 0, \\ \end{array}
$$
for almost surely  $(X,Y)$ , where  $\eta_{\ell_1,\ell_2}(j)$  and  $\omega_{\gamma}(j)$  are the limits of  $T_{\ell_1,\ell_2}^n (j) / [m_n\mathcal{D}_1'(G_n,G_*)]$  and  $S_{\gamma}^{n}(j) / [m_{n}\mathcal{D}_{1}^{\prime}(G_{n},G_{*})]$ , respectively, for all  $j\in [k_{*}]$ ,  $0\leq 2|\ell_1| + \ell_2\leq 2$  and  $0\leq |\gamma |\leq 1$ . Here, at least one among  $\eta_{\ell_1,\ell_2}(j)$  and  $\omega_{\gamma}(j)$  is different from zero. On the other hand, since the set
$$
\begin{array}{l} \mathcal {W} _ {1}: = \left\{X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} \left(Y \mid \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right): j \in [ k _ {*} ], 0 \leq 2 | \ell_ {1} | + \ell_ {2} \leq 2 \right\} \\ \cup \left\{X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) g _ {G _ {*}} (Y | X): j \in [ k _ {*} ], 0 \leq | \gamma | \leq 1 \right\}, \tag {25} \\ \end{array}
$$
is linearly independent (see Lemma 2 at the end of this proof), we obtain that  $\eta_{\ell_1,\ell_2}(j) = \omega_\gamma (j) = 0$  for all  $j\in [k_{*}]$ $0\leq 2|\ell_1| + \ell_2\leq 2$  and  $0\le |\gamma |\le 1$  , which is a contradiction.
Thus, we reach the local inequality in 15, that is, there exists  $\varepsilon' > 0$  that satisfies
$$
\inf_ {G \in \mathcal {E} _ {k _ {*}} (\Theta): \mathcal {D} _ {1} (G, G _ {*}) \leq \varepsilon^ {\prime}} \mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] / \mathcal {D} _ {1} (G, G _ {*}) > 0.
$$
Global version: Thus, it suffices to prove its following global inequality:
$$
\inf  _ {G \in \mathcal {E} _ {k _ {*}} (\Theta):} \mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] / \mathcal {D} _ {1} \left(G, G _ {*}\right) > 0. \tag {26}
$$
$$
\mathcal {D} _ {1} (G, G _ {*}) > \varepsilon^ {\prime}
$$
Assume by contrary that there exists a sequence  $G_{n}^{\prime} \in \mathcal{E}_{k_{*}}(\Theta)$  that satisfies
$$
\left\{ \begin{array}{l} \lim  _ {n \to \infty} \mathbb {E} _ {X} [ V (g _ {G _ {n} ^ {\prime}} (\cdot | X), g _ {G _ {*}} (\cdot | X)) ] / \mathcal {D} _ {1} (G _ {n} ^ {\prime}, G _ {*}) = 0, \\ \mathcal {D} _ {1} (G _ {n} ^ {\prime}, G _ {*}) > \varepsilon^ {\prime}. \end{array} \right.
$$
Therefore, we obtain that  $\mathbb{E}_X[V(g_{G_n'}(\cdot |X),g_{G_*(\cdot |X))]\to 0$  as  $n\to \infty$ . Since the set  $\Theta$  is compact, we are able to replace the sequence  $G_{n}^{\prime}$  by its subsequence which converges to some mixing measure  $G^{\prime}\in \mathcal{E}_{k_{*}}(\Theta)$  such that  $\mathcal{D}(G',G_{*}) > \varepsilon^{\prime}$ . Then, by the Fatou's lemma, we get
$$
\lim  _ {n \rightarrow \infty} \mathbb {E} _ {X} \left[ V \left(g _ {G _ {n} ^ {\prime}} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] \geq \frac {1}{2} \int \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \left| g _ {G _ {n} ^ {\prime}} (Y | X) - g _ {G _ {*}} (Y | X) \right| d (X, Y),
$$
$$
\int \left| g _ {G ^ {\prime}} (Y | X) - g _ {G _ {*}} (Y | X) \right| \mathrm {d} (X, Y) = 0
$$
Thus, we obtain that  $g_{G'}(Y|X) = g_{G_*(Y|X)}$  for almost surely  $(X,Y)$ . Now that the softmax gating Gaussian mixture of experts is identifiable up to a translation (see Proposition 1), the mixing measure  $G'$  admits the form  $G' = \sum_{i=1}^{k_*} \exp(\beta_{0\tau(i)}^* + t_1) \delta_{(\beta_{1\tau(i)}^* + t_2, a_{\tau(i)}^*, b_{\tau(i)}^*, \sigma_{\tau(i)}^*)}$  for some  $t_1 \in \mathbb{R}$  and  $t_2 \in \mathbb{R}^d$ , where  $\tau$  is some permutation of the set  $\{1, 2, \ldots, k\}$ . This leads to the fact that  $\mathcal{D}_1(G', G_*) = 0$ , which contradicts the hypothesis  $\mathcal{D}_1(G', G_*) > \varepsilon' > 0$ . Hence, we obtain the inequality in equation (14).
To complete the proof, we will show the previous claim regarding the independence of elements in  $\mathcal{W}_1$  in the following lemma:
Lemma 2. The set  $\mathcal{W}_1$  defined in equation (25) is linearly independent w.r.t  $X$  and  $Y$ .
Proof of Lemma 2. Assume that the following holds for almost surely  $(X,Y)$ :
$$
\begin{array}{l} \sum_ {j = 1} ^ {k _ {*}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 0} ^ {2} \eta_ {\ell_ {1}, \ell_ {2}} (j) \cdot X ^ {\ell_ {1}} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) \\ + \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \gamma | = 0} ^ {1} \omega_ {\gamma} (j) \cdot X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) g _ {G _ {*}} (Y | X) = 0, \\ \end{array}
$$
where  $\eta_{\ell_1,\ell_2}(j)\in \mathbb{R}$  and  $\omega_{\gamma}(j)\in \mathbb{R}$ . Then, we need to show that  $\eta_{\ell_1,\ell_2}(j) = \omega_{\gamma}(j) = 0$ , for all  $j\in [k_{*}]$ ,  $0\leq 2|\ell_1| + \ell_2\leq 2$  and  $0\leq |\gamma |\leq 1$ . The above equation is equivalent to
$$
\sum_ {j = 1} ^ {k _ {*}} \sum_ {\zeta = 0} ^ {1} \Big [ \sum_ {\ell_ {2} = 0} ^ {2 - 2 \zeta} \eta_ {\zeta , \ell_ {2}} (j) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) + \omega_ {\zeta} (j) g _ {G _ {*}} (Y | X) \Big ] X ^ {\zeta} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) = 0,
$$
for almost surely  $(X,Y)$ . Since  $\beta_{11}^{*},\ldots,\beta_{1k_{*}}^{*}$  are  $k_{*}$  distinct values, we get that the set  $\{\exp ((\beta_{1j}^{*} + t_{2})^{\top}X):j\in [k_{*}]\}$  is linearly independent, which implies that
$$
\sum_ {\zeta = 0} ^ {1} \Big [ \sum_ {\ell_ {2} = 0} ^ {2 - 2 \zeta} \eta_ {\zeta , \ell_ {2}} (j) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) + \omega_ {\zeta} (j) g _ {G _ {*}} (Y | X) \Big ] X ^ {\zeta} = 0,
$$
for all  $j \in [k_{*}]$  for almost surely  $(X, Y)$ . Obviously, the above equation is a polynomial of  $X \in \mathcal{X}$ , where  $\mathcal{X}$  is a compact subset of  $\mathbb{R}^d$ . Then, we achieve that
$$
\sum_ {\ell_ {2} = 0} ^ {2 - 2 \zeta} \eta_ {\zeta , \ell_ {2}} (j) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) + \omega_ {\zeta} (j) g _ {G _ {*}} (Y | X) = 0,
$$
for all  $j \in [k_{*}]$  and  $\zeta \in \{0,1\}$ , for almost surely  $(X,Y)$ . Again, as  $(a_j^*, b_j^*, \sigma_j^*)$  for  $j \in [k_{*}]$  are  $k_{*}$  distinct tuples, we have that  $((a_j^*)^\top X + b_j^*, \sigma_j^*)$  for  $j \in [k_{*}]$  are also  $k_{*}$  distinct tuples for almost surely  $X$ . Therefore,  $\left\{\frac{\partial^{\ell_2} f}{\partial h_1^{\ell_2}} (Y | (a_j^*)^\top X + b_j^*, \sigma_j^*), g_{G_*(Y|X)}\right\}$  is a linearly independent set. As a result,  $\eta_{\ell_1,\ell_2}(j) = \omega_\gamma(j) = 0$  for all  $j \in [k_{*}]$ ,  $0 \leq 2|\ell_1| + \ell_2 \leq 2$  and  $0 \leq |\gamma| \leq 1$ .
Hence, the proof is completed.