SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
H _ {B} (t, \widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta , t), \| \cdot \|) \leq H _ {B} (t, \mathcal {P} _ {k} (\Theta , t), h)
$$
for any  $t > 0$ , it follows from equation (33) that
$$
\mathcal {J} _ {B} (\delta , \widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta , \delta)) \leq \int_ {\delta^ {2} / 2 ^ {1 3}} ^ {\delta} H _ {B} ^ {1 / 2} (t, \mathcal {P} _ {k} (\Theta , t), h)   \mathrm {d} t \vee \delta \lesssim \int_ {\delta^ {2} / 2 ^ {1 3}} ^ {\delta} \log (1 / t) d t \vee \delta ,
$$
where we apply the upper bound of a bracketing entropy in Lemma 3 (cf. the end of this proof) in the second inequality. Let  $\Psi(\delta) = \delta \cdot [\log(1/\delta)]^{1/2}$ , we have  $\Psi(\delta)/\delta^2$  is a non-increasing function of  $\theta$ . Moreover, the above equation deduces that  $\Psi(\delta) \geq \mathcal{I}_B(\delta, \widetilde{\mathcal{P}}_k^{1/2}(\Theta, \delta))$ . Additionally, let  $\delta_n = \sqrt{\log(n)/n}$ , we have that  $\sqrt{n}\delta_n^2 \geq c\Psi(\delta_n)$  for some universal constant  $c$ . As all the assumptions are met, Theorem 3 gives us
$$
\mathbb {P} (\mathbb {E} _ {X} [ h (g _ {\widehat {G} _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)) ] > C (\log (n) / n) ^ {1 / 2}) \lesssim \exp (- c \log (n)),
$$
for some universal constant  $C$  that depends only on  $\Theta$ .
For completion, we will provide the result regarding the upper bound of a bracketing entropy in the following lemma:
Lemma 3. Assume that  $\Theta$  is a bounded set, then the following inequality holds true for any  $0\leq \varepsilon \leq 1 / 2$ :
$$
H _ {B} (\varepsilon , \mathcal {P} _ {k} (\Theta), h) \lesssim \log (1 / \varepsilon).
$$
Proof of Lemma 3. Firstly, we will establish an upper bound for the univariate Gaussian density  $f(Y|a^{\top}X + b,\sigma)$ . Since both  $\mathcal{X}$  and  $\Theta$  are bounded sets, there exist positive constants  $\kappa, u, \ell$  such that  $-\kappa \leq a^{\top}X + b \leq \kappa$  and  $\ell \leq \sigma \leq u$ . As a result,
$$
f (Y | a ^ {\top} X + b, \sigma) = \frac {1}{\sqrt {2 \pi h _ {2}}} \exp \left(- \frac {(Y - h _ {1}) ^ {2}}{2 h _ {2}}\right) \leq \frac {1}{\sqrt {2 \pi \ell}}.
$$
For any  $|Y| \geq 2\kappa$ , we have that  $\frac{(Y - h_1)^2}{2h_2} \geq \frac{Y^2}{8u}$ , which leads to
$$
f (Y | a ^ {\top} X + b, \sigma) \leq \frac {1}{\sqrt {2 \pi \ell}} \exp \left(- \frac {Y ^ {2}}{8 u}\right).
$$
Putting the above results together, we obtain that  $f(Y|a^{\top}X + b,\sigma) \leq K(Y|X)$ , where we define  $K(Y|X) := \frac{1}{\sqrt{2\pi\ell}}\exp \left(-\frac{Y^2}{8u}\right)$  if  $|Y| \geq 2\kappa$ , and  $K(Y|X) := \frac{1}{\sqrt{2\pi\ell}}$  otherwise.
Subsequently, let  $\eta \leq \varepsilon$ , we assume that the set  $\mathcal{P}_k(\Theta)$  has an  $\eta$ -cover (under  $\ell_1$ -norm) denoted by  $\{\pi_1,\ldots ,\pi_N\}$ , where  $N\coloneqq N(\eta ,\mathcal{P}_k(\Theta),\| \cdot \| _1)$  is known as the  $\eta$ -covering number of  $\mathcal{P}_k(\Theta)$ . Then, we will build up the brackets of the form  $[\nu_i(Y|X),\mu_i(Y|X)]$  for all  $i\in [N]$  as follows:
$$
\nu_ {i} (Y | X) := \max  \left\{\pi_ {i} (Y | X) - \eta , 0 \right\},
$$
$$
\mu_ {i} (Y | X) := \max  \left\{\pi_ {i} (Y | X) + \eta , K (Y | X) \right\}.
$$
Consequently, it can be checked that  $\mathcal{P}_k(\Theta) \subset \bigcup_{i=1}^N [\nu_i(Y|X), \mu_i(Y|X)]$  with a note that  $\mu_i(Y|X) - \nu_i(Y|X) \leq \min\{2\eta, K(Y|X)\}$ . Next, for each  $i \in [N]$ , we attempt to give an upper bound for
$$
\begin{array}{l} \| \mu_ {i} - \nu_ {i} \| _ {1} = \int_ {| Y | <   2 \kappa} \left(\mu_ {i} (Y | X) - \nu_ {i} (Y | X)\right) d (X, Y) + \int_ {| Y | \geq 2 \kappa} \left(\mu_ {i} (Y | X) - \nu_ {i} (Y | X)\right) d (X, Y) \\ \leq R \eta + \exp \left(- \frac {R ^ {2}}{2 u}\right) \leq R ^ {\prime} \eta , \\ \end{array}
$$
where  $R \coloneqq \max \{2\kappa, \sqrt{8u}\} \log(1 / \eta)$  and  $R'$  is some positive constant. By definition of the bracketing entropy, since  $H_B(R' \eta, \mathcal{P}_k(\Theta), \| \cdot \|_1)$  is the logarithm of the smallest number of brackets of size  $R' \eta$  necessary to cover  $\mathcal{P}_k(\Theta)$ , we achieve that
$$
H _ {B} \left(R ^ {\prime} \eta , \mathcal {P} _ {k} (\Theta), \| \cdot \| _ {1}\right) \leq \log N = \log N (\eta , \mathcal {P} _ {k} (\Theta), \| \cdot \| _ {1}).
$$
Assume that the following upper bound for the covering number  $\log N(\eta, \mathcal{P}_k(\Theta), \| \cdot \|_1) \lesssim \log (1 / \eta)$  holds true (proof is provided below), then the above result leads to
$$
H _ {B} \left(R ^ {\prime} \eta , \mathcal {P} _ {k} (\Theta), \| \cdot \| _ {1}\right) \lesssim \log (1 / \eta).
$$
By selecting  $\eta = \varepsilon / R'$ , we receive that  $H_B(\varepsilon, \mathcal{P}_k(\Theta), \| \cdot \|_1) \lesssim \log(1 / \varepsilon)$ . Furthermore, since the Hellinger distance is upper bounded by the  $L^1$ -norm, we reach the desired conclusion:
Upper bound of the covering number. For completion, we will establish the following upper bound for the covering number, i.e.,
$$
\log N (\eta , \mathcal {P} _ {k} (\Theta), \| \cdot \| _ {1}) \lesssim \log (1 / \eta).
$$
Let us denote  $\Delta := \{(\beta_0, \beta_1) \in \mathbb{R} \times \mathbb{R}^d : (\beta_0, \beta_1, a, b, \sigma) \in \Theta\}$  and  $\Omega := \{(a, b, \sigma) \in \mathbb{R}^d \times \mathbb{R} \times \mathbb{R}_+ : (\beta_0, \beta_1, a, b, \sigma) \in \Theta\}$ . Since  $\Theta$  is a compact set,  $\Delta$  and  $\Omega$  are also compact. Thus, we can find  $\eta$ -covers  $\Delta_\eta$  and  $\Omega_\eta$  for  $\Delta$  and  $\Omega$ , respectively. It can be verified that  $|\Delta_\eta| \leq O(\eta^{-(d + 1)k})$  and  $|\Omega_\eta| \lesssim O(\eta^{-(d + 2)k})$ .
For each mixing measure  $G = \sum_{i=1}^{k} \exp(\beta_{0i}) \delta_{(\beta_{1i}, a_i, b_i, \sigma_i)} \in \mathcal{O}_k(\Theta)$ , we consider another one denoted by  $\widetilde{G} := \sum_{i=1}^{k} \exp(\beta_{0i}) \delta_{(\overline{a}_i, \overline{a}_i, \overline{b}_i, \overline{\sigma}_i)}$ , where  $(\overline{a}_i, \overline{b}_i, \overline{\sigma}_i) \in \Omega_\eta$  such that  $(\overline{a}_i, \overline{b}_i, \overline{\sigma}_i)$  are the closest to  $(a_i, b_i, \sigma_i)$  in that set for all  $i \in [k]$ . In addition, we also take into account the following mixing measure  $\overline{G} := \sum_{i=1}^{k} \exp(\overline{\beta}_{0i}) \delta_{(\overline{\beta}_{1i}, \overline{a}_i, \overline{b}_i, \overline{\sigma}_i)}$ , where  $(\overline{\beta}_{0i}, \overline{\beta}_{1i}) \in \Delta_\eta$  are the closest to  $(\beta_{0i}, \beta_{1i})$  in that set. We can verify that the conditional density  $g_{\overline{G}}$  belongs to the following set:
$$
\mathcal {R} := \left\{g _ {G} \in \mathcal {P} _ {k} (\Theta): \left(\beta_ {0 i}, \beta_ {1 i}\right) \in \Delta_ {\eta}, \left(a _ {i}, b _ {i}, \sigma_ {i}\right) \in \Omega_ {\eta}, \forall i \in [ k ] \right\}.
$$
Let us denote  $\mathrm{Softmax}(\beta_{1i}^{\top}X + \beta_{0i})\coloneqq \frac{\exp(\beta_{1i}^{\top}X + \beta_{0i})}{\sum_{j = 1}^{k}\exp(\beta_{1j}^{\top}X + \beta_{0j})}$ . From the formulation of  $\widetilde{G}$ , we get the following bounds:
$$
\begin{array}{l} \| g _ {G} - g _ {\widetilde {G}} \| _ {1} \leq \sum_ {i = 1} ^ {k} \int_ {\mathcal {X}} \operatorname {S o f t m a x} \left(\beta_ {1 i} ^ {\top} X + \beta_ {0 i}\right) \left| f (Y | \left(a _ {i}\right) ^ {\top} X + b _ {i}, \sigma_ {i}) - f (Y | (\bar {a} _ {i}) ^ {\top} X + \bar {b} _ {i}, \bar {\sigma} _ {i}) \right| \mathrm {d} X \\ \leq \sum_ {i = 1} ^ {k} \int_ {\mathcal {X}} \left| f (Y | (a _ {i}) ^ {\top} X + b _ {i}, \sigma_ {i}) - f (Y | (\bar {a} _ {i}) ^ {\top} X + \bar {b} _ {i}, \bar {\sigma} _ {i}) \right| d X \\ \lesssim \sum_ {i = 1} ^ {k} \left(\left\| a _ {i} - \bar {a} _ {i} \right\| + \left| b _ {i} - \bar {b} _ {i} \right| + \left| \sigma_ {i} - \bar {\sigma} _ {i} \right|\right) \\ \lesssim \eta , \tag {34} \\ \end{array}
$$
where the second inequality follows from the facts that  $\mathcal{X}$  is a bounded set. Note that Softmax is a Lipschitz function with Lipschitz constant  $L\geq 0$ . Additionally, since  $\mathcal{X}$  is a bounded set, there exists a constant  $B > 0$  such that  $\| X\| \leq B$  for any  $X\in \mathcal{X}$ . As a result, we get
$$
\begin{array}{l} \| g _ {\widetilde {G}} - g _ {\overline {{G}}} \| _ {1} \leq \sum_ {i = 1} ^ {k} \int_ {\mathcal {X}} \left| \operatorname {S o f t m a x} \left(\beta_ {1 i} ^ {\top} X + \beta_ {0 i}\right) - \operatorname {S o f t m a x} \left(\bar {\beta} _ {1 i} ^ {\top} X + \bar {\beta} _ {0 i}\right) \right| f (Y | (\bar {a} _ {i}) ^ {\top} X + \bar {b} _ {i}, \bar {\sigma} _ {i}) \mathrm {d} X \\ \lesssim L \sum_ {i = 1} ^ {k} \int_ {\mathcal {X}} \left(\| \beta_ {1 i} - \bar {\beta} _ {1 i} \| \cdot \| X \| + | \beta_ {0 i} - \bar {\beta} _ {0 i} |\right) d X \\ \leq L k \eta (B + 1), \tag {35} \\ \end{array}
$$
where the second inequality follows from the fact that Softmax is a Lipschitz function and the Gaussian density  $f(Y|(\overline{a}_i)^\top X + \overline{b}_i, \overline{\sigma}_i)$  is bounded. Putting the bounds in equations (34) and (35) together with the triangle inequality, we receive that
$$
\left\| g _ {G} - g _ {\widetilde {G}} \right\| _ {1} \leq \left\| g _ {G} - g _ {\widetilde {G}} \right\| _ {1} + \left\| g _ {\widetilde {G}} - g _ {\overline {{G}}} \right\| _ {1} \lesssim \eta ,
$$
which means that  $\mathcal{R}$  is an  $\eta$ -cover (not necessarily smallest) of the metric space  $(\mathcal{P}_k(\Theta), \| \cdot \|_1)$ . By definition of the covering number, we know that
$$
N (\eta , \mathcal {P} _ {k} (\Theta), \| \cdot \| _ {1}) \leq | \mathcal {R} | = | \Delta_ {\eta} | \times | \Omega_ {\eta} | \leq \mathcal {O} (\eta^ {- (d + 1) k}) \cdot \mathcal {O} (\eta^ {(- d + 2) k}) \leq \mathcal {O} (\eta^ {- (2 d + 3) k}),
$$
which implies that,
$$
\log N (\eta , \mathcal {P} _ {k} (\Theta), \| \cdot \| _ {\infty}) \lesssim \log (1 / \eta).
$$
First of all, let us recall the system of polynomial equations of interest here:
$$
\sum_ {j = 1} ^ {m} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1}, \ell_ {2}}} \frac {p _ {5 j} ^ {2} p _ {1 j} ^ {\alpha_ {1}} p _ {2 j} ^ {\alpha_ {2}} p _ {3 j} ^ {\alpha_ {3}} p _ {4 j} ^ {\alpha_ {4}}}{\alpha_ {1} ! \alpha_ {2} ! \alpha_ {3} ! \alpha_ {4} !} = 0, \tag {36}
$$
where  $\mathcal{I}_{\ell_1,\ell_2} = \{\alpha = (\alpha_1,\alpha_2,\alpha_3,\alpha_4)\in \mathbb{N}^d\times \mathbb{N}^d\times \mathbb{N}\times \mathbb{N}\colon \alpha_1 + \alpha_2 = \ell_1$ $\alpha_{3} + 2\alpha_{4} = \ell_{2} - |\alpha_{2}|.\}$  for any  $(\ell_1,\ell_2)\in \mathbb{N}^d\times \mathbb{N}$  such that  $0\leq |\ell_1|\leq r,0\leq \ell_2\leq r - |\ell_1|$  and  $|\ell_1| + \ell_2\geq 1$
In this proof, we denote  $p_{1j} = (p_{1j1}, p_{1j2}, \ldots, p_{1jd})$  and  $p_{2j} = (p_{2j1}, p_{2j2}, \ldots, p_{2jd})$ .
When  $m = 2$ : By observing a portion of the above system when  $\ell_1 = \mathbf{0}_d$ , which is given by
$$
\sum_ {j = 1} ^ {m} \sum_ {\alpha_ {3} + 2 \alpha_ {4} = \ell_ {2}} \frac {p _ {5 j} ^ {2} p _ {3 j} ^ {\alpha_ {3}} p _ {4 j} ^ {\alpha_ {4}}}{\alpha_ {3} ! \alpha_ {4} !} = 0, \quad \ell_ {2} = 1, 2, \dots , r. \tag {37}
$$
It follows from Proposition 2.1 in [28] that the smallest natural number  $r$  such that the system (37) does not have any non-trivial solutions when  $m = 2$  is  $r = 4$ . It is worth noting that a solution of the system 37 is considered non-trivial in [28] if all the values of  $p_{5j}$  are different from zero, whereas at least one among  $p_{3j}$  is non-zero, which aligns with our definition of non-trivial solutions for the system (36). Thus, we get  $\bar{r}(m) \leq 4$ , and it suffices to demonstrate that  $\bar{r}(m) > 3$ . Indeed, when  $r = 3$ , the system in equation (36) can be written as follows:
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j l} = 0 \forall l \in [ d ], \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {3 j} = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} (p _ {2 j u} + p _ {1 j v} p _ {3 j}) = 0 \forall u, v \in [ d ],
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j u} p _ {1 j v} = 0 \forall u, v \in [ d ], \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2} p _ {3 j} ^ {2} + p _ {4 j}\right) = 0, \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{3 !} p _ {3 j} ^ {3} + p _ {3 j} p _ {4 j}\right) = 0,
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j u} p _ {1 j v} p _ {1 j l} = 0 \forall u, v, l \in [ d ], \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2} p _ {1 j u} p _ {1 j v} p _ {3 j} + p _ {1 j l} p _ {2 j \tau}\right) = 0 \forall u, v, l, \tau \in [ d ], \\ \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2} p _ {1 j u} \cdot p _ {3 j} ^ {2} + p _ {1 j v} p _ {4 j} + p _ {2 j l} p _ {3 j}\right) = 0 \forall u, v, l, \tau \in [ d ]. \tag {38} \\ \end{array}
$$
It can be seen that the following is a non-trivial solution of the above system:  $p_{5j} = 1$ ,  $p_{1j} = p_{2j} = \mathbf{0}_d$  for all  $j \in [m]$ ,  $p_{31} = 1$ ,  $p_{32} = -1$ ,  $p_{41} = p_{42} = -\frac{1}{2}$ . Therefore, we obtain that  $\bar{r}(m) > 3$ , which leads to  $\bar{r}(m) = 4$ .
When  $m = 3$ : Note that  $\bar{r}(m)$  is a monotonically increasing function of  $m$ . Therefore, it follows from the previous result that  $\bar{r}(m) > \bar{r}(2) = 4$ , or equivalently,  $\bar{r}(m) \geq 5$ . Additionally, according to Proposition 2.1 in [28], we deduce that  $\bar{r}(m) \leq 6$  based on the reduced system in equation (37). Thus, we only need to show that  $\bar{r}(m) > 5$ . Indeed, the system (36) when  $r = 5$  is a combination of
the system in equation (38) and the following system:
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} p _ {1 j u} p _ {1 j v} p _ {1 j l} p _ {1 j \tau} = 0 \forall u, v, l, \tau \in [ d ], \quad \sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{4 !} p _ {3 j} ^ {4} + \frac {1}{2 !} p _ {3 j} ^ {2} p _ {4 j} + \frac {1}{2 !} p _ {4 j} ^ {2}\right) = 0,
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{3 !} p _ {1 j u} p _ {3 j} ^ {3} + p _ {1 j v} p _ {3 j} p _ {4 j} + \frac {1}{2 !} p _ {2 j l} p _ {3 j} ^ {2} + p _ {2 j \tau} p _ {4 j}\right) = 0 \forall u, v, l, \tau \in [ d ],
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{3 !} p _ {1 j u _ {1}} p _ {1 j u _ {2}} p _ {1 j u _ {3}} p _ {3 j} + \frac {1}{2 !} p _ {1 j v _ {1}} p _ {1 j v _ {2}} p _ {2 j v _ {3}}\right) = 0 \forall (u _ {i}) _ {i = 1} ^ {3}, (v _ {i}) _ {i = 1} ^ {3} \in [ d ] ^ {3},
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2 ! 2 !} p _ {1 j u _ {1}} p _ {1 j u _ {2}} p _ {3 j} ^ {2} + \frac {1}{2 !} p _ {1 j u _ {3}} p _ {1 j u _ {4}} p _ {4 j} + p _ {1 j u _ {5}} p _ {1 j u _ {6}} p _ {3 j}\right) = 0 \forall (u _ {i}) _ {i = 1} ^ {6} \in [ d ] ^ {6},
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \prod_ {i = 1} ^ {5} p _ {1 j u _ {i}} = 0 \forall (u _ {i}) _ {i = 1} ^ {5} \in [ d ] ^ {5},
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{5 !} p _ {3 j} ^ {5} + \frac {1}{3 !} p _ {3 j} ^ {3} p _ {4 j} + \frac {1}{2 !} p _ {3 j} p _ {4 j} ^ {2}\right) = 0,
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{4 !} p _ {1 j u _ {1}} p _ {3 j} ^ {4} + \frac {1}{2 !} p _ {1 j u _ {2}} p _ {3 j} ^ {2} p _ {4 j} + \frac {1}{2 !} p _ {1 j u _ {3}} p _ {4 j} ^ {2} + \frac {1}{3 !} p _ {2 j u _ {4}} p _ {3 j} ^ {3} + p _ {2 j u _ {5}} p _ {3 j} p _ {4 j}\right) = 0
$$
$$
\forall (u _ {i}) _ {i = 1} ^ {5} \in [ d ] ^ {5},
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{4 !} \prod_ {i = 1} ^ {4} p _ {1 j u _ {i}} p _ {3 j} + \frac {1}{3 !} \prod_ {i = 5} ^ {7} p _ {1 j u _ {i}} p _ {2 j u _ {8}}\right) = 0 \forall (u _ {i}) _ {i = 1} ^ {8} \in [ d ] ^ {8},
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{2 ! 3 !} \prod_ {i = 1} ^ {2} p _ {1 j u _ {i}} p _ {3 j} ^ {3} + \frac {1}{2 !} \prod_ {i = 3} ^ {4} p _ {1 j u _ {i}} p _ {3 j} p _ {4 j} + p _ {1 j u _ {5}} p _ {2 j u _ {6}} \left(\frac {1}{2} p _ {3 j} ^ {2} + p _ {4 j}\right) + \frac {1}{2 !} \prod_ {i = 7} ^ {8} p _ {2 j u _ {i}} p _ {3 j}\right) = 0
$$
$$
\forall (u _ {i}) _ {i = 1} ^ {8} \in [ d ] ^ {8},
$$
$$
\sum_ {j = 1} ^ {m} p _ {5 j} ^ {2} \left(\frac {1}{3 ! 2 !} \prod_ {i = 1} ^ {3} p _ {1 j u _ {i}} p _ {3 j} ^ {2} + \frac {1}{3 !} \prod_ {i = 4} ^ {6} p _ {1 j u _ {i}} p _ {4 j} + \frac {1}{2 !} p _ {1 j u _ {7}} p _ {2 j u _ {8}} p _ {3 j} + \frac {1}{2 !} p _ {1 j u _ {9}} \prod_ {i = 1 0} ^ {1 1} p _ {2 j u _ {i}}\right) = 0
$$
$$
\forall (u _ {i}) _ {i = 1} ^ {1 1} \in [ d ] ^ {1 1}.
$$
We can verify that the following is a non-trivial solution of this system:
$$
p _ {5 j} = 1, \quad p _ {1 j} = p _ {2 j} = \mathbf {0} _ {d}, \quad \forall j \in [ m ],
$$
$$
p _ {3 1} = \frac {\sqrt {3}}{3}, \quad p _ {3 2} = - \frac {\sqrt {3}}{3}, \quad p _ {3 3} = 0,
$$
$$
p _ {4 1} = p _ {4 2} = - \frac {1}{6}, \quad p _ {4 3} = 0.
$$
Hence, we conclude that  $\bar{r}(m) = 6$ .
In this appendix, we conduct a simulation study to empirically validate our theoretical results on the convergence rates of maximum likelihood estimation (MLE) in the softmax gating Gaussian mixture of experts established in Theorem 1 and Theorem 2.
C.1 Numerical Schemes
We illustrate the heterogeneity convergence rates of the MLE under the softmax gating Gaussian mixture of experts via exact-fitted and over-fitted models, which correspond to the exact-fitted
and over-fitted settings described in Sections 3.1 and 3.2, respectively. For each case, we let  $X$  be uniformly distributed over  $[0,1]$  and we generate observations  $Y$  from the conditional density  $g_{G_*(Y|X)}$  of softmax gating Gaussian mixture of experts model in equation (1). Here, the true mixing measure  $G_* = \sum_{i=1}^{k_*} \exp(\beta_{0i}^*) \delta_{(\beta_{1i}^*, a_i^*, b_i^*, \sigma_i^*)}$ , where  $k_* = 2$ , is specified as follows:
$$
\beta_ {0 1} ^ {*} = - 8, \quad \beta_ {1 1} ^ {*} = 2 5, \quad a _ {1} ^ {*} = - 2 0, \quad b _ {1} ^ {*} = 1 5, \quad \sigma_ {1} ^ {*} = 0. 3,
$$
$$
\beta_ {0 2} ^ {*} = 0, \quad \beta_ {1 2} ^ {*} = 0, \quad a _ {2} ^ {*} = 2 0, \quad b _ {2} ^ {*} = - 5, \quad \sigma_ {2} ^ {*} = 0. 4.
$$
Then, we compute the MLE  $\widehat{G}_n$  w.r.t. a number of components  $k$  for each sample. For both of these settings, we choose  $k\in \{k_{*},k_{*} + 1,k_{*} + 2\}$ . In order to perform the MLE, we use a numerical scheme based on the EM algorithm [12] similar to the one used by Chamroukhi et al. [7, 8]. Note that the main difference with a classical EM is in the maximization step, as there are no closed formulas for updating the softmax gating parameters  $(\beta_{0i},\beta_{1i}),\forall i\in [k]$ . For this purpose, following the results of Chamroukhi et al. [7, 8], see also [38, 10, 23], we use a multi-class iterative reweighted least-squares algorithm. All code for our simulation study below was written in Python 3.9.13 on a standard Unix machine.
We choose the convergence criteria  $\epsilon = 10^{-6}$  and 2000 maximum EM iterations. Our goal is to illustrate the theoretical properties of the estimator  $\widehat{G}_n$ . Therefore, we have initialized the EM algorithm in a favourable way. More specifically, we first randomly partitioned the set  $\{1,\dots ,k\}$  into  $k_{*}$  index sets  $J_{1},\ldots ,J_{k_{*}}$ , each containing at least one point, for any given  $k$  and  $k_{*}$  and for each replication. Finally, we sampled  $\beta_{1j}^{*}$  (resp.  $a_j^*,b_j^*,\sigma_j^*)$  from a unique Gaussian distribution centered on  $\beta_{1t}^{*}$  (resp.  $a_t^*,b_t^*,\sigma_t^*$ ), with vanishing covariance so that  $j\in J_{t}$ .
C.2 Empirical Convergence Rates
The empirical mean of discrepancies  $\mathcal{D}_1$  and  $\mathcal{D}_2$  between  $\widehat{G}_n$  and  $G_{*}$ , and the choice of  $k$  for exact-fitted and over-fitted models are reported in Figures 1 and 2, respectively. It can be observed that the average discrepancies from  $\widehat{G}_n$  to  $G_0$  vanish at a rate of  $\mathcal{O}(n^{-1/2})$  up to a logarithmic factor, as envisaged by Theorems 1 and 2. Although these empirical rates of convergence are similar for the two models, they imply that the convergence behaviour of the individual fitted parameters is very different in an over-fitted setting, which was already discussed in more detail in the Section 3.2.
C.2.1 Exact-fitted Model
We generate 40 samples of size  $n$  for each setting, given 200 different choices of sample size  $n$  between  $10^{2}$  and  $10^{5}$ . The empirical parametric convergence rate of the MLE to  $G_{*}$  under the metric  $\mathcal{D}_1$  in Figure 1 is consistent with the theoretical rates of estimating the true parameters  $\exp(\beta_{0i}^*)$ ,  $\beta_{1i}^*$  (up to translation),  $a_i^*$ ,  $b_i^*$ ,  $\sigma_i^*$  for  $i \in [k_*]$ , which are of order  $\mathcal{O}(n^{-1/2})$  up to logarithmic factors.
C.2.2 Over-fitted Model
In the over-fitted setting, we generate 40 samples of size  $n$  for each setting, given 200 different choices of sample size  $n$  between  $n_{\mathrm{min}} \approx 14 * 10^3$  for  $k = k_{*} + 1$ ,  $n_{\mathrm{min}} \approx 27 * 10^3$  for  $k = k_{*} + 2$  and  $n_{\mathrm{max}} = 10^5$ . To the best of our knowledge, there is still a lack of theoretical understanding of EM performance, in particular an established algorithm that enjoys global convergence for the parameter estimation of the over-fitted softmax gating Gaussian mixture of experts. The most related theoretical results are only for the mixture of expert with covariate-free gating networks in [41, 40, 42]. This explains why in Figure 2 for over-fitted setting, we have not plotted the error bar due to the instability of the EM algorithm for finding the global solution. Moreover, the sample size must be large enough so that the empirical behaviour of the MLE from the EM algorithm matches the theoretical rate of order  $\mathcal{O}(n^{-1/2})$  up to a logarithmic term.
Figure 1: Log-log scaled plots of the simulation results for exact-fitted setting. We compute the estimator  $\widehat{G}_n$  on 40 independent samples of size  $n$  between  $10^2$  and  $10^5$ . We plot its mean discrepancy from the true mixing measure in red, with error bars representing two empirical standard deviations. We also plot the least-squares fitted linear regression line of these points in a black dash-dotted line.
(a) Over-fitted,  $k = 3$ ,  $n_{\mathrm{min}} = 14070$
(b) Over-fitted,  $k = 4$ ,  $n_{\min} = 26733$
Figure 2: Log-log scaled plots of the empirical mean of the discrepancy  $\mathcal{D}_2$  between  $\widehat{G}_n$  and  $G_*$  (red curves) and least-squares fitted linear regression (black dash-dotted lines) are shown using 40 independent sample sizes  $n$  between  $n_{\mathrm{min}}$  and  $10^5$ .