SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this proof, we adapt the framework in Appendix A.1 to the setting of Theorem 2. However, since the arguments utilized for the global version part remain the same (up to some changes of notations) for the over-fitted settings, they will not be presented here again. Thus, we focus only on proving the following local inequality:
$$
\lim  _ {\varepsilon \rightarrow 0} \inf  _ {\begin{array}{c}G \in \mathcal {O} _ {k} (\Theta):\\\mathcal {D} _ {2} (G, G _ {*}) \leq \varepsilon\end{array}} \mathbb {E} _ {X} \left[ V \left(g _ {G} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} _ {2} (G, G _ {*}) > 0. \tag {27}
$$
Assume that the above claim is not true, then there exists a sequence of mixing measures  $G_{n} \coloneqq \sum_{i=1}^{k_{n}} \exp(\beta_{0i}^{n}) \delta_{(\beta_{1i}^{n}, a_{i}^{n}, b_{i}^{n}, \sigma_{i}^{n})} \in \mathcal{O}_{k}(\Theta)$  such that
$$
\mathbb {E} _ {X} \left[ V \left(g _ {G _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} _ {2} \left(G _ {n}, G _ {*}\right)\rightarrow 0,
$$
$$
\mathcal {D} _ {2} (G _ {n}, G _ {*}) \to 0,
$$
when  $n$  tends to infinity. Since the proof argument is asymptotic, we also assume that  $k_{n} = k^{\prime}$  for all  $n\geq 1$ . Following the proof argument of Theorem 1 in Appendix A.1, we also assume that the Voronoi cells  $\mathcal{A}_j = \mathcal{A}_j^n$  does not change with  $n$  for all  $j\in [k_{*}]$ . Additionally, since  $\mathcal{D}_2(G_n,G_*)\to 0$ , we have  $(a_i^n,b_i^n,\sigma_i^n)\rightarrow (a_j^*,b_j^*,\sigma_j^*)$  for any  $i\in \mathcal{A}_j$  as  $n$  approaches infinity. Furthermore, there exist  $t_1\in \mathbb{R}$  and  $t_2\in \mathbb{R}^d$  such that  $\sum_{i\in \mathcal{A}_j}\exp (\beta_{0i}^n)\to \exp (\beta_{0j}^* +t_1)$  and  $\beta_{1i}^{n}\to \beta_{1j}^{*} + t_{2}$  for any  $i\in \mathcal{A}_j$  and  $j\in [k_{*}]$ . Then, we can upper bound the Voronoi loss function  $\mathcal{D}_2$  as  $\mathcal{D}_2(G_n,G_*)\leq \mathcal{D}_2'(G_n,G_*)$ , where
$$
\begin{array}{l} \mathcal {D} _ {2} ^ {\prime} (G _ {n}, G _ {*}) := \sum_ {j: | \mathcal {A} _ {j} | > 1} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) \Big (\| (\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n}, \Delta b _ {i j} ^ {n}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |)} + \| (\Delta a _ {i j} ^ {n}, \Delta \sigma_ {i j} ^ {n}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |) / 2} \Big) \\ + \sum_ {j: | \mathcal {A} _ {j} | = 1} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) \| (\Delta_ {t _ {2}} \beta_ {1 i j}, \Delta a _ {i j} ^ {n}, \Delta b _ {i j} ^ {n}, \Delta \sigma_ {i j} ^ {n}) \| + \sum_ {j = 1} ^ {k _ {*}} \Big | \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}) \Big |, \\ \end{array}
$$
$$
\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n} := \beta_ {1 i} ^ {n} - \beta_ {1 j} ^ {*} - t _ {2}, \quad \Delta a _ {i j} ^ {n} := a _ {i} ^ {n} - a _ {j} ^ {*},
$$
$$
\Delta b _ {i j} ^ {n} := b _ {i} ^ {n} - b _ {j} ^ {*}, \quad \Delta \sigma_ {i j} ^ {n} := \sigma_ {i} ^ {n} - \sigma_ {j} ^ {*}.
$$
Recall that  $\mathbb{E}_X[V(g_{G_n}(\cdot |X),g_{G_*(\cdot |X))] / \mathcal{D}_2(G_n,G_*)\to 0$  as  $n\to \infty$  , which leads to
$$
\mathbb {E} _ {X} \left[ V \left(g _ {G _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right)\right] / \mathcal {D} _ {2} ^ {\prime} \left(G _ {n}, G _ {*}\right)\rightarrow 0.
$$
In this step, we decompose the quantity  $Q_{n} = \left[\sum_{j=1}^{k_{*}} \exp\left(\left(\beta_{1j}^{*} + t_{2}\right)^{\top} X + \beta_{0j}^{*} + t_{1}\right)\right] \cdot \left[g_{G_{n}}(Y|X) - g_{G_{*}}(Y|X)\right]$  with abuse of notations in Appendix A.1 as follows:
$$
\begin{array}{l} Q _ {n} = \sum_ {j = 1} ^ {k _ {*}} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) \Big [ u (Y | X; \beta_ {1 i} ^ {n}, a _ {i} ^ {n}, b _ {i} ^ {n}, \sigma_ {i} ^ {n}) - u (Y | X; \beta_ {1 j} ^ {*} + t _ {2}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}) \Big ] \\ - \sum_ {j = 1} ^ {k _ {*}} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) \Big [ v (Y | X; \beta_ {1 i} ^ {n}) - v (Y | X; \beta_ {1 j} ^ {*} + t _ {2}) \Big ] \\ + \sum_ {j = 1} ^ {k _ {*}} \Big (\sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}) \Big) \Big [ u (Y | X; \beta_ {1 j} ^ {*} + t _ {2}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}) - v (Y | X; \beta_ {1 j} ^ {*} + t _ {2}) \Big ], \\ := A _ {n} + B _ {n} + E _ {n}, \\ \end{array}
$$
where we denote  $u(Y|X; \beta_1, a, b, \sigma) \coloneqq \exp(\beta_1^\top X) f(Y|a^\top X + b, \sigma)$  and  $v(Y|X; \beta_1) \coloneqq \exp(\beta_1^\top X) g_{G_n}(Y|X)$ . Since each Voronoi cell  $\mathcal{A}_j$  possibly has more than one element, we continue to decompose  $A_n$  and  $B_n$  as follows:
$$
\begin{array}{l} A _ {n} = \sum_ {j: | \mathcal {A} _ {j} | = 1} \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i} ^ {n}\right) \left[ u \left(Y | X; \beta_ {1 i} ^ {n}, a _ {i} ^ {n}, b _ {i} ^ {n}, \sigma_ {i} ^ {n}\right) - u \left(Y | X; \beta_ {1 j} ^ {*} + t _ {2}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right) \right] \\ + \sum_ {j: | \mathcal {A} _ {j} | > 1} \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i} ^ {n}\right) \left[ u (Y | X; \beta_ {1 i} ^ {n}, a _ {i} ^ {n}, b _ {i} ^ {n}, \sigma_ {i} ^ {n}) - u (Y | X; \beta_ {1 j} ^ {*} + t _ {2}, a _ {j} ^ {*}, b _ {j} ^ {*}, \sigma_ {j} ^ {*}) \right] \\ := A _ {n, 1} + A _ {n, 2}, \\ \end{array}
$$
$$
\begin{array}{l} B _ {n} = - \sum_ {j: | \mathcal {A} _ {j} | = 1} \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i} ^ {n}\right) \left[ v (Y | X; \beta_ {1 i} ^ {n}) - v (Y | X; \beta_ {1 j} ^ {*} + t _ {2}) \right] \\ - \sum_ {j: | \mathcal {A} _ {j} | > 1} \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i} ^ {n}\right) \left[ v (Y | X; \beta_ {1 i} ^ {n}) - v (Y | X; \beta_ {1 j} ^ {*} + t _ {2}) \right] \\ := B _ {n, 1} + B _ {n, 2}. \\ \end{array}
$$
Now, we apply the first-order Taylor expansions to two terms  $A_{n,1}$  and  $B_{n,1}$  as in equations (17) and (19), while for  $A_{n,2}$  and  $B_{n,2}$ , we use the Taylor expansions of orders  $\bar{r}(|\mathcal{A}_j|)$  and 2, respectively, for each  $j: |\mathcal{A}_j| > 1$  as follows:
$$
\begin{array}{l} A _ {n, 2} = \sum_ {j: | \mathcal {A} _ {j} | > 1} \sum_ {i \in \mathcal {A} _ {j}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 1} ^ {2 \bar {r} (| \mathcal {A} _ {j} |)} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1}}, \ell_ {2}} \frac {\exp (\beta_ {0 i} ^ {n})}{2 ^ {\alpha_ {4}} \alpha !} (\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n}) ^ {\alpha_ {1}} (\Delta a _ {i j} ^ {n}) ^ {\alpha_ {2}} (\Delta b _ {i j} ^ {n}) ^ {\alpha_ {3}} (\Delta \sigma_ {i j} ^ {n}) ^ {\alpha_ {4}} \\ \times X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) \cdot \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) + R _ {3} (X, Y), \\ \end{array}
$$
$$
B_{n,2} = -\sum_{j:|\mathcal{A}_{j}| > 1}\sum_{i\in \mathcal{A}_{j}}\sum_{|\gamma | = 1}^{2}\frac{\exp(\beta_{0i}^{n})}{\gamma!} (\Delta_{t_{2}}\beta_{1ij}^{n})^{\gamma}X^{\gamma}\exp ((\beta_{1j}^{*} + t_{2})^{\top}X)g_{G_{n}}(Y|X) + R_{4}(X,Y).
$$
Here,  $\mathcal{I}_{\ell_1,\ell_2}:= \left\{\alpha = (\alpha_i)^4\in \mathbb{N}^d\times \mathbb{N}^d\times \mathbb{N}\times \mathbb{N}: \alpha_1 + \alpha_2 = \ell_1,\alpha_3 + 2\alpha_4 = \ell_2 - |\alpha_2|\right\}$  for any  $(\ell_1,\ell_2)\in \mathbb{N}^2$  such that  $1\leq 2|\ell_1| + \ell_2\leq 2\bar{r} (|\mathcal{A}_j|)$ , while  $R_{3}(X,Y)$  and  $R_{4}(X,Y)$  are Taylor remainders such that  $R_{p}(X,Y) / \mathcal{D}_{2}^{\prime}(G_{n},G_{*})\to 0$  when  $n\to \infty$  for  $p\in \{3,4\}$ .
As a result,  $Q_{n}$  can be represented as
$$
\begin{array}{l} Q _ {n} = \sum_ {j = 1} ^ {k _ {*}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 0} ^ {2 \bar {r} (| \mathcal {A} _ {j} |)} T _ {\ell_ {1}, \ell_ {2}} ^ {n} (j) \cdot X ^ {\ell_ {1}} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) \\ + \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \gamma | = 0} ^ {1 + \mathbf {1} _ {\{| A _ {j} | > 1 \}}} S _ {\gamma} ^ {n} (j) \cdot X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) g _ {G _ {n}} (Y | X) + \sum_ {\rho = 1} ^ {4} R _ {\rho} (X, Y), \tag {28} \\ \end{array}
$$
with coefficients  $T_{\ell_1,\ell_2}^n (j)$  and  $S_{\gamma}^{n}(j)$  being defined for any  $j\in [k_{*}]$ ,  $0\leq 2|\ell_1| + \ell_2\leq 2$  and  $0\le |\gamma |\le 1$  as
$$
T _ {\ell_ {1}, \ell_ {2}} ^ {n} (j) = \left\{ \begin{array}{l l} \sum_ {i \in \mathcal {A} _ {j}} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1}, \ell_ {2}}} \frac {\exp (\beta_ {0 i} ^ {n})}{2 ^ {\alpha_ {4}} \alpha !} (\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n}) ^ {\alpha_ {1}} (\Delta a _ {i j} ^ {n}) ^ {\alpha_ {2}} (\Delta b _ {i j} ^ {n}) ^ {\alpha_ {3}} (\Delta \sigma_ {i j} ^ {n}) ^ {\alpha_ {4}}, (\ell_ {1}, \ell_ {2}) \neq (\mathbf {0} _ {d}, 0), \\ \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) - \exp (\beta_ {0 j} ^ {*} + t _ {1}), & (\ell_ {1}, \ell_ {2}) = (\mathbf {0} _ {d}, 0); \end{array} \right.
$$
$$
S _ {\gamma} ^ {n} (j) = \left\{ \begin{array}{l l} - \sum_ {i \in \mathcal {A} _ {j}} \frac {\exp (\beta_ {0 i} ^ {n})}{\gamma !} (\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n}) ^ {\gamma}, & | \gamma | \neq 0, \\ - \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) + \exp (\beta_ {0 j} ^ {*} + t _ {1}), & | \gamma | = 0. \end{array} \right.
$$
Next, we will show that not all the quantities  $T_{\ell_1,\ell_2}^n (j) / \mathcal{D}_2'(G_n,G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}_{2}^{\prime}(G_{n},G_{*})$  go to 0 as  $n\to \infty$ . Assume that all of them vanish when  $n$  tends to infinity. Then, by arguing similarly as in equations (22) and (23), we obtain that
$$
\begin{array}{l} \frac {1}{\mathcal {D} _ {2} ^ {\prime} \left(G _ {n} , G _ {*}\right)} \cdot \left[ \sum_ {j = 1} ^ {k _ {*}} \Big | \sum_ {i \in \mathcal {A} _ {j}} \exp \left(\beta_ {0 i} ^ {n}\right) - \exp \left(\beta_ {0 j} ^ {*} + t _ {1}\right) \right| \\ \left. + \sum_ {j: | \mathcal {A} _ {j} | = 1} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i} ^ {n}) \| (\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n}, \Delta a _ {i j} ^ {n}, \Delta b _ {i j} ^ {n}, \Delta \sigma_ {i j} ^ {n}) \| \right] \to 0. \\ \end{array}
$$
Putting the above limit and the formulation of  $\mathcal{D}_2(G_n, G_*)$  together, we deduce that
$$
\frac {1}{\mathcal {D} _ {2} ^ {\prime} (G _ {n} , G _ {*})} \cdot \sum_ {j: | \mathcal {A} _ {j} | > 1} \sum_ {i \in \mathcal {A} _ {j}} \exp (\beta_ {0 i}) \Big (\| (\Delta_ {t _ {2}} \beta_ {1 i j} ^ {n}, \Delta b _ {i j} ^ {n}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |)} + \| (\Delta a _ {i j} ^ {n}, \Delta \sigma_ {i j} ^ {n}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |) / 2} \Big) \not \to 0,
$$
which indicates that there exists some index  $j^{*} \in [k_{*}]$  such that  $|\mathcal{A}_{j^*}| > 1$  and
$$
\frac {1}{\mathcal {D} _ {2} ^ {\prime} (G _ {n} , G _ {*})} \cdot \sum_ {i \in \mathcal {A} _ {j *}} \exp (\beta_ {0 i}) \Big (\| (\Delta_ {t _ {2}} \beta_ {1 i j *} ^ {n}, \Delta b _ {i j *} ^ {n}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |)} + \| (\Delta a _ {i j *} ^ {n}, \Delta \sigma_ {i j *} ^ {n}) \| ^ {\bar {r} (| \mathcal {A} _ {j} |) / 2} \Big) \not \to 0,
$$
for all  $t_2 \in \mathbb{R}^d$ . Without loss of generality, we may assume that  $j^* = 1$ . Recall that for  $(\ell_1, \ell_2) \in \mathbb{N}^d \times \mathbb{N}$  such that  $1 \leq |\ell_1| + \ell_2 \leq \bar{r}(|\mathcal{A}_1|)$ , we have  $T_{\ell_1, \ell_2}^n(1) / \mathcal{D}_2'(G_n, G_*) \to 0$  as  $n \to \infty$ . Thus, by dividing this ratio and the left hand side of the above equation and let  $t_2 = 0$ , we obtain that
$$
\frac {\sum_ {i \in \mathcal {A} _ {1}} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1} , \ell_ {2}}} \frac {\exp \left(\beta_ {0 i} ^ {n}\right)}{2 ^ {\alpha_ {4}} \alpha !} \left(\Delta_ {t _ {2}} \beta_ {1 i 1} ^ {n}\right) ^ {\alpha_ {1}} \left(\Delta a _ {i 1} ^ {n}\right) ^ {\alpha_ {2}} \left(\Delta b _ {i 1} ^ {n}\right) ^ {\alpha_ {3}} \left(\Delta \sigma_ {i 1} ^ {n}\right) ^ {\alpha_ {4}}}{\sum_ {i \in \mathcal {A} _ {1}} \exp \left(\beta_ {0 i} ^ {n}\right)\left(\| \left(\Delta_ {t _ {2}} \beta_ {1 i 1} ^ {n}, \Delta b _ {i 1} ^ {n}\right) \| ^ {\bar {r} (| \mathcal {A} _ {1} |)} + \| \left(\Delta a _ {i 1} ^ {n}, \Delta \sigma_ {i 1} ^ {n}\right) \| ^ {\bar {r} (| \mathcal {A} _ {1} |) / 2}\right)} \rightarrow 0, \tag {29}
$$
for all  $(\ell_1,\ell_2)$  such that  $1\leq |\ell_1| + \ell_2\leq \bar{r} (|\mathcal{A}_1|)$
Let us define  $\overline{M}_n \coloneqq \max \{\| \Delta_{t_2} \beta_{1i1}^n \|, \| \Delta a_{i1}^n \|^{1/2}, |\Delta b_{i1}^n|, |\Delta \sigma_{i1}^n|^{1/2} : i \in \mathcal{A}_1\}$  and  $\overline{\beta}_n \coloneqq \max_{i \in \mathcal{A}_1} \exp(\beta_{0i}^n)$ . Note that the sequence  $\exp(\beta_{0i}^n) / \overline{\beta}_n$  is bounded, therefore, we can replace it by its subsequence that has a positive limit  $p_{5i}^2 \coloneqq \lim_{n \to \infty} \exp(\beta_{0i}^n) / \overline{\beta}_n$ . Thus, at least one among  $p_{5i}^2$ , for  $i \in \mathcal{A}_1$ , equals 1.
In addition, we also define
$$
\left(\Delta_ {t _ {2}} \beta_ {1 i 1} ^ {n}\right) / \overline {{M}} _ {n} \to p _ {1 i}, \quad \left(\Delta a _ {i 1} ^ {n}\right) / \overline {{M}} _ {n} \to p _ {2 i},
$$
$$
\left(\Delta b _ {i 1} ^ {n}\right) / \bar {M} _ {n} \rightarrow p _ {3 i}, \quad \left(\Delta \sigma_ {i 1} ^ {n}\right) / \left[ 2 \bar {M} _ {n} \right]\rightarrow p _ {4 i}.
$$
Here, at least one of  $p_{1i}, p_{2i}, p_{3i}$  and  $p_{4i}$  for  $i \in \mathcal{A}_1$  equals either 1 or -1. Next, we divide both the numerator and the denominator of the ratio in equation (29) by  $\overline{\beta}_n \overline{M}_n^{\ell_1 + \ell_2}$ , and then achieve the following system of polynomial equations:
$$
\sum_ {i \in \mathcal {A} _ {1}} \sum_ {\alpha \in \mathcal {I} _ {\ell_ {1}, \ell_ {2}}} \frac {1}{\alpha !} \cdot p _ {5 i} ^ {2} p _ {1 i} ^ {\alpha_ {1}} p _ {2 i} ^ {\alpha_ {2}} p _ {3 i} ^ {\alpha_ {3}} p _ {4 i} ^ {\alpha_ {4}} = 0,
$$
for all  $(\ell_1,\ell_2)\in \mathbb{N}^d\times \mathbb{N}$  such that  $1\leq |\ell_1| + \ell_2\leq \bar{r} (|\mathcal{A}_1|)$ . However, based on the definition of  $\bar{r} (|\mathcal{A}_1|)$ , the above system has no non-trivial solutions, which is a contradiction. Thus, not all the quantities  $T_{\ell_1,\ell_2}^n (j) / \mathcal{D}_2'(G_n,G_*)$  and  $S_{\gamma}^{n}(j) / \mathcal{D}_{2}^{\prime}(G_{n},G_{*})$  go to 0 as  $n\to \infty$ .
Subsequently, we denote by  $m_{n}$  be the maximum of the absolute values of those quantities. Based on the result in Step 2, we know that  $1 / m_{n} \neq \infty$ . Then, by applying the Fatou's lemma as in equation (24), we get that  $Q_{n} / [m_{n}\mathcal{D}_{2}^{\prime}(G_{n},G_{*})]\to 0$  as  $n\to \infty$  for almost surely  $(X,Y)$ . It follows
from the decomposition of  $Q_{n}$  in equation (28) that
$$
\begin{array}{l} \sum_ {j = 1} ^ {k _ {*}} \sum_ {2 | \ell_ {1} | + \ell_ {2} = 0} ^ {2 \bar {r} (| \mathcal {A} _ {j} |)} \eta_ {\ell_ {1}, \ell_ {2}} (j) \cdot X ^ {\ell_ {1}} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} (Y | (a _ {j} ^ {*}) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}) \\ + \sum_ {j = 1} ^ {k _ {*}} \sum_ {| \gamma | = 0} ^ {1 + \mathbf {1} _ {\{| A _ {j} | > 1 \}}} \omega_ {\gamma} (j) \cdot X ^ {\gamma} \exp ((\beta_ {1 j} ^ {*} + t _ {2}) ^ {\top} X) g _ {G _ {*}} (Y | X) = 0, \\ \end{array}
$$
for almost surely  $(X,Y)$ , where  $\eta_{\ell_1,\ell_2}(j)$  and  $\omega_{\gamma}(j)$  denote the limits of  $T_{\ell_1,\ell_2}^n (j) / [m_n\mathcal{D}_2'(G_n,G_*)]$  and  $S_{\gamma}^{n}(j) / [m_{n}\mathcal{D}_{2}^{\prime}(G_{n},G_{*})]$  as  $n\to \infty$ , respectively, for all  $j\in [k_{*}]$ ,  $0\leq 2|\ell_1| + \ell_2\leq 2\bar{r} (|\mathcal{A}_j|)$  and  $0\le |\gamma |\le 1 + \mathbf{1}_{\{|A_j| > 1\}}$ . By definition, at least one among  $\eta_{\ell_1,\ell_2}(j)$  and  $\omega_{\gamma}(j)$  is different from zero. Nevertheless, as the set
$$
\begin{array}{l} \mathcal {W} _ {2}: = \left\{X ^ {\ell_ {1}} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) \frac {\partial^ {\ell_ {2}} f}{\partial h _ {1} ^ {\ell_ {2}}} \left(Y \mid \left(a _ {j} ^ {*}\right) ^ {\top} X + b _ {j} ^ {*}, \sigma_ {j} ^ {*}\right): j \in [ k _ {*} ], 0 \leq 2 | \ell_ {1} | + \ell_ {2} \leq 2 \bar {r} (| \mathcal {A} _ {j} |) \right\} \\ \cup \left\{X ^ {\gamma} \exp \left(\left(\beta_ {1 j} ^ {*} + t _ {2}\right) ^ {\top} X\right) g _ {G _ {*}} (Y | X): j \in [ k _ {*} ], 0 \leq | \gamma | \leq 1 + \mathbf {1} _ {\{| A _ {j} | > 1 \}} \right\}, \tag {30} \\ \end{array}
$$
is linearly independent w.r.t  $X$  and  $Y$  (proof can be done similarly to Lemma 2), it follows that
$$
\eta_ {\ell_ {1}, \ell_ {2}} (j) = \omega_ {\gamma} (j) = 0,
$$
for all  $j\in [k_{*}]$ $0\leq 2|\ell_1| + \ell_2\leq 2\bar{r} (|\mathcal{A}_j|)$  and  $0\le |\gamma |\le 1 + \mathbf{1}_{\{|A_j| > 1\}}$  , which is a contradiction. Hence, we achieve the inequality in equation (27), and complete the proof.
B Proofs of Auxiliary Results
In this appendix, we provide proofs for the results of Proposition 1 and Proposition 2 in Appendix B.1 and Appendix B.2, respectively, while we leave that for Lemma 1 in Appendix B.3.
B.1 Proof of Proposition 1
Given the notations in Proposition 1, assume that the equation  $g_{G}(Y|X) = g_{G^{\prime}}(Y|X)$  holds true, that is,
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \frac {\exp \left(\left(\beta_ {1 i}\right) ^ {\top} X + \beta_ {0 i}\right)}{\sum_ {j = 1} ^ {k} \exp \left(\left(\beta_ {1 j}\right) ^ {\top} X + \beta_ {0 j}\right)} \cdot f (Y | \left(a _ {i}\right) ^ {\top} X + b _ {i}, \sigma_ {i}) \\ = \sum_ {i = 1} ^ {k ^ {\prime}} \frac {\exp \left(\left(\beta_ {1 i} ^ {\prime}\right) ^ {\top} X + \beta_ {0 i} ^ {\prime}\right)}{\sum_ {j = 1} ^ {k ^ {\prime}} \exp \left(\left(\beta_ {1 j} ^ {\prime}\right) ^ {\top} X + \beta_ {0 j} ^ {\prime}\right)} \cdot f (Y | \left(a _ {i} ^ {\prime}\right) ^ {\top} X + b _ {i} ^ {\prime}, \sigma_ {i} ^ {\prime}), \tag {31} \\ \end{array}
$$
for almost surely  $(X,Y)$ . Then, it follows from the identifiability of the location-scale Gaussian mixtures [56, 57] that the number of components and the weight set of the mixing measure  $G$  equal to those of its counterpart  $G'$ , i.e.  $k = k'$  and
$$
\left\{\frac {\exp ((\beta_ {1 i}) ^ {\top} X + \beta_ {0 i})}{\sum_ {j = 1} ^ {k} \exp ((\beta_ {1 j}) ^ {\top} X + \beta_ {0 j})}: i \in [ k ] \right\} \equiv \left\{\frac {\exp ((\beta_ {1 i} ^ {\prime}) ^ {\top} X + \beta_ {0 i} ^ {\prime})}{\sum_ {j = 1} ^ {k} \exp ((\beta_ {1 j} ^ {\prime}) ^ {\top} X + \beta_ {0 j} ^ {\prime})}: i \in [ k ] \right\},
$$
for almost surely  $X$ . For simplicity, we may assume that
$$
\frac {\exp \left(\left(\beta_ {1 i}\right) ^ {\top} X + \beta_ {0 i}\right)}{\sum_ {j = 1} ^ {k} \exp \left(\left(\beta_ {1 j}\right) ^ {\top} X + \beta_ {0 j}\right)} = \frac {\exp \left(\left(\beta_ {1 i} ^ {\prime}\right) ^ {\top} X + \beta_ {0 i} ^ {\prime}\right)}{\sum_ {j = 1} ^ {k} \exp \left(\left(\beta_ {1 j} ^ {\prime}\right) ^ {\top} X + \beta_ {0 j} ^ {\prime}\right)},
$$
for all  $i \in [k]$ . Since the softmax function is invariant to translation, we get that  $\beta_{0i} = \beta_{0i}' + t_1$  and  $\beta_{1i} = \beta_{1i}' + t_2$  for some  $t_1 \in \mathbb{R}$  and  $t_2 \in \mathbb{R}^d$ . Therefore, equation (31) reduces to
$$
\sum_ {i = 1} ^ {k} \exp \left(\beta_ {0 i}\right) u \left(Y \mid X; \beta_ {1 i}, a _ {i}, b _ {i}, \sigma_ {i}\right) = \sum_ {i = 1} ^ {k} \exp \left(\beta_ {0 i}\right) u \left(Y \mid X; \beta_ {1 i}, a _ {i} ^ {\prime}, b _ {i} ^ {\prime}, \sigma_ {i} ^ {\prime}\right), \tag {32}
$$
for almost surely  $(X,Y)$ , where  $u(Y|X;\beta_1,a,b,\sigma) \coloneqq \exp(\beta_1^\top X)f(Y|a^\top X + b,\sigma)$  for all  $i \in [k]$ . Next, we will partition the index set  $[k]$  into  $q$  subsets  $U_1, U_2, \ldots, U_q$  such that for each  $\ell \in [q]$ , we have  $\exp(\beta_{0i}) = \exp(\beta_{0i'})$  for any  $i, i' \in U_\ell$ . As a result, equation (32) can be rewritten as
$$
\sum_ {\ell = 1} ^ {q} \sum_ {i \in U _ {\ell}} \exp (\beta_ {0 i}) u (Y | X; \beta_ {1 i}, a _ {i}, b _ {i}, \sigma_ {i}) = \sum_ {\ell = 1} ^ {q} \sum_ {i \in U _ {\ell}} \exp (\beta_ {0 i}) u (Y | X; \beta_ {1 i}, a _ {i} ^ {\prime}, b _ {i} ^ {\prime}, \sigma_ {i} ^ {\prime}),
$$
for almost surely  $(X,Y)$ . Given the above equation, for each  $\ell \in [q]$ , we obtain that
$$
\left\{\left(\left(a _ {i}\right) ^ {\top} X + b _ {i}, \sigma_ {i}\right): i \in U _ {\ell} \right\} \equiv \left\{\left(\left(a _ {i} ^ {\prime}\right) ^ {\top} X + b _ {i} ^ {\prime}, \sigma_ {i} ^ {\prime}\right): i \in U _ {\ell} \right\},
$$
for almost surely  $X$ , which directly leads to
$$
\left\{\left(a _ {i}, b _ {i}, \sigma_ {i}\right): i \in U _ {\ell} \right\} \equiv \left\{\left(a _ {i} ^ {\prime}, b _ {i} ^ {\prime}, \sigma_ {i} ^ {\prime}\right): i \in U _ {\ell} \right\}.
$$
WLOG, we assume that  $(a_{i},b_{i},\sigma_{i}) = (a_{i}^{\prime},b_{i}^{\prime},\sigma_{i}^{\prime})$  for all  $i\in U_{\ell}$ . Consequently,
$$
\sum_ {\ell = 1} ^ {q} \sum_ {i \in U _ {\ell}} \exp (\beta_ {0 i}) \delta_ {\{\beta_ {1 i}, a _ {i}, b _ {i}, \sigma_ {i} \}} = \sum_ {\ell = 1} ^ {q} \sum_ {i \in U _ {\ell}} \exp (\beta_ {0 i} ^ {\prime} + t _ {1}) \delta_ {\{\beta_ {1 i} ^ {\prime} + t _ {2}, a _ {i} ^ {\prime}, b _ {i} ^ {\prime}, \sigma_ {i} ^ {\prime} \}},
$$
or equivalently,  $G \equiv G_{t_1,t_2}^{\prime}$ . Hence, the proof is completed.
B.2 Proof of Proposition 2
Our proof will be based on the convergence rates of density estimation from MLE in Theorem 7.4 in [58]. Before stating this result here, let us introduce some necessary notations. Firstly, let  $\mathcal{P}_k(\Theta)$  be the set of conditional densities of all mixing measures in  $\mathcal{O}_k(\Theta)$ , i.e.,  $\mathcal{P}_k(\Theta) \coloneqq \{g_G(Y|X): G \in \mathcal{O}_k(\Theta)\}$ . Additionally, we define
$$
\widetilde {\mathcal {P}} _ {k} (\Theta) := \left\{g _ {(G + G _ {*}) / 2} (Y | X): G \in \mathcal {O} _ {k} (\Theta) \right\},
$$
$$
\widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta) := \{g _ {(G + G _ {*}) / 2} ^ {1 / 2} (Y | X): G \in \mathcal {O} _ {k} (\Theta) \}.
$$
Next, for each  $\delta > 0$ , the Hellinger ball centered around the conditional density  $g_{G_*(Y|X)}$  and intersected with the set  $\widetilde{\mathcal{P}}_k^{1/2}(\Theta)$  is denoted by
$$
\widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta , \delta) := \left\{g ^ {1 / 2} \in \widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta): h (g, g _ {G _ {*}}) \leq \delta \right\}.
$$
Finally, in order to measure the size of the above set, [58] proposes using the following quantity:
$$
\mathcal {J} _ {B} \left(\delta , \widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta , \delta)\right) := \int_ {\delta^ {2} / 2 ^ {1 3}} ^ {\delta} H _ {B} ^ {1 / 2} \left(t, \widetilde {\mathcal {P}} _ {k} ^ {1 / 2} (\Theta , t), \| \cdot \|\right) d t \vee \delta , \tag {33}
$$
where  $H_{B}(t,\widetilde{\mathcal{P}}_{k}^{1 / 2}(\Theta ,t),\| \cdot \|)$  denotes the bracketing entropy [58] of  $\widetilde{\mathcal{P}}_k^{1 / 2}(\Theta ,u)$  under the 2-norm, and  $t\vee \delta \coloneqq \max \{t,\delta \}$ . Now, we are ready to recall the statement of Theorem 7.4 in [58]:
Theorem 3 (Theorem 7.4, [58]). Take  $\Psi(\delta) \geq \mathcal{J}_B(\delta, \widetilde{\mathcal{P}}_k^{1/2}(\Theta, \delta))$  that satisfies  $\Psi(\delta)/\delta^2$  is a nonincreasing function of  $\delta$ . Then, for some universal constant  $c$  and for some sequence  $(\delta_n)$  such that  $\sqrt{n}\delta_n^2 \geq c\Psi(\delta_n)$ , we achieve that
$$
\left. \mathbb {P} \left(\mathbb {E} _ {X} \left[ h \left(g _ {\widehat {G} _ {n}} (\cdot | X), g _ {G _ {*}} (\cdot | X)\right) \right] > \delta\right) \leq c \exp \left(- \frac {n \delta^ {2}}{c ^ {2}}\right), \right.
$$
for all  $\delta \geq \delta_{n}$
The proof of this theorem can be seen in [58].
Proof of Proposition 2. Back to our main proof, since