SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] L. Bordes, C. Delmas, and P. Vandekerkhove. Semiparametric estimation of a two-component mixture model where one component is known. Scandinavian journal of statistics, 33(4):733-752, 2006.
[2] L. Bordes, S. Mottelet, and P. Vandekerkhove. Semiparametric estimation of a two-component mixture model. The Annals of Statistics, 34(3):1204-1232, 2006.
[3] T. Cai, X. J. Jeng, and J. Jin. Optimal detection of heterogeneous and heteroscedastic mixtures. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5):629-662, 2011.
[4] T. Cai, J. Jin, and M. G. Low. Estimation and confidence sets for sparse normal mixtures. The Annals of Statistics, 35(6):2421-2449, 2007.
[5] T. Cai and Y. Wu. Optimal detection of sparse mixtures against a given null distribution. IEEE Transactions on Information Theory, 60(4):2217 - 2232, 2014.
[6] G. Casella and R. L. Berger. Statistical inference. Cengage Learning, 2021.
[7] H. Chen and J. Chen. Tests for homogeneity in normal mixtures in the presence of a structural parameter. Statistica Sinica, 13:351-365, 2003.
[8] H. Chen, J. Chen, and J. D. Kalbfleisch. A modified likelihood ratio test for homogeneity in finite mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(1):19-29, 2001.
[9] J. Chen. Optimal rate of convergence for finite mixture models. Annals of Statistics, 23(1):221-233, 1995.
[10] J. Chen, P. Li, and Y. Fu. Inference on the order of a normal mixture. Journal of the American Statistical Association, 107:1096-1105, 2012.
[11] W. G. Cochran. The  $\chi 2$  test of goodness of fit. The Annals of mathematical statistics, pages 315-345, 1952.
[12] N. Deb, S. Saha, A. Guntuboyina, and B. Sen. Two-component mixture model in the presence of covariates. Journal of the American Statistical Association, 117(540):1820-1834, 2022.
[13] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, Sept. 1977.
[14] D. Do, N. Ho, and X. Nguyen. Beyond black box densities: Parameter learning for the deviated components. arXiv preprint arXiv:2202.02651, 2022.
[15] D. Donoho and J. Jin. Higher criticism for detecting sparse heterogeneous mixtures. Annals of Statistics, 32(3):962-994, 2004.
[16] S. Gadat, J. Kahn, C. Marteau, and C. Maugis-Rabusseau. Parameter recovery in two-component contamination mixtures: The  $l^2$  strategy. In Annales de l'Institut Henri Poincaré, Probabilités et Statistiques, volume 56, pages 1391-1418. Institut Henri Poincaré, 2020.
[17] E. Giné and R. Nickl. Mathematical foundations of infinite-dimensional statistical models. Cambridge university press, 2021.
[18] M. Hardt and E. Price. Tight bounds for learning a mixture of two gaussians. In STOC, 2015.
[19] P. Heinrich and J. Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. Annals of Statistics, 46(6A):2844-2870, 2018.
[20] N. Ho and L. Nguyen. Singularity structures and impacts on parameter estimation in finite mixtures of distributions. SIAM Journal on Mathematics of Data Science, 1(4):730-758, 2019.
[21] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistics, 44:2726-2755, 2016.
[22] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, 10:271-307, 2016.
[23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
[24] H. Kasahara and K. Shimotsu. Non-parametric identification and estimation of the number of components in multivariate mixtures. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1):97-111, 2014.
[25] H. Kasahara and K. Shimotsu. Testing the number of components in normal mixture regression models. Journal of the American Statistical Association, 110(512):1632-1645, 2015.
[26] P. Li and J. Chen. Testing the order of a finite mixture. Journal of the American Statistical Association, 105(491):1084-1092, 2010.
[27] Q. Liu, J. Lee, and M. Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 276–284. PMLR, 20–22 Jun 2016.
[28] X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. Annals of Statistics, 4(1):370-400, 2013.
[29] R. Patra and B. Sen. Estimation of a two-component mixture model with applications to multiple testing. J. R. Stat. Soc. Series B Stat. Methodol., 78(4):869-893, 2016.
[30] R. Patra and B. Sen. Estimation of a two-component mixture model with applications to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(4):869-893, 2016.
[31] A. Schrab, B. Guedj, and A. Gretton. KSD aggregated goodness-of-fit test. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[32] S. Talts, M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman. Validating bayesian inference algorithms with simulation-based calibration, 2018.
[33] S. van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
[34] N. Verzelen and E. Arias-Castro. Detection and feature selection in sparse mixture models. Annals of Statistics, 45(5):1920-1950, 2017.
[35] C. Villani. Topics in Optimal Transportation. American Mathematical Society, 2003.
[36] Y. Wu and P. Yang. Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics, 48:1987-2007, 2020.
[37] J. Yang, V. Rao, and J. Neville. A stein-papangelou goodness-of-fit test for point processes. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 226–235. PMLR, 16–18 Apr 2019.
Supplement to "Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models"
In this supplementary material, we present additional results and proofs. The minimax lower bounds and convergence rates of parameter estimation in the non-distinguishable settings are presented in Section A. The general theory for the convergence rates of densities and their proofs can be found in Appendix B. Proofs of the geometric inequalities that relate the convergence of density estimation to that of parameter estimation are in Section C, while those for minimax lower bounds and convergence rates of parameter estimation are left in Appendix D. Then, we provide a necessary lemma for those results along with its proof in Appendix E. Finally, some discussion about the general setting of the paper is presented in Section F, followed by a set of simulations to support the developed theory.
A Minimax Lower Bounds and Convergence Rates of Parameter Estimation under the Non-distinguishable Settings
Theorem A.1. (Strongly identifiable and non-distinguishable settings) Assume that classes of densities  $h_0$  and  $f$  satisfy the conditions in Theorem 3.5. We define
$$
\Xi_{1}(l_{n}):= \Bigg\{G = (\lambda ,\mu ,\Sigma)\in \Xi :\frac{l_{n}}{\min_{\substack{1\leq i\leq d_{1}\\ 1\leq u,v\leq d_{2}}}\{|(\Delta \mu)_{i}|^{2},|(\Delta \Sigma)_{uv}|^{2}\} \sqrt{n}}\leq \lambda \Bigg\},
$$
for any sequence  $\{l_n\}$ . Then, we achieve
(a) (Minimax lower bound) Assume that  $f$  satisfies assumption S.1 in Theorem 4.1. Then for any  $r < 1$  and sequence  $\{l_n\}$ , there exist two universal positive constants  $c_1$  and  $c_2$  such that
$$
\inf _ {\widehat {G} _ {n} \in \Xi} \sup _ {G \in \Xi_ {1} (l _ {n})} \mathbb {E} _ {p _ {G}} \left(\lambda^ {2} \| (\Delta \mu , \Delta \Sigma) \| ^ {2} \| (\widehat {\mu} _ {n}, \widehat {\Sigma} _ {n}) - (\mu , \Sigma) \| ^ {2}\right) \geq c _ {1} n ^ {- 1 / r},
$$
$$
\inf  _ {\widehat {G} _ {n} \in \Xi} \sup  _ {G \in \Xi_ {1} (l _ {n})} \mathbb {E} _ {p _ {G}} \left(\| \Delta \mu , \Delta \Sigma) \| ^ {4} | \widehat {\lambda} _ {n} - \lambda | ^ {2}\right) \geq c _ {2} n ^ {- 1 / r}.
$$
(b) (MLE rate) Let  $\widehat{G}_n$  be the MLE defined in equation (3), and the family  $\{p_G : G \in \Xi\}$  satisfies condition A2. Then, for any sequence  $\{l_n\}$  such that  $l_n / \log n \to \infty$ ,
$$
\sup  _ {G _ {*} \in \Xi_ {1} (l _ {n})} \mathbb {E} _ {p _ {G _ {*}}} \left((\lambda^ {*}) ^ {2} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {2} \| (\widehat {\mu} _ {n}, \widehat {\Sigma} _ {n}) - (\mu^ {*}, \Sigma^ {*}) \| ^ {2}\right) \lesssim \frac {\log^ {2} n}{n},
$$
$$
\sup  _ {G _ {*} \in \Xi_ {1} (l _ {n})} \mathbb {E} _ {p _ {G _ {*}}} \left(\| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {4} | \widehat {\lambda} _ {n} - \lambda^ {*} | ^ {2}\right) \lesssim \frac {\log^ {2} n}{n}.
$$
Proof of Theorem A.1 is in Appendix D.2. The results of part (b) are the generalization of those in Theorem 3.1 and Theorem 3.2 in [16] to the setting of strongly identifiable in the second-order kernel. The condition regarding the lower bound of  $\lambda$  in the formation of  $\Xi_1(l_n)$  is necessary to guarantee that  $(\widehat{\mu}_n,\widehat{\Sigma}_n)$  and  $\widehat{\lambda}_n$  are consistent estimators of  $(\mu^{*},\Sigma^{*})$  and  $\lambda^*$  respectively. In particular, from the results in equation (34) of the proof of Theorem A.1, we have for any  $G_{*}\in \Xi$  that
$$
\mathbb {E} _ {p _ {G *}} (\lambda^ {*}) ^ {2} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {2} \| (\widehat {\mu} _ {n}, \widehat {\Sigma} _ {n}) - (\mu^ {*}, \Sigma^ {*}) \| ^ {2} \lesssim \frac {\log^ {2} n}{n}.
$$
Therefore, for any  $1 \leq i \leq d_{1}$  and  $1 \leq u, v \leq d_{2}$  we get
$$
\mathbb {E} _ {p _ {G *}} \left\{\left(\frac {(\Delta \widehat {\mu} _ {n}) _ {i}}{(\Delta \mu^ {*}) _ {i}} - 1\right) ^ {2} \right\} \lesssim \frac {\log^ {2} n}{n (\lambda^ {*}) ^ {2} \left\{(\Delta \mu^ {*}) _ {i} \right\} ^ {4}},
$$
$$
\mathbb {E} _ {p _ {G ^ {*}}} \left\{\left(\frac {(\Delta \widehat {\Sigma} _ {n}) _ {u v}}{(\Delta \Sigma^ {*}) _ {u v}} - 1\right) ^ {2} \right\} \lesssim \frac {\log^ {2} n}{n (\lambda^ {*}) ^ {2} \left\{(\Delta \Sigma^ {*}) _ {u v} \right\} ^ {4}}.
$$
It indicates that
$$
\frac {\log n}{\sqrt {n} \lambda^ {*} \min  _ {1 \leq i \leq d _ {1} , 1 \leq u , v \leq d _ {2}} \left\{\left| (\Delta \mu^ {*}) _ {i} \right| ^ {2} , \left| (\Delta \Sigma^ {*}) _ {i} \right| ^ {2} \right\}} \rightarrow 0
$$
for the left-hand-side terms of the above display to go to 0 for all  $1 \leq i \leq d_{1}$  and  $1 \leq u, v \leq d_{2}$ .
The results of Theorem A.1 imply that as long as the kernel functions are strongly identifiable in the second order, the convergence rates of  $\widehat{\mu}_n$  to  $\mu^{*}$  and  $\widehat{\Sigma}_n$  to  $\Sigma^{*}$  are similar, which depend on the vanishing rate of  $(\lambda^{*})^{2} \left\| (\Delta \mu^{*}, \Delta \Sigma^{*}) \right\|^{2}$  to 0. In our next result of location-covariance multivariate Gaussian distribution, we will demonstrate that such uniform convergence rates of different parameters no longer hold.
Theorem A.2. (Weakly identifiable and non-distinguishable settings) Assume that  $f$  is a family of location-covariance multivariate Gaussian distributions, and  $h_0(x) = f(x|\mu_0,\Sigma_0)$  for some  $(\mu_0,\Sigma_0)\in \Theta \times \Sigma$ . We define
$$
\Xi_ {2} (l _ {n}) := \left\{G = (\lambda , \mu , \Sigma) \in \Xi : \frac {l _ {n}}{\underset {1 \leq i \leq d , 1 \leq u , v \leq d} {\min } \{| (\Delta \mu) _ {i} | ^ {4} , | (\Delta \Sigma) _ {u v} | ^ {2} \}} \sqrt {n} \leq \lambda \right\},
$$
for any sequence  $\{l_n\}$ . Then, the following holds:
(a) (Minimax lower bound) For any  $r < 1$  and sequence  $\{l_n\}$ , there exist two universal positive constants  $c_1$  and  $c_2$  such that
$$
\begin{array}{l} \inf _ {\widehat {G} _ {n} \in \Xi} \sup _ {G \in \Xi_ {2} (l _ {n})} \mathbb {E} _ {p _ {G}} \left(\lambda^ {2} \left\{\| \Delta \mu \| ^ {4} + \| \Delta \Sigma \| ^ {2} \right\} \left\{\| \widehat {\mu} _ {n} - \mu \| ^ {4} + \| \widehat {\Sigma} _ {n} - \Sigma \| ^ {2} \right\}\right) \geq c _ {1} n ^ {- 1 / r}, \\ \inf  _ {\widehat {G} _ {n} \in \Xi} \sup  _ {G \in \Xi_ {2} (l _ {n})} \mathbb {E} _ {p _ {G}} \left(\left\{\| \Delta \mu \| ^ {8} + \| \Delta \Sigma \| ^ {4} \right\} | \widehat {\lambda} - \lambda | ^ {2}\right) \geq c _ {2} n ^ {- 1 / r}. \\ \end{array}
$$
(b) (MLE rate) Let  $\widehat{G}_n$  be the estimator defined in (3). Then, for any sequence  $\{l_n\}$  such that  $l_{n} / \log n\to \infty$  the following holds
$$
\begin{array}{l} \sup _ {G _ {*} \in \Xi_ {2} (l _ {n})} \mathbb {E} _ {p _ {G *}} \left((\lambda^ {*}) ^ {2} \left\{\| \Delta \mu^ {*} \| ^ {4} + \| \Delta \Sigma^ {*} \| ^ {2} \right\} \left\{\| \widehat {\mu} _ {n} - \mu^ {*} \| ^ {4} + \| \widehat {\Sigma} _ {n} - \Sigma^ {*} \| ^ {2} \right\}\right) \lesssim \frac {\log^ {2} n}{n}, \\ \sup _ {G _ {*} \in \Xi_ {2} (l _ {n})} \mathbb {E} _ {p _ {G *}} \left(\left\{\| \Delta \mu^ {*} \| ^ {8} + \| \Delta \Sigma^ {*} \| ^ {4} \right\} | \widehat {\lambda} _ {n} - \lambda^ {*} | ^ {2}\right) \lesssim \frac {\log^ {2} n}{n}. \\ \end{array}
$$
Proof of Theorem A.2 is in Appendix D.3. A few comments are in order:
(i) Similar to the argument after Theorem A.1, the condition regarding  $\lambda$  in the formulation of  $\Xi_2(l_n)$  is to guarantee that  $(\widehat{\mu}_n,\widehat{\Sigma}_n)$  and  $\widehat{\lambda}_n$  are consistent estimators of  $(\mu^{*},\Sigma^{*})$  and  $\lambda^{*}$ , respectively.
(ii) The results of part (b) indicate that the convergence rate of estimating  $\Sigma^{*}$  is generally much faster than that of estimating  $\mu^{*}$  regardless of the circumstance of  $(\lambda^{*})^{2}\left\{\| \Delta \mu^{*}\|^{4} + \| \Delta \Sigma^{*}\|^{2}\right\}$ . The non-uniformity of these convergence rates is mainly due to the structure of the heat partial differential equation, where the second-order derivative of the location parameter and the first-order derivative of covariance parameter correlate.
(iii) From the results of part (b), it is clear that when  $\| \Delta \mu^{*}\| +\| \Delta \Sigma^{*}\| \nrightarrow 0$ , i.e.,  $(\mu^{*},\Sigma^{*})\to (\overline{\mu},\overline{\Sigma})\neq (\mu_0,\Sigma_0)$ , and  $\lambda^*\not\to 0$ , the convergence rate of  $\widehat{\lambda}_n$  to  $\lambda^{*}$  is  $n^{-1 / 2}$ . Furthermore, by using the result from part (a) of Proposition C.4 we can verify that
$$
\sup  _ {G _ {*}} \mathbb {E} _ {p _ {G _ {*}}} \left(\left(\lambda^ {*}\right) ^ {2} \left\{\| \widehat {\mu} _ {n} - \mu^ {*} \| ^ {2} + \| \widehat {\Sigma} _ {n} - \Sigma^ {*} \| ^ {2} \right\}\right) \lesssim \frac {\log^ {2} n}{n},
$$
where the supremum is taken over  $\{G_{*}\in \Xi_{2}(l_{n}):\mathcal{K}(G_{*},\overline{G})\leq \epsilon \}$ , and  $\overline{G} = (\overline{\lambda},\overline{\mu},\overline{\Sigma})$ ,  $\lambda^{*}\rightarrow \overline{\lambda}$  and  $\epsilon$  is some sufficiently small positive constant. Since  $\overline{\lambda}\neq 0$ , we achieve the optimal convergence rate  $n^{-1 / 2}$  of estimating  $(\mu^{*},\Sigma^{*})$  within a sufficiently small neighborhood of  $\overline{G}$  under metric  $\kappa$ . These results imply that even though the convergence rate of estimating  $G_{*}$  may be extremely slow
when  $G_{*}$  moves over the whole space  $\Xi_2(l_n)$  (global convergence), such convergence rate can be at standard rate  $n^{-1/2}$  when  $G_{*}$  moves within a sufficiently small neighborhood of some appropriate parameters  $\overline{G}$  (local convergence).
As we have seen from the convergence rate results from location-covariance multivariate Gaussian distributions, the heat PDE structure plays a key role in the slow convergence rates of location and covariance parameters as well as the mismatch of orders of these rates.
B Convergence Rate of Density Estimation
B.1 General Theory and Proof of Theorem 2.3
We now describe the convergence rate of density estimation under the Hellinger distance in detail and give a general result for the multivariate deviated model. We recall some popular notions in Empirical Process theory as follows. An  $\epsilon$ -net for a metric space  $(\mathcal{P}, d)$  is a collection of balls with radius  $\epsilon$  (with respect to metric  $d$ ) having union contains  $\mathcal{P}$ . The minimal cardinality of such  $\epsilon$ -nets is called the covering number and denoted by  $N(\epsilon, \mathcal{P}, d)$ . The logarithm of  $N(\epsilon, \mathcal{P}, d)$  is called the entropy number and is denoted by  $H(\epsilon, \mathcal{P}, d)$ . The bracketing number  $N_B(\epsilon, \mathcal{P}, d)$  is the minimal number  $n$  such that there exists  $n$  pairs  $(\underline{f}_i, \overline{f}_i)_{i=1}^n$  such that  $\underline{f}_i < \overline{f}_i$ ,  $d(\underline{f}_i, \overline{f}_i) < \epsilon$ , and their union covers  $\mathcal{P}$ . The logarithm of  $N_B(\epsilon, \mathcal{P}, d)$  is called the bracketing entropy number and is denoted by  $H_B(\epsilon, \mathcal{P}, d)$ . In the following discussion, if  $\mathcal{P}$  is a family of density and we omit  $d$ , we understand that  $d$  is the distance associated with  $L^2(m)$ , where  $m$  is the Lebesgue measure.
Denote  $\mathcal{P}(\Xi) = \{p_G : G \in \Xi\}$  and  $\overline{\mathcal{P}}(\Xi) = \{(p_{G_+} + p_G) / 2 : G \in \Xi\}$  for the fixed true parameter  $G_*$ . The convergence rate can be deduced from the complexity of the set:
$$
\overline {{\mathcal {P}}} ^ {1 / 2} (\Xi , \epsilon) = \left\{\bar {p} _ {G} ^ {1 / 2}: G \in \Xi , h \left(\bar {p} _ {G}, p _ {G _ {*}}\right) \leq \epsilon \right\}, \tag {7}
$$
where for any  $G \in \Xi$ , we denote  $\bar{p}_G \coloneqq (p_G + p_{G_*)} / 2$ . We measure the complexity of this class through the bracketing entropy integral
$$
\mathcal {J} _ {B} (\epsilon , \bar {\mathcal {P}} ^ {1 / 2} (\Xi , \epsilon)) = \int_ {\epsilon^ {2} / 2 ^ {1 3}} ^ {\epsilon} H _ {B} ^ {1 / 2} (u, \bar {\mathcal {P}} ^ {1 / 2} (\Xi , \epsilon)) d u \vee \epsilon , \tag {8}
$$
where  $H_{B}(\epsilon, \mathcal{P})$  denotes the  $\epsilon$ -bracketing entropy number of a metric space  $\mathcal{P}$ . We recall assumption A2:
A2. Given a universal constant  $J > 0$ , there exists  $N > 0$ , possibly depending on  $\Theta$  and  $k$ , such that for all  $n \geq N$  and all  $\epsilon > (\log n / n)^{1/2}$ ,
$$
\mathcal {J} _ {B} (\epsilon , \overline {{\mathcal {P}}} ^ {1 / 2} (\Xi , \epsilon)) \leq J \sqrt {n} \epsilon^ {2}.
$$
Theorem B.1. Assume that Assumption A2 holds, and let  $k \geq 1$ . Then, there exists a constant  $C > 0$  depending only on  $\Theta$  and  $k$  such that for all  $n \geq 1$ ,
This result can be obtained by modifying the proof of Theorem 7.4 in [33]. Recall that we defined the function class
$$
\bar {\mathcal {P}} ^ {1 / 2} (\Xi , \epsilon) = \left\{\bar {p} _ {G} ^ {1 / 2}: G \in \Xi , h \left(\bar {p} _ {G}, p _ {G _ {*}}\right) \leq \epsilon \right\}, \tag {9}
$$
where for any  $G \in \Xi$ , we write  $\bar{p}_G = (p_G + p_{G_*)} / 2$ , and measure the complexity of this class through the bracketing entropy integral
$$
\mathcal {J} _ {B} (\epsilon , \overline {{\mathcal {P}}} ^ {1 / 2} (\Xi , \epsilon), \nu) = \int_ {\epsilon^ {2} / 2 ^ {1 3}} ^ {\epsilon} \sqrt {\log N _ {B} (u , \overline {{\mathcal {P}}} ^ {1 / 2} (\Xi , u) , \nu)} d u \vee \epsilon ,
$$
where  $N_B(\epsilon, X, \eta)$  denotes the  $\epsilon$ -bracketing number of a metric space  $(X, \eta)$  and  $\nu$  is the Lebesgue measure. We denote by  $P_G$  the distribution corresponding to the density  $p_G$ . The technique to prove this theorem is to bound the convergence rate by the increments of an empirical process:
$$
\nu_ {n} (G) = \sqrt {n} \int_ {\{p _ {G _ {*}} > 0 \}} \frac {1}{2} \log \frac {\overline {{p}} _ {G}}{p _ {G _ {*}}} d (P _ {n} - P _ {G _ {*}}),
$$
where  $P_{n} = \frac{1}{n}\sum_{i = 1}^{n}\delta_{X_{i}}$  is the empirical measure  $(X_{1},\ldots ,X_{n}\stackrel {iid}{\sim}p_{G_{*}})$ . We first recall Theorem 5.11 in [33] with the notations adapted from our setting:
Theorem B.2. Let  $R > 0$ ,  $k \geq 1$ , and  $\mathcal{G}$  be a subset of  $\Xi$ , which contains  $G_*$ . Given  $C_1 < \infty$ , for all  $C$  sufficiently large, and for  $n \in \mathbb{N}$  and  $t > 0$  satisfying
$$
t \leq \sqrt {n} ((8 R) \wedge \left(C _ {1} R ^ {2}\right)), \tag {10}
$$
$$
t \geq C ^ {2} \left(C _ {1} + 1\right) \left(R \vee \int_ {t / \left(2 ^ {6} \sqrt {n}\right)} ^ {R} H _ {B} ^ {1 / 2} \left(\frac {u}{\sqrt {2}}, \overline {{\mathcal {P}}} ^ {1 / 2} (\Xi , R), \nu\right) d u\right), \tag {11}
$$
$$
\mathbb {P} _ {\lambda^ {*} G _ {*}} \left(\sup  _ {G \in \mathcal {G}, h (\bar {p} _ {G}, p _ {G _ {*}}) \leq R} | \nu_ {n} (G) | \geq t\right) \leq C \exp \left(- \frac {t ^ {2}}{C ^ {2} \left(C _ {1} + 1\right) R ^ {2}}\right). \tag {12}
$$
Now we proceed to prove Theorem 2.3, the proof is divided into three parts: Bounding the tail probability of  $h(p_{\hat{G}_n}, p_{G_*})$  by sums of empirical processes increments using the chaining technique, bounding the empirical processes increments using Theorem B.2, and bounding the expectation of  $h(p_{\hat{G}_n}, p_{G_*})$  using its tail probability.
Step 1 (Bounding the tail probability  $h(p_{\tilde{G}_n}, p_{G_*})$  by sums of empirical processes increments): Firstly, by Lemma 4.1 and 4.2 of [33], we have
$$
\frac {1}{1 6} h ^ {2} (p _ {\hat {G} _ {n}}, p _ {G _ {*}}) \leq h ^ {2} (\overline {{p}} _ {\hat {G} _ {n}}, p _ {G _ {*}}) \leq \frac {1}{\sqrt {n}} \nu_ {n} (\hat {G} _ {n}).
$$
Hence, for any  $\delta >\delta_{n}\coloneqq (\log n / n)^{1 / 2}$ , we have
$$
\begin{array}{l} \mathbb {P} _ {G _ {*}} \left(h \left(p _ {\hat {G} _ {n}}, p _ {G _ {*}}\right) \geq \delta\right) \leq \mathbb {P} _ {G _ {*}} \left(\nu_ {n} \left(\hat {\lambda} _ {n} \hat {G} _ {n}\right) - \sqrt {n} h ^ {2} \left(\bar {p} _ {\hat {G} _ {n}}, p _ {G _ {*}}\right) \geq 0, \right. \\ \left. h \left(\bar {p} _ {\hat {G} _ {n}}, p _ {G _ {*}}\right) \geq \delta / 4\right) \\ \leq \mathbb {P} _ {G _ {*}} \left(\sup  _ {G: h (\bar {p} _ {G}, p _ {G _ {*}}) \geq \delta / 4} \left[ \nu_ {n} (G) - \sqrt {n} h ^ {2} \left(\bar {p} _ {G}, p _ {G _ {*}}\right) \right] \geq 0\right) \\ \leq \sum_ {s = 0} ^ {S} \mathbb {P} _ {G _ {*}} \left(\sup  _ {G: 2 ^ {s} \delta / 4 \leq h (\bar {p} _ {G}, p _ {G _ {*}}) \leq 2 ^ {s + 1} \delta / 4} | \nu_ {n} (G) | \geq \sqrt {n} 2 ^ {2 s} (\delta / 4) ^ {2}\right) \\ \leq \sum_ {s = 0} ^ {S} \mathbb {P} _ {G _ {*}} \left(\sup  _ {G: h (\bar {p} _ {G}, p _ {G _ {*}}) \leq 2 ^ {s + 1} \delta / 4} | \nu_ {n} (G) | \geq \sqrt {n} 2 ^ {2 s} (\delta / 4) ^ {2}\right), \\ \end{array}
$$
where  $S$  is a smallest number such that  $2^S\delta /4 > 1$ , as  $h(\overline{p}_G,p_{G*})\leq 1$ . Now we will bound each term above using Theorem B.2.
Step 2 (Bounding the empirical processes increments using Theorem B.2): In Theorem B.2, choose  $R = 2^{s + 1}\delta$ ,  $C_1 = 15$  and  $t = \sqrt{n} 2^{2s}(\delta /4)^2$ , we can readily check that Condition (10) satisfies (because  $2^{s - 1}\delta /4\leq 1$  for all  $s = 0,\dots ,S$ ). Condition (11) satisfies thanks to Assumption A3:
$$
\begin{array}{l} \int_ {t / (2 ^ {6} \sqrt {n})} ^ {R} H _ {B} ^ {1 / 2} \left(\frac {u}{\sqrt {2}}, \mathcal {P} ^ {1 / 2} (\Xi , R), \nu\right) d u \vee 2 ^ {s + 1} \delta = \sqrt {2} \int_ {R ^ {2} / 2 ^ {1 3}} ^ {R / \sqrt {2}} H _ {B} ^ {1 / 2} \left(u, \mathcal {P} ^ {1 / 2} (\Xi , R), \nu\right) d u \vee 2 ^ {s + 1} \delta \\ \leq 2 \mathcal {J} _ {B} (R, \mathcal {P} ^ {1 / 2} (\Xi , R), \nu) \\ \leq 2 J \sqrt {n} 2 ^ {2 s + 1} \delta^ {2} = 2 ^ {6} J t. \\ \end{array}
$$
So the conclusion of Theorem B.2 gives us
$$
\mathbb {P} _ {G _ {*}} \left(h \left(p _ {\hat {G} _ {n}}, p _ {G _ {*}}\right) > \delta\right) \leq C \sum_ {s = 0} ^ {\infty} \exp \left(\frac {2 ^ {2 s} n \delta^ {2}}{J ^ {2} 2 ^ {1 4}}\right) \leq c \exp \left(\frac {n \delta^ {2}}{c ^ {2}}\right), \tag {13}
$$
where  $c$  is a large constant that does not depend on  $G_{*}$ .
Step 3 (Implying the bound on supremum of expectation): Thus, we have