SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.3. Assume that  $f$  is distinguishable from  $h_0$  up to the first order. Furthermore,  $f$  admits uniform Lipschitz condition up to the first order. For any  $G$  and  $G_*$ , we define
$$
\mathcal {K} (G, G _ {*}) := | \lambda - \lambda^ {*} | + (\lambda + \lambda^ {*}) \| (\mu , \Sigma) - (\mu^ {*}, \Sigma^ {*}) \|.
$$
Then, the following holds:
$$
C. \mathcal {K} (G, G _ {*}) \leq V \left(p _ {G}, p _ {G _ {*}}\right) \leq C _ {1}. \mathcal {K} (G, G _ {*}),
$$
for all  $G$  and  $G_{*}$ , where  $C$  and  $C_1$  are two positive constants depending only on  $\Theta$ ,  $\Omega$ , and  $h_0$ .
See Appendix C.1 for the proof of Theorem 3.3. Since the MLE approach yields the convergence rate  $n^{-1/2}$  up to some logarithmic factor for  $p_{G_*}$  under the first order uniform Lipschitz condition of  $f$ , the result of Theorem 3.3 directly yields the convergence rate  $n^{-1/2}$  up to some logarithmic factor for  $G_*$  under metric  $\mathcal{K}$ . This entails that the estimation of weight  $\lambda_*$  converges at rate  $n^{-1/2}$  up to some logarithmic factor while the convergence rate of estimating  $(\mu^*, \Sigma^*)$  is typically much slower than  $n^{-1/2}$  as it depends on the rate of convergence of  $\lambda^*$  to 0 (cf. Theorem 4.1).
3.2 Non-distinguishable Settings
When  $f$  is not distinguishable to  $h_0$  up to the first order, the bound in Theorem 3.3 may not hold in general. In this section, we investigate the inverse bounds under the specific settings of non-distinguishable in the first-order models when  $h_0$  belongs to the family  $f(\cdot |\mu ,\Sigma)$ , i.e.,  $h_0(x) = f(x|\mu_0,\Sigma_0)$  for some  $(\mu_0,\Sigma_0)\in \Theta \times \Sigma$ . Our studies are divided into two separate regimes of  $f$ : the first setting is when  $f$  is strongly identifiable in the second order (cf. Definition 3.4), while the second setting is when it is not. For the simplicity of the presentation in the paper, we define  $(\Delta \mu ,\Delta \Sigma) = (\mu -\mu_0,\Sigma -\Sigma_0)$  for any element  $(\mu ,\Sigma)\in \Theta \times \Omega$ .
Definition 3.4 (Strong Identifiability). We say that  $f$  is strongly identifiable in the second order if  $f$  is twice differentiable in  $(\mu, \Sigma)$  and the following holds:
D2. For any positive integer  $k$ , given  $k$  distinct pairs  $(\mu_1, \Sigma_1), \ldots, (\mu_k, \Sigma_k)$ , if we have  $\alpha_{\eta}^{(i)}$  such that
$$
\sum_ {\ell = 0} ^ {2} \sum_ {| \eta | = \ell} \sum_ {i = 1} ^ {k} \alpha_ {\eta} ^ {(i)} \frac {\partial^ {| \eta | f}}{\partial \mu^ {\eta_ {1}} \partial \Sigma^ {\eta_ {2}}} (x | \mu_ {i}, \Sigma_ {i}) = 0,
$$
for almost all  $x\in \mathcal{X}$ , then  $\alpha_{\eta}^{(i)} = 0$  for all  $i\in [k]$  and  $|\eta |\leq 2$ .
3.2.1 Strongly Identifiable Settings
Now, we have the following result regarding the lower bound of  $V(p_{G}, p_{G_{*}})$  under the strongly identifiable settings of  $f$ .
Theorem 3.5. Assume that  $h_0(x) = f(x|\mu_0,\Sigma_0)$  for some  $(\mu_0,\Sigma_0)\in \Theta \times \Sigma$  and  $f$  is strongly identifiable in the second order and admits uniform Lipschitz condition up to the second order. Furthermore, we denote
$$
\begin{array}{l} \mathcal {D} (G, G _ {*}) := \lambda \| (\Delta \mu , \Delta \Sigma) \| ^ {2} + \lambda^ {*} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {2} - \min  \left\{\lambda , \lambda^ {*} \right\} \left(\| (\Delta \mu , \Delta \Sigma) \| ^ {2} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {2}\right) \\ + \left(\lambda \| (\Delta \mu , \Delta \Sigma) \| + \lambda^ {*} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \|\right) \| (\mu , \Sigma) - (\mu^ {*}, \Sigma^ {*}) \| \\ \end{array}
$$
for any  $G$  and  $G_{*}$ . Then, there exists a positive constant  $C$  depending only on  $\Theta$ ,  $\Omega$ , and  $(\mu_0, \Sigma_0)$  such that  $V(p_G, p_{G_*}) \geq C \mathcal{D}(G, G_*)$ , for all  $G$  and  $G_{*}$ .
The proof of Theorem 3.5 and the second-order uniform Lipschitz condition are deferred to Appendix C.2. Several remarks regarding Theorem 3.5 are in order:
(i) For any  $G$  and  $G_{*}$ , by defining
$$
\begin{array}{l} \overline {{\mathcal {D}}} (G, G _ {*}) := | \lambda^ {*} - \lambda | \| (\Delta \mu , \Delta \Sigma) \| \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| \\ + \| (\mu , \Sigma) - (\mu^ {*}, \Sigma^ {*}) \| \left(\lambda \| (\Delta \mu , \Delta \Sigma) \| + \lambda^ {*} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \right\Vert) \\ \end{array}
$$
we can verify that  $1/2 \leq \mathcal{D}(G, G_{*}) / \overline{\mathcal{D}}(G, G_{*}) \leq 2$ , i.e.,  $\mathcal{D}(G, G_{*}) \asymp \overline{\mathcal{D}}(G, G_{*})$ . The reason that we prefer to use the formation of  $\mathcal{D}(G, G_{*})$  over that of  $\overline{\mathcal{D}}(G, G_{*})$  is not only due to the convenience of the proof argument of Theorem 3.5 later in Appendix C but also due to its partial connection with Wasserstein metric that we are going to discuss in the next remark.
(ii) When  $f$  is a multivariate location family and is identical to  $h_0$ , i.e.,  $\mu_0 = 0$ , it was demonstrated recently in [16] that
$$
V \left(p _ {G}, p _ {G _ {*}}\right) \gtrsim | \lambda - \lambda^ {*} | \| \mu \| \| \mu^ {*} \| + \left(\lambda^ {*} \| \mu^ {*} \| + \lambda \| \mu \|\right) \| \mu - \mu^ {*} \|, \tag {5}
$$
which is also the key result for establishing the convergence rates of parameter estimation in their work. However, their proof technique only works for the location family and it is unclear what is the sufficient condition for the family of density functions beyond the location family such that the inequality (5) will hold. As the location family is strongly identifiable in the second order, we can verify that the lower bound in Theorem 3.5 and inequality (5) are in fact similar. Therefore, the result in Theorem 3.5 gives a generalization of inequality (5) in [16] under the strongly identifiable in the second order setting of  $f$ .
(iii) As indicated in [16], we can further lower bound the right-hand side of inequality (5) in terms of the second order Wasserstein metric  $W_{2}$  [35] between  $G$  and  $G_{*}$  when we present  $G$  and  $G_{*}$  as two discrete probability measures with two components. In particular, with an abuse of the notations we denote that  $G = (1 - \lambda)\delta_{(\mu_0,\Sigma_0)} + \lambda \delta_{(\mu ,\Sigma)}$  and  $G^{*} = (1 - \lambda)\delta_{(\mu_{0},\Sigma_{0})} + \lambda \delta_{(\mu^{*},\Sigma^{*})}$ , i.e., we think of  $G$  and  $G_{*}$  as two mixing measures with one fixed atom to be  $(\mu_0,\Sigma_0)$ . In light of Lemma E.1 in Appendix E, we have
$$
\begin{array}{l} W _ {2} ^ {2} (G, G _ {*}) \asymp \lambda \| (\Delta \mu , \Delta \Sigma) \| ^ {2} + \lambda^ {*} \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {2} \\ - \min  \left\{\lambda , \lambda^ {*} \right\} \left(\| (\Delta \mu , \Delta \Sigma) \| ^ {2} + \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| ^ {2}\right) + \min  \left\{\lambda , \lambda^ {*} \right\} \| \| (\mu , \Sigma) - (\mu^ {*}, \Sigma^ {*}) \| ^ {2}. \\ \end{array}
$$
Therefore,  $\mathcal{D}(G,G_{*})$  and  $W_2^2 (G,G_*)$  share the similar term  $\lambda \| (\Delta \mu ,\Delta \Sigma)\| ^2 +\lambda^*\| (\Delta \mu^*,\Delta \Sigma^*)\| ^2 -$ $\min \{\lambda ,\lambda^{*}\} \left(\| (\Delta \mu ,\Delta \Sigma)\| ^2 +\| (\Delta \mu^*,\Delta \Sigma^*)\| ^2\right)$  in their formulations. However, as  $\lambda \| (\Delta \mu ,\Delta \Sigma)\| +$ $\lambda^{*}\| (\Delta \mu^{*},\Delta \Sigma^{*})\| \geq \min \{\lambda ,\lambda^{*}\} \| |\langle \mu ,\Sigma \rangle -(\mu^{*},\Sigma^{*})|\| ,$  the remaining term in  $\mathcal{D}(G,G_{*})$  is stronger than that of  $W_2^2 (G,G_*)$  . Moreover, as  $\lambda = \lambda^{*}$  , we further obtain that
$$
\mathcal {D} (G, G _ {*}) / W _ {2} ^ {2} (G, G _ {*}) \asymp (\| (\Delta \mu , \Delta \Sigma) \| + \| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \|) / \| (\mu , \Sigma) - (\mu^ {*}, \Sigma^ {*}) \|.
$$
Hence, as long as the right-hand side term in the above display goes to  $\infty$ , i.e.,  $||(\Delta \mu + \Delta \mu^{*}, \Delta \Sigma + \Delta \Sigma^{*})|| \to 0$ , we have  $\mathcal{D}(G, G_{*}) / W_{2}^{2}(G, G_{*}) \to \infty$ . This strong refinement of the Wasserstein metric is due to the special structure of  $G$  and  $G_{*}$  as one of their components is always fixed to be  $(\mu_0, \Sigma_0)$ .
(iv) Under the setting when  $G_{*}$  is varied,  $d_{1} = 1$ , and  $d_{2} = 0$ , by means of Fatou's lemma the result from Theorem 4.6 in [19] yields  $V(p_{G}, p_{G_{*}}) \geq C'$ .  $W_{3}^{3}(G, G_{*})$  if the kernel density function  $f$  is 4-strongly identifiable (cf. Definition 2.2 in [19]) and satisfies uniform Lipschitz condition up to the fourth order where  $C'$  is some positive constant depending only on  $G$  and  $G_{*}$ . Since  $\mathcal{D}(G, G_{*}) \gtrsim W_{2}^{2}(G, G_{*}) \geq W_{1}^{2}(G, G_{*}) \gtrsim W_{3}^{3}(G, G_{*})$ , it indicates that the bound in Theorem 3.5 is much tighter than this bound. The loss of efficiency in this bound is again due to the special structures of  $G$  and  $G_{*}$  as one of their components is always fixed to be  $(\mu_{0}, \Sigma_{0})$ .
Unlike the convergence rate results from the strongly distinguishable in the first order setting between  $f$  and  $h_0$  in Theorem 3.3, the convergence rate of  $\lambda^*$  under the setting of Theorem 3.5 depends on the rate of convergence of  $\| (\Delta \mu^*, \Delta \Sigma^*) \|^2$  to 0 (cf. Theorem A.1). Additionally, the convergence rate of estimating  $(\mu^*, \Sigma^*)$  will be determined based on the convergence rates of  $\lambda^*$  and  $(\Delta \mu^*, \Delta \Sigma^*)$  to 0.
3.2.2 Weakly Identifiable Settings
Thus far, as  $h_0$  belongs to the family  $f$ , our results regarding the lower bounds between  $p_G$  and  $p_{G_0}$  under Total Variation distance rely on the strongly identifiable in the second order assumption of kernel  $f$ . However, there are various families of density functions that do not satisfy such an assumption, which we refer to as the weakly identifiable condition. To illustrate the non-uniform natures of  $V(p_G,p_{G*})$  under the weakly identifiable condition of  $f$ , we consider specifically a popular setting of  $f$  in this section: multivariate location-covariance Gaussian kernel.
Location-covariance multivariate Gaussian kernel: As indicated in the previous work in the literature [7, 25, 21], if  $f$  is a family of multivariate location-covariance Gaussian distributions in  $d$  dimension, it exhibits the heat partial differential equation (PDE) with respect to the location and covariance parameter  $\frac{\partial^2 f}{\partial \mu \partial \mu^\top} (x|\mu, \Sigma) = 2 \frac{\partial f}{\partial \Sigma} (x|\mu, \Sigma)$ , for any  $x \in \mathbb{R}^d$  and  $(\mu, \Sigma) \in \Theta \times \Omega$ . We can verify that this structure leads to the loss of the second-order strong identifiability condition of the Gaussian kernel. Note that, the PDE structure of the Gaussian kernel has been shown to lead to very slow convergence rates of parameter estimation under general over-fitted Gaussian mixture models (cf. Theorem 1.1 in [21]). For the setting of the multivariate deviated model, since the parameters  $\lambda^*$  and  $(\mu^*, \Sigma^*)$  are allowed to vary with the sample size, we may expect that the estimation of these parameters will also suffer from the very slow rate. In fact, we achieve the following lower bound of  $V(p_G, p_{G_*})$  under the multivariate location-covariance Gaussian kernel.
Theorem 3.6. Assume that  $h_0(x) = f(x|\mu_0,\Sigma_0)$  for some  $(\mu_0,\Sigma_0)\in \Theta \times \Sigma$  and  $f$  is a family of multivariate location-covariance Gaussian distributions. We denote
$$
\begin{array}{l} \mathcal {Q} (G, G _ {*}) := \lambda (\| \Delta \mu \| ^ {4} + \| \Delta \Sigma \| ^ {2}) + \lambda^ {*} (\| \Delta \mu^ {*} \| ^ {4} + \| \Delta \Sigma^ {*} \| ^ {2}) \\ - \min  \left\{\lambda , \lambda^ {*} \right\} \left(\| \Delta \mu \| ^ {4} + \| \Delta \Sigma \| ^ {2} + \| \Delta \mu^ {*} \| ^ {4} + \| \Delta \Sigma^ {*} \| ^ {2}\right) \\ + \Bigg (\lambda (\| \Delta \mu \| ^ {2} + \| \Delta \Sigma \|) + \lambda^ {*} (\| \Delta \mu^ {*} \| ^ {2} + \| \Delta \Sigma^ {*} \|) \Bigg) \Bigg (\| \mu - \mu^ {*} \| ^ {2} + \| \Sigma - \Sigma^ {*} \| \Bigg), \\ \end{array}
$$
for any  $G$  and  $G_{*}$ . Then, we can find a positive constant  $C$  depending only on  $\Theta$ ,  $\Omega$ , and  $(\mu_0, \Sigma_0)$  such that  $V(p_G, p_{G_*}) \geq C \mathcal{Q}(G, G_*)$ , for any  $G$  and  $G_*$ .
See Appendix C.3 for the proof of Theorem 3.6. A few comments with Theorem 3.6 are in order.
(i) Different from the formulation of  $\mathcal{D}(G,G_{*})$  in Theorem 3.5 where we have the same power between  $\mu$  and  $\Sigma$ , there is a mismatch of power between  $\| \Delta \mu \| ^2$ ,  $\| \Delta \mu^{*}\|^{2}$  and  $\| \Delta \Sigma \|, \| \Delta \Sigma^{*}\|$  in the formulation of  $\mathcal{Q}(G,G_{*})$ . This interesting phenomenon is mainly due to the structure of the heat equation where the second-order derivative of the location parameter and the first-order derivative of the covariance parameter is linearly dependent.
(ii) If we denote  $\mathcal{Q}'(G, G_{*}) \coloneqq \lambda (\|\Delta \mu\|^{4} + \|\Delta \Sigma\|^{2}) + \min \{\lambda, \lambda^{*}\} \left(\|\mu - \mu^{*}\|^{4} + \|\Sigma - \Sigma^{*}\|^{2}\right) + \lambda^{*}(\|\Delta \mu^{*}\|^{4} + \|\Delta \Sigma^{*}\|^{2}) - \min \{\lambda, \lambda^{*}\} \left(\|\Delta \mu\|^{4} + \|\Delta \Sigma\|^{2} + \|\Delta \mu^{*}\|^{4} + \|\Delta \Sigma^{*}\|^{2}\right)$ , then we can verify that  $\mathcal{Q}(G, G_{*}) \gtrsim \mathcal{Q}'(G, G_{*})$  for any  $G, G_{*}$ . If we treat  $G$  and  $G_{*}$  as two-components measures as in the remark (iii) after Theorem 3.5, we would have
$$
\mathcal {Q} ^ {\prime} (G, G _ {*}) \asymp W _ {4} ^ {4} \left(G _ {1}, G _ {1, *}\right) + W _ {2} ^ {2} \left(G _ {2}, G _ {2, *}\right), \tag {6}
$$
where  $G_{1} = (1 - \lambda)\delta_{\mu_{0}^{\prime}} + \lambda \delta_{\mu}$ ,  $G_{2} = (1 - \lambda)\delta_{\Sigma_{0}^{\prime}} + \lambda \delta_{\Sigma}$  and similarly for  $G_{1,*}$  and  $G_{2,*}$ . Here,  $(\mu_0,\Sigma_0) = (\mu_0',\Sigma_0')$ , and  $W_{2},W_{4}$  are respectively second and fourth order Wasserstein metrics. The formulations of  $\mathcal{Q}'(G,G_{*})$ , therefore, can be thought of as a combination of two Wasserstein metrics: one is with only parameter  $\mu$  and another one is only with parameter  $\Sigma$ . The division into two Wasserstein metrics can be traced back again to the PDE structure of the heat equation.
If  $\lambda = \lambda^{*}$  and  $(\| \Delta \mu \| ^2 +\| \Delta \mu^*\| ^2 +\| \Delta \Sigma \| +\| \Delta \Sigma^*\|) / (\| \mu -\mu^*\| ^2 +\| \Sigma -\Sigma^*\|)\to \infty$  , we will have that  $\mathcal{Q}(G,G_{*}) / \mathcal{Q}^{\prime}(G,G_{*})\rightarrow \infty$  . It proves that the result from Theorem 3.6 under the multivariate setting of Gaussian kernel is a strong refinement of the summation of Wasserstein metrics regarding location and covariance parameter in equation (6).
A consequence of Theorem 3.6 is that the convergence rate of estimating  $\lambda^{*}$  is determined by  $\| \Delta \mu^{*}\|^{4} + \| \Delta \Sigma^{*}\|^{2}$ , instead of  $\| (\Delta \mu^{*},\Delta \Sigma^{*})\|^{2}$  as in the strongly identifiable setting of  $f$ . Furthermore, we also encounter a phenomenon that the rate of convergence of estimating  $\Sigma^{*}$  is much faster than that of estimating  $\mu^{*}$ . In particular, estimating  $\Sigma^{*}$  depends on the rate in which  $\lambda^{*}(\| \mu^{*}\|^{2} + \| \Sigma^{*}\|)$  converges to 0 while estimating  $\mu^{*}$  relies on square root of this rate (cf. Theorem A.2).
4 Minimax Lower Bounds and Convergence Rates of Parameter Estimation
In this section, we study the convergence rates of MLE  $\widehat{G}_n$  as well as minimax lower bounds of estimating  $G_*$  under various settings of  $h_0$  and  $f$ . Due to space constraints, we present the theory in the distinguishable regime of  $h_0$  and  $f$ . Non-distinguishable cases, though more interesting, are deferred to Appendix A.
Theorem 4.1. (Distinguishable settings) Assume that classes of densities  $h_0$  and  $f$  satisfy the conditions in Theorem 3.3. Then, we achieve that
(a) (Minimax lower bound) Assume that  $f$  satisfies the following assumption S.1:
$$
\sup  _ {\| (\mu , \Sigma) - \left(\mu^ {\prime}, \Sigma^ {\prime}\right) \| \leq c _ {0}} \int \frac {\left(\partial^ {| \alpha |} f (x | \mu , \Sigma) / \partial \mu^ {\alpha_ {1}} \partial \Sigma^ {\alpha_ {2}}\right) ^ {2}}{f (x | \mu^ {\prime} , \Sigma^ {\prime})} d x <   \infty \text {f o r s o m e s u f f i c i e n t l y s m a l l} c _ {0} >
$$
0, where  $\alpha_{1} \in \mathbb{N}^{d_{1}}, \alpha_{2} \in \mathbb{N}^{d_{2}}$  in the partial derivative of  $f$  take any combination such that  $|\alpha| = |\alpha_{1}| + |\alpha_{2}| \leq 1$ .
Then for any  $r < 1$ , there exist two universal positive constants  $c_{1}$  and  $c_{2}$  such that
$$
\begin{array}{l} \inf_{\widehat{G}_{n}\in \Xi}\sup_{G\in \Xi}\mathbb{E}_{p_{G}}\bigg(\lambda^{2}\| (\widehat{\mu}_{n},\widehat{\Sigma}_{n}) - (\mu ,\Sigma)\|^{2}\bigg)\geq c_{1}n^{-1 / r}, \\ \inf  _ {\widehat {G} _ {n} \in \Xi} \sup  _ {G \in \Xi} \mathbb {E} _ {p _ {G}} \left(\left| \widehat {\lambda} _ {n} - \lambda \right| ^ {2}\right) \geq c _ {2} n ^ {- 1 / r}. \\ \end{array}
$$
Here, the infimum is taken over all sequences of estimates  $\widehat{G}_n = (\widehat{\lambda}_n,\widehat{\mu}_n,\widehat{\Sigma}_n)$
(b) (MLE rate) Let  $\widehat{G}_n$  be the MLE defined in equation (3), and the family  $\{p_G : G \in \Xi\}$  satisfies condition A2. Then, we have the convergence rate for the MLE:
$$
\begin{array}{l} \sup  _ {G _ {*} \in \Xi} \mathbb {E} _ {p _ {G _ {*}}} \left(\left(\lambda^ {*}\right) ^ {2} \| \left(\widehat {\mu} _ {n}, \widehat {\Sigma} _ {n}\right) - \left(\mu^ {*}, \Sigma^ {*}\right) \| ^ {2}\right) \lesssim \frac {\log^ {2} n}{n}, \\ \sup  _ {G _ {*} \in \Xi} \mathbb {E} _ {p _ {G _ {*}}} \left(\left| \widehat {\lambda} _ {n} - \lambda^ {*} \right| ^ {2}\right) \lesssim \frac {\log^ {2} n}{n}. \\ \end{array}
$$
Proof of Theorem 4.1 is in Appendix D.1. The results of Theorem 4.1 imply that even though we still can estimate  $\lambda^{*}$  at the standard rate  $n^{-1 / 2}$ , the convergence rate of  $(\widehat{\mu}_n,\widehat{\Sigma}_n)$  to  $(\mu^{*},\Sigma^{*})$  strictly depends on the vanishing rate of  $\lambda^{*}$  to 0. Therefore, the convergence rate of estimating  $(\mu^{*},\Sigma^{*})$  can be generally slower than  $n^{-1 / 2}$  as long as  $\lambda^{*}$  goes to 0 at a rate slower than  $n^{-1 / 2}$ .
We can also use the geometric inequalities developed in Section 3.2 to investigate the behaviors of  $\widehat{G}_n$  in the non-distinguishable settings. We will further see how the non-identifiability and singularity of the model affect the convergence rate for density estimation. Due to space constraints, the results for this setting are presented in Appendix A.
We now demonstrate the convergence rates of parameter estimation in the strongly distinguishable setting, where the choice of  $h_0$  is a standard Cauchy distribution, and  $f(\cdot |\mu ,\sigma^2)$  is the normal distribution with mean  $\mu$  and variance  $\sigma^2$ . The additional experiments with the non-distinguishable setting are deferred to Appendix F.
Assume that  $X_{1},\ldots ,X_{n}$  are i.i.d. samples drawn from the true density function  $p_{G_*}$  with  $G_{*} = (\lambda^{*},\mu^{*},(\sigma^{*})^{2})$  and we obtain the MLE  $(\hat{\lambda}_n,\hat{\mu}_n,\hat{\sigma}_n^2)$ . We consider two following cases:
(i)  $\lambda^{*} = 0.5$ ,  $\mu^{*} = 2.5$ ,  $(\sigma^{*})^{2} = 0.25$
(ii)  $\lambda^{*} = 0.5 / n^{1 / 4}$ ,  $\mu^{*} = 2.5$ ,  $(\sigma^{*})^{2} = 0.25$
Two histograms for samples from the density  $p_{G_*}$  with  $n = 10000$  corresponding to the above two cases are illustrated in Figure 1(a) and 2(a). For each case, we take into account multiple sample
(a) Histogram
(b) Rate of  $\hat{\lambda}_n$
(c) Rate of  $\widehat{\mu}_n$
(d) Rate of  $\hat{\sigma}_n^2$
Figure 1: Case (i)  $\lambda^{*} = 0.5$
(b) Rate of  $\widehat{\lambda}_n$
Figure 2: Case (ii)  $\lambda^{*} = 0.5 / n^{1 / 4}$ .
sizes  $n$  ranging from  $10^{2}$  to  $10^{4}$ . For each sample size  $n$ , we calculate the MLE  $(\hat{\lambda}_n, \hat{\mu}_n, \hat{\sigma}_n^2)$  via the EM algorithm [13] and measure the errors  $|\widehat{\lambda}_n - \lambda^*|, |\widehat{\mu}_n - \mu^*|$ , and  $|\widehat{\sigma}_n^2 - (\sigma^*)^2|$ . We repeat this procedure 64 times and plot the mean (blue dot) and quartile error bars (yellow bar) of the logarithm of estimation errors against the log of  $n$ . Theorem 4.1 suggests that the log convergence rate of  $\lambda^*$  is of order  $-1/2$  for all cases, and so is the rate for  $(\mu, (\sigma)^2)$  in the first case. Meanwhile, the convergence rates of  $(\mu^*, (\sigma^*)^2)$  in the second case are slower, which is in the order of  $-1/4$ . The empirical rates found in the experiments match this theoretical result, where the least square line shows that the logarithm of the rate for estimating  $\mu^*$  in case (i) is  $-0.5$  and that of the case (ii) is  $-0.27 \approx -1/4$  (similar for  $(\sigma^*)^2$ ).
We once again emphasize that this interesting phenomenon of the rates of convergence is due to the singularity and identifiability of the multivariate deviated model. Our theory and simulation have accurately shown quantitative convergence rates for parameter estimation when  $\lambda^{*}$  near the singularity point 0, where all pairs of  $(\mu^{*},\Sigma^{*})$  in model (2) give the same model. Together with the non-distinguishable settings, we provide a comprehensive study of the large-sample theory for this type of model, thanks to the newly developed notion of distinguishability that helps to control the linear independent relation between  $h_0$  and  $f$ . The developed optimal minimax lower bounds and convergence rates will certainly help Machine Learning practitioners understand better the role of sample sizes in the accuracy of estimation in the multivariate deviated model, and are also inspired theorists to study more about the estimation rate of complex hierarchical/mixture models.
In this paper, we establish the uniform rate for estimating true parameters in the multivariate deviated model by using the maximum likelihood estimation (MLE) method. During our derivation, we have to overcome two major obstacles, which are firstly the interaction between the known function  $h_0$  and the Gaussian density  $f$ , and secondly the likelihood of the deviated proportion  $\lambda^*$  vanishing to either one or zero. To this end, we introduce a notion of distinguishability to control the linearly independent relation between two functions  $h_0$  and  $f$ . Finally, we achieve the optimal convergence rate of the MLE under both the distinguishable and non-distinguishable settings.