Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models
Authors: Dat Do
Huy Nguyen
Khai Nguyen
Nhat Ho
Venue: NeurIPS 2023 poster (2023)
Keywords: Mixture Model
Minimax Rate
Maximum Likelihood Estimation

Abstract: We study the maximum likelihood estimation (MLE) in the multivariate deviated model where the data are generated from the density function $(1-\lambda^{\ast})h_{0}(x)+\lambda^{\ast}f(x|\mu^{\ast}, \Sigma^{\ast})$ in which $h_{0}$ is a known function, $\lambda^{\ast} \in [0,1]$ and $(\mu^{\ast}, \Sigma^{\ast})$ are unknown parameters to estimate. The main challenges in deriving the convergence rate of the MLE mainly come from two issues: (1) The interaction between the function $h_{0}$ and the density function $f$; (2) The deviated proportion $\lambda^{\ast}$ can go to the extreme points of $[0,1]$ as the sample size tends to infinity. To address these challenges, we develop the \emph{distinguishability condition} to capture the linear independent relation between the function $h_{0}$ and the density function $f$. We then provide comprehensive convergence rates of the MLE via the vanishing rate of $\lambda^{\ast}$ to zero as well as the distinguishability of two functions $h_{0}$ and $f$.

Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models
Dat Do*
Huy Nguyen
Department of Statistics University of Michigan at Ann Arbor Ann Arbor, MI 48109 dodat@umich. edu
Department of Statistics and Data Sciences The University of Texas Austin Austin, TXX 78712 huymmLutexas.edu
Khai Nguyen
Nhat
Department of Statistics and Data Sciences The University of Texas at Austin Austin, TX 78712 khainbeutexas edu
Department of Statistics and Data Sciences The University of Texas at Austin Austin, TX 78712
minhnhat@utexean edu
Abstract
We study the maximum likelihood estimation (MLE) in the multivariate deviated model where the data generated from the density function (1 x hho + X f=l(=" E²) in which ho is a known function, X [0, 1] and (x",")) are
unknown parameters estimate. The main challenges in deriving the convergence rate of the MIEE mainly come from two issues: (1) The interaction between the function ho and the density function f: (2) The deviated proportion X can go to the extreme points of [0, as the sample size tends to infinity. To address these challenges, develop the distinguishability condition capture the linear independent relation between the function hn and the density function f. We then provide comprehensive convergence rates the MLE via the vanishing rate of X to zero as well as the distinguishability of two functions hy and f.
Introduction
The goodness-of-frt test ||| is one of the foundational tools in statistics with several applications in data-driven scientific fields, namely kernel Stein discrepancy [1], point processes [37] and Bayesian statistics (32. etc. Given sample set of data and pre-specified distribution with density function hg. the test indicates whether the samples are reasonably distributed according to he (null hypothesis) or to another family of distributions {p(-10) : E θ} (alternative hypothesis). It is worth noting that knowledge about the null hypothesis distribution can come from prior knowledge of scientists. A key to understanding the statistical efficiency of testing is via the likelihood ratio and the maximum likelihood estimation (MLE) methods. 66]
While traditional testing problems often assume the null distribution An = p(-10)) and the alternative one p(-10) are from single simple family of distributions such exponential families, there also many problems in science require to test hgg against the alternative f(-10) that can be deviated from ho by distribution from a potentially different family. Specifically. in this paper, we consider the family of distributions named nultivariate deviated model with density functions defined as follows:
Pa(x)= = (1 A)ho(r) + Af(x/u,, E),
(1)
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

where E Rd, G := (X, E) are the model's parameters with À E [0.1] being the deviated proportion (from hio) and (Hi, Σ) E θ 52 are parameters of vector-matrix family of distributions f. where O C. R2 and f2 C gaxd being compact. When À 0, this recovers the null hypothesis distribution ho-
The deviated model can be motivated by many applications in science. For instance, in microarray data analysis, it can be used to detect differentially expressed genes under two or more conditions [ 2]. where hu is the uniform distribution and f(11p, Σ) is required to estimate. Many other applications can be seen in many contamination problems in astronomy and biology [29]. Besides, the deviated model can also be viewed as low-rank adaptation model in the domain adaptation problem (23], where ho is a pre-trained model on large data, and f is a simpler component to be estimated from the smaller data domain. Our goal in this paper is study the parameter estimation
rate of the deviated model.
Problem setup. Suppose that we observe i.idd samples Xi,... Xn from the true multivariate deviated model:
Pa. (z):= = (o)))()) + X "(<l"",E")
(2)
where G= (x", ) are true but unknown parameters with X * O. Throughout the paper, we allow G. to change with the sample size n (see Appendix F.I for a discussion). To facilitate our presentation, we suppress the dependence of G. on n. and then estimate G. from the data. The main focus of this paper to establish both uniform convergence rate and minimax rate for parameter estimation via the MLE approach, which is given by:
G E arg max
Σ
log (X:),
(3)
GEE
where G= :Rn En) and E = [01]
Contribution. There are two main challenges in studying the convergence rate of the MLE G.,: (1) The interaction between the function ho and the density function f. e.g., hg belongs to the family of f and (u², Σ*) approaches ho as the sample size n goes to infinity; (2) The deviated proportion X can go to the extreme points of [0, 1] as the sample size goes to infinity and make the estimation become more challenging, because when X 0, all the parameters (μ² Σ*) yield the same model. To address these singularity and identifiability issues, we first develop the distinguishability condition to capture the linear independent relation between the function hn and the density function f. We then study the optimal convergence rate of parameters under both distinguishable and non-distinguishable settings of the multivariate deviated model. Our theoretical results can be summarized follows:
1. Distinguishable settings: We demonstrate that as long as the function hg and the density function f are distinguishable, the convergence rate of Xn to X is (n-1/22 while (Tin S..) converges to (n") )²) at a rate determined by the vanishing rate of X as follows:
X II Σn.) (M", | On 0/2-1
It indicates that if X* goes O, the convergence rate of estimating (p", E") is slower than the parametric rate.
2. don-distin settings: When ho and I are not distinguishable, it becomes complicated to capture the convergence rate of the MIE. To shed light on the behaviors of the MLE under the non-distinguishable settings of multivariate deviated model, we specifically study the settings when ho belongs to the same family as f. namely, (.) = f(lpo, Mo) for some (Ho, Eo). To precisely characterize the rates of the MLE under this setting, consider the second-order strong identifiability of f, which requires the linear independence up to second-order derivatives of f with respect to its parameters. The second-order identifiability had also been considered in the literature to investigate
the convergence rate of parameter estimation in finite mixtures 99 28 22 21 20 19].
2.1. Strongly identifiable and non-distinguishable settings: When f is strongly identifiable in the second order, demonstrate that ||()/* AΣ*) X11 O(n -1/2) and
(I,, +
(10)|++I[[AA))) |I 1), E.) - "u'""">| =

---

where Au := 1 Mo and AE Σ Eg. It indicates that the convergence rate of Xn to X depends on that of (u* ) to (no, Eoo while the convergence rate of (An, Σn) to (u², -n)) depends on both the rate of X* to and the rate of ("²,>")) to (No, 2o). These results are strictly different from those in the distinguishable settings, which is mainly due to the non-distinuuishabbiity between ho and f.
2.2. Weakly identifiable and non-distinguishabl settings: When f is weakly identifiable, i.e., is not strongly identifiable in the second order, we specifically consider the popular setting when f is the density of multivariate Gaussian distribution. The loss of the strong identifiability of the Gaussian distribution is due to the following partial differential equation (PDE) between the location and scale parameters (the heat equation):
as (z) (z),, = +c (z| Σ).
0pôμu
Due the above PDE, the convergence rate of the MLE under this setting exhibits very dif- ferent behaviors from those under the strongly identifiable setting. In particular, we prove that [|Ap"]|* + | |2] AA X (nm1/22 and X'(1A,""1 1 147,,11 12 +1= +| ΙΛ DI - +11 - O
Notably, there is a mismatch in the orders of convergence rates of the location vector and covariance matrix. Furthermore, the rate of the deviated mixing proportion also depends on different orders of μ² to pr and Σ to Ep. Such rich behaviors of the MLE are mainly due to the PDE between the location and scale parameters.
Comparing to moment methods. We would like to remark that the results for the MLE under the non- distinguishable settings in the paper (much) tighter than those obtained from moment methods for a general mixture of two components in the literature. In particular, when f is multivariate Gaussian distribution with fixed covariance matrix, i.e., f is strongly identifiable in the second order and do not estimate Σ" an application of the results with moment methods from [6) to the deviated
models leads to ||مμ*)] 11 [| - X-1 = On 1133 and >* |'I|mmmmmem - 1 l| = O(n Onn-1/2,, which are
much slower compared to the results for the MLE in the strongly identifiable and nonndistinguishable settings, where (Am 14 m.moment) denote moment estimators of X and μ²"
When f is a multivariate Gaussian density and we estimate both the location vector and covari- ance matrix, i.e., f weakly identifiable, an adaptation of the moment estimators from the sem- inal work [8]]
to the multivariate deviated models shows that I*ll mmmoment 1 ||6 (1/2))), X I1| Smoment 1 = On (m122 and (|) ||6 + [ momeent - - = O(n -1/2), where Ammoment Ammomenn Symoment) are moment estimators of (A*, ",,,,"""" ).). These results are also much slower than those of the MLE in weakly identifiable settings.
Other related work. The hypothesis testing and MLE problem related to the multivariate deviated model had been considered in previous work, including the problem of detecting sparse homoge- 34,, the problem of determining the number 25 and the problem of multiple testing 30 12]. In particular, 4 considers testing problem for the deviated model with ho NO, 1) and f = N(p", 1) being one- dimensional Gaussian distributions. They show that no test can reliably detect A = 0 against X > 0 if A = o(n 1/2), while the Likelihood Ratio test can consistently do it when >"'² 2 n-1/2+e for any >> However, no guarantee for estimation of X and μ² is provided. In the same setting where is the density of a location Gaussian distribution, the convergence rate of parameter estimation in the deviated model had been studied the work of [6]. Since the location Gaussian distribution is a special case of the strongly identifiable distribution, our result in the strongly identifiable and non- distinguishable settings is a generalization of the results in [16], but with a different proof technique as their proof technique relies strictly on the properties of the location Gaussian distribution.
neous and heteroscedastic mixtures 14 15
of components 8
26
10
24
Organization. The paper is organized follows. In Section 2 we provide background on the identifiaiility and density estimation rate of the multivariate deviated model. Then, we establish the lower bounds of the Total Variation distance between two densities in terms of loss functions among parameters under both the distinguishable and dis-innuiishhblee settings in Section 3 Next, we characterize the convergence rates of parameter estimation as well as derive the corresponding minimax lower bounds in Section 4 In Section 5
we carry out simulation study to empirically verify our theoretical results before concluding the paper in Section ( Rigorous proofs and additional results are deferred to the supplementary material.
3

---

Notations. For any a, b E R. we denote a V vb: == max {a. and A bb ;= min {a. b). Next, we say that hg identical to f if hn(a)) = f( En) for some (Ho, Ea) E e Σ. For each parameter G' E E, let Eyg be the expectation taken with respect to product measure with density PG- Lastly, for any two density functions and q (with respect to the Lehesgue measure m), the Total Variation distance between them is given by ((p, q) == 2 [p(x)) g(x))dm(x)) while we define their squared Hellinger distance as (2 g) == 2 J pxx - v /q(x)]]d((())
2 Preliminaries
2.1
Identifiability Condition
Our principal goal in this paper is to assess the statistical efficiency of parameter estimation from the MLE method. To do that, we should be able guarantee the parameter identifiability of the deviated model (2). i.e., if (<)(
PG. (x) for almost surely æ E X' where G (X, μ,, Σ), then G = G..
That identifiability condition leads to the following notion of distinguishability between the density function (-) and the family of density functions {f(-lu, Σ): (p, Σ) E θ x Ω2}.
Definition 2.1 (Distingushaablitt)) We say that the family of density functions (.f(-l,, (1, Σ) x Q} (or in short, f) is distinguishable from hg if the following holds:
A1. For any two distinct components (μ1, Σn) and (u2. Σ2), if we have real coefficients Tι for 1 S i S 3 such that 77 < 0 and mı f (x) μ1, )n) + 72 ( u2) 22) + (z)) = 0, for almost surely x E Rd, then 771 = 7/2 = 7Js = O.
We can verify that as long as f distinguishable from ho. the parameter identifiability of our multivariate deviated model follows. In particular, assume that there exists G = (A, Σ) such that
(1 (1-X*) h=x)))((>)> + X f(lpp" Σ") = (1 + (f(z),,) E),
for almost surely I E X. The above equation is equivalent to (A - X (z)()z)) + f Σ²) (zlu, Σ) = 0. Assume that f is distinguishable from ho, then equation 44 indicates that if (n, Σ) * (n", E²). As a result, equation (4) becomes (A- /o((() (A* fz Σ) = 0. By applying the distinguishability condition again, we get A = > Therefore, the multivariate deviated model is identifiable.
we have À = x = O. Since X * 0 from our assumption, we obtain that (v. Σ)- =
In the following example, we will verify the distinguishability condition in Definition 2.1 given some specific choices of function ho and density f.
Example 2.2. (a) Assume that belongs
location family of density functions, i.e., fal Σ) fxx( - 14) for all x where Σ is a fixed covariance matrix. ho # fxx for almost surely x E X, then f is distinguishable from hg-
(b) When ho is a finite mixture of multivariate Gaussian densities and f belongs to a class of multivariate Student's density functions with any fixed odd degree of freedom 1, we get that is distinguishable from Ag-
(c) When f is identical to hg, then f is not distinguishable from ho-
2.2 Convergence Rate of Density Estimation
Our strategy to obtain the convergence rate of the MLE G, is by first establishing the convergence rate of density PG., and then studying the geometric inequalities between the parameter space and density space. For the former, the standard method is to use the empirical process theory [17 [33] while for the latter step, investigate those inequalities under various settings of distinguishability in Section[3] Due to space constraints and the popularity of empirical process theory, we choose to informally present a main result for yielding the parametric convergence rate for density estimation in this section. For full explanation and definition, readers are referred to Appendix B The convergence rate for density estimation be characterized by bounding the complexity of the parameter space =
via a function called bracketing entropy integral Jn(e, 1/2 (E, <)) (ef. equation
Theorem 2.3. Assume the following assumption holds:

---

A2. Given a universal constant J > 0, there exists N > O, possibly depending on E, such that for all n 2 N and all c /n) 1/2 we have JB(C, T (E,)) S J Jyne..
Then, there exists a constant C > O depending only on such that for all n 2 1,
sup Epo. M(P., PP.) S C G.EE
Therefore, in order to get the convergence rate for density estimators based on the MLE method, we only need check Assumption A2, which holds true for several parametric models [3]. For model, we give an example that it holds for a general class of f and hg-
Proposition 2.4. Suppose that both θ and St are compact, and (f(xlu,>) .) ; 0, E Q} is a vector-matrix family of densities being uniformly bounded, Lipschitz, and light tail, iee there exists constants M, L,, B, b1, b2, by >> O such that [f E) I M, (x/l,, Σ) - f(x E') S
T(Ix== p'll + | III) for all x E TR.", and
[f(z,, 23) S. b1 exp(-b2 '8)
||
for all (p. Σ) E Additionally, if the density ho is bounded, then the corresponding multivariate deviated model defined in equation (1) satisfies assumption A2.
Example 2.5. We can check that the location-scale Gaussian density [r] Σ) with E E Ω having eigenvalues bounded below by positive constant satisfies the condition of Proposition 2.4 This condition for ho is mild and is satisfied by most distributions such as Gaussian and t-distribution.
3 From the Convergence Rate of Densities to Rate of Parameters
The objective of this section is to develop a general theory according to which small distance between Pc and Pc. under the Hellinger distance (or Total Variation distance) would imply that and G. are also close under appropriate distance where G (X, μa, Σ) and G. = (X* p E"). By combining those results with Theorem 2.3 we can obtain the convergence rate for parameter estimation (cf. Section 4p. The distinguishability condition between ho and f implicitly requires that PG PG. would entail = G.; however, to obtain quantitative bounds for their Total Variation distance, we need stronger notions of both distinguishability and classical parameter identifiability, involve higher order derivatives of the densities hn and f. taken with respect to mixture model parameters. Throughout the rest of this section, we denote G (A, u4, Σ) and G. (X'; M". E²).
which
3.1 Distinguishable Settings
Definition 3.1 (First-order Distinguishabillty). We say that f is distinguishable from ho up the first order if f is differentiable in (14, E), and the following holds:
DI. For any component (n', E' θ x Ω if we have real coefficients y, for all a (a1, a2) E No x x2 |al = |ail + |a2l < such that
alal
(ho(z) + Σ
To dy ag (mln', 2') =0
aa
for all x E X, then 17 = 0 for all |a| < 1.
We can verify that the examples from part (a) and part (b) of Example 2.2 satisfy the first-order distinguishability condition. Next, we introduce a notion of uniform Lipschitz condition in the following definition.
Definition 3.2 (Uniform Lipschitz). We say that / addmiits uniform Lipschitz condition up to the first order if the following holds: there are positive constants 51, 62 such that for any R1, RR, Rs > 0, m e Rd, 72 E xd2, R1 S 1/2 (Σz) < 21/22 (E2) < R2, || ll. || < Rs, f1, O.E,, Eg f we can find positive constants C(R1, R2) and C(Ra) such that for all x E X,
(of
@ff
0f
(c|p2, E) 0μ
(a) (x/ Σ)
< C(RR, Re)l uell |
Dp
tr ((2) (01Re, 21)
0f
0f
72 ) C(Ra) - 1
(m| 2nح
5

---

Now, we have the following results characterizing the behavior of V (pG. PG.) regarding the variation of G and G..
Theorem 3.3. Assume that I is distinguishable from ho up to the first order. Furthermore, f admits uniform Lipschitz condition up to the first order. For any G and Gu,, we define
KC(G,G.) : ==) 1 (*+
En)
Then, the following holds:
C.K G.) S (pc.Pc..) S C,K,G
for all and G., where and C₁ are two positive constants depending only on θ Q and ho-
See Appendix C.1 for the proof of Theorem 3.3 Since the MLE approach yields the convergence rate n 1/2 11/2 up to some logarithmic factor for PG. under the first order uniform Lipschitz condition of f. the result of Theorem 3.3 directly yields the convergence rate n 1/2 up to some logarithmic factor for G. under metric IC. This entails that the estimation of weight X, converges at rate n -1/2 up to some logarithmic factor while the convergence rate of estimating (n²,, E²) is typically much slower than 1122 as it depends the rate of convergence of X (cf. Theorem 4.1
3.2 Non-distinguishable Settings
When f is not distinguishable to hg up to the first order, the bound in Theorem in general. In this section, we investigate the inverse bounds under the specific settings of non-
3.3 may not hold
distinguishable in the first-order models when hg belongs to the family f(|lu, 2), i.e., ho(x)) [= Σo) for some (No, Σo) E θ x Σ. Our studies are divided into two separate regimes of f: the first setting when f is strongly identifiable in the second order (cf. Definition 3.4), while the second setting is when it is not. For the simplicity of the presentation in the paper, define (Aμ, AΣ) = (μ- Ho: Σ Σo) for any element (p. Σ) E e x Ω.
Definition 3.4 (Strong Identifiability), We say that f is strongly identifiable in the second order if is twice differentiable in (n, Σ) and the following holds:
D2. For any positive integer ks given & distinct pairs (m1, Ei)...., such that
..... (Mi, Ex), if we have an (i)
k
a
(2lps, Ex) = 0,
8pm mym
2= in|=ti=1
(8) for almost all x E X, then on = 0 for all i E [k] and Inl S 2.
3.2.1 Strongly Identifiable Settings
Now, we have the following result regarding the lower bound of (pa.Pa.) under the strongly identifiable settings of f.
Theorem 3.5. Assume that ha(xx) = f(xx po, Ea) for some (po: Eo) E θ and I is strongly identifiable in the second order and admits uniform Lipschitz condition up to the second order. Furthermore, we denote
D(G,G.) = Al(AA,, A + X (I A - min {A, }*} (II(A,, A,)) + (Al[Ap,, AE)I + *"I((p" AE")I) E) (1""E"))
for any G and G.. Then, there exists a positive constant C depending only on θ Q and (No, En) such that (Pp.PG..) 2 C.D(G,G,, for all G and G..
The proof of Theorem 3.5 and the second-order uniform Lipschitz condition are deferred Ap- pendix C.2 Several remarks regarding Theorem 3.5 are in order:
(i) For any G and G., by defining
D(G, G.) = ^X- AII(A,, A
+ [1(,,, - (")))))) AΣJI + AAE")))

---

S, D)) D(G, The reason that we prefer to use the formation of DiG, G.) over that of DiG G.) is not only due to the convenience of the proof argument of Theorem 3.5 later in Appendix C but also due to its partial connection with Wasserstein metric that we are going to discuss in the next remark.
we can verify that 1/2 &
(ii) When f is multivariate location family and is identical to ha, i.e., No 0, it was demonstrated recently in [6] that
V(pa.pa.) 2 AA |||||||| (A*]) || AlIe|DII 1²"l
which is also the key result for establishing the convergence rates of parameter estimation in their work. However, their proof technique only works for the location family and it is unclear what is the sufficient condition for the family of density functions beyond the location family such that the inequality (5) will hold. As the location family is strongly identifiable in the second order, we can verify that the lower bound in Theorem 3.5 and inequality (5) are in fact similar. Therefore, the result in Theorem 3.5 gives generalization of inequality (5 in [16 under the strongly identifiable in the second order setting of f.
further lower bound the right-hand side of inequality 5 in terms of the second order Wasserstein metric W2 35 between G and G. when we present G and G_ as two discrete probability measures with two components. In particular, with abuse of the notations we denote that G = (1 me + X6 and G = (1 - 2) + X ,E',, i.e. we think of G and G. as two mixing measures with one fixed atom to be (No, Ea). In light of Lemma E.1 in Appendix E we have
(iii) As indicated in [16).
G.) x Al, AEJI + X'I(App, |
- min (A. [(AA,, + II(A,", )E)) + min (2, [1]((, Σ) - -,,,E""|1" E')
Therefore, D(G, and W:3(G, G.) share the similar term AI,, AΣ)I + II 1
min fA, A*} [(A,, 11 | in their formulations. However, as A/ AΣ||+
I'Ap",, AE 2 min { (1,,
x) (n²², El the remaining term in (DG,G.) is stronger
than that of W (G, G.). Moreover, as = x", we further obtain that
G G.G = (I(As, AΣ + [|A,,", /E//11/ E) - E")].
Hence, long the hind side term in the above display goes to oo, i.e., |1 + Ap", AE AE"I|| + O, we have D(G, G+) (G. G.) + co. This strong refinement of the Wasserstein metric is due to the special structure of G and G, as one of their components is always fixed to be (Po, En).
(iv) Under the setting when G. is varied, d1 1, and d2 = 0. by means of Fatou's lemma the result from Theorem 4.6 in 19 yields V (pG. pc.) > C. WG G.) if the kernel density function is 4-strongly identifiable (cf. Definition 2.2 in 11) and satisfies uniform Lipschitz condition up to the fourth order where C" is some positive constant depending only on G and G.. Since D(G,G. G.) 2 W2 Gx) > WF G.) Z (G G.), it indicates that the bound in Theorem 3.5 much tighter than this bound. The loss of efficiency in this bound is again due to the special structures of G and G. as one of their components is always fixed to be (Ho, Eo).
Unlike the convergence rate results from the strongly distinguishable in the first order setting between I and ho in Theorem 3.3] the convergence rate of X under the setting of Theorem 3.5 depends on the rate of convergence of [[ (Ap", AΣ*) I12 0 (cf. Theorem A.1 Additionally, the convergence rate of estimating (μ² Σ") will be determined based on the convergence rates of X* and (Ayr", AE") to O.
3.2.2 Weakly Identifiable Settings
Thus far, as hy belongs to the family f. our results regarding the lower bounds between PG and PG, under Total Variation distance rely on the strongly identifiable in the second order assumption of kernel f. However, there are various families of density functions that do not satisfy such an assumption, which we refer to as the weakly identifiable condition. To illustrate the non-uniform natures of V (PG, under the weakly identifiable condition of f, we consider specificaly popular setting of f in this section: multivariate Docation-covariane Gaussian kernel.
7

---

Location-covariance multivariate Gaussian kernel: As indicated in the previous work in the literature 7 25 2n, if f a family of multivariate oocaion-covariiace Gaussian distributions in
25
dimension, it exhibits the heat partial differential equation (PDE) with respect to the location and
f
0f (xlps, E), for any x E I2 and (H, 2) E + x S2 We 0
covariance parameter 8p0uu (z/p, 2) -
verify that this structure leads to the loss of the second-order strong identifiability condition of the Gaussian kemel.. Note that, the PDE structure of the Gaussian kernel has been shown to lead to very slow convergence rates of parameter estimation under general over-fitted Gaussian mixture models (cf. Theorem 1.1 in (2I)) For the setting of the multivariate deviated model, since the parameters X and (μ², , Σ²) are allowed vary with the sample size, may expect that the estimation of these parameters will also suffer from the very slow rate. In fact, we achieve the following lower bound of V (pa. PG.) under the multivariate pecatio--ovaarinee Gaussian kernel.
Theorem 3.6. Assume that ho(x)) = f= for
for some (no, Ea) E E and is a family of multivariate location-covariane Gaussian distributions. We denote
Q XI1A || + X ( I1 |4
min (A, A ||سس|| | + A| I + [AET)))
*(X1IAn| +IAEI) + + 11 + (AA (11-- - Σ11
for any G and G.. Then, we can find a positive constant C depending only on O Q and (po. En) such that V (pG.PG.) 2 C.Q(G, ). for any G and G..
See Appendix C.3 for the proof of Theorem 3.6 A few comments with Theorem 3.6 are in order.
(i) Different from the formulation of D(G, G.) in Theorem 3.5 where we have the same power
between μ and Σ, there is a mismatch of power between |Apl 2, A and || ||ΣΣ*| in the formulation of Q(G, G.). This interesting phenomenon is mainly due to the structure of the heat equation where the second-order derivative of the location parameter and the first-order derivative of the covariance parameter linearly dependent.
(ii) If we denote Q' G.) == A(Il + ||EE|)? + min {A, (|m- 101l" + || -E---11))
||A + !| + A1p11 + | then we can X1 + |3 min }
verify that Q(G, G.) X (G, G.) for any G, G.. If we treat G and G. as two-components measures as in the remark (iii) after Theorem 3.5 we would have
G'G x WI G1s) + W3
(6)
where G1 = (1 (1- 2)6H + A6 G2 = (1 A)oz; + Aō and similarly for G1,, and G2... Here, (Po. 2n) = (M6, M), and W2, W4 are respectively second and fourth order Wasserstein metrics. The formulations of Q' G.), therefore. can be thought of as a combination of two Wasserstein metrics: one is with only parameter and another one is only with parameter Σ. The division into two Wasserstein metrics can be traced back again to the PDE structure of the heat equation.
If = X3 and (|Ap|)2 2 |Ap 12+ | | that Q(G, G.) G.) + co. It proves that the result from Theorem 3.6 under the multivariate setting of Gaussian kernel is strong refinement of the summation of Wasserstein metrics regarding
(1) 1= + ||ا E I]) - oc, we will have
location and covariance parameter in equation (6)
A consequence of Theorem 3.6 is that the convergence rate of estimating X is determined by as in the strongly identifiable setting of f. Further- more, we also encounter phenomenon that the rate of convergence of estimating Σ* is much faster than that of estimating 1²" particular, estimating Σ* depends on the rate in which [[|++]| [[>*))))))) converges to 0 while estimating μ relies square root of this rate (cf. Theorem A.2
||بي" 1 + ||, instead of T " ||2
8

---

Minimax Lower Bounds and Convergence Rates of Parameter Estimation
In this section, we study the convergence rates of MLE G. well as minimax lower bounds of estimating G. under various settings of lig and f. Due to space constraints, we present the theory in the distinguishable regime of ho and f. Non-distinguishable cases, though more interesting, are deferred to Appendix A
Theorem 4.1. (Distinguishable settings) Assume that classes of densities ho and f satisfy the conditions in Theorem 3.3 Then, we achieve that
(a) (Minimax lower bound) Assume that J satisfies the following assumption S.I:
alal S(a 2
(S.1)
dx oo for some sufficiently small co where 01 E No. 02 Nda in the partial derivative of take any combination such that |a| = |ail lazl 1.
sup 11(,,)))- (w.)))=
[(alu', (ell'',) E')
Then for any < there exist two universal positive constants eı and e2 such that
(2 En) (e, 21)1)) 2 cin c/n-1rr
inf sup Epa
G, GES
inf sup Epc ( -21² 2 c2n c2n 1//
G,E3 GEB
Here, the infimum is taken over all sequences of estimates G = (Rn, Puas En).
(b) (MLE rate) Let Gn be the MLE defined in equation (3 and the family {pa G E} satisfies condition A2. Then, we have the convergence rate for the MLE:
(A2))21(77 S.m) -1=,,"
x log*
sup Epo- G.EZ
72
sup Epc. -App) G.éS
log" 2 n
n
Proof of Theorem 4.1 in Appendix D.I] The results of Theorem 4.1 imply that even though we still can estimate X at the standard rate n 1/2 the convergence rate of (Pn, En) to (4* E*) strictly depends the vanishing rate of X to O. Therefore, the convergence rate of estimating (μ²", E*) can be generally slower than 1/2 as long X goes to 0 at rate slower than n-1/2
We can also use the geometric inequalities developed in Section 3.2 investigate the behaviors of G in the non-distinguishable settings. We will further see how the -i-iieiiiifibbliity and singularity of the model affect the convergence rate for density estimation. Due to space constraints, the results for this setting are presented in Appendix
Experiments
We now demonstrate the convergence rates of parameter estimation in the strongly distinguishable setting, where the choice of ho is a standard Cauchy distribution, and f(|lu, o²) the normal distribution with mean and variance z², The additional experiments with the non-distinguishable setting are deferred to Appendix(F
Assume that X,.... Xx, are i.i.d. samples drawn from the true density function Pc. with G, = (X*, p"" (0*) and obtain the MIEE (2n, Pn, o2)). We consider two following cases:
(i) X- 0.5, μ² 2.5, (a ) 0.25;
(ii) X" 0.5/n2/4
2.5. (a*)² 0.25.
Two histograms for samples from the density Pc. with n 10000 corresponding to the above two
cases illustrated in Figure [ and 2 a). For each case, we take into account multiple sample

---

1- 056.1 6.32
1 1--55---.8
1- 051x 64
(a) Histogram
(b) A
(c) Rate Jin
Rate
Figure 1: Case (i) 0.5.
4 022
(a) Histogram
(5) Rate of A.
(c) Rate of Po
(d) Rate of
Figure Case (ii) X 0.5/n
sizes n ranging from 102 10 For each sample size n, we calculate the MLE (An Pr, d.) via the EM algorithm [13] and measure the errors {A, - >²I, 17,, 1411, and ? ( We repeat this procedure 64 times and plot the mean (blue dot) and quartile error bars (yellow bar) of the logarithm of estimation errors against the log of n. Theorem 4.1 suggests that the log convergence rate of X is of order 1/22 for all cases, and convergence rates of (n², (a*)²) in the second case are slower, which is in the order of -144. The empirical rates found in the experiments match this theoretical result, where the least square line shows that the logarithm of the rate for estimating μ in case (i) is -0.5 and that of the case (ii) is -0.27 1/4 (similar for (o*)²)
the rate for (u, (a) in the first case. Meanwhile, the
We once again emphasize that this interesting phenomenon of the rates of convergence is due to the singularity and identifiability of the multivariate deviated model. Our theory and simulation have accurately shown quantitative convergence rates for parameter estimation when X near the singularity point where all pairs of (n" >)*) in model ( give the same model. Together with the non-distinguishable settings, we provide comprehensive study of the large-sample theory for this type of model, thanks to the newly developed notion of distinguishability that helps to control the linear independent relation between hy and f. The developed optimal minimax lower bounds and convergence rates will certainly help Machine Learning practitioners understand better the role of sample sizes in the accuracy of estimation in the multivariate deviated model, and are also inspired theorists to study more about the estimation rate of complex hierarchical/mixture models.
6 Conclusion
In this paper, we establish the uniform rate for estimating true parameters in the multivariate deviated model by using the maximum likelihood estimation (MLE) method. During our derivation, we have to overcome two major obstacles, which are firstly the interaction between the known function ho and the Gaussian density f, and secondly the likelihood of the deviated proportion X vanishing to either one or zero. To this end, we introduce notion of distinguishability to control the linearly independent relation between two functions hg and f. Finally, we achieve the optimal convergence rate of the MLE under both the distinguishable and non-distinguishable settings.
Acknowledgements
NH acknowledges support from the NSF IFMI. 2019844 and the NSF Institute for Foundations of Machine Learning.
10

---

References
[1] L. Bordes, C. Delmas, and Vandekerkhove. Semiparametric estimation of a two-component mixture model where one component is known. Scandinavian journal of stafistics, 33(4):733- 752, 2006.
[2] I.. Bordes, S. Mottelet, and P. Vandekerkhove. Semiparametric estimation of a two-component mixture model. The Annals of Statistics, 34(3):1204-1232, 2006.
[3] T. Cai, X. J. Jeng, and J. Jin. Optimal detection of heterogeneous and heteroscedastic mixtures. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5):629-662. 2011.
[4] I. Cai, J. Jin, and M. G. Low. Estimation and confidence sets for sparse normal mixtures. The Annals of Statistics, 35(6):2421-2445) 2007.
[5] T. Cai and Y. Wu. Optimal detection of sparse mixtures against given null distribution. lEEE Transactions on Information Theory, 60(4):2217 2232, 2014.
[6] G. Casella and R. L. Berger. Statistical inference. Cengage Learning, 2021.
[7] H. Chen and J. Chen, Tests for homogeneity in normal mixtures in the presence of structural parameter. Statistica Sinica, 13:351-363 2003.
[8] H. Chen, J. Chen, and D. Kalbfteisch. A modified likelihood ratio test for homogeneity in finite mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(1):19-22) 2001.
[9] J. Chen. Optimal rate of convergence for finite mixture models. Annals of Statistics, 23(1)2221 233, 1995.
[10] J. Chen, P. Li, and Y. Fu. Inference on the order of a normal mixture. Journal of the American Statistical Association, 107:1096-1100 1105, 2012.
[11] W. G. Cochran. The x2 test of goodness of fit. The Annals of mathematical statistics, pages 315-345, 1952.
[12] N. Deb, Saha, A. Guntuboyina, and B. Sen. Two-component mixture model the presence of covariates. Journal of the American Statistical Association, (17(550):1220-185 2022.
[13] A. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1:1122, Sept. 1977.
[14] D. Do, N. Ho, and X. Nguyen. Beyond black box densities: Parameter learning for the deviated components. arXiv preprint urXiv:2202.02651, 2022.
[15] D. Donoho and Jin. Higher criticism for detecting sparse heterogeneous mixtures. Annals of Statistics, 32(3):962-994, 2004.
[16] Gadat, Kahn, C. Marteau, and C. Maugis-Rabusseau Parameter recovery in two-component contamination mixtures: The 12 strategy. In Annales de l'Institut Henri Poincaré, Probabilités Statistiques, volume 56, pages 1391-1418. Institut Henri Poincaré, 2020.
[17] E. Giné and R. Nickl. Mathematical foundations of infinite- statistical models. Cambridge university press, 2021.
[18] M. Hardt and E. Price. Tight bounds for learning a mixture of two gaussians. In STOC, 2015.
[19] Heinrich and J. Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. Annals of Statistics, (16666)):2444287777 2018.
[20] N. Ho and I.. Nguyen. Singularity structures and impacts on parameter estimation in finite mixtures of distributions. SIAM Journal on Mathematics of Data Science, ((4):730-758, 2019.

---

[21] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistics, 44:2726-2755 2016.
[22] N. Ho and X. Nguyen. On strong identifiability and convergence rates parameter estimation in finite mixtures. Electronic Journal of Statistics, 10:271-307, 2016.
[23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09668 2021.
[24] H. Kasahara and K. Shimotsu. Non-parametric identification and estimation of the number of components in multivariate mixtures. Journal of the Royal Statistical Society: Series (Statistical Methodology), 76(1):97-111, 2014.
[25] H. Kasahara and K. Shimotsu. Testing the number of components in normal mixture regression models. Journal of the American Statistical Association, 1006512)1632-1665 2015.
[26] Li and J. Chen. Testing the order of a finite mixture. Journal of the American Statistical Association, (5(491):1084-1092 2010.
[27] Q. Liu, J. Lee, and M. Jordan. A kernelized stein discrepancy for goodness-of-fi tests. In Pro- ceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 276-284. PMLR, 20-22 Jun 2016.
[28] X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. Annals of Statistics, 4(1):370-400, 2013.
[29] R. Patra and B. Sen. Estimation of two-component mixture model with applications to multiple testing. J. R. Stat. Soc. Series B Stat. Methodol., 78(4):869-88) -93, 2016.
[30] R. Patra and B. Sen. Estimation of two-component mixture model with applications to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(4):869 893, 2016.
[31] A. Schrab, B. Guedj, and A. Gretton. KSD aggregated goodness-of-fit test. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[32] S. Talts, M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman. Validating bayesian inference algorithms with simulation- based calibration, 2018
[33] van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
[34] N. Verzelen and E. Annals of Statistics, 15200:1920 2017.
Detection and feature selection in sparse mixture models.
[35] C.
Villani. Topics in Optimal Transportation. American Mathematical Society, 2003.
[36] Y. Wu and P. Yang. Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics, 48:1987-200 -00777 2020.
[37] Yang, V. Rao, and J. Neville. stein- seein-papangelou goodness-of-frt test for point processes. In Proceedings of the Secoodd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 226-235, PMLR, 16-18 Apr 2019.
12

---

Supplement to "Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models"
In this supplementary material, we present additional results and proofs. The minimax lower bounds and convergence rates parameter estimation in the non-distinguishble settings are presented in Section A The general theory for the convergence rates of densities and their proofs can be found in Appendix B Proofs of the geometric inequalities that relate the convergence of density estimation to that of parameter estimation in Section C while those for minimax lower bounds and convergence rates of parameter estimation are left in Appendix [ Then, we provide a necessary lemma for those results along with its proof in Appendix Finally, some discussion about the general setting of the paper is presented in Section followed by set of simulations to support the developed theory.
A Minimax Lower Bounds and Convergence Rates of Parameter Estimation under the Non-distinguishable Settings
Theorem A.I. (Strongly identifiable and non-distinguishable settings) Assume that classes of densities by and I satisfy the conditions Theorem 3.5 We define
(1.)) { G EE
la ((AE)) P/ SA snJ.
min Isisd 1Sw,edd
for any sequence {br}- Then, we achieve
(a) (Minimax lower bound) Assume that J satisfies assumption S.1 in Theorem 4.1 Then for any
< 1 and sequence (In), there exist two universal positive constants ei and 02 such that
Epc (X]I(AAAA En)) (v,, E)1²) 2 c/n
inf
sup
G.E3 (!.)
Epc (I4M A2II 1p2 uP)2 ozn 1/r
sup G, (lEi(l.)
inf
(b) (MLE rate) Let Gn be the MLE defined in equation
B.. and the family {PG G E} satisfies condition A2. Then, for any sequence {Ia} such that Ln/ log n + co,
Epo. (A*))I((А"" ( En) - (H", ((",)")13)
log"
sup G.E3,(b))
<
n
sup Epc. (I(4N1, AE"),, xmp) x (L.).
log" n
n
Proof of Theorem A.1 1 is in Appendix D.2 The results of part (b) are the generalization of those in Theorem 3.1 and Theorem 3.2 in [6] to the setting of strongly identifiable in the second-order kernel. The condition regarding the lower bound of À in the formation of E1 (ln) is necessary to guarantee that (M., E..) and X, are consistent estimators of (μ²,, Σ²) and X respectively. In particular, from the results equation (34) of the proof of Theorem A.I we have for any G. that
log n
E Epc. (A*) ,A )1 Ex)) -
[u) <
Therefore, for any S S d1 and S u, S dg we get
Epc. {
((A7n)):
log²2 n n²>²)² (AAr)))) log²
(A)r") -1
(AE uv (AD')) Jun
})}
2
-1)
Epc.
n(>*)² ((EE( )
13

---

It indicates that
log
VnA
min
Isigd, Su.
for the left-hand-side terms of the above display to go to for all si d1 and S u, S d2.
The results of Theorem A. imply that as long as the kernel functions are strongly identifiable in the second order, the convergence rates of fn to μº and Σn to Σ" are similar, which depend on the vanishing rate of (X* ""I AE") O. our next result of Gaussian distribution, we will demonstrate that such uniform convergence rates of different parameters no longer hold.
covariance multivariate
Theorem A.2. (Weakly identifiable and non-distinguishable settings) Assume that f is family of location-covarian multivariate Gaussian distributions, and ho = f( Eo) for some (no, Eo) E @ E. We define
(t.))) = { G E)
S 5
min
((A)),
1sidd.5u.edd
for any sequence }. Then, the following holds:
(a) (Minimax lower bound) For any < 1 and sequence {In), there exist two universal positive constants 61 and C2 such that
inf sup Epc (2 (2 || [12mn wll' + I1 -2112) G.E3GEE
cin
inf sup Epa GEEg(L))
[1Alll]) + [ 1 11------PP---pp 20)22o211
ozn c2n-1/1
(b) (MLE rate) Let Gn be the estimator defined in Then, for any sequence {nat such that ln/ logn + oo the following holds
sup Epo- (A492 (1A +1 {1 G.E3a(In)
1=1l 11 + 11En - -20113] s
log* n
+| 12n - 1xP) 5
log
sup G.eF,(1.)
Epc.
Proof of Theorem A.2 is in Appendix D.3 A few comments are in order:
(i) Similar the argument after Theorem A.1 the condition regarding À in the formulation of E2(h,)
to guarantee that (7n, S.,) and Xn are consistent estimators of (jr", E²*) and X*, respectively.
(ii) The results part (b) indicate that the convergence rate of estimating X* is generally much faster than that of estimating μ² regardless of the circumstance of (X*)² [||A,r"]** +| The non-uniformity of these convergence rates is mainly due to the structure of the heat partial differential equation, where the second-orded derivative of the location parameter and the first-order derivative of covariance parameter correlate.
(iii) From the results of part (b), it is clear that when ||p'1| + |AΣ"| A 0, i.e. (x²,E*) (n, Σ) * (No, Eo), and X A O, the convergence rate of 7n to 112 Furthermore, by using the result from part (a) of Proposition C.4 we can verify that
(ar)3 1p 1²11 + SSS -E1P)
log²
sup Epe. G_
where the supremum is taken over {G. (2(()) K(G., < c). and 7 = (1, and is some sufficiently small positive constant. Since X 0. we achieve the optimal convergence rate -1/2 of estimating (n²", E²) within a sufficiently small neighborhood of G under metrie AC. These results imply that even though the convergence rate of estimating G. may be extremely slow
X,
14

---

when G. moves over the whole space Ey(4,) (global convergence), such convergence rate can be at standard rate n 1/2 when G. moves within sufficiently small neighborhood of some appropriate parameters G (local convergence).
As we have seen from the convergence rate results from location-covariance multivariate Gaussian distributions, the heat PDE structure plays a key role in the slow convergence rates of location and covariance parameters well as the mismatch of orders of these rates.
B
Convergence Rate of Density Estimation
B.1 General Theory and Proof of Theorem 2.3
We now describe the convergence rate of density estimation under the Hellinger distance in detail and give general result for the multivariate deviated model. We recall some popular notions in Empirical Process theory as follows. An c-net for metric space (P. d) is collection of balls with radius e (with respect to metric d) having union contains P. The minimal cardinality of such e-nets is called the covering number and denoted by (e,, d). The logarithm of ((e, P, d) is called the entropy number and is denoted by (e,, P.d). The bracketing number N,,e,P,dl is the minimal number nt such that there exists n. pairs (f f()=1 such that f, f. ;. < e, and their union covers P. The logarithm of Ne P, dd is called the bracketing entropy number and is denoted by (e, P, d). In the following discussion, if P is family of density and omit d, understand that d is the distance associated with (m), where m is the Lebesgue measure.
Denote P(3) = {PG= : E E} and P(=) = {(PG. + pG)/2 2: : G E E} for the fixed true parameter G.. The convergence rate can be deduced from the complexity of the set:
(E,)= {
1/2 E E, a(pa.Pa.) s
PG
where for any G E E, we denote PG := (PG + pG.)/2. We measure the complexity of this class through the bracketing entropy integral
Jnle, 12
|
11/22,,, (u, 1/2 (E,e))du Ve,
(8)
22/223
where Hs(e, denotes the c-bracketing entropy number of a metric space P. We recall assumption A2:
A2. Given a universal constant 0, there exists > 0, possibly depending on O and k, such that for all n 2 N and all (logn/n)1/2
Jnle, 1212
(E,c)) & June..
Theorem B.1. Assume that Assumption A2 holds, and let k 2 1. Then, there exists a constant C depending only on O and i such that for all n 2 1,
sup Epa. hPa,, .P..) S CVlogn/n. G_EE
This result can be obtained by modifying the proof of Theorem 7.4 in 33 Recall that we defined the function class
1/12 p' = PG (pd
1/2 G h(Fa. Pc.) se)
where for any GE we write Pa = (PG + PG...) //2, and measure the complexity of this class through the bracketing entropy integral
Jnle, J/ (E,0,,)
Vlog Na (', u) ve,
J2
where NB x) denotes the c- c-bracketing number of metric space (X, 1)) and is the Lebesgue measure. We denote by P& the distribution corresponding to the density pc. The technique to prove this theorem is bound the convergence rate by the increments of empirical process:
Vr (G) vn /
1
Pa d(Pn Pa.),
= log
(p0.>0)
2
PG.
15

---

where Pa - dx, is the empirical measure (X)..... Xn itd PG.). We first recall Theorem 5.11 in 33 with the notations adapted from our setting:
Theorem B.2. Let R O, k 2 1, and G be subset of which contains G.. Given C₁ for all C sufficiently large, and for nt E IN and t >> O satisfying
I (8)) ^ (C,
(10)
and
RV f
$ $ (C +1)
11
u
(E,)(v))) du
(11)
V22
we have
P..G.
sup Po.
Mn [n. 2t 2t
Cep (TOa(C +
(T +1
(12)
Now we proceed prove Theorem 2.3 the proof is divided into three parts: Bounding the tail probability of h(GG. PG.) by sums of empirical processes increments using the chaining technique, bounding the empirical processes increments using Theorem B.2, and bounding the expectation of h(PG. PP..) using its tail probability.
Step 1 (Bounding the tail probability h(po. PG.) by sums of empirical processes increments):
Firstly, by Lemma 4.1 I and 4.2 of [33, we have
1 h22 (G.. po.) S ((PG,.Pa.) S Vn (C.).
1
16
Hence, for any 5 > 6n = (log n/n) 1/2, we have
Pa. (h(PG,, PG.) 2 8) < Pc. Gn) Vnh PG.) 20,
h(PG. PG.) 2 6/4
< Pc.
sup
[v,(G) - -
( (nn 20
G:M )25/4
<
Pc.
sup 1 (G)] 2 (
==0
Pc.
sup (SM.PP.:P..{2.t1...
[vn(G) (6/4))
==0
where is a smallest number such that 2/8/4 > 1, as h (PG.PG.) S 1. Now we will bound each term above using Theorem B.2
Step 2 (Bounding the empirical processes increments using Theorem B.2): In Theorem B.2
choose R = 2-+18, Ci = 15 and L = Vn122=66 (6/4)², we can readily check that Condition (TO)
satisfies (because 2--18/4 < 1 for all O,..., S). Condition (1] satisfies thanks to Assumption
A3:
R/V2
11
1/2
u
(1,
1)),v v du v v2
11 (u, (u,p1/2(E,11) R), du v 2
11/(20/m) (m)
222
R2//213
< 2J R, pb (1, R),v)
< 2J nn222+188 = Jt. Jt.
So the conclusion of Theorem B.2 gives us
n2
n282 S cexp
Pa. (h(Pa, PP..) 5)
exp
J2214
C²
(13)
where is a large constants that does not depend on G..
16

---

Step 3 (Implying the bound on supremum of expectation): Thus, we have
Eh(pe,, .P.) 1 P(h(PG.. Pc.) > 8)d6 S on + exp
nd²
(-70)
S čdn
8n
for some does depend on A G.. Hence, we finally proved that sup Ba, Bo,h, .P.)) S C Cy/log n/n.
G_E3
As consequence, we obtain the conclusion of the theorem.
B.2 Proof for Proposition 2.4
We further introduce some more notations that are required for the proof. Let (e,, P(E), || [lle) be
the e-covering number of (P(E), (1aa)) and NB(e, P(E), h) be the bracketing number of P
measured by Hellinger metric h. (c, P(E), = log (c P(E), h) called the bracketing entropy of ((=) under metric h. We want show that
JB(C, P 1 (E,c), = (Te
(6, (E.6), L'm)) v8
S Vnee,
(14)
for all > N large enough and e > (logn/n)!/ We proceed to show that claim
14 will be proved
log,N(e, P(E)), |+ log(1/e), Hp(c, P()), h) S (1g()/)),
(15)
(16)
and then prove claim (15) and 16
Because 23,2(),)) C (E) and from the defini-
Proof of that claim 16 implies claim tion of Hellinger distance,
14
HB(6,p' (E),)) = HB( 8 P).
HB(6,
HB
/2"
Now use the fact that for densities fi, f2, we have 1²((f f2)/2, (f2 + f+)/2) < (f,(f, one can readily check that H V2 h) < then
Hence, if claim (66) holds true,
HB(6 P 11 (). p) < H8 P(E), h) / log(1/8),
which implies that
JB(E, p 1/2 (2, elog 13 222 < nE?
for all > (log (/n)2/))//2 Hence, claim 14 is proved.
Proof of claim As X E [0, 1], we can choose an n-net for it with the cardinality no more Similarly, because θ and Ω are compact, we can cover them by hypercube [-a, ajdh and
than
[-b, b. d2ddd Hence, there exists c- nets for them with the cardinality no more than 2a d
and
d3 Let S be the Cartesian product of them. We have log |S| S log(1/e) and for every G (A, . E E, there exists G' = (X, p', (ip'""") E') E S such that - X1I, 1/'l || K' By triangle inequalities,
2b
<
PPG po al < I X| + II + ([(()), - [(ll))
thanks to the uniform bounded and Lipchitz assumptions. Hence,
log (e, (1.11x)) log(1/e).
17

---

Proof of claim (16 Now, from the entropy number, we are going to bound the bracketing number, we let ?7 S which will be chosen later. Let fi..... fN be a ne-et for P(E), where f((x)
(1 - (x)) + (xlps, E,). Let
{ b1 expe(p-bp ||=/1""),, M,
| 2 B1, otherwise
H(x)
(17)
an envelop for f(rlu, 2.). We can construct brackets lpt 1 follows.
p (x) (1- + X, max Σ.) 17, 0,.0,
p (x) (1 (1- ).)o(()) X, min E.) + n,. (x))
Because for each f P()), there fa such that |ls - fill 7 we have p s S p.. Moreover, for any B 2 B,
(
J ( pb) S Ao
/
1 2nda +
Hxx)dz)
||
S InB" + +B" exp (exp(1bog (-b2B"*)
(18)
where spherical coordinate to have
T/2 T(d/2+1)
de
B3a TB"
and
f [|x] 2B
exp (-bz [1=
ydt exp (-b2rda) dr
r2B
a/b3-1 exp(p - u)du
(change of variable u = -b2221))
bybg
//bs
Bbb
exp(-B3"")
byb2 1/b
Hence, in (18. if we choose B = (glog(1/n))))) then
J -
(
-
d/bs
pd S n
log
(19)
Therefore, there exists a positive constant which does not depend on v) such that
Hg(cnlog(1/n) P(E)) S log(1/n).
Let c (nog((1/))) we have log(1/c) log(1/)). which combines with inequality |I-I|, S h2 (e M))),)) < Hn(e?, (())), (1 log(1/e?) Z log(1/e).
leads to
Thus, we have proved claim (16).
C Proofs for Geometric Inverse Bounds
C.1 Proof of Theorem
The second inequality in Theorem 3.3 is straightforward from the equivalent form of W, (G. G.) in
Lemma E.T (see Appendix FD. Therefore, we will only focus on establishing the first inequality in that theorem. We start with the following key result:
Proposition C.I. Given the assumptions in Theorem 3.5 and G (λ, Fr, E) such that X E [0, and (n, Σ) can be equal to (Mo, Eo). Then, huve
lim inf { ((pG.PG.). KIG, G.G.G { K(G.G.)
18

---

Proof. The high level idea of the proof of Proposition C.3 is to utilize the Taylor expansion techniques previously employed in (9 28 [2 Indeed, following Fatou's 's argument from Theorem 3.1 in it suffices to demonstrate that
22 to obtain the conclusion of Proposition
C.3
|Pp - PG. K(G,G.)
lim inf < G,G.
K(G, V VVGG,
0.
Assume that the above conclusion does not hold. It implies that can find two sequences G,, (nn Mni En) and G. (Ni, M 27, such that K(G,, G) + O, XC(G,,, G) - 0. and l|pG, PG... loo.. /olllo///// (K(Gn G.,n) + 0 as n + co. Now, we only consider the most challenging setting of (pn,En) and (1C Σn)) when they share the same limit point (μ²², Σ¹). The other settings of these two components can be argued in the same fashion. Here, (n', Σ') is not necessarily equal to (No, Σo) or (1, Σ) as Am, Ar, can go O or 1 in the limit. Under that setting, by means of Taylor expansion up to the first order we obtain
PG. (x) -G.... (x)
(Λ An) Eo)
(2lppn ΣY]] + An ( En. f(z/p En)
K[Gnn G.
K(Gn, G.m)
(A Xn) 14o En) sallin [7)1
K(Gnn G.
(Un M 0= (In - En a!
glalf
Or (xlps, M2) + R1(x))
K((Gn, G..))
where Rx) is Taylor remainder and a (a1, a2) in the summation of the second equality satisfies an =a1 ,....,dd E Nd, 02 (a Ndaxda |a|
dy (1)
(1)
E
Que and
(2)
1gupedd2
a! = II ai
11) II and As f admits the first order uniform Lipschitz condition, have isu,odde
R1 (x) = Oll(pnn En) - (Mh, ++) for some > O, which implies that
Xn (=))///) G.,a) O (Mm En) (Pi E7)l1))
as n + co. Therefore, we can treat [pc, (I) PG... (r))/K((Gn Gm,n) as the linear combination of glalf ho 0o 2o) and dya (alpm 27)) when |a| 1. Assume that the coefficients of these terms
Of Of (alPo. Eo), and @En
go to O. Then, by studying the coefficients of (o 0o Eo), 8p:-
(0l Eo).
achieve
(X; An)/KK(G, G.m)) + 0, -M. G.,n) + 0, (En - un) Ga,n) - for all < i S di and 1 S u, S d2 where (a); denotes the i-th element of vector and Ann denotes the (u, v)-th element of matrix A. It would imply that Gu,n) O.
(An + (m 2n) (Hi,
Therefore, we achieve
1= |>n Aml + (An + (3))((E; (nig [551 /K(GGn Ga,n) -
glad f
a contradiction. Therefore, not all the coefficients of (6. En) and 0pmigr (xl Σi) go to O. If denote m. be the maximum of the absolute values of the coefficients of ho (ar(0n En) gialf
and aumr 05 (slma Ei), then we get 1/m,, A oo as + oo, i.., 1/m, is uniformly bounded.
Hence, we achieve for all x that
PG. (x) PG... (x) K(G G.,n)
glal
- ns ( Ex) +
Σ Ta Dym 02 (rlp', Σ' 0
mi
lals1
for some coefficients 1) and such that they are not all O. However, as f is distinguishable from ho up to the first order, the above equation indicates that 1 = To = for all (o) S 1, contradiction. As consequence, we achieve the conclusion of the proposition.
D
19

---

Now, assume that the conclusion of Theorem (.3) does not hold. It implies that we can find two sequences G2, and G'., such that An Por. Po Po:.,. lll/K( ,.,,)) 0 as n + cc. Since Q and Si are two bounded subsets, can find subsequences of G, and G such that K(G G1) and K.(G", +,7 G2) vanish to 0 as > oc where G1. G2 are some discrete measures having one component to be (Ho. Eg). Because An - O, we obtain Vipe, PP:... 0 as n + co. By means of Fatou's lemma, we have
0 = lim [IPOEr (1)o,,())-oor por... (z)) de 2 lim inf (Par, (x) - pot,,. (x)) dr = ViP,, (I), Pa, (z)) n=+50
n=+00
Due to the fact that f is distinguishable from bn up to the first order, the above equation implies that T1 = Gzz However, from the result of Proposition C.I regardless of the value T1 we would have
A₂ A 0 as + which is contradiction. Therefore, we obtain the conclusion of the theorem.
C.2 Proof of Theorem 3.5
Prior to presenting the proof of Theorem Lipschitz:
3.5 we introduce the definition of second-order uniform
Definition C.2 (Second-order Uniform Lipschitz). We say that f is uniformly Lipschitz up the second order if the following holds: there are positive constants 03, 64 such that for any R4, Rs. Rs > Vdd (E) < Rs. ||0]| & Rs. 0,, 02 E θ, E,, Eg E Ω
0, m E IRd, 72 E Rd2xds, R4 S (E) S there are positive constants C1 depending on (R1, Rs) and C2 depending on Rg such that for all x E X.
(
O 002
8 (x|01,
jJ 002
71
2)
(r 22 1)
71
$ 1-11-100 - 11112111113
Ir
(ir
(x/0, OE
0f
Σ) 72))) O
a (
Ir
Of
(x10,, Σ2) 72 ) 72)
LOE
0
- <c Mg 1 17/2
Now, we are back to the main proof. Utilizing the same Fatou's argument as that of Proposition to achieve the conclusion of the first inequality in Theorem 3.5 it suffices to demonstrate the following result
Proposition C.3. Given the assumptions in Theorem 3.5 and G = (Λ, n. Σ) such that X E 10, 1] and
(T, )) can be identical to (Ho, Mo). Then, the following holds
(a) If (uo, Ea) + (n,, E) and X 0, then
|IPa- - pa KIG,G)WKG, G,G.G K(G.G.)
lim inf
(b) If (No, En) = Ep, or (no, Σo) # (1,, E) and X = 0, then
ipa-po.lles D(G,G) G) DDG. DIG.GVDG..C)S 0.
lim inf G.G.
{ D(G.G.)
Proof. The proof of part (a) is essentially similar that of Proposition C.I therefore, we only provide the proof for the challenging settings of part (b). Here, we only consider the setting that (Po, Ex) = (n, Σ) as the proof for other possibilities of (No, Eo) can be argued in the similar fashion. Under this assumption, (no. En) = (Ho, Eo). G = (X. Go. Eo). and hx (. En) = f(x 10. Eo) for all x E X. Assume that the conclusion of Proposition CinittionCC3cee does not hold. It implies that we can find two sequences
Gn (Az, Mn, En) and G.n. (Nr Mh. Ee)) such that D(G,,, G) X, ll (Apn, AE,) 12 0. D(G G) = >x,]((App
AE;)|" + O. and ||pG. PG... Do....loo/D((C DD(G,, G.,,. + 0 the transparency of presentation, we denote An = [1(A A B, = [[(A,, [[AA);, AE and C, = 11 E.)) (Ki, X; = ||(A/n, AΣ,.) - (Apr, A Now, we have three main cases regarding the convergence behaviors of (Mn, En) and (14, )x))
cc. For
20

---

Both A - O and B, + 0. i... (Um, En) and (p Σn) vanish to (po. Σo) as - cc. Due to the symmetry between Xn and , we assume without loss of generality that X 2 Xn for infinite values of Without loss of generality, we replace these subsequences of Gn; Ga,n by the whole sequences of G, and G.,n. Now, the formulation of D(Gn, G.,a) is
Case 1:
D G. = (Xn - An) AnAA + A,, Cn.
Now, by means of Taylor expansion up to the second order, get
pG. (x) D(Gn, G.,n)
PG., (x)
(Λ; An)[f(2luuo Σo) - f(x M; + An Mmn Ex) - f(z/// 21
D(Gn, G.,))
2 (-A (- )
glals
(A -n))
jpm 00 (2/ xnn) + Ri (1(
a!
|0/==1
D G.,n)
(Ap Apt) (AEn A
glalf
Ap
(alph, En) + R2(xx)
dpmn Mya
D G.,.)
where (x) and R2(x) are Taylor remainders that satisfy Rí (z) = O and R2(x) = O(C2+7) for some positive number 7 due the second order uniform Lipschitz condition of kernel since density function f. From the formation of (G G.n)), since A, + Bn 2 C, (triangle inequality), as An 0 and B,, 0 is clear that
(An XWIR (2) G.,a) S (1) (x)) (x) - + 0
Xm ( G. s R2(x))/ ((A,, + B,)Cn) o -0(221//2 = O(C3) - 0
as n + oo for all X. Therefore, we achieve for all x E X that
(An- - - X2]lRi (x) An|Ra(x))) ( /D((Gn G.,) 0.
Hence, glaly
treat (x) Po... (z)) D(n, G.,n) as linear combination of
0pm (rln,, Σy) for all x and a = (a1, 02) such that I |a| s 2. Assume that all the coefficients of these terms go to 0 as co. By studying the vanishing behaviors of the coefficients
flal of dy 0gaz (x) P 27) |a| 1, we achieve the following limits
+ An(AA))): A(( /D(GG, G..)) +0 (An(AE)) X1 Boaim//DiGG. /D(G Ge,n) G.,n)
for all 1 S i S di and 1 S u, d2 where (a); denotes the i-th element of vector and Agu denotes the (u, v)-th element of matrix A. Furthermore, for any 1 < i, S (i and j can be equal),
glal
the coefficient of aym 00 (x)M², Ex)) when (a1); = (a11) = and 02 leads to
(A - An)(A + An - Apt),(Ap - -Awhis /D(Gn, Ga,)) +0. (20)
When =j, the above limits lead to
(x; An) ((A,)))) + An ((Apn A(G)))3 /D(Cn: Go,n) - O.
Therefore, we would have
(x= - An)1AAHH11 + Am Apn - APtlI D(Gnn,G.ni + O.
(21)
21

---

Now, as (An(S/m)) - -xi(d(t)s) /D(Gn, Gr O we obtain that
An (n(A))))(An))))) -AA0)))) /D(Gn, G.m)) 0,
An (A(()))(A))) -XI(A) ) Gon) 7 0.
(22)
Plugging the results from
22 into
20
ultimately achieve for any i,j I d that
x A, (An))) Ga,a)
(23)
Using the results from
20 and
23
would have
An (Ap pn) (Ann
Apnld
(12 (n) - 0,
D(Gn; Go,n)
D(Gm, Go,n)
X (AAp)))(AH Aptnss D(Gn, G.,n)
(X; (n)( (AAM)) +0
D(Gns Gayn)
for any 1 i, S d. Therefore, it leads to
Σ Am An ANG)s IsiSd
An Σ 1(Apa)il
E 1A ApGidd
Isisa
1535a
+0
D(Gn. G..n)
D(Gr, G.,r)
Σ AlllAA)ell(AAa Apt)al ISAEEd
X, Σ ((up)) ISiSd
Σ |(Att - APGNd
IsiSd
+
D( G.m)
D(Gn, Ga,n)
The above results mean that
An mallApallllen Apt Apt / DD Ga,n) - 0, 14 Aph - A4 D(Gn, Ga,a) - 0. (24)
By applying the above argument with the coefficients of a
0pai 05 (x) Mi;) when a₁ = 0 and for any two pairs v1), (u2, 12) (not neccessarily distinct) such that 1 S u1, U2 01, 12 I d or (a1); = and (az)uv = for any 1 < S and 1 S u, v S d, we respectively obtain that
(ag)uim (az)uam
(A; - M + An AAE - AE D G.,n) - 0,
Am. AR,, |,| AE, /D( G.,n) 0, X A AEE, A//I//D(Gn G.,n) -
X,, || A,ر, | I/D(/G. G.,n) + 0, A | - AE; I/D(Gn Ga,n) + 0. (25)
Combining the results from 21 and 25
leads to
D(G,, G. G. G.,.) + 0,
which is a contradiction. As a consequence, not all the coefficients of glal
Sn - (slu4, E go to
|a] < 2. Follow the argument of Proposition
C.I by denoting m, be the maximum of the
absolute values of the coefficients of glal
(ale xx) we achieve for all x that
1 PG., (x) - PG... (I)
ala (0l Σx) =
My. W3 (G,, G.,n)
[x/=1
where To R are some coefficients such that not all of them are 0. Due to the second order identifiability condition of f, the above equation implies that Ta = 0 for all a such that |o|| = which is contradiction. As a consequence, Case cannot happen.
22

---

Case 2: Exactly one of An and Bn goes to O. i.e., there exists at least one component among (pr En) and (LG, 2n) that does not converge (Nn, Eo) as n + cc. Due to the symmetry of A, and Bn, we assume without loss of generality that A, A 0 and B, - O, which is equivalent to (Pm, Em) - (j', Σ²) + (no, Σo) while (177 2x)) + (No, En) as n - co. We denote
( G. 12 An + An + Br.
Since [PG,. (z) (z)-PG., [z)l/D((G D( G,,n) + O, we achieve that [pc, (a) Pc.,, n (z)l/D'(G,, Gwn) - for all as D(G G.,n) 2 DD D'G, G.m)). By means of Taylor expansion up to the first order, we have
PG.. (x)
PG.. (I) D',,,, G.,n)
(X X Eo) - S(a// En,] + An End ( En)
D' G.,n)
(-Ang) ai (-AE))
0f
(2 -n))
(z/) Enn)
[in) + Ri (1)
10|==1
D'(Gn Gann)
Xas(a)/mas Fmn) Xn (xlpis En)
DiGG, G.
where R (I) is Taylor remainder that satisfies (Ar Am some positive number y > O. Since (Mm, En, and (ny, En) do not have the same limit, they will be different when n is large enough, i.e., n 2 M' for some value of M'. Now, as 2 M', [PG, (z)
G..) + 0 for
PG... (x)) (G, G. becomes a linear combination of 0f
0y (xlo) x7) for all |a| 1 and
f( 2, If all of the coefficients of these terms go to O, we would have A,/ DD G.,n) + O, (A%
/D G. - O, and (A; - (- ) uu /D' (G,, G. + for all 1 < i d1 and I < d2. It would imply that (A; An)B,, / (G,, G.,a) -
A..n/DD((Ca G.,n) + 0, and An B G.,n) + 0. These results lead to
= - 12m
Am B. + An + Bm, /D' Ga,n)= - O,
a contradiction. Therefore, not all the coefficients of 0ʃ 0umrd (alp Σi) and f( En) go to 0. By defining m/, to be the maximum of these coefficients, we achieve for all x that
1 PG. (x) - PG... 1/ [2:/wo, Σo) + Σ oa |a|=0
alais
pp 05 (wlpa E') 0,
m/, D',Gn, G.,n)
where o/ and Tr are coefficients such that not all of them are 0, which is a contradiction to the first order identifiability of f. As consequence, Case 2 cannot hold.
Case 3: Both An and B, do not go to 0, i... (Um, En) and (u 2x) do not converge to (Ho, Eo) as n -> co. Since D,, G., S K[G,,, G.,n) 1an X (A,, X;,C,, and [pc.. (x) PG... (z))/D(G,,, G.,,) + O, we achieve that [pc. (x) pG. (z))/X((G,, Ga.n) + for all æ. From here, by using the same argument as that of the proof of Proposition C.I we also reach the contradiction. Therefore, Case 3 cannot happen.
In sum, we achieve the conclusion of the proposition.
C.3 Proof of Theorem 3.6
For the simplicity of proof argument, will only consider the univariate setting of Gaussian kernel, i.e., when both μ and Σ = o² are scalars. The argument for the multivariate setting of Gaussian kernel can be argued in the rather similar fashion, which is omitted. Throughout this proof. we denote == 2². Now, according to the proof argument of Proposition and and Proposition C.3 to achieve the conclusion of the theorem suffices to demonstrate the following result:
Proposition C.4. Given G (). p. v) such that X E [0, 1] and (n, v) can be identical to (p), Then, the following holds
23

---

(a) If (Ho, vo) + (7, T) and x> 0, then
{ IIpG - Pc. po V(G,G)VK(GG.. s 20.
lim inf G,G,
G.)
(b) (No, vo) = (n, v) or (o) # (1, and X = 0, then
Q(G,G) QGG,, 3) - Q(G,G.)
- 1pa-vG.
lim inf
Proof. We will only provide the proof for part (b) since the proofs for part (a) can be argued in similar
fashion that of Proposition C.I Assume that the conclusion of Proposition C.4 does not hold.
implies that we can find two sequences G, = (An Pn: Un) and Gz.n = (X Pn 2%) such that Q G) - 0. Q(G.,, an G) - O, and IlPG. - Pc... oloo/ (Q G.,n) - 0 as n + co. Due to the symmetry between An and X we can assume without loss of generality that X An. Therefore, we achieve that
(G G..n) -A, X.)((AMI + +1auP)) (A + I
+ X 11 + (Ar x
U/ 1 1 + 2// 2/
In this proof, we only consider the scenario when (1 Aun) - 0 and (|Ap Ae
since the arguments for other settings of these two terms are similar to those of Case 2 and Case
in the proof of Proposition C.3 As being indicated in Section B.2.2] the univariate Gaussian kernel
contains the partial differential equation structure 02
0s
(1r|u, u) Dr (sllu,, v) for all μ E Q and
v E Ω. Therefore, for any = ((1,, 02) we can check that
89 mn Doaz (x)μu, v) 20z Ons (z)p,))
alaا
where 3 = (01 200y. Now, by means of Taylor expansion up to the fourth order, obtain
Σ
alals
(Ar --aa)
Opan dvez (xlu w1)) + R (xx)
PG, (x)
PG... (I)
1a|=1
an laz!
Q(Gn, G.,n)
Q G.,n)
(Apn Apt) (Ann Ang)
glalf
An
Σ
3pm droz (xlpa, vi) + R2(x))
a1laz!
n1=1
Q(Gn; Ga,n)
(X (
+ An (pn ( - Au;)
20 las !an! G.,a)
B=1 11:00
0 f
(IIKC x + (X; An + R2(x)
x
dnd
Q G.,n.
where R1 (x), R2 (x) are Taylor remainders and the range a1 a2 in the summation the second equality satisfies / = an 202. As Gaussian kernel admits fourth-order uniform Lipschitz condition, is clear that
(\ Am) ( An (x))) O(I(A,), Ac;) + [1 v.)) - (1h, 17) +
QGG G.,))
n do for some 7 0. Therefore, we can consider [po,, (x). - PC.... ( G.,a) I0 f linear combination of 0 (xln v for 1 < B < all of the coefficients of these terms go to
then we obtain
(A An) (-)(-A())))))) (- (-Ao + (Aun Apt) (Au, - A
Σ
2/22 laz!
L.g 01
0
(G G.,z)
for any 1 5 B < 8. Now, divide our argument with Lg into two key cases
24

---

Case 1: (A.((Sppi ( + A1 (1 ( (nncler Mil on -12)) + co.
It implies that as is large enough, we would have
Q(Gn. G.,)) & (A% Am) ( |A ?)) An(1A/m Aprl 2 Annn 1.
Combining the above result with 43 - 0 for all 1 3 S 8, we get
(1 - An) An)(-An (-An,) Ja jaz + X, (Api Apj)as (Avn A2
Σ H3 a1 a1,99
2 a !az!
(2; Am. 11 + A P) + Am Aphn Aup 4 + Ann - A 13)
Note that, when the denominator of the above limits is (X; - )n) / [+) Am An Aptl Ac, - - [t). the technique for studying the above system of limits with this denominator has been considered in Proposition 2.3 in [2T]. However, since the current denominator of H3 strongly dominates by the previous denominator, we must develop more sophisticated control of H3 as S f I 8 to obtain a concrete understanding of their limits. Due to the symmetry between X Xn and An. we assume without loss of generality that X Xn I X, for all (by the subsequence argument). We have two possibilities regarding An and X
Case 1.:: (Λ - Am)//An A aas n - oc. Under that setting, we define P.. max {); - An An and
M2 = max { AAnhl AuT Auml. |4 A Aun [1/2]
Additionally, we (A;, - An)/Pr + An/ & + -ai, (Ap, Aun)/MM + 02 A /M - -21,, and (Arn Av both 61, 62 are different from 0. Now, by dividing both the numerators and the denominators of H as S 3 S 4 by pr, Ma. we achieve the following system of polynomial equations
2b2. From here, at least one among 01 02, b,, b2 and
c c2 =0
(cfa &a3)) + bū, + cbb22 =0
3! (da)) 2a + cta + CBazb2 =
1 (Gat 2 + (cfaßb cz + 1 (c-b) -1)) = 0.
1
4.
2!
As being indicated in Proposition 2.1 in (1 this system will only admits the trivial solution, i.c.,
a1 02 by which a contradiction. Therefore, Case 1.1 cannot happen.
Case 1.2: (An - An)/An + 0, i.e., Xi/An + 1, as + co. Under that setting, if Mn E
max { A1mn - A Avn Aut 1/23. then we have
X, M = max - (X, Am - A, Au A,.l,, - deu?]
By dividing both the numerator and the denominator of H, by >,, Mns given that the new denominator of H1 goes to O. its new numerator also goes to O, i.e., we obtain
(A;
(pA))/))/// M,, + (Ap,, Ap +
Since (A; - Ann//An - 0 and Aما < M,, we have (Λ; - -..) (An, Mx) - 0. Therefore, we have (Ait, A/x,)) /M, - O. With the previous results, by dividing both the numerator and the denominator of Hz by M2 and given that the new denominator goes to 0, we have
(X, (-)(- { M + (Au, - A0 + O.
As (A:=-a)(-Aot) M + 0 (due to the assumption of M,), we get (Au, A 0.
These results imply that
max {1 A11 | AU,
Au - 0,
M2
25

---

which is a contradiction. Therefore, we would only have M,, E max { (AAr)) |Au; 11/3. For the simplicity of the proof, we only consider the setting when M, = | for all n (by subsequence argument). The setting that Ma = |/ for all n can be argued in the similar fashion. Now, if have
max (ASM - Ap A - A0
MM, A 0,
then by dividing the
numerator
and denominator of
H₃ with
An (max {1Ammn Aph Au A 1/2))) as S S 2, would achieve
max [AuR A,,1 Au Ae
max {1 A,t Aun Aug + 0,
contradiction. Therefore, must have
max (1SHA -ptl. Ac, - 1/2 //,, +
(26)
as 72 + oc. Now, we further divide the argument under that setting of M, into two small cases
Case 1.2.1: (A )n) S An Am A for all n (by subsequence argument). Since
M, A we would have (1, - A S Ap Ap l for all n and 1 S <1 S 4. From here, we obtain that
(A% Xn. Ay
Anl A p
0,
Xn 1m1Apm Mn 1
Am Apn Appl
(1; X, (.)
(A; >n) Ap
Xm AnlApn - App 0.
Am Ayn AuA
If Apn Aptl//A A 1/2 As O, by diving both the numerator and the denominator of H1 by
An App, A and given that the new denominator goes to 0, the new numerator must converge to
0, i.c. have
(A; - An) (nn Apú)) -1. -1.
However, since we have (Apm Apt Ap + O. the above result would imply that
(X; An)) (Am/A APt oc,
which is contradiction the assumption of Case 1.2.1.1. As consequence, must have Ay, - Aprl//Ar, Au,, 1/2 + O. Now, we also have that
(A; An) Ap
An/AUn Augl²
An Ann A 1/2
Anl Apt 1/2
(2; An) A
(2% An.) An
0.
An Aun - Augl 1/2
An - Aug
for all 1 i 5. Without loss generality, we assume that Av, Au for all n. We denote (-Anū) = (Av, - Au;) and Au; = (Av, - Aej)) for all n. From the result of (26). we would have |/' + co. Given the above results, by dividing the numerators and the denominators of H3 by (Dn, Au;) 8/2 for any B < 3, we would have the new denominators go Therefore, all the new numerators of these Hg also go to 0, i.e. we achieve the following system of limits
x Am
X An
X An
(qpt)
0.
{(
3 +
0,
0.
An
An
2
)
Since 1g|| + xo, the last limit the above system implies that (Λı 2n)
+ 9b
/An - O.
Combining this result with the second limit the above system yields that (> An (n)( ///mn 3/2 + O, which cannot happen. Therefore, Case 1.2.1 does not hold.
26

---

Case 1.2.2: - A > An - Aynll for all (by subsequence argument). If
(A, -,,) Ap S A, /,,, 1 for all n, the by using the same argument as that of Case 1.2.1.
we quickly achieve the contradiction. Therefore, we must have (2, - -x,) A > Xm Av, Aw 2 my
m2(AAppp (Ap,, ) and (Avn Ar = mg (Ay, Since M2 | we would have [m," S 1 for all 1 - S 3. Denote m - my for all S cist (by subsequence argument). The results of (26) lead to my mg = 0. Now by dividing both
Denote (Ayu, Ae²)) m -- (-A
the numerator and denominator of Hg by (Λ; (-)(=pp for any 1 s B < 4, the new
denominators of [l do not go to oc, we would also achieve that the new numerators of H [3] go to i.e. the following system of limits hold
X
An
Aa An (mp))
X; An
1
mi
+0, 1+
+ m2 An
+
m3 - 0,
An
An
1+ Xn
An (mnf)3) /6+
Xo An
m² +
mîms
12 0,
An
An
1+ x An (mp)* 244 An
X,
An (m²)
\; An (mz)² An
m2
+
/4+
(m²)²
+
/8 + O.
An
Combining with m O, the first and third limit of the above system of limits imply that m2 -/33. From here, the second and fourth limit yields that 1/6 + m2 + m2/2 = O, which is a contradiction. Therefore, Case 1.2.2 cannot hold.
11 (v + AM + (Aalnnn M2l + Unn - 021) - co.
Case 2:
We define
(G G2,n) = (2% 1 (Aun))(( -1 An(lApmn (m 11
+ 2 (Amil)
(I p/g 1-/2 10a w
We will demonstrate that Q(G,, G.nn)= x Q(G,,, G.,n). In fact, from the above formulation of (G G.,n), we would have that
(G, G.,n) s 2(x; - A1A - Apt + Aoj + AU, -
+|
+ Am An +1 + A5 Am [ - -Mil 10x - 2/2
< 2Q(Gn, G.,..)
where the first inequality is due the triangle inequality and basic inequality (a ()+ < 2(a² b2) and the second inequality is due to the following result
(Λ AM A Acti))
Ni [mn
1
11 2n
On the other hand, we also have that
(Gn,n, G.ma
2 (2% - Am) 1 2 +1Aunl
1M4 11711 2 1Up 0,11)
- An An ( X; (axl)) ( - 1 2 |1/ 1
@G.n G/,n)/2
where the last incquality is due to triangle inequality and basic inequality (a + 6)2 2(a² + b2). Therefore, we conclude that Q(Gn,, G..)) x (GGns G.n)). Now, since H3 0 for all 1 < 3 < 8, we would have that
(2; An)) Ap )on (- Au; yan X, (A,m. Apt) (Du, )
Σ 01.00
2 jeslaylay laz!
F8
0.
(G G.,))
Similar to Case 1, under Case 2 we also consider two distincts setting of / Ann
27

---

Case 2.1: Xn/An An A cc. Under this case, we denote
Mr max {1an Avr,, Aut Ao
From the assumption of Case 2, we would have
Aynn A,, O, AAv, AeNl/M, 0.
(27)
Due to the symmetry between (IA (A, and "0,41,,A,;;)) assume without loss of generality that M max [14 | Au Under that assumption, we have two distinct cases
Case 2.1.1: Mr An, 2 for all n (by the subsequence argument). From (27p. have | Ap, 1/ + 0, i.e., Apn/ Au" + 1. To be able to utilize the assumptions of Case 2, we will need study the formulations of F3 more deeply. In fact, when 3 1 simple calculation yields
A1 == Ap X, ( G.,n) + 0.
When 5 = 2, we have
F2 ( (n) An(Ap APT)) + (X -
+ An Arj 0.
Q(Gn G.,n)
Combining with the result of A1, it clear that
(A An) (Apn - APE) (A) Ap) Q(Gn,G...
Q(Gns G.,z)
Combining the above result with F-2 O, we would have
A2 (\x An) An N, An 0.
@(Gn, G.,))
Now, we have two small cases
Case 2.1.1.1: Au,/(Ann) + 0 n co. From since we have A0 + it implies that Av (Ap)) ? + O. Since Aptn/ AyC, Now, from the formulations of Q(Gn, G.,n) we have
1, we also have that Ae, (Ap()) + 0.
(Λ, An) Au, v / Q(Gn Gr.n) < Aual/ 12 0,
(Ar A,. A / G.,n) S A Apm 1 O,
(A; An (A,..) ( G.,n) / Ap, -
(28)
From the result that A2 + by multiplying A2 with Apt, Aut we would also have that
(x - (A), AA X Ae + O.
(29)
QGGn, G.m)
As \r A oo, we have two distinct settings of X²
Case 2.1.1.1.1: >; A 1. Using the result from 28 and the fact that Apn./ Apt, - 1, would obtain that
AA,
A Ga,n) + 0.
Combining the above result with (29) it leads to
(A (A,, QQG,, G..)) + 0.
(30)
Combining
28 and 30 we would achieve that
(X - Am) | A
|A 0.
QGGn, G.m))
28

---

From the formulation of (G Gr,n), the above limit implies that
E (n A (A + + [Inn 12/1 1- + 10/2 177l
@G Gu,n)
Due to the previous assumptions, we obtain that E max (An Apr,)? , An ) Avz)]
Q G.mn
By combining the results from
28 and 30%
we can verify that
(X, Xn)
(AAp)) + (Aun)
An (1 Appi)
+0
Q(Gns G.,n)
Q G.,r)
An (Apn)) pp ) (A
- Avi))
(Ar An (An A + O.
2(Gn,: G.,n.)
Q(Gn, G.n))
Therefore, we achieve F: 0, which is contradiction. As consequence, Case 2.1.1.1.1 cannot happen.
Case 2.1.1.1.2:
/n
1. Under this case, if we have
AmlAAu - Ap Am|AU, Au
max
000
(A; A (A; - Am AM)
then we will achieve that
- - A
(A; - An) 0. Am
(A,- An,, /O G.m) S min
A Ayn App
Aur
From here, by using the same argument as that of Case 2.1.1.1.1, we will obtain E - 0, which is contradiction. Therefore, we would have that
A - A >, A
max
A 00.
(31)
(X; A 12, (Aa - Am) |m 1
With that assumption, it leads QGG,, G.,m) x (X; (n (A x (2; -2;- A ( Ap,,/ Apn, - 1. Now, denote Apn, Apt T Aμū and Au Au; (Ap,)) From the assumption of Case 2.1.1.1, we would have that Ti + O and 7 + 0. By dividing both the numerator and the denominator of F3 by (A; (A)()))))))))))))))))))))))))) 3 the new denominators of F3 goes to O, we also obtain the numerator of this term goes to 0, i.e., the following holds
{11
An X Am (T
An
/6 +
T + 0.
2(1; Xx)
From (1)
have that An An(7T)²/(A;; An) A 0e and An (\; -An) A 0o. Therefore, since
Ti + 0 and Th + O, we would achieve that An (T By plugging these results the above limit, it implies that 11/6 O, which is contradiction. As consequence, Case 2.1.1.1.2 cannot hold.
An) - 0 and >n An) + 0.
Case 2.1.1.2: Av, A that / An > 1 as the argument for other settings of that ratio can be argued in the similar fashion. Since we have |Av, /vt A + 0. it leads Aut (Ap,,) A O. Combining with Apn/ Ap, + 1, implies that as n is large enough we would have
co. Under that case, will only consider the setting
max (AAp,) (A) S min (1Au,), Av
(32)
According the formulation of (GG,, G.,,), we achieve
(X; - An, < min X, Awn Amjil
(\; - Ae () An)lNol
(X; Am) A
@(Gn, Ga,n)
d Ar, A
An An 11
(A; An An))
x; Am A B.
29

---

If have B + O, we would get (A, Xn) O. Combining with (32) we can check
g G.,a)
that all the results in 28 30 and (31 hold. With similar argument Case 2.1.1.1, we achieve
@G (f,n)/((G,, G.,n) O, contradiction. Therefore, we must have B A 0. It implies that as n is large enough we must have
max (1a 3 max {1Ammn APT IAe, Au. Aol & (X, An) min {|Amnl, Ar
Furthermore, as (A; An)/An O, we obtain A A Autl co and A App 1 + 0c, i.c. Aun/Art - 1 With all of these results, (X, Av Denote (Ar, Ae;) RT)) Av,l. (Ayr, Ap)) kg) Ae; 1/2 and An
check that ( G.,n) X
kg] Au;11/2
1/ for all n. From all the assumptions we have thus far, get Kj 0, kg - 0. and [k A
oo. Additionally, as B A O, we further have >., 1/AA, - An) A oc and Am (((2))//0 / An) A 00.
By dividing both the numerator and the denominator of F3 and Fa respectively by (A >,) Am 3//2
and (Λ; Xn) Ae 2 as the new denominators of F3, F4 do not go infinity, obtain the new numerators of these terms go to O, i.c., the following holds
{
An
An
(kg)
(k2)
/6
In +
/2 - 0,
A An
Am
An - Am
An
An
(Ag)*+
(kT)
124+ (kg)
k (Ki
/44
1 +
(kT)) //8 -
X,
Am
Am
With the assumptions with kyp k2. and kg, we would have An
An
An
0,
KP + 0,
(k1)) +0
A
A An
x An
for any Si and S S If we denote kg - ki, by combining all the above results we achieve the following system of equations
- /6 + / = 0, k / 1/8 O,
which does not admit solution, contradiction. Hence, Case 2.1.1.2 cannot hold.
Case 2.1.2: M A for all n (by the subsequence argument). From 27 we would have
|Au, Ae; Pell/AAB, Au, under this case is rather similar to that of Case 2: therefore, we only sketch the key steps. By using
i.e., A - 1, and A4,,, /
0. The argument
the result that A + and A2 - would obtain that
(A; - An (Apm)² 34 + 3(An5)
(A: - An)(A)))) + (Apa - APO) (4 G.,w)
24 G.,
(A% An) AunAv, - AUCdEn Au,AC
(x;; - (An;) + (n - Aug)?
3D(Gn,Ga..
8Q(G G.,.)
(2n - Xn)
Apn APT + AC,
(Api ( Art)
4Q(Gn, G..a)
4Q G.. GoM.
As F4 - 0, we equivalently have
24Q G.... Apt An
(Aur)² SAHAA +3(A
A = (N, Xa)
APr -2(A(C))
Apn A Dej + AUn 0.
8Q(G,,, G.,n)
Under Case 2.1.2, we only consider the setting when (A 2//2nn - 0 as other settings of this term can be argued in the similar fashion that of Case 2.2. Since Apn A/ + 0.
30

---

we have (A Aen, - + O. As Aun Ar, - 1, we also further have that
+ 0 and
(a,,)\/An,, 0. Therefore, we have Au Ae, + and Ap,. Ae 0. Now, from the formulation of Q(G,, G.,n), we achieve
(A, - Xn) Ap) A ( G.,n) ) - A / 7 0,
(Λr Xn) Ae (A /Q ... - Ap, //AA,,,, 7 0,
(Ni An) Au, (A / G..a) < App /2 P/Auup + 0,
(X) Amm
Q G..) Apm Ap A + 0,
(Λ, pm A (Q G.,n) 1 A A I/ O.
Combining these results with A4 + O, we achieve (A; Xn An) ( G.,nn - 0. From here, casily verify that all the results in (31) hold. Thus, by using the same argument that of Case 2.1.1, we would get Q(G,, G.,))/ (G., G.,n) - 0, contradiction. As consequence, Case 2. 1.2 cannot happen.
Case 2.2: X,/An - CC. Remind that M2 - max {14 I |4 |Ar We can verify that [(G,, G.,n) S (M 4 By dividing both the numerator and the denominator of A and A2 respectively by Xr (M2))/2 and XA M,, given that the new denominators go 0 would obtain the new numerators also go to 0, i.e., we have the following results
An A (M
ApT (MM/(MM)) - 0.
- Ap, + An Ao, A, / [X,,M, + 0.
Since An + O, the first limit implies that APA/M + 0. Combining this result with the second limit, we obtain A /M, - O. Therefore, we would have M₁, max {14p,l?, Av., Without loss of generality, argued in the similar fashion. With these assumptions, | - l //, A oc, i.e., as is large enough we get AA, Auy I Ap, Now, we have two distinct cases
assume that Mr, || as the argument for other possibility of M', can be
Case 2.2.1: X max 1 IA5i)) /(nn/4mmll))) + CC Due to this assumption, we can check that as is large enough, Q(G., Ga,a) x A/ max [2,, Aej]]. If max (14,E, |Ar]]]
Ap, for all then by dividing both the numerator and denomina- tor of A, by Xr, Ap, given that the new denominator of A₁ goes to 0, its new numerator must go to i.e., we have
An An (X,, +1
which cannot hold since M/A,,1///XX Au, 19) 0 (assumption of Case 2.2.1) and Ap,, A/ + Therefore, we must have max | | A for all By dividing both the numerator and denominator of A₂ by 2, Aui as the new denominator of Ag goes to 0, we would have
(A; Am AK
+ Add
0.
A;
An Aval
\m Amm 12
Since
<
0 and (X; - An)/2; + 1, the above limit shows that
X Augl
X; Aenl
+ 1. Since (A,,) / + co, it implies that (ARG)²/Ae) + 0. Now. by combing the result that A₁ - and Ae + since F'3 + we can verify that it is equivalent to
Ap, A Ae
(>; A.)
2ApH)
/3 (X; An)
Ag =
0.
Q(Gn, G..))
By dividing both the numerator and the denominator of Ag by >; Ar,,,,, obtain
(X% An, A 24
13 (N; A
0.
S,A,,,
31

---

As (Ap))) + and Ap, Ae - 1, the above limit leads to Au Avn (Ap, Aen) - -1/3. Now, by studying A, + with the assumption that @(G,,, Gu,n) x X pm 1 we eventually get the equation 1/24 - 1/12 0, which is contradiction. Therefore, Case 2.2.1 cannot hold.
Case 2.2.2: X; max (I 1eppa.//Aiaa | /An/Apal A 30. Therefore, as n is large enough, we would have X, max {1A | S (Am/App/9) 1)) Hence, we achieve under this case that Q(G G...)) An/Apn1 Denote Au =1 Ap AU, L2 (A,,)) and Av 19 (A From the assumptions of Case 2.2.2, we would have ! + 0 and L + 0 while 12 A oc. Additionally, X, max (47)2, [ AA, A O. By dividing the numerators and denominators of A, by Xm (n()))
for 1 < S achieve the following system of limits
X
(2 - An
A
X An
-
17
1 + 0,
An + 12
0,
11
- 0.
(33)
Xn
An
An
3
As I7 + the first limit in the above system implies that (11) + If we have max (27)) [2]]] 12712 for all the previous result would mean that // 0. There- fore, the second limit in (33) demonstrates that 1 - -1. However, plugging these results to the third limit in this system would yield 1/3 1 O, which is a contradiction. Hence, must have
X
+ a as + oc, the first and second limit in 33 lcads to 17 - 1. With this result, the third limit in this system shows that a = 2/3. With these results, by dividing both the numerator and denominator of A by A, (Apt) )+, quickly achieve the equation 1/24 5/72 O, which is contradiction. Therefore, Case 2.2.2 cannot hold.
max ((9) 1151 = [/ for all Under this setting, by denoting A,
a1il
In sum, not all the coefficients of Ond (x (c ( as 1 < |3| < 8 go to O. From here, by using the
same argument that of Proposition C and Proposition C.3 achieve the result of part (b) of the proposition. As a consequence, we reach the conclusion of the theorem.
D Proofs for Convergence Rates of Parameter Estimation and Minimax Lower Bounds
In this appendix, we provide the proofs for the convergence rates of the MLE as well as the corre- sponding minimax lower bounds introduced in S
D.1 Proof of Theorem
(a) For any G1 G1 M1, Ex) and G2 G2(Ae, μ2, Σ2). we denote the following distance
di G2)
Aull(n Σ1) (122 Ez)ll:
(x G2)
11--222
Even though (G,, G2) is proper distance, is clear that d(G1, G2) is not symmetric and only satisfies a weak triangle inequality, i.c. we have
di (G1, G3) + di (G2, G3) 2 min {di (G1 G2), di (G2, G1))
Therefore, will utilize the modification of Le Cam method for nonsymmetric loss in Lemma 6.1 of [16] to deal with such distance. We start with the following proposition
Proposition D.I. Given that f satisfies assumption (S.I) in Theorem that
we achieve for any r <
(i) lim G1=(A E.), E.) (h(pc,,Pc,)//TCC, G2) di (G1, G2) <e)=
inf
(ii) lim inf e00 GI (4ipc..Pc.)//5(CG dh G2) =e]=0.
32

---

Proof. (i) For any sequences G1,= = (An HI,n, E1,n) and 2n (An, M2,n: E2,1), we have
$ An I (pGi. (z) - PG2,n (x))
dx
f(z1 m2,1)
An /
(f(x)u E1,n) [ E2,n)) dx
f(z) U2, 22,1)
where the first inequality is due to v /PG,,. (x) + VVV PG n (x) > Ann [af(x>/H2, Ez,n.. By Taylor expansion up the first order, we have
(Fi. - H2,m) (E1,1 E2,n) 0f
[1 E1m) - f( m2n)
(x) (2//22, E2,n
a1laz!
|a|=1
(Nin - 12,n 122,) (21,)) E2,n )a2
0s
ppo 0502 (zluzn + 1(11, - M2n), E2, + 1(E1,,n - Σ2,0))
a1loz!
|a|=1
Now, by choosing ( Ei,,) (M2,n, 21,,)
o. and [1 E1.n)
(p2,m; E2,n
0 and using condition (S.1), we can easily verify that h(pG,.. PG2 PG2.)/d/ff (G G2,n) O. Therefore, achieve the conclusion of part (i).
(ii) The argument for this part is essentially similar to that in part (i). In fact, for any two sequences GI, = (AI, Mn, En, and G2 (22,n: Ma, Enn, we also obtain
h2 (PG, :PO
(A1, 22,))=22
(ho(z)po, Eo) f(zl/un En))² dr ho(2|po, Eo) + f(z/ En)
d3 (G' 1, G2,nn)
(1 -111n) A A1,n
2 2(A1.n )2
(1 >1,n) A1,n
By choosing (A1. - )2,1)² / {(1 - >1,n) A A1, - O, we also achieve the conclusion of part (ii).
Now, given G. (4*, 34"" E*) and r < 1. Let Co be any fixed constant. According to part (i)
of Proposition D.1 for any sufficiently small 0, there exists G², (A",i] Ei) such that d, (G., G..) = (G, G.) = and h(pa., Pc;) I Coc". By means of Lemma 6.1 of (16), we
achieve
inf sup Epo (X31((m, En) (M, E)13) G_
e² (1 2 V-V
2
2
where Pa. denotes the density of the n-iid sample XI...., Xn. From there.
(P I AIPG.,Pö:
v1 - (1- h Pc:))
V (1
C2e2rjn (1-Gee2)a
Hence, we obtain
62
inf sup Ep G,e3GEs
X En) (1,
2 V 1- (1 (1-622r.. C2erjn
By choosing 2r
we achieve
nh
inf sup Epc ()²]((2m, En) (m, E)1E² 2 1/n
G. Gez
for any r where ei is some positive constant. Using the similar argument, with the result of (ii)
in Proposition D.I also immediately obtain the result inf sup Epc ( x12) 2 cyn 1/r
G.EE GES
As consequence, reach the conclusion of part (a) of the theorem.
33

---

(b) The proof of this part is a direct consequence of Theorem 3.3
and Theorem 2.3 Indeed, for
Gn (An: Pn, Enn being the MLE as in equation we have
Eyc. (1ān-- - X-1
(m En) (n* 21)
Thm 3
E Eva V(PG, PG.) Epa. h(pG, PG.)
Thm 2
log
V1i
Because all inequalities are uniform in G., we achieve the conclusion of part (b) of the theorem.
D.2 Proof of Theorem A.1
(a) Similar to the proof argument of part (a) of Theorem 4.1 we define
d3(G), G2) A A1 AΣ )I)llll [[ 21) (u2, 21)11
dG G2)
A\ >2 (A, AET)
for any G = (),,, M1, Σn) and G2 = G2 U22 Σg). clear that both d3(G,, G2) and (G G2) still satisfy weak triangle inequality. To achieve the conclusion of this part, it suffices to demonstrate the following results
(i) There exists IWO sequences
(An. Egnn G1. = (An M1, E1,0) E
(1,))) and and
Gz.
E1n), such that dg(GL,n G2,, J.PG...:GG...1 (of,.)/d((Gi,n G2 a oC.
(ii) There exists two sequences Gi,n = (A1, En) E (In) and G2 = (22, May En) E
(1,) such that da (Gi, G2,,) - and h(po, PG: ( G2,n) as n + 0o.
for any r The proof argument for the above results can proceed in similar fashion as that of Proposition D.I: therefore, it is omitted. We achieve the conclusion of part (a) of the theorem.
(b) Combining the result of Theorem 3.5 and the fact that D.G x D(G, G.) for any and G.,
we immediately achieve the following convergence rates
sup Epc G.E3
(A)))
",,")"
(l En) (M", 2²1
log" n
log n
sup Epc.
45.) |1 - x
(34)
G.eB
clear that the second result 34 does not match with the second result the conclusion of part (b) of the theorem. To circumvent this issue, we utilize the fact that G. E (1,). Indeed, notice
that (Dns Em) - (4²", E* (Ap,, AE (Ap", AE"). ), we have
Epc- AApi ALT) (Ap", AE*)
log² m(X-)?](Ap> AE )| 0.
sup G.E=
[(Ap', I|?
(35)
Hence, by the AM-GM inequality, we have
Ec., (( AS,)) -
2 2 AA,," ||*)| Epc. An - ~²- Epc. |l(ATin, AZx) - (Axr" (A ーー-
Epc. MApm:AEin (Apr, AΣ") (nn - 201²
2 "E*))
Epc. On 2 22022
|1(A,, 1
-2)2
(36)
uniformly in G., where in the last inequality we use uniformly bounded by 2. Hence,
(35 combining with the fact that 1An X²1 is
Epp-- [I(A/*, A 1 xT) S Epec (I(A7T. AE MFPI(A
log log²(n)
n
which is the conclusion of the theorem.
34

---

D.3 Proof of Theorem A.2
(a) Similar to the proof argument of part (a) of Theorem we define
ds G2) = A111(11. Σ1} - (M2, E2]l
do(Gi, G2) = |A - A2 b2ll(( (A,, AEI)
for any Gi = (Ai,,, ,i, Σ1) and G2 G2(A2, μ2, Σ2). is clear that da(Gi, G2) satisfies weak triangle inequality while dg(G,, G2) no longer satisfies weak triangle inequality. In particular, have
(G, G3) + ds (G2, Ga) 2 min (ds G2), ds dG( G1)
8
A close investigation of Lemma 6.1 of [6 reveals that modified Le Cam method still works under this setting of ds metric. More specifically, for any 6 > 0 the following holds
inf sup Epc (dB(C,C.)) G.EEGGEA((M)
6²
1 ViG
128
where G1, G2 E2(hn) such that ds(G), G2) A (G,, G2) 2 e/4. From here, to achieve the conclusion of part (a), it suffices to demonstrate for any < 1 that
(i) There exists two sequences Gi,
(Nu, Mi,n, E2,1) E G2, = (An:H2. Σgn) E (1())) such that do ds(G1,ns G2,n) + and h(po1...PG PG2.. (d55 G2,z) 11 1 xc.
Ea(h,)
and
(ii) There exists sequences Gin (A1, Mn; En) E E2(In) and G2 (N2,ns Pny Em.
E/ (1%) such that de (G,,; G2,u) - and h(pe;,, PGG,, (ddg(G,,ns G2,n)
DC.
quickly verify the above results. As consequence, we reach the conclusion of part (a) of the theorem.
Following the proof argument of Proposition D.I
(b) From the discussion after Theorem
3.5 we can show that:
Q G.) A->*]([|) ||
(| 1"1EE- DII X|TT:///AAATT.T.TT...TT/TTTAAAAX
II).
Hence, from Theorem 3.6 combining with Theorem 2.3 we have
log2(n)
sup Ep. (11)
- -1" |Ι
+ +1 x
n
log log2(m) (n)
sup Epc. 15n - ~ |I, "' S G.
Similar to the proof of Theorem A.1 and with the definition of (2(42), we have
Epc, A5n X+1 (1 I1 A z (1 || 17n x
uniformly G, (22)) Hence,
log (m) Epc. PS, X1 11
sup G.EE.(1))
Y2
As consequence, we obtain the conclusion of the theorem.
E Proofs for Auxiliary Results
Lemma E.1. For any 2 we define
D,GG,..) Al((A, AEJI A*IKA AETII"
min {,, (I(A,, + AEDII - -I(,,)) (4r", )r")"
for any G and G.. Then, we have W,(GG x D.( for any r 2 where W,, is the r-th order Wasserstein distance.
35

---

Proof. Without loss of generality, we assume throughout the lemma that À < X". Therefore, we
obtain from the formulation of (, G.) that
D, G.) = (X* - A)) AE"II" + A)I(, [""E"")"""
Direct computation of W" (G, G.) yields three distinct cases:
Case 1: I1(Au, AE) 1r + I1 AEMII = 11 2) - (,,, """"" then
(G, AIl AEJI" + "I"I(I"",""II
min A (II(A,, + I1(Ap", AΣ"III 11(,, 2) - -(4,, ((")")
G.).
Case 2: If 11 AEII + "IA)""AE"II" 1164 - and A S then
WF(G,G.) AIK(A, AEJI" + X AEDII
= (A' -)IIIAI ,A") + A(I(Ap, + AE))IT)
From Stuchy-Schartz's inequality, we have AE)ح + (II,,,A"") (u* E". Therefore, under Case 2 we have (A,, AT) I[AA,, AE )Ir (1, Σ) (n" E',l which directly implies that W7 G.) = (.,
2
(1, 2)
Case 3: If I1(A,, AE)I + " < M) - (1', E")""" and À + X > 1, then
(F G.)
(1-
(A AE")II"
+ (A X- -))((,, (M"'
(A*
(1xx))((AA AEJI" AE")
+ (A* + - -1)11,)), - E"","")
Since I[(Ap, AE + (1 A x [[1,,) - "",,"),"", we achieve
(1 - (')'III(( AMII x x(1- X²) 2) - "")'"
Therefore, also have WTIG, G.) D, G.) under Case 3.
Combining the results from these cases, we reach the conclusion of the lemma.
F Discussion and Additional Experiments
F1 Parameter Changes with the Sample Size
In statistics and machine learning, researchers often want to know how many samples are enough to achieve some pre-specified error for the estimation of parameter 0 in the fitted model. In the language of probability, we want to find an inequality such as E' 0ll < C' * rate(n), where rate(n) is a decreasing sequence in n and C' does not depend on Usually, in the parametric models, we have rate(n) /T, and therefore it takes C C²/e² samples achieve average error in estimation. In complex models such as hierarchical models or the multivariate deviated model that we consider in this paper, difficulties arise because of the singularity and identifiability of the model. For example, in Eq. (2), if X = 0, any pair of (p"E"" yields the same model. Hence, when X Rs O, should harder to estimate (n² E*,, and researchers may need more samples to have an accurate estimation for them. Notably, we have shown, for example in Theorem 4.1 the precise dependence of the convergence rate of (n*", x²)
on the magnitude of x= In particular, we have
E T - X0 C logn
EX- | Sia) - (1r", Σー> < coogn Vn
logn
Vn
where C' is constant that does not depend on x", E* and n. Hence, one can have a good estimation (with error e) for X" with C2/e /e2 samples, while he needs C2 /(e X+)² samples to achieve such good estimation for (μ², E.). The simulation studies in the next section will make this clearer.
36

---

y=
-0.37
kogp -271
(2) Rate of X
(h) Rate of Pr,
(c) Rate of d
Figure 3: Case \ 0.5/n
-0.4
y- 0.02 x -0.99
- 0.01 x 0.52
-20
19970
l0gin
l0go1
Rate An
(b) Rate
Rate of 2
Figure 4: Case X = 0.5 0.5/n
F.2 Additional Experiments for the Distinguishable Settings
We have seen in the main text that in two cases where X" is either fixed or decreasing with rate n -1/4, the convergence rate of X is C' x n 1/2 where the constants C' is the same for both cases. The convergence rate for (μ² ) is C x n- -14 for the latter case, which is much slower than the parametric rate the former case. This phenomenon quite rare for parametric models. We want to further bring readers' attention to two more extreme cases:
1/2
1. X 0.5/n increases;
2. X 0/5/n increases,
where we consider the same (u", E') with the experiments in the main text. The convergence rate (X, 14, o²) in both cases in the log domain can be seen Figure and Figure Hence, in cases, the rate of convergence for X always of order 1/2 meanwhile, the rate for (n* E*) becomes slower as X tends to 0 faster. From the theoretical result, when X 0.5/n3// we expect the rate for (μ², E*) to be of order n 11/8, which is demonstrated Figure (b)&())) At the extreme case = 0.5/n 1/2, it is even impossible to recover (μ*, Σ²) as This suggests practitioners collect more data when 2, is small to have a good estimate for (μ² E*). Finally, in the case Xn is extremely small (of order n 1122,, we suggest not to report the estimated values (7n, 5,), as they are highly uncertain.
oo (cf. Figure 4 (
F.3 Non-distinguishable Settings
Finally, we consider the weakly identifiable and non-distinguishable setting here to demonstrate that the convergence rate for X* can be slower than the parametric rate when f is near hg- Let both ho and f belong to the location-scale Gaussian family. ho(z) = f 1) and consider two cases of (X*, p=", ))) 0.25, μ² 0. are fixed and a² = n 1/8 as n increases;
2. X 0.25, o² 1. are fixed and μ² n -1/8 as increases; 1/8
37

---

y .0.50
y=
-10
(3) Rate of X
(h) Rate of P.,
(c) Rate of d
Figure Case 1: p=" Pi and (o*)² of in the rate n 1--1- 1/8
0.03 -0.8
00.15 -027
-0.49 0.3
2.8
10g011
335
ligh1
(a) Rate of An
(b) Rate Aa
(c) Rate of 8
Figure 6: Case 2: 0² oo and
- po in the rate 1/8
| Recall that we have proved that |1 X 0 and X (1
|| +| the orders of convergence rates of the location and scale parameter. Notably, the rate of convergence for X also depends on the rate Ap" and A(o²)² - 0. The experiments do support this theoretical
-
0( where there is mismatch in
finding, where we have the rate for X" is = n 1/4 is the first case (as |3 O(n-1/4) and
does not convergence the second case X = n -1/4 is the first case (as | = (22 The mismatch rate for 1,7, =I"|l and || (a (1 can also be seen clearly in the second case, where the rate for the scale parameter is still of the parametric rate, whereas it is slower for the mean.
38


Reviews:
[{"review_id": "0lv7jtNxrD", "reviewer": "Reviewer_1EFM", "summary": "This paper tackles the issue of parameter estimation in the deviated Gaussian mixture of experts problem using the Maximum Likelihood Estimation (MLE) method. The authors propose new distances and analyze the convergence of MLE under distinguishable and non-distinguishable conditions.", "questions": "\n1. what is the mini-max rate of this problem? how close is the current rate to the min-max rate? \n\n2. What is the difference and technical novelty compared to https://huynm99.github.io/Deviated_MoE.pdf\n", "limitations": "NA", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is in relatively good shape. The results seem to be solid. ", "weaknesses": "The major weakness is the novelty. \n\nThis paper basically considers a much simpler case than the paper https://huynm99.github.io/Deviated_MoE.pdf. \n\nThey consider multiple $k$ while this paper considers a single $k$. \n\nThe definitions, results,  organization, and even the notations are almost the same. For example, the hellinger distance and TV distance (quite strange to me but adopted by both papers interesting), although this paper changes the distance from $D$. \n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The major weakness is the novelty.**\n\nThanks for your comment. We would like to refer you to the General Response section for our elaboration on the novelty of this paper.\n\n**Q2: This paper basically considers a much simpler case than the paper [1]. In particular, the authors in that paper consider multiple $k$ while this paper considers a single $k$.**\n\nThanks for your comment. However, we gracefully disagree with the viewpoint that the problem considered our paper is a simpler case than that in [1] for the following reasons: \n\nBased on the formulations of the models considered in two papers (ours and [1]), it might seem that our model is simpler as we test whether data are sampled from a distribution with known density $h_0$ (null hypothesis) or to another single distribution (alternative hypothesis), whereas the authors in [1] also test the previous null hypothesis but against another alternative hypothesis which says that data are generated from a mixture of experts. \n\nNevertheless, it turns out to be untrue as the nature of two papers are different. In particular, in [1], the ground-truth parameters are not assumed to change with the sample size, therefore, the main goal is to derive the point-wise convergence rates for parameter estimation. By contrast, we do impose this assumption on our problem setup as mentioned in Question 1. Thus, the objective of our paper is to characterize the uniform convergence rates, which is demanding but more precise than the point-wise counterparts. \n\nHence, our work is not a simpler case of [1], but it lays a foundation to capture uniform convergence rates for parameter estimation as well as minimax lower bounds in the deviated Gaussian mixture of experts. However, these directions are beyond the scope of our paper and we leave them for future work.\n\n**Q3: The definitions, results, organization, and even the notations of this paper and [1] are almost the same.**\n\nThanks for your comment, but we gracefully disagree with this claim for the following reasons:\n\n1) Regarding the definitions and notations: notions introduced in our paper such as identifiability and distinguishability or technical tools like the Hellinger distance and Total Variation distance are commonly used in the literature of mixture models to capture the convergence rates for parameter estimation, namely [2, 3]. Thus, we strongly believe that the usage of those ingredients should be considered normal rather than as a weakness.\n\n2) Regarding the results: as we stated in our responses to Question 1 and Question 2, the results introduced in this paper are novel and totally different from those in [1].\n\n3) Regarding the organization: our paper is organized in a different way compared to [1]. In particular, while we present the Total Variation lower bounds in Section 3, and then ntroduce uniform and minimax rates in Section 4, [1] respectively provides the point-wise rates under distinguishable and non-distinguishable settings in those sections. Moreover, our Section 5 is devoted for experiments, whereas [1] use this section for proof sketch.\n\n\n**Q4: What is the mini-max rate of this problem? how close is the current rate to the min-max rate?**\n\nThanks for your questions. Actually, we already provided the result of minimax lower bounds in Section 4 and Appendix A of our paper. For example, under the distinguishable settings, the minimax lower bounds are given line 316 indicates that the minimax rate of $\\widehat{\\lambda}_n$ is of order $\\mathcal{O}(n^{-1/2r})$ for $r<1$. As a consequence, the convergence rate of $\\widehat{\\lambda}_n$ to $\\lambda^{\\ast}$, which is of order $\\mathcal{O}(n^{-1/2})$, is sharp.\n\n**Q5: What is the difference and technical novelty compared to [1]?**\n\nThanks for your question. Firstly, we would like to refer the reviewer to Question 1 for the difference between our paper and [1]. Secondly, we will elaborate on the technical novelty in our paper as follows: \n\n**Uniform Convergence Rates:** under the distinguishable settings, although our work and [1] both point out that the estimation rate for $\\lambda^{\\ast}$ is of order $\\mathcal{O}(n^{-1/2})$, we arrive at different rates for estimating $(\\mu^{\\ast},\\Sigma^{\\ast})$ from that in [1] because of the sample size dependence assumption. Specifically, while the authors of [1] show that this rate is of order $\\mathcal{O}(n^{-1/2})$, we demonstrate that it should rather be slower since it is actually determined by the convergence rate of $\\lambda^{\\ast}$ to zero via the following bound: \n\n$\\lambda^{\\ast}||(\\widehat{\\mu}_{n}-\\mu^{\\ast},$\n\n$\\widehat{\\Sigma}_{n}-\\Sigma^{\\ast})||$\n\n$=\\mathcal{O}(n^{-1/2})$.\n\nIt is clear that these rates are sophisticated and able to highlight the implicit interactions between the convergence rates of various parameter estimations, which remains missing in [1]. To achieve the above rates, we have to face many challenging settings in our proofs. For instance, we first need to make sure that two sequence $G_{n}$ and $G_{\\ast,n}$ converge to the same limit $\\overline{G}$ under the proposed loss functions. Furthermore, there is still a possibility that the last two components of $G_{n}$ or $G_{\\ast,n}$ may not converge to those of $\\overline{G}$ under the $2$-norm. Thus, it takes us greater effort to consider all these possible scenarios than in [1] where the authors only need to control the convergence of $G_n$ to $G_{\\ast}$.  \n\n**References**\n\n[1] H. Nguyen, K. Nguyen, N. Ho. On Parameter Estimation in Deviated Gaussian Mixture of Experts.\n\n[2] S. Gadat, J. Kahn, C. Marteau, and C. Maugis-Rabusseau. Parameter recovery in two-component\ncontamination mixtures: The l2 strategy. In Annales de l’Institut Henri Poincaré, Probabilités\net Statistiques, volume 56, pages 1391–1418. Institut Henri Poincaré, 2020.\n\n[3] D. Do, L. Do, and X. Nguyen. Strong identifiability and parameter learning in regression with\nheterogeneous response. arXiv preprint arXiv:2212.04091, 2022."}, {"review_id": "UoZaTzpZy9", "reviewer": "Reviewer_9DGR", "summary": "The paper studies the problem of parameter recovery in the multivariate deviated model where the data is generated according to the following distribution:\n$$\n    (1 - \\lambda) h_0 (x) + \\lambda f(x | \\mu, \\sigma)\n$$\nwhere $f$ belongs to a mean-variance family and $h_0$ is known. One prominent example of such a family is the family of Gaussian distributions.\n\nThe paper studies the recovery problem under three settings. The first is the distinguishable setting where $h_0$ and the density $f$ are distinguishable (essentially $h_0$ cannot be written as a linear combination of two distributions from the family the family) where statistical recovery is guaranteed at a $\\sqrt{n}$ rate. In the second setting of strong identifiability, $f$ is assumed to belong to the mean-variance family and here the convergence rates depend on the closeness of $\\mu_0, \\Sigma_0$, the parameters of $h_0$, to $\\mu, \\Sigma$, the parameters of the unknown mixture component. Finally, in the third setting where the mean-variance family is the Gaussian distribution where different convergence behavior is observed from the strongly identifiable setting where a second order PDE guarantees improved performance.\n\nFrom a technical standpoint, the algorithm (MLE) is analyzed in the following way. First, they show that under some mild assumptions on the function class, the MLE solutions approximate the distribution in Hellinger-distance. Then, by noting the relationship between the Hellinger and TV distances, the paper then shows that the for this class of distributions, the distance between the parameters is upper bounded by a constant multiple of the TV distance. The first step follows by standard empirical process theory. The second, however, relies on some intricate recently developed machinery. Roughly speaking, one first shows that the TV and parameter-distances approximate each other locally where the limit of the neighborhood is taken to $0$. Subsequently, a short analytic argument leads to a global approximation guarantee. This technique, while inspired prior work, still takes significant care to execute. \n\nOverall, the results in the paper are interesting and the technical contributions seem strong. The fact that the statistical performance in this setting may be distinguished from algorithms that operate on mixture models where both components are unknown is also intriguing. However, Theorem 3.6 is only proved for the univariate setting (Appendix C3) while the rest of the paper focuses on the multivariate setting.\n", "questions": "See main review", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "See main review", "weaknesses": "See main review", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Theorem 3.6 is only proved for the univariate setting (Appendix C3) while the rest of the paper focuses on the multivariate setting.**\n\nThanks for your comment. Although the proof of Theorem 3.6 is only proved for the univariate setting, it can be adapted to high-dimensional settings along with some according changes of notations to those settings. Since there are too many scenarios arising in that proof, we consider the univariate setting to avoid the unnecessary complexity of notations which might make the main arguments implicit and the proof untidy. "}, {"review_id": "0wHAMvzaNZ", "reviewer": "Reviewer_bb5i", "summary": "## Summary\n\nThe authors study the minimax rate for parameter recovery in deviated multivariate models.\nIn this setting, we observe samples from a mixture (of *unknown* weight \\lambda) of a \"null\" distribution h_0 and a distribution from a parametric family f( | \\mu, \\Sigma).\nThe goal is to recover from n samples the (\\lambda, \\mu, \\Sigma).\n\n## Contribution\n\nThe authors study the MLE performance in various regimes (depending on how \"far\" is h_0 from the parametric family of distributions.\nTheir analysis is very tight, leading to obtaining the minimax rates.\n", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "I like the result: it is mathematically clean and leads to tight results. It is always nice to see the minimax rates for new problems.", "weaknesses": "I am unfortunately failing to conclude the position of the paper in the literature. \nAre the authors the first to obtain results in this setting?\nIf yes, please explain (much) more why studying the model is interesting/important.\nIf not, please compare thoroughly the results with the existing ones.\n\nAlso, are the techniques used new in any way? Or are the results simply an application of known techniques?\n\nAll in all, I think the paper needs a much more thorough literature review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Are the authors the first to obtain results in this setting? If yes, please explain why studying the model is important. If not, please compare thoroughly the results with the existing ones.**\n\nThanks for your questions. We provide below a more thorough comparison with other existing results and hope that it helps to clarify the position of this paper in the literature. \n\n1. This model gained some of its early interest from testing and detection problems in biology [1], where the authors consider the semi-parametric setting:\n$$p^*(\\cdot) = \\lambda^* h_0(\\cdot) + (1-\\lambda^*) f^*(\\cdot - \\mu_2^*),$$\nwhere $h_0$ is known, and $\\lambda^*$, $\\mu^*$, $f^*$ are to be estimated from the data. The density $f$ is further assumed to be symmetric but non-parametric. Then the identifiability, consistency and asymptotic normality of parameters are studied.\n\n2. In [2], the authors consider the parametric location deviated model:\n$$p^*(\\cdot) = (1-\\lambda^*) N(\\cdot | 0, 1) + \\lambda^* N(\\cdot | \\mu^*, 1),$$\nwhere the density $N$ is standard Gaussian, the true parameter $\\lambda^*$ and $\\mu^*$ are of interest and would be recovered given data. They specifically consider $l^2$ estimation and provide the uniform bound for parameter estimation (i.e., holds for all $(\\lambda^*, \\mu^*)$, which is \nsimilar to our result). It is worth noting that the setting in our paper is more general in the sense that the known component $N(0, 1)$ can be assumed to be any density $h_0$ and the estimated component $N(\\mu^*, 1)$ can be any vector-matrix family $f(\\cdot | \\mu^*, \\Sigma^*)$.\n\n3. In [3, 4, 5], a similar setting is considered:\n$$p^*(\\cdot) = (1- \\epsilon_n) N(\\cdot | 0, 1) + \\epsilon_n N(\\cdot | \\mu_n, 1),$$\nin which the main question is: How small $\\epsilon_n$ can be compared to the sample size $n$ in order to reliably test (or \ndetect) $\\epsilon > 0$ against $\\epsilon = 0$. In the so-called dense setting, where they assume $\\epsilon_n \\asymp n^{-\\beta}$ for $\\beta\\in (0,1/2)$, it is possible to do so when $\\epsilon_n \\mu_n \\gtrsim (\\log(n)/n)^{1/2}$, which matches the result that we have in Theorem A.1. We go beyond this setting by also considering varying covariance of the second component $N(\\cdot | \\mu_n, \\sigma_n^2)$. Theorem A.3 implicitly says that it is possible to estimate parameters when $\\epsilon_n \\mu_n^2 \\gtrsim (\\log n/n)^{1/2}$ and $\\epsilon_n \\sigma_n^2 \\gtrsim (\\log n/n)^{1/2}$ Additional experiments in Section F.2 and F.3 also support this finding. \n\n4. Although this model is statistically interesting and worth studying in the classical sense, please note that its idea is still used in the Machine Learning community [9, 10]. Specifically, $h_0$ can be a pre-trained Large Model, and we wish to adapt it to some specific downstream task. A popular solution is to consider the deviated model with small deviated weight $\\lambda^*$ and a simple model $f(\\cdot | \\mu^*, \\Sigma^*)$, which costs less to train compared to $h_0$. By doing that, it does not require to re-train $h_0$ but still can borrow some knowledge learned from it. We hope the theoretical result that we build in our model can shed some light to the estimation problem for this Low-rank adaptation technique.\n\n**Q2: Are the techniques used new in any way? Or are the results simply an application of known techniques?**\n\nThe core of the techniques in our paper is from [6, 7, 8], which allows us to develop lower bounds of density distance by parameter distance using the notion of \"identifiability\" or \"distinguishability.\" A novel technique here is we look at the convergence rate of each parameter in detail instead of relying on the Wasserstein distance, which could not capture the specific rate for each parameter. Besides, the novelty that we wish to highlight is that our bounds are uniform in the parameter space $(\n\\lambda^*, \\mu^*, \\Sigma^*)$. Because of that, we can detect whether $(\\mu^*, \\Sigma^*)$ is estimable when $\\lambda^*\\to 0$, which is a singular point of the model (i.e., when $\\lambda^*=0$, every pair of $(\\mu^*, \\Sigma^*)$ give the same density). Both the minimax rate and the convergence rate of $(\\mu^*, \\Sigma^*)$ are then developed in this setting. \n\n**Q3: I think the paper needs a much more thorough literature review.**\n\nThanks for your comment. We hope that the answer to your first question can partially help to clarify the current literature. Also, kindly refer to a more detailed review of our general answer. We will include them in the revision of the paper.  \n\n[1] L. Bordes. Semiparametric estimation of a two-component mixture model where one component is known. Scandinavian journal of statistics, 2006.\n\n[2] S. Gadat. Parameter recovery in two-component contamination mixtures: The l2 strategy. In Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, volume 56, pages 1391–1418. Institut Henri Poincaré, 2020.\n\n[3] T. Cai. Optimal detection of heterogeneous and heteroscedastic mixtures. Journal of the Royal Statistical Society: Series B,  2011.\n\n[4] T. Cai. Estimation and confidence sets for sparse normal mixtures. Annals of Statistics, 2007.\n\n[5] T. Cai. Optimal detection of sparse mixtures against a given null distribution. IEEE Transactions on Information Theory, 2014.\n\n[6] Chen, J. H. Optimal rate of convergence for finite mixture models. Annals of Statistics, 1995\n\n[7] N. Ho. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistic, 2016.\n\n[8] P. Heinrich and J. Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. Annals of Statistics, 2018.\n\n[9] Hu, Edward J. Lora: Low-rank adaptation of large language models. arXiv preprint, 2021.\n\n[10] Jiang, Ziyu. DnA: Improving Few-Shot Transfer Learning with Low-Rank Decomposition and Alignment. In European Conference on Computer Vision, 2022."}, {"review_id": "KjSzvD0uBq", "reviewer": "Reviewer_UU3o", "summary": "In this paper, the authors establish the rate for estimating true parameters in the multivariate deviated model by using the MLE method. They mainly try to address two challenges encountered in deriving the rate of convergence for MLE estimators, i.e. 1) the interaction between the null hypothesis density $h_0$ and the alternative density function $f$, 2) the likelihood of the deviated proportion $\\lambda$ vanishing to either extreme points of the interval [0, 1]. To this end, they develop the distinguishability condition to capture the linear independent relation between the function $h_0$ and the density function $f$, and derived the optimal convergence rate of the MLE under both distinguishable and non-distinguishable settings.", "questions": "How does this paper compare with the current literature on heterogeneous mixture detection in terms of assumptions and results?\n\nCan the results in Section 3.2.2 be extended to non-Gaussian distributions for f?", "limitations": "No potential negative societal impact", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and effectively presents the problem setup, theoretical framework, and main results. The definitions and explanations of key concepts are well presented. The authors address a fundamental statistical problem and provide insights into the behavior of the MLE in the multivariate deviated model. The derived convergence rates and minimax rates contribute to the understanding of parameter estimation and hypothesis testing in complex data scenarios.", "weaknesses": "I feel that in the paper the comparison to existing literature is a bit limited. Particularly, how does this paper compare with the current literature on heterogeneous mixture detection?\nIn the experiment section, it seems that the setup is a bit limited with f being Gaussian. It would be great if the authors could show some more numerical results with more expanded scenarios. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: How does this paper compare with the current literature on heterogeneous mixture detection in terms of assumptions and results?**\n\nThanks for your question. Different from the heterogeneous mixture detection literature where most of the results are under specific settings of $h_{0}$ and $f(x|\\theta, \\Sigma)$, in our paper we are able to provide general theories for more general settings of $h_{0}$ and $f(x|\\theta, \\Sigma)$. In particular, we develop the novel notion of distinguishability between $h_{0}$ and $f(x|\\theta, \\Sigma)$, which allows us to characterize the minimax uniform rates of parameter estimation under the general distinguishable settings of these functions. For the settings when these functions are not distinguishable, our results also cover a wide range of $h_{0}$ and $f(x|\\theta, \\Sigma)$ in practice, including strongly and weakly identifiable cases (e.g., multivariate Gaussian distribution and multivariate Student's t distribution with general covariance matrices). \n\n**Q2: In the experiment section, it seems that the setup is a bit limited with $f$ being Gaussian. It would be great if the authors could show some more numerical results with more expanded scenarios.**\n\nThanks for your question. It is indeed true that the mixture model can be used with a much wider range of kernel densities. Although the Gaussian kernel is arguably the most popular, the mixture of Gamma distributions is also of interest when it comes to modeling heterogeneous data on the real line. We can consider the true generative model:\n$$p^*(x) = (1-\\lambda^*) G(x | \\alpha_0, \\beta_0) + \\lambda^* G(x | \\alpha^*, \\beta^*), $$\nwhere the shape-rate density Gamma is defined by:\n$$G(x|\\alpha, \\beta) = \\dfrac{\\beta^{\\alpha}}{\\Gamma(\\alpha)} x^{\\alpha - 1} e^{-\\beta x}, \n\\quad \\alpha, \\beta > 0.$$\nThe model is known to be strongly identifiable when $\\alpha^* \\neq \\alpha_0$ or $|\\beta^* - \\beta_0| \\neq 1$. So that the result in Theorem A.1 applies in this case. We will provide a simulation study for this model in Appendix F. \n\n**Q3: Can the results in Section 3.2.2 be extended to non-Gaussian distributions for $f$?**\n\nThanks for your question. Most kernel densities, such as multivariate Student's t distribution and multivariate Laplace distribution, satisfy the second-order identifiability so that the results in Section 3.2.1 apply to them. For weakly identifiable families, the most popular kernel may be the location-scale Gaussians. Other examples are the Gamma distribution that we discussed previously or the skew normal distribution (cf. [1]). For Gamma distribution, it is interesting that the Gamma densities are only weakly identifiable in a zero Lebesgue measure set so that it is not of the main focus of our paper where we consider convergence rate from testing problems' perspective so that $\\lambda^* \\approx 0$, $\\alpha^* \\approx \\alpha_0$, and $\\beta^* \\approx \\beta_0$. \n\nFor skew-normal distribution, it consists of three parameters: the location, scale, and skewness (shape) parameter. When the skewness parameter is 0, the skew-normal distribution becomes the normal distribution. This distribution is useful for modeling asymmetric data. The skew-normal distribution possesses more complex algebraic structures among the location, scale, and skewness parameters (via partial differential equations of the skew-normal distribution with respect to these parameters) than those in the location-scale Gaussian distribution (c.f. equations (2) and (3) in [1]). Therefore, the theoretical results in Section 3.2.2. will be richer and more complicated for skew-normal distribution than those for Gaussian distribution. We leave detailed theoretical analyses of both Gamma and skew-normal distributions for future work. \n\n**References**\n\n[1] N. Ho and L. Nguyen. Singularity structures and impacts on parameter estimation in finite mixtures of distributions. SIAM Journal on Mathematics of Data Science, 2019."}, {"review_id": "FCsPeMkcE2", "reviewer": "Reviewer_HL4W", "summary": "The paper studies the optimal rate for multivariate deviated models. Specifically, they consider the model $(1-\\lambda) h(x) + \\lambda f(x|\\mu,\\Sigma)$, where $h$ is known and the goal is to estimate the other parameters. The authors propose to use the notion of *distinguishability* and study the convergence rate of parameters using MLE under both distinguishable and non-distinguishable cases. The authors present three pairs of upper and lower bounds for distinguishable, non-distinguishable but $f$ is strongly-identifiable, and distinguishable but $f$ is a family of location-covariance multivariate Gaussian distributions. Experiments are also provided to corroborate their theoretical results.", "questions": "- I totally understand the logic behind the organization of the sections but I would suggest shortening Section 3 and moving the results in Appendix A to the main content. Theorem 3.3/3.5/3.6 seem to be intermediate results for proving the convergence rates with artificial distances $\\mathcal{K}$, $\\mathcal{D}$, and $\\mathcal{G}$. So I don't quite understand why they should take up nearly three pages, forcing the main results to be postponed to the appendix.\n- Say we would like to adopt the model to fit some data in practice. How should we obtain $h_0$ which is claimed to be known in this paper? Or $h_0$ could just be good enough and then this model may take over? Would you please comment on how the results may provide possible guidance to the methodology?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is clear. The notations are very consistent for such a lengthy work.\n- There are plenty of explanations and discussions around each main result, making it an interesting paper to read.\n- To the best of my knowledge, the proofs are technically sound and highly sophisticated. The notion of distinguishability and identifiability seems very suitable and intuitive.\n- Lower bounds are also presented and match their convergence rate for all three cases considered.", "weaknesses": "- The idea of distinguishability is not wholly novel. Similar notions have been used in [1] for a different model, which in turn is derived from the notion of *identifiability* adopted in [2] and many other previous works.\n- The section for related works is very concise. \n- Given the classical parametric setting and the MLE estimation, I am not very sure if this paper would be a good match for neurips rather than a more statistically-focused journal. \n\n[1] Do, Dat, Nhat Ho, and XuanLong Nguyen. \"Beyond black box densities: Parameter learning for the deviated components.\" Advances in Neural Information Processing Systems 35 (2022): 28167-28178.\n[2] Nguyen, XuanLong. \"Convergence of latent mixing measures in finite and infinite mixture models.\" (2013): 370-400.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The idea of distinguishability is not wholly novel. Similar notions have been used in [1] for a different model, which in turn is derived from the notion of identifiability adopted in [2] and many other previous works.**\n\nThank you for raising this concern. We would like to emphasize that the other main contribution of the paper is to develop \"uniform\" convergence rates for parameters $(\\lambda^*, \\mu^*, \\Sigma^*)$, compared to \"point-wise\" convergence rates from previous work. To see further why this is significant for hierarchical/mixture models such as the one in this paper, let us reconsider the deviated model:\n$$p^*(x) = (1-\\lambda^*) h_0(x) + \\lambda^* f(x|\\mu^*, \\Sigma^*).$$\nSuppose that $\\lambda^* > 0$ is fixed, $\\mu^*, \\Sigma^*$ are fixed, and $h_0$ is distinguishable from $f$. Using the proof technique from previous work; it is possible to prove that the MLE $(\\widehat{\\lambda}_n, \\widehat{\\mu}_n, \\widehat{\\Sigma}_n)$ satisfies:\n$$\\mathbb{E} |\\widehat{\\lambda}_n - \\lambda^*| \\leq C \\left(\\dfrac{\\log(n)}{n}\\right)^{1/2}, \\mathbb{E} \\|\\widehat{\\mu}_n - \\mu^*\\| \\leq C \\left(\\dfrac{\\log(n)}{n}\\right)^{1/2}, $$\nand \n$$\\mathbb{E} \\|\\widehat{\\Sigma}_n - \\Sigma^*\\| \\leq C \\left(\\dfrac{\\log(n)}{n}\\right)^{1/2},$$\nwhere $C$ is a constant that depends on $(\\lambda^*, \\mu^*, \\Sigma^*)$ and does not depend on $n$. This is called point-wise convergence rates. However, when $\\lambda^* = 0$, any pair of $(\\mu^*, \\Sigma^*)$ will give the same density, so parameter estimation is not possible. Hence, there must be a transition of convergence rates for $(\\mu^*, \\Sigma^*)$ as $\\lambda^* \\to 0$. We demonstrate this point precisely in Theorem 4.1, where the main result claims that\n$$\\mathbb{E}||(\\widehat{\\mu}_n, \\widehat{\\Sigma}_n) - (\\mu^*, \\Sigma^*)|| \\leq C \\dfrac{1}{\\lambda^*}\\left(\\dfrac{\\log n}{n}\\right)^{1/2},$$\nwhere $C$ is a constant does not depend on $(\\lambda^*, \\mu^*, \\Sigma^*)$ and $n$. Hence, the convergence rates of $\\mu^*$ and $\\Sigma^*$\nare slower as $\\lambda^*\\to 0$. This uniform convergence rate allows scientists to obtain more concise confident intervals for parameter estimation and design better experiments in real life.\n\nTo obtain those uniform bounds, the bounds between density distances and parameter distances that we develop in Section 3 need to be more refined compared to Wasserstein distances in the previous work. It leads to the divergence $\\mathcal{K}, \\mathcal{D}$ and $\\mathcal{G}$, which seem to be quite artificial but technically meaningful. The novelty in the presented technique lies in the careful examination of the bound for each element $\\lambda^*$, $\\mu^*$, and $\\Sigma^*$.\n\nIn the paper, there are more challenging settings where $h_0$ can belong to the same family of distributions as $f$ and when the distinguishable condition does not hold. Each leads to different convergence rates for $\\lambda^*, \\mu^*$, and $\\Sigma^*$. Then those uniform convergence rates are supported by simulation studies in Section 5 and Appendix F. The rates we obtain are even better than the rates obtained by using the method of moments in the literature. Please kindly refer to the general comment for more details.\n\n**Q2: The section for related works is very concise.**\n\nThis is also a common concern raised by other reviewers. We decided to use the general response to explain the literature further. Please kindly refer to that. We will add the detailed literature review to the revision version.\n\n**Q3: Given the classical parametric setting and the MLE estimation, I am not very sure if this paper would be a good match for neurips rather than a more statistically-focused journal.**\n\nThanks for your comment. In our opinion, MLE is the principle estimation for several Machine Learning models including Diffusion models and Transformers, therefore it is not outdated. The idea of deviated models is actually can be used in the big data regime, where $h_0$ may be a pre-trained large model such as Transformer, and $f(\\cdot | \\mu, \\Sigma)$ is a small, low-rank model that is trained to adapt to some downstream task while freezing $h_0$ in training. The deviated proportion $\\lambda^*$ can be studied to check how much the model needs to update to study the new and smaller downstream task. This idea is popular in Domain Adaptation recently [1, 2].\n\n**Q4. I would suggest shortening Section 3 and moving the results in Appendix A to the main content. Theorem 3.3/3.5/3.6 seem to be intermediate results for proving the convergence rates with artificial distances $\\mathcal{K}, \\mathcal{D}$ and $\\mathcal{G}$.**\n\nThanks for your suggestion. We will consider it and edit our paper accordingly.\n\n**Q5: Assume we would like to adopt the model to fit some data in practice. How should we obtain $h_0$ which is claimed to be known in this paper? \n\nIn practice, we can see $h_0$ as rising from the domain adaptation problem, where it is estimated from a relevant data set. We then try to modify $h_0$ by a distribution in the vector-matrix family $f(\\cdot | \\mu, \\Sigma)$ to estimate the density of the data set that we are working with [1, 2]. Also, kindly refer to the general response for another example in multiple testing problems, where we consider the distribution of the $p$ value that is obtained from numerous (independent) hypotheses tests. Hence, $h_0$ is uniform distribution on $[0, 1]$. The distribution under $H_1$ is unknown and required to estimate using the deviated component $f^*(x)$.\n\n**References**\n\n[1] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).\n\n[2] Jiang, Ziyu, Tianlong Chen, Xuxi Chen, Yu Cheng, Luowei Zhou, Lu Yuan, Ahmed Awadallah, and Zhangyang Wang. \"DnA: Improving Few-Shot Transfer Learning with Low-Rank Decomposition and Alignment.\" In European Conference on Computer Vision, pp. 239-256. Cham: Springer Nature Switzerland, 2022."}]

Author rebuttal:
**General Response**

Dear AC and reviewers, 

We would like to express our gratitude for your constructive reviews, which help us improve our work significantly. There are two common concerns about the literature on the deviated models and the novelty of our paper. Thus, we dedicate this general response to clarify these concerns and then include them in the revision of our paper. 

**I. Literature of the deviated models**

Let us consider the general model: 
$$p^{\ast}(x) = \lambda^{\ast} h_0(x) + (1-\lambda^{\ast}) f^{\ast}(x),$$
where $h_0$ is known and $(\lambda^{\ast}, f^{\ast})$ are to be estimated from data. Several attempts to provide theoretical guarantees and algorithms for this problem have been made in the literature. Starting from multiple-testing problems (microarray analysis,
neuroimaging) [1, 2], the p-values, obtained from the numerous (independent) hypotheses tests, are uniformly distributed on $[0,1]$, under hypothesis $H_0$. Hence $h_0$ is the uniform distribution, and $f^*$ is required to be estimated. When $f^*$ is assumed to be symmetric and non-parametric, [3] provides identifiability and consistency results of the parameter estimation. In the parametric setting of $f^*$, the related work is listed below.

1. [4] considers this model specifically when $h_0 = N(0, 1)$ and $f^* = N(\mu^*, 1)$ are normal distributions. In the setting $\lambda^* = n^{-\beta}$ where $\beta \in (0, 1/2)$, they prove that no test can reliably detect $\lambda^* = 0$ against $\lambda^* \neq 0$ when  $\lambda^* \mu^* = o(n^{-1/2})$, while the Likelihood Ratio Test can consistently do it when $\lambda^* \mu^* \gtrsim n^{-1/2+\epsilon}$ for any $\epsilon > 0$. However, no guarantee for estimation of $\lambda^*$ and $\mu^*$ is provided.

2. The uniform convergence of estimating $\lambda^*$ and $\mu^*$ is then revisited in [5], in the same setting, where it provides minimax rate and uniform convergence rates for both $\lambda^*$ and $\mu^*$ under the $l^2$ estimation strategy. They prove the tight convergence rate for $\lambda^*$ and $\mu^*$ when $\lambda^* |\mu^*| \gtrsim n^{-1/2 + \epsilon}$ and $|\mu^*| \gtrsim n^{-1/4}$. However, their technique heavily relies on the properties of the location Gaussian family, which might be difficult to generalize to other settings of kernel densities.

3. When $f^* = N(\mu^*, \Sigma^*)$, it is possible to derive the estimation rate for $(\lambda^*, \mu^*, \Sigma^*)$ from results for the general mixture of two components in the literature. However, those bounds are often less sharp compared to the bound that we developed in our paper due to the lack of information about the known component. 
In particular, when estimating $\mu^*$ with $\Sigma^*$ being fixed, an application of the results with moment methods from [7] to the deviated models leads to $||\Delta \mu^{\ast}||^{3} |\lambda_{n}^{\text{moment}}-\lambda^{\ast}| = \mathcal{O}(n^{-1/2})$ and $\lambda^{\ast} ||\mu_{n}^{\text{moment}} - \mu^{\ast}||^3 = \mathcal{O}(n^{-1/2})$, which are much slower compared to the results for the MLE in the strongly identifiable and non-distinguishable settings in our work, where $(\lambda_{n}^{\text{moment}}, \mu_{n}^{\text{moment}})$ denote moment estimators of $\lambda^{\ast},\mu^{\ast}$. When we estimate both $\mu^*$ and $\Sigma^*$, an adaptation of the moment estimators from the seminal work [6] to the multivariate deviated models show $(||\Delta \mu^{\ast}||^{6} + ||\Delta \Sigma^{\ast}||^{3})  |\lambda_{n}^{\text{moment}}-\lambda^{\ast}| = \mathcal{O}(n^{-1/2})$, which is also slower than those of the MLE in weakly identifiable setting.

**II. The novelty of our paper**

**1. Novel settings:** We allow ground-truth parameters $G_*=(\lambda^{\ast},\mu^{\ast},\Sigma^{\ast})$ to change with the sample size $n$, which is closer to practical settings than assumptions in previous work [8]. However, this induces two main obstacles: 
 (i) $h_0$ may belong to the distribution family of $f$, which leads to some interaction between these densities;
 (ii) The deviated proportion $\lambda^{\ast}$ can go to zero as the sample size is sufficiently large. Then, any pairs of $(\mu^{\ast},\Sigma^{\ast})$ will induce the same model $h_0$, which makes the parameter estimation more challenging.

**2. Uniform Convergence Rates:** Since true parameters may vary with the sample size, the convergence rates of parameter estimations in our work are uniform rather than point-wise, as in [8]. Additionally, these rates are able to capture the interaction between the convergences of parameter estimations.

**3. Minimax Lower Bounds:** Finally, we determine minimax lower bounds under both distinguishable and non-distinguishable settings. Based on these lower bounds, we deduce that our derived convergence rates are sharp.


[1] B. Efron. Empirical Bayes analysis of a microarray experiment. Journal of the American statistical association 96, no. 456 (2001): 1151-1160.

[2] S. Robin. "A semi-parametric approach for mixture models: Application to local false discovery rate estimation." Computational statistics \& data analysis 51, no. 12 (2007): 5483-5493.

[3] L. Bordes. Semiparametric estimation of a two-component mixture model where one component is known. Scandinavian journal of statistics, 33(4):733–
752, 2006.

[4] T. Cai. Optimal detection of heterogeneous and heteroscedastic mixtures.
Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5):629–662,
2011.

[5] S. Gadat. Parameter recovery in two-component contamination mixtures: The l2 strategy. In Annales de l’Institut Henri Poincaré, Probabilitéset Statistiques, volume 56, pages 1391–1418. Institut Henri Poincaré, 2020.

[6] M. Hardt. Tight bounds for learning a mixture of two gaussians. In STOC, 2015.

[7] Y. Wu. Optimal estimation of Gaussian mixtures via denoised method of moments.
The Annals of Statistics, 48:1987–2007, 2020.

[8] H. Nguyen. On Parameter Estimation in Deviated Gaussian Mixture of Experts.

Meta-review: The reviewers have a mixed opinion on this work. One reviewer maintains in their review and throughout the discussion phase that the submission is too similar to another paper. Other reviews support acceptance of the paper. I believe the authors have very clearly explained why their submission differs from the other paper in important ways; therefore, I do not think the other paper is relevant for the decision-making process on the submission and will focus on the other reviews.

I will briefly summarize the positives:
- The technical contributions are strong; the proofs have a high level of sophistication.
- The case when distinguishability does not hold seems very challenging, and the authors' results here are impressive.
- The bounds holding uniformly over the parameter space and so also give results when $\lambda^*$ goes to zero, which is quite interesting.
- There are potential applications in domain adaptation.

One reviewer wondered if NeurIPS is a good venue for this work, as compared to a stats journal. I believe results like this do have a home in NeurIPS.

The main downside is (as mentioned by several reviewers) the inadequacy of the related work section. The authors have said they will add a detailed literature review in the revised version.

This looks like a solid paper and is a welcome addition to the conference proceedings. As promised, please be sure to add the detailed literature review; it is vital for contextualizing the results.
Decision: Accept (poster)