Title: Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Mixture Model

Minimax Rate

Maximum Likelihood Estimation
Abstract: We study the maximum likelihood estimation (MLE) in the multivariate deviated model where the data are generated from the density function $(1-\lambda^{\ast})h_{0}(x)+\lambda^{\ast}f(x|\mu^{\ast}, \Sigma^{\ast})$ in which $h_{0}$ is a known function, $\lambda^{\ast} \in [0,1]$ and $(\mu^{\ast}, \Sigma^{\ast})$ are unknown parameters to estimate. The main challenges in deriving the convergence rate of the MLE mainly come from two issues: (1) The interaction between the function $h_{0}$ and the density function $f$; (2) The deviated proportion $\lambda^{\ast}$ can go to the extreme points of $[0,1]$ as the sample size tends to infinity. To address these challenges, we develop the \emph{distinguishability condition} to capture the linear independent relation between the function $h_{0}$ and the density function $f$. We then provide comprehensive convergence rates of the MLE via the vanishing rate of $\lambda^{\ast}$ to zero as well as the distinguishability of two functions $h_{0}$ and $f$.

Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models

Abstract

We study the maximum likelihood estimation (MLE) in the multivariate deviated model where the data are generated from the density function (1 -λ ∗ ) h 0 ( x ) + λ ∗ f ( x | µ ∗ , Σ ∗ ) in which h 0 is a known function, λ ∗ ∈ [0 , 1] and ( µ ∗ , Σ ∗ ) are unknown parameters to estimate. The main challenges in deriving the convergence rate of the MLE mainly come from two issues: (1) The interaction between the function h 0 and the density function f ; (2) The deviated proportion λ ∗ can go to the extreme points of [0 , 1] as the sample size tends to infinity. To address these challenges, we develop the distinguishability condition to capture the linear independent relation between the function h 0 and the density function f . We then provide comprehensive convergence rates of the MLE via the vanishing rate of λ ∗ to zero as well as the distinguishability of two functions h 0 and f .

1 Introduction

The goodness-of-fit test [11] is one of the foundational tools in statistics with several applications in data-driven scientific fields, namely kernel Stein discrepancy [27, 31], point processes [37] and Bayesian statistics [32], etc. Given a sample set of data and a pre-specified distribution with density function h 0 , the test indicates whether the samples are reasonably distributed according to h 0 ( null hypothesis ) or to another family of distributions { p ( ·| θ ) : θ ∈ Θ } ( alternative hypothesis ). It is worth noting that knowledge about the null hypothesis distribution can come from prior knowledge of scientists. A key to understanding the statistical efficiency of testing is via the likelihood ratio and the maximum likelihood estimation (MLE) methods. [6].

While traditional testing problems often assume the null distribution h 0 = p ( ·| θ 0 ) and the alternative one p ( ·| θ ) are from a single simple family of distributions such as exponential families, there are also many problems in science require to test h 0 against the alternative f ( ·| θ ) that can be deviated from h 0 by a distribution from a potentially different family. Specifically, in this paper, we consider the family of distributions named multivariate deviated model with density functions defined as follows:

where x ∈ R d , G := ( λ, µ, Σ) are the model's parameters with λ ∈ [0 , 1] being the deviated proportion (from h 0 ) and ( µ, Σ) ∈ Θ × Ω are parameters of a vector-matrix family of distributions f , where Θ ⊂ R d and Ω ⊂ R d × d being compact. When λ = 0 , this recovers the null hypothesis distribution h 0 .

The deviated model can be motivated by many applications in science. For instance, in microarray data analysis, it can be used to detect differentially expressed genes under two or more conditions [1, 2], where h 0 is the uniform distribution and f ( ·| µ, Σ) is required to estimate. Many other applications can be seen in many contamination problems in astronomy and biology [29]. Besides, the deviated model can also be viewed as a low-rank adaptation model in the domain adaptation problem [23], where h 0 is a pre-trained model on large data, and f is a simpler component to be estimated from the smaller data domain. Our goal in this paper is to study the parameter estimation rate of the deviated model.

Problem setup. Suppose that we observe n i.i.d. samples X 1 , . . . , X n from the true multivariate deviated model:

̸

where G ∗ := ( λ ∗ , µ ∗ , Σ ∗ ) are true but unknown parameters with λ ∗ = 0 . Throughout the paper, we allow G ∗ to change with the sample size n (see Appendix F.1 for a discussion). To facilitate our presentation, we suppress the dependence of G ∗ on n , and then estimate G ∗ from the data. The main focus of this paper is to establish both a uniform convergence rate and minimax rate for parameter estimation via the MLE approach, which is given by:

where ̂ G n := ( ̂ λ n , ̂ µ n , ̂ Σ n ) and Ξ := [0 , 1] × Θ × Ω .

Contribution. There are two main challenges in studying the convergence rate of the MLE ̂ G n : (1) The interaction between the function h 0 and the density function f , e.g., h 0 belongs to the family of f and ( µ ∗ , Σ ∗ ) approaches h 0 as the sample size n goes to infinity; (2) The deviated proportion λ ∗ can go to the extreme points of [0 , 1] as the sample size goes to infinity and make the estimation become more challenging, because when λ ∗ = 0 , all the parameters ( µ ∗ , Σ ∗ ) yield the same model. To address these singularity and identifiability issues, we first develop the distinguishability condition to capture the linear independent relation between the function h 0 and the density function f . We then study the optimal convergence rate of parameters under both distinguishable and non-distinguishable settings of the multivariate deviated model. Our theoretical results can be summarized as follows:

Distinguishable settings: We demonstrate that as long as the function h 0 and the density function f are distinguishable, the convergence rate of ̂ λ n to λ ∗ is O ( n -1 / 2 ) while ( ̂ µ n , ̂ Σ n ) converges to ( µ ∗ , Σ ∗ ) at a rate determined by the vanishing rate of λ ∗ as follows:

It indicates that if λ ∗ goes to 0, the convergence rate of estimating ( µ ∗ , Σ ∗ ) is slower than the parametric rate.

Non-distinguishable settings: When h 0 and f are not distinguishable, it becomes complicated to capture the convergence rate of the MLE. To shed light on the behaviors of the MLE under the non-distinguishable settings of multivariate deviated model, we specifically study the settings when h 0 belongs to the same family as f , namely, h 0 ( . ) = f ( . | µ 0 , Σ 0 ) for some ( µ 0 , Σ 0 ) . To precisely characterize the rates of the MLE under this setting, we consider the second-order strong identifiability of f , which requires the linear independence up to second-order derivatives of f with respect to its parameters. The second-order identifiability had also been considered in the literature to investigate the convergence rate of parameter estimation in finite mixtures [9, 28, 22, 21, 20, 19].

2.1. Strongly identifiable and non-distinguishable settings: When f is strongly identifiable in the second order, we demonstrate that ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ 2 | ̂ λ n -λ ∗ | = O ( n -1 / 2 ) and

where ∆ µ := µ -µ 0 and ∆Σ := Σ -Σ 0 . It indicates that the convergence rate of ̂ λ n to λ ∗ depends on that of ( µ ∗ , Σ ∗ ) to ( µ 0 , Σ 0 ) while the convergence rate of ( ̂ µ n , ̂ Σ n ) to ( µ ∗ , Σ ∗ ) depends on both the rate of λ ∗ to 0 and the rate of ( µ ∗ , Σ ∗ ) to ( µ 0 , Σ 0 ) . These results are strictly different from those in the distinguishable settings, which is mainly due to the non-distinguishability between h 0 and f .

2.2. Weakly identifiable and non-distinguishable settings: When f is weakly identifiable, i.e., it is not strongly identifiable in the second order, we specifically consider the popular setting when f is the density of a multivariate Gaussian distribution. The loss of the strong identifiability of the Gaussian distribution is due to the following partial differential equation (PDE) between the location and scale parameters (the heat equation):

Due to the above PDE, the convergence rate of the MLE under this setting exhibits very dif- ferent behaviors from those under the strongly identifiable setting. In particular, we prove that [

Notably, there is a mismatch in the orders of convergence rates of the location vector and covariance matrix. Furthermore, the rate of the deviated mixing proportion also depends on different orders of µ ∗ to µ 0 and Σ ∗ to Σ 0 . Such rich behaviors of the MLE are mainly due to the PDE between the location and scale parameters.

Comparing to moment methods. Wewould like to remark that the results for the MLE under the nondistinguishable settings in the paper are (much) tighter than those obtained from moment methods for a general mixture of two components in the literature. In particular, when f is multivariate Gaussian distribution with fixed covariance matrix, i.e., f is strongly identifiable in the second order and we do not estimate Σ ∗ , an application of the results with moment methods from [36] to the deviated models leads to ∥ ∆ µ ∗ ∥ 3 | λ moment n -λ ∗ | = O ( n -1 / 2 ) and λ ∗ ∥ µ moment n -µ ∗ ∥ 3 = O ( n -1 / 2 ) , which are much slower compared to the results for the MLE in the strongly identifiable and non-distinguishable settings, where ( λ moment n , µ moment n ) denote moment estimators of λ ∗ and µ ∗ .

When f is a multivariate Gaussian density and we estimate both the location vector and covariance matrix, i.e., f is weakly identifiable, an adaptation of the moment estimators from the seminal work [18] to the multivariate deviated models shows that λ ∗ ∥∥ ˜ µ moment n -µ ∗ ∥ 6 = O ( n -1 / 2 ) , λ ∗ ∥∥ ˜ Σ moment n -Σ ∗ ∥ 3 = O ( n -1 / 2 ) and ( ∥ ∆ µ ∗ ∥ 6 + ∥ ∆Σ ∗ ∥ 3 ) | ˜ λ moment n -λ ∗ | = O ( n -1 / 2 ) , where ( ˜ λ moment n , ˜ µ moment n , ˜ Σ moment n ) are moment estimators of ( λ ∗ , µ ∗ , Σ ∗ ) . These results are also much slower than those of the MLE in weakly identifiable settings.

Other related work. The hypothesis testing and MLE problem related to the multivariate deviated model had been considered in previous work, including the problem of detecting sparse homogeneous and heteroscedastic mixtures [14, 15, 4, 3, 5, 34], the problem of determining the number of components [8, 26, 10, 24, 25], and the problem of multiple testing [30, 12]. In particular, [4] considers testing problem for the deviated model with h 0 = N (0 , 1) and f = N ( µ ∗ , 1) being onedimensional Gaussian distributions. They show that no test can reliably detect λ ∗ = 0 against λ ∗ > 0 if λ ∗ µ ∗ = o ( n -1 / 2 ) , while the Likelihood Ratio test can consistently do it when λ ∗ µ ∗ ≳ n -1 / 2+ ϵ for any ϵ > 0 . However, no guarantee for estimation of λ ∗ and µ ∗ is provided. In the same setting where f is the density of a location Gaussian distribution, the convergence rate of parameter estimation in the deviated model had been studied in the work of [16]. Since the location Gaussian distribution is a special case of the strongly identifiable distribution, our result in the strongly identifiable and nondistinguishable settings is a generalization of the results in [16], but with a different proof technique as their proof technique relies strictly on the properties of the location Gaussian distribution.

Organization. The paper is organized as follows. In Section 2, we provide background on the identifiability and density estimation rate of the multivariate deviated model. Then, we establish the lower bounds of the Total Variation distance between two densities in terms of loss functions among parameters under both the distinguishable and non-distinguishable settings in Section 3. Next, we characterize the convergence rates of parameter estimation as well as derive the corresponding minimax lower bounds in Section 4. In Section 5, we carry out a simulation study to empirically verify our theoretical results before concluding the paper in Section 6. Rigorous proofs and additional results are deferred to the supplementary material.

Notations. For any a, b ∈ R , we denote a ∨ b := max { a, b } and a ∧ b := min { a, b } . Next, we say that h 0 is identical to f if h 0 ( x ) = f ( x | µ 0 , Σ 0 ) for some ( µ 0 , Σ 0 ) ∈ Θ × Σ . For each parameter G ∈ Ξ , let E p G be the expectation taken with respect to product measure with density p G . Lastly, for any two density functions p and q (with respect to the Lebesgue measure m ), the Total Variation distance between them is given by V ( p, q ) := 1 2 ∫ | p ( x ) -q ( x ) | dm ( x ) , while we define their squared Hellinger distance as h 2 ( p, q ) := 1 2 ∫ [ √ p ( x ) -√ q ( x )] 2 dm ( x ) .

2 Preliminaries

2.1 Identifiability Condition

Our principal goal in this paper is to assess the statistical efficiency of parameter estimation from the MLE method. To do that, we should be able to guarantee the parameter identifiability of the deviated model (2), i.e., if p G ( x ) = p G ∗ ( x ) for almost surely x ∈ X where G = ( λ, µ, Σ) , then G ≡ G ∗ . That identifiability condition leads to the following notion of distinguishability between the density function h 0 ( · ) and the family of density functions { f ( ·| µ, Σ) : ( µ, Σ) ∈ Θ × Ω } .

Definition 2.1 ( Distinguishability). We say that the family of density functions { f ( ·| µ, Σ) , ( µ, Σ) ∈ Θ × Ω } (or in short, f ) is distinguishable from h 0 if the following holds:

A1. For any two distinct components ( µ 1 , Σ 1 ) and ( µ 2 , Σ 2 ) , if we have real coefficients η i for 1 ≤ i ≤ 3 such that η 1 η 2 ≤ 0 and η 1 f ( x | µ 1 , Σ 1 ) + η 2 f ( x | µ 2 , Σ 2 ) + η 3 h 0 ( x ) = 0 , for almost surely x ∈ R d , then η 1 = η 2 = η 3 = 0 .

We can verify that as long as f is distinguishable from h 0 , the parameter identifiability of our multivariate deviated model follows. In particular, assume that there exists G = ( λ, µ, Σ) such that

̸

for almost surely x ∈ X . The above equation is equivalent to ( λ -λ ∗ ) h 0 ( x ) + λ ∗ f ( x | µ ∗ , Σ ∗ ) -λf ( x | µ, Σ) = 0 . Assume that f is distinguishable from h 0 , then equation (4) indicates that if ( µ, Σ) = ( µ ∗ , Σ ∗ ) , we have λ = λ ∗ = 0 . Since λ ∗ = 0 from our assumption, we obtain that ( µ, Σ) = ( µ ∗ , Σ ∗ ) . As a result, equation (4) becomes ( λ -λ ∗ ) h 0 ( x ) + ( λ ∗ -λ ) f ( x | µ, Σ) = 0 . By applying the distinguishability condition again, we get λ = λ ∗ . Therefore, the multivariate deviated model (2) is identifiable.

̸

In the following example, we will verify the distinguishability condition in Definition 2.1 given some specific choices of function h 0 and density f .

̸

Example 2.2. (a) Assume that f belongs to a location family of density functions, i.e., f ( x | µ, Σ) = f Σ ( x -µ ) for all x where Σ is a fixed covariance matrix. If h 0 ( x ) = f ( x ) for almost surely x ∈ X , then f is distinguishable from h 0 .

(b) When h 0 is a finite mixture of multivariate Gaussian densities and f belongs to a class of multivariate Student's density functions with any fixed odd degree of freedom ν > 1 , we get that f is distinguishable from h 0 .

(c) When f is identical to h 0 , then f is not distinguishable from h 0 .

2.2 Convergence Rate of Density Estimation

Our strategy to obtain the convergence rate of the MLE ̂ G n is by first establishing the convergence rate of density p ̂ G n and then studying the geometric inequalities between the parameter space and density space. For the former, the standard method is to use the empirical process theory [17, 33], while for the latter step, we investigate those inequalities under various settings of distinguishability in Section 3. Due to space constraints and the popularity of empirical process theory, we choose to informally present a main result for yielding the parametric convergence rate for density estimation in this section. For full explanation and definition, readers are referred to Appendix B. The convergence rate for density estimation can be characterized by bounding the complexity of the parameter space Ξ

via a function called bracketing entropy integral J B ( ϵ, P 1 / 2 (Ξ , ϵ )) (cf. equation (8)).

Theorem 2.3. Assume the following assumption holds:

A2. Given a universal constant J > 0 , there exists N > 0 , possibly depending on Ξ , such that for all n ≥ N and all ϵ > (log( n ) /n ) 1 / 2 , we have J B ( ϵ, P 1 / 2 (Ξ , ϵ )) ≤ J √ nϵ 2 .

Then, there exists a constant C > 0 depending only on Ξ such that for all n ≥ 1 ,

Therefore, in order to get the convergence rate for density estimators based on the MLE method, we only need to check Assumption A2, which holds true for several parametric models [33]. For our model, we give an example that it holds for a general class of f and h 0 .

Proposition 2.4. Suppose that both Θ and Ω are compact, and { f ( x | µ, Σ) : µ ∈ Θ , Σ ∈ Ω } is a vector-matrix family of densities being uniformly bounded, Lipschitz, and light tail, i.e. there exists constants M,L,B,b 1 , b 2 , b 3 > 0 such that | f ( x | µ, Σ) | ≤ M, | f ( x | µ, Σ) -f ( x | µ ′ , Σ ′ ) | ≤ L ( ∥ µ -µ ′ ∥ + ∥ Σ -Σ ′ ∥ ) for all x ∈ R d , and

for all ( µ, Σ) ∈ Θ × Ω . Additionally, if the density h 0 is bounded, then the corresponding multivariate deviated model defined in equation (1) satisfies assumption A2.

Example 2.5. We can check that the location-scale Gaussian density f ( x | µ, Σ) with Σ ∈ Ω having eigenvalues bounded below by a positive constant satisfies the condition of Proposition 2.4. This condition for h 0 is mild and is satisfied by most distributions such as Gaussian and t-distribution.

3 From the Convergence Rate of Densities to Rate of Parameters

The objective of this section is to develop a general theory according to which a small distance between p G and p G ∗ under the Hellinger distance (or Total Variation distance) would imply that G and G ∗ are also close under appropriate distance where G = ( λ, µ, Σ) and G ∗ = ( λ ∗ , µ ∗ , Σ ∗ ) . By combining those results with Theorem 2.3, we can obtain the convergence rate for parameter estimation (cf. Section 4). The distinguishability condition between h 0 and f implicitly requires that p G = p G ∗ would entail G = G ∗ ; however, to obtain quantitative bounds for their Total Variation distance, we need stronger notions of both distinguishability and classical parameter identifiability, ones which involve higher order derivatives of the densities h 0 and f , taken with respect to mixture model parameters. Throughout the rest of this section, we denote G = ( λ, µ, Σ) and G ∗ = ( λ ∗ , µ ∗ , Σ ∗ ) .

3.1 Distinguishable Settings

Definition 3.1 ( First-order Distinguishability). We say that f is distinguishable from h 0 up to the first order if f is differentiable in ( µ, Σ) , and the following holds:

D1. For any component ( µ ′ , Σ ′ ) ∈ Θ × Ω , if we have real coefficients η, τ α for all α = ( α 1 , α 2 ) ∈ N d 1 × N d 2 × d 2 , | α | = | α 1 | + | α 2 | ≤ 1 such that

for all x ∈ X , then η = τ α = 0 for all | α | ≤ 1 .

We can verify that the examples from part (a) and part (b) of Example 2.2 satisfy the first-order distinguishability condition. Next, we introduce a notion of uniform Lipschitz condition in the following definition.

Definition 3.2 ( Uniform Lipschitz). We say that f admits uniform Lipschitz condition up to the first order if the following holds: there are positive constants δ 1 , δ 2 such that for any R 1 , R 2 , R 3 > 0 , γ 1 ∈ R d 1 , γ 2 ∈ R d 2 × d 2 , R 1 ≤ λ 1 / 2 min (Σ 1 ) ≤ λ 1 / 2 max (Σ 2 ) ≤ R 2 , ∥ µ 1 ∥ , ∥ µ 2 ∥ ≤ R 3 , µ 1 , µ 2 ∈ Θ , Σ 1 , Σ 2 ∈ Ω , we can find positive constants C ( R 1 , R 2 ) and C ( R 3 ) such that for all x ∈ X ,

Now, we have the following results characterizing the behavior of V ( p G , p G ∗ ) regarding the variation of G and G ∗ .

Theorem 3.3. Assume that f is distinguishable from h 0 up to the first order. Furthermore, f admits uniform Lipschitz condition up to the first order. For any G and G ∗ , we define

Then, the following holds:

for all G and G ∗ , where C and C 1 are two positive constants depending only on Θ , Ω , and h 0 .

See Appendix C.1 for the proof of Theorem 3.3. Since the MLE approach yields the convergence rate n -1 / 2 up to some logarithmic factor for p G ∗ under the first order uniform Lipschitz condition of f , the result of Theorem 3.3 directly yields the convergence rate n -1 / 2 up to some logarithmic factor for G ∗ under metric K . This entails that the estimation of weight λ ∗ converges at rate n -1 / 2 up to some logarithmic factor while the convergence rate of estimating ( µ ∗ , Σ ∗ ) is typically much slower than n -1 / 2 as it depends on the rate of convergence of λ ∗ to 0 (cf. Theorem 4.1).

3.2 Non-distinguishable Settings

When f is not distinguishable to h 0 up to the first order, the bound in Theorem 3.3 may not hold in general. In this section, we investigate the inverse bounds under the specific settings of nondistinguishable in the first-order models when h 0 belongs to the family f ( ·| µ, Σ) , i.e., h 0 ( x ) = f ( x | µ 0 , Σ 0 ) for some ( µ 0 , Σ 0 ) ∈ Θ × Σ . Our studies are divided into two separate regimes of f : the first setting is when f is strongly identifiable in the second order (cf. Definition 3.4), while the second setting is when it is not. For the simplicity of the presentation in the paper, we define (∆ µ, ∆Σ) = ( µ -µ 0 , Σ -Σ 0 ) for any element ( µ, Σ) ∈ Θ × Ω .

Definition 3.4 ( Strong Identifiability). We say that f is strongly identifiable in the second order if f is twice differentiable in ( µ, Σ) and the following holds:

D2. For any positive integer k , given k distinct pairs ( µ 1 , Σ 1 ) , . . . , ( µ k , Σ k ) , if we have α ( i ) η such that

for almost all x ∈ X , then α ( i ) η = 0 for all i ∈ [ k ] and | η | ≤ 2 .

3.2.1 Strongly Identifiable Settings

Now, we have the following result regarding the lower bound of V ( p G , p G ∗ ) under the strongly identifiable settings of f .

Theorem 3.5. Assume that h 0 ( x ) = f ( x | µ 0 , Σ 0 ) for some ( µ 0 , Σ 0 ) ∈ Θ × Σ and f is strongly identifiable in the second order and admits uniform Lipschitz condition up to the second order. Furthermore, we denote

for any G and G ∗ . Then, there exists a positive constant C depending only on Θ , Ω , and ( µ 0 , Σ 0 ) such that V ( p G , p G ∗ ) ≥ C. D ( G,G ∗ ) , for all G and G ∗ .

The proof of Theorem 3.5 and the second-order uniform Lipschitz condition are deferred to Appendix C.2. Several remarks regarding Theorem 3.5 are in order:

(i) For any G and G ∗ , by defining

we can verify that 1 / 2 ≤ D ( G,G ∗ ) / D ( G,G ∗ ) ≤ 2 , i.e., D ( G,G ∗ ) ≍ D ( G,G ∗ ) . The reason that we prefer to use the formation of D ( G,G ∗ ) over that of D ( G,G ∗ ) is not only due to the convenience of the proof argument of Theorem 3.5 later in Appendix C but also due to its partial connection with Wasserstein metric that we are going to discuss in the next remark.

(ii) When f is a multivariate location family and is identical to h 0 , i.e., µ 0 = 0 , it was demonstrated recently in [16] that

which is also the key result for establishing the convergence rates of parameter estimation in their work. However, their proof technique only works for the location family and it is unclear what is the sufficient condition for the family of density functions beyond the location family such that the inequality (5) will hold. As the location family is strongly identifiable in the second order, we can verify that the lower bound in Theorem 3.5 and inequality (5) are in fact similar. Therefore, the result in Theorem 3.5 gives a generalization of inequality (5) in [16] under the strongly identifiable in the second order setting of f .

(iii) As indicated in [16], we can further lower bound the right-hand side of inequality (5) in terms of the second order Wasserstein metric W 2 [35] between G and G ∗ when we present G and G ∗ as two discrete probability measures with two components. In particular, with an abuse of the notations we denote that G = (1 -λ ) δ ( µ 0 , Σ 0 ) + λδ ( µ, Σ) and G ∗ = (1 -λ ) δ ( µ 0 , Σ 0 ) + λδ ( µ ∗ , Σ ∗ ) , i.e., we think of G and G ∗ as two mixing measures with one fixed atom to be ( µ 0 , Σ 0 ) . In light of Lemma E.1 in Appendix E, we have

Therefore, D ( G,G ∗ ) and W 2 2 ( G,G ∗ ) share the similar term λ ∥ (∆ µ, ∆Σ) ∥ 2 + λ ∗ ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ 2 -min { λ, λ ∗ } ( ∥ (∆ µ, ∆Σ) ∥ 2 + ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ 2 ) in their formulations. However, as λ ∥ (∆ µ, ∆Σ) ∥ + λ ∗ ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ ≥ min { λ, λ ∗ } ∥∥ ( µ, Σ) -( µ ∗ , Σ ∗ ) ∥ , the remaining term in D ( G,G ∗ ) is stronger than that of W 2 2 ( G,G ∗ ) . Moreover, as λ = λ ∗ , we further obtain that

Hence, as long as the right-hand side term in the above display goes to ∞ , i.e., || (∆ µ +∆ µ ∗ , ∆Σ+ ∆Σ ∗ ) || → 0 , we have D ( G,G ∗ ) /W 2 2 ( G,G ∗ ) → ∞ . This strong refinement of the Wasserstein metric is due to the special structure of G and G ∗ as one of their components is always fixed to be ( µ 0 , Σ 0 ) .

(iv) Under the setting when G ∗ is varied, d 1 = 1 , and d 2 = 0 , by means of Fatou's lemma the result from Theorem 4.6 in [19] yields V ( p G , p G ∗ ) ≥ C ′ .W 3 3 ( G,G ∗ ) if the kernel density function f is 4-strongly identifiable (cf. Definition 2.2 in [19]) and satisfies uniform Lipschitz condition up to the fourth order where C ′ is some positive constant depending only on G and G ∗ . Since D ( G,G ∗ ) ≳ W 2 2 ( G,G ∗ ) ≥ W 2 1 ( G,G ∗ ) ≳ W 3 3 ( G,G ∗ ) , it indicates that the bound in Theorem 3.5 is much tighter than this bound. The loss of efficiency in this bound is again due to the special structures of G and G ∗ as one of their components is always fixed to be ( µ 0 , Σ 0 ) .

Unlike the convergence rate results from the strongly distinguishable in the first order setting between f and h 0 in Theorem 3.3, the convergence rate of λ ∗ under the setting of Theorem 3.5 depends on the rate of convergence of ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ 2 to 0 (cf. Theorem A.1). Additionally, the convergence rate of estimating ( µ ∗ , Σ ∗ ) will be determined based on the convergence rates of λ ∗ and (∆ µ ∗ , ∆Σ ∗ ) to 0.

3.2.2 Weakly Identifiable Settings

Thus far, as h 0 belongs to the family f , our results regarding the lower bounds between p G and p G 0 under Total Variation distance rely on the strongly identifiable in the second order assumption of kernel f . However, there are various families of density functions that do not satisfy such an assumption, which we refer to as the weakly identifiable condition. To illustrate the non-uniform natures of V ( p G , p G ∗ ) under the weakly identifiable condition of f , we consider specifically a popular setting of f in this section: multivariate location-covariance Gaussian kernel.

Location-covariance multivariate Gaussian kernel: As indicated in the previous work in the literature [7, 25, 21], if f is a family of multivariate location-covariance Gaussian distributions in d dimension, it exhibits the heat partial differential equation (PDE) with respect to the location and covariance parameter ∂ 2 f ∂µ∂µ ⊤ ( x | µ, Σ) = 2 ∂f ∂ Σ ( x | µ, Σ) , for any x ∈ R d and ( µ, Σ) ∈ Θ × Ω . We

can verify that this structure leads to the loss of the second-order strong identifiability condition of the Gaussian kernel. Note that, the PDE structure of the Gaussian kernel has been shown to lead to very slow convergence rates of parameter estimation under general over-fitted Gaussian mixture models (cf. Theorem 1.1 in [21]). For the setting of the multivariate deviated model, since the parameters λ ∗ and ( µ ∗ , Σ ∗ ) are allowed to vary with the sample size, we may expect that the estimation of these parameters will also suffer from the very slow rate. In fact, we achieve the following lower bound of V ( p G , p G ∗ ) under the multivariate location-covariance Gaussian kernel.

Theorem 3.6. Assume that h 0 ( x ) = f ( x | µ 0 , Σ 0 ) for some ( µ 0 , Σ 0 ) ∈ Θ × Σ and f is a family of multivariate location-covariance Gaussian distributions. We denote

for any G and G ∗ . Then, we can find a positive constant C depending only on Θ , Ω , and ( µ 0 , Σ 0 ) such that V ( p G , p G ∗ ) ≥ C. Q ( G,G ∗ ) , for any G and G ∗ .

See Appendix C.3 for the proof of Theorem 3.6. A few comments with Theorem 3.6 are in order.

(i) Different from the formulation of D ( G,G ∗ ) in Theorem 3.5 where we have the same power between µ and Σ , there is a mismatch of power between ∥ ∆ µ ∥ 2 , ∥ ∆ µ ∗ ∥ 2 and ∥ ∆Σ ∥ , ∥ ∆Σ ∗ ∥ in the formulation of Q ( G,G ∗ ) . This interesting phenomenon is mainly due to the structure of the heat equation where the second-order derivative of the location parameter and the first-order derivative of the covariance parameter is linearly dependent.

verify that Q ( G,G ∗ ) ≳ Q ′ ( G,G ∗ ) for any G,G ∗ . If we treat G and G ∗ as two-components measures as in the remark (iii) after Theorem 3.5, we would have

where G 1 = (1 -λ ) δ µ ′ 0 + λδ µ , G 2 = (1 -λ ) δ Σ ′ 0 + λδ Σ and similarly for G 1 , ∗ and G 2 , ∗ . Here, ( µ 0 , Σ 0 ) = ( µ ′ 0 , Σ ′ 0 ) , and W 2 , W 4 are respectively second and fourth order Wasserstein metrics. The formulations of Q ′ ( G,G ∗ ) , therefore, can be thought of as a combination of two Wasserstein metrics: one is with only parameter µ and another one is only with parameter Σ . The division into two Wasserstein metrics can be traced back again to the PDE structure of the heat equation.

If λ = λ ∗ and ( ∥ ∆ µ ∥ 2 + ∥ ∆ µ ∗ ∥ 2 + ∥ ∆Σ ∥ + ∥ ∆Σ ∗ ∥ ) / ( ∥ µ -µ ∗ ∥ 2 + ∥ Σ -Σ ∗ ∥ ) →∞ , we will have that Q ( G,G ∗ ) / Q ′ ( G,G ∗ ) →∞ . It proves that the result from Theorem 3.6 under the multivariate setting of Gaussian kernel is a strong refinement of the summation of Wasserstein metrics regarding location and covariance parameter in equation (6).

A consequence of Theorem 3.6 is that the convergence rate of estimating λ ∗ is determined by ∥ ∆ µ ∗ ∥ 4 + ∥ ∆Σ ∗ ∥ 2 , instead of ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ 2 as in the strongly identifiable setting of f . Furthermore, we also encounter a phenomenon that the rate of convergence of estimating Σ ∗ is much faster than that of estimating µ ∗ . In particular, estimating Σ ∗ depends on the rate in which λ ∗ ( ∥ µ ∗ ∥ 2 + ∥ Σ ∗ ∥ ) converges to 0 while estimating µ ∗ relies on square root of this rate (cf. Theorem A.2).

4 Minimax Lower Bounds and Convergence Rates of Parameter Estimation

In this section, we study the convergence rates of MLE ̂ G n as well as minimax lower bounds of estimating G ∗ under various settings of h 0 and f . Due to space constraints, we present the theory in the distinguishable regime of h 0 and f . Non-distinguishable cases, though more interesting, are deferred to Appendix A.

Theorem 4.1. (Distinguishable settings) Assume that classes of densities h 0 and f satisfy the conditions in Theorem 3.3. Then, we achieve that

(a) (Minimax lower bound) Assume that f satisfies the following assumption S.1:

0 , where α 1 ∈ N d 1 , α 2 ∈ N d 2 in the partial derivative of f take any combination such that | α | = | α 1 | + | α 2 | ≤ 1 .

Then for any r < 1 , there exist two universal positive constants c 1 and c 2 such that

Here, the infimum is taken over all sequences of estimates ̂ G n = ( ̂ λ n , ̂ µ n , ̂ Σ n ) .

(b) (MLE rate) Let ̂ G n be the MLE defined in equation (3) , and the family { p G : G ∈ Ξ } satisfies condition A2. Then, we have the convergence rate for the MLE:

Proof of Theorem 4.1 is in Appendix D.1. The results of Theorem 4.1 imply that even though we still can estimate λ ∗ at the standard rate n -1 / 2 , the convergence rate of ( ̂ µ n , ̂ Σ n ) to ( µ ∗ , Σ ∗ ) strictly depends on the vanishing rate of λ ∗ to 0. Therefore, the convergence rate of estimating ( µ ∗ , Σ ∗ ) can be generally slower than n -1 / 2 as long as λ ∗ goes to 0 at a rate slower than n -1 / 2 .

We can also use the geometric inequalities developed in Section 3.2 to investigate the behaviors of ̂ G n in the non-distinguishable settings. We will further see how the non-identifiability and singularity of the model affect the convergence rate for density estimation. Due to space constraints, the results for this setting are presented in Appendix A.

5 Experiments

We now demonstrate the convergence rates of parameter estimation in the strongly distinguishable setting, where the choice of h 0 is a standard Cauchy distribution, and f ( ·| µ, σ 2 ) is the normal distribution with mean µ and variance σ 2 . The additional experiments with the non-distinguishable setting are deferred to Appendix F.

Assume that X 1 , . . . , X n are i.i.d. samples drawn from the true density function p G ∗ with G ∗ = ( λ ∗ , µ ∗ , ( σ ∗ ) 2 ) and we obtain the MLE ( ˆ λ n , ˆ µ n , ˆ σ 2 n ) . We consider two following cases:

Two histograms for samples from the density p G ∗ with n = 10000 corresponding to the above two cases are illustrated in Figure 1(a) and 2(a). For each case, we take into account multiple sample

(a) Histogram

(b) Rate of

̂

λ

n

(c) Rate of

µ

n

(d) Rate of

σ

2

n

C1-

y =-0.54x+-0.37

y=-0.5×+-0.36

y=-0.51x+-0.41

400

1.4

350

1.6

(l( ,0)  l)6o)

=1A

300

-“yl)6o

T

,n °)6o}

-1.8

250

-2.0

2.0

200

2.2

-2.0

2.2

150

100

-2.4

-2.2

=2.4

50

9'2=

=2.6

0

8'2

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.8

2.25

2.50

2.75

log(n)

00T

3.25

3.50

3.75

4.00

(a) Histogram

(b) Rate of

̂

λ

n

(c) Rate of

̂

µ

n

(d) Rate of

̂

σ

2

n

-1.0

y =-0.52x+-0.28

y = -0.27 × + -0.04

y =-0.26×+ 0.14

300

-1.2

0.2

℃'0=

200

250

-°yl)6o

-1.6

1,rf  “fl)6o)

0.6

(1( ,0)  l)6o)

0.6

1.8

0.8

150

2.0

1.0

0.8

100

-2.2

1.0

50

2.4

9'2-

0

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

00T

3.25

3.50

3.75

4.00

2.25

2.50

2.75

log(n)

00E

3.25

3.50

3.75

4.00

sizes n ranging from 10 2 to 10 4 . For each sample size n , we calculate the MLE ( ˆ λ n , ˆ µ n , ˆ σ 2 n ) via the EM algorithm [13] and measure the errors | ̂ λ n -λ ∗ | , | ̂ µ n -µ ∗ | , and | ̂ σ 2 n -( σ ∗ ) 2 | . We repeat this procedure 64 times and plot the mean (blue dot) and quartile error bars (yellow bar) of the logarithm of estimation errors against the log of n . Theorem 4.1 suggests that the log convergence rate of λ ∗ is of order -1 / 2 for all cases, and so is the rate for ( µ, ( σ ) 2 ) in the first case. Meanwhile, the convergence rates of ( µ ∗ , ( σ ∗ ) 2 ) in the second case are slower, which is in the order of -1 / 4 . The empirical rates found in the experiments match this theoretical result, where the least square line shows that the logarithm of the rate for estimating µ ∗ in case (i) is -0.5 and that of the case (ii) is -0.27 ≈ -1 / 4 (similar for ( σ ∗ ) 2 ).

We once again emphasize that this interesting phenomenon of the rates of convergence is due to the singularity and identifiability of the multivariate deviated model. Our theory and simulation have accurately shown quantitative convergence rates for parameter estimation when λ ∗ near the singularity point 0 , where all pairs of ( µ ∗ , Σ ∗ ) in model (2) give the same model. Together with the non-distinguishable settings, we provide a comprehensive study of the large-sample theory for this type of model, thanks to the newly developed notion of distinguishability that helps to control the linear independent relation between h 0 and f . The developed optimal minimax lower bounds and convergence rates will certainly help Machine Learning practitioners understand better the role of sample sizes in the accuracy of estimation in the multivariate deviated model, and are also inspired theorists to study more about the estimation rate of complex hierarchical/mixture models.

6 Conclusion

In this paper, we establish the uniform rate for estimating true parameters in the multivariate deviated model by using the maximum likelihood estimation (MLE) method. During our derivation, we have to overcome two major obstacles, which are firstly the interaction between the known function h 0 and the Gaussian density f , and secondly the likelihood of the deviated proportion λ ∗ vanishing to either one or zero. To this end, we introduce a notion of distinguishability to control the linearly independent relation between two functions h 0 and f . Finally, we achieve the optimal convergence rate of the MLE under both the distinguishable and non-distinguishable settings.

References

L. Bordes, C. Delmas, and P. Vandekerkhove. Semiparametric estimation of a two-component mixture model where one component is known. Scandinavian journal of statistics , 33(4):733752, 2006.

L. Bordes, S. Mottelet, and P. Vandekerkhove. Semiparametric estimation of a two-component mixture model. The Annals of Statistics , 34(3):1204-1232, 2006.

T. Cai, X. J. Jeng, and J. Jin. Optimal detection of heterogeneous and heteroscedastic mixtures. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 73(5):629-662, 2011.

T. Cai, J. Jin, and M. G. Low. Estimation and confidence sets for sparse normal mixtures. The Annals of Statistics , 35(6):2421-2449, 2007.

T. Cai and Y. Wu. Optimal detection of sparse mixtures against a given null distribution. IEEE Transactions on Information Theory , 60(4):2217 - 2232, 2014.

G. Casella and R. L. Berger. Statistical inference . Cengage Learning, 2021.

H. Chen and J. Chen. Tests for homogeneity in normal mixtures in the presence of a structural parameter. Statistica Sinica , 13:351-365, 2003.

H. Chen, J. Chen, and J. D. Kalbfleisch. A modified likelihood ratio test for homogeneity in finite mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 63(1):19-29, 2001.

J. Chen. Optimal rate of convergence for finite mixture models. Annals of Statistics , 23(1):221233, 1995.

J. Chen, P. Li, and Y . Fu. Inference on the order of a normal mixture. Journal of the American Statistical Association , 107:1096-1105, 2012.

W. G. Cochran. The χ 2 test of goodness of fit. The Annals of mathematical statistics , pages 315-345, 1952.

N. Deb, S. Saha, A. Guntuboyina, and B. Sen. Two-component mixture model in the presence of covariates. Journal of the American Statistical Association , 117(540):1820-1834, 2022.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, Sept. 1977.

D. Do, N. Ho, and X. Nguyen. Beyond black box densities: Parameter learning for the deviated components. arXiv preprint arXiv:2202.02651 , 2022.

D. Donoho and J. Jin. Higher criticism for detecting sparse heterogeneous mixtures. Annals of Statistics , 32(3):962-994, 2004.

S. Gadat, J. Kahn, C. Marteau, and C. Maugis-Rabusseau. Parameter recovery in two-component contamination mixtures: The l 2 strategy. In Annales de l'Institut Henri Poincaré, Probabilités et Statistiques , volume 56, pages 1391-1418. Institut Henri Poincaré, 2020.

E. Giné and R. Nickl. Mathematical foundations of infinite-dimensional statistical models . Cambridge university press, 2021.

M. Hardt and E. Price. Tight bounds for learning a mixture of two gaussians. In STOC , 2015.

P. Heinrich and J. Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. Annals of Statistics , 46(6A):2844-2870, 2018.

N. Ho and L. Nguyen. Singularity structures and impacts on parameter estimation in finite mixtures of distributions. SIAM Journal on Mathematics of Data Science , 1(4):730-758, 2019.

N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistics , 44:2726-2755, 2016.

N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics , 10:271-307, 2016.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

H. Kasahara and K. Shimotsu. Non-parametric identification and estimation of the number of components in multivariate mixtures. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76(1):97-111, 2014.

H. Kasahara and K. Shimotsu. Testing the number of components in normal mixture regression models. Journal of the American Statistical Association , 110(512):1632-1645, 2015.

P. Li and J. Chen. Testing the order of a finite mixture. Journal of the American Statistical Association , 105(491):1084-1092, 2010.

Q. Liu, J. Lee, and M. Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 276-284. PMLR, 20-22 Jun 2016.

X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. Annals of Statistics , 4(1):370-400, 2013.

R. Patra and B. Sen. Estimation of a two-component mixture model with applications to multiple testing. J. R. Stat. Soc. Series B Stat. Methodol. , 78(4):869-893, 2016.

R. Patra and B. Sen. Estimation of a two-component mixture model with applications to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 78(4):869-893, 2016.

A. Schrab, B. Guedj, and A. Gretton. KSD aggregated goodness-of-fit test. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

S. Talts, M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman. Validating bayesian inference algorithms with simulation-based calibration, 2018.

S. van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

N. Verzelen and E. Arias-Castro. Detection and feature selection in sparse mixture models. Annals of Statistics , 45(5):1920-1950, 2017.

C. Villani. Topics in Optimal Transportation . American Mathematical Society, 2003.

Y. Wu and P. Yang. Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics , 48:1987-2007, 2020.

J. Yang, V. Rao, and J. Neville. A stein-papangelou goodness-of-fit test for point processes. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89 of Proceedings of Machine Learning Research , pages 226-235. PMLR, 16-18 Apr 2019.

Supplement to 'Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models'

In this supplementary material, we present additional results and proofs. The minimax lower bounds and convergence rates of parameter estimation in the non-distinguishable settings are presented in Section A. The general theory for the convergence rates of densities and their proofs can be found in Appendix B. Proofs of the geometric inequalities that relate the convergence of density estimation to that of parameter estimation are in Section C, while those for minimax lower bounds and convergence rates of parameter estimation are left in Appendix D. Then, we provide a necessary lemma for those results along with its proof in Appendix E. Finally, some discussion about the general setting of the paper is presented in Section F, followed by a set of simulations to support the developed theory.

A Minimax Lower Bounds and Convergence Rates of Parameter Estimation under the Non-distinguishable Settings

Theorem A.1. (Strongly identifiable and non-distinguishable settings) Assume that classes of densities h 0 and f satisfy the conditions in Theorem 3.5. We define

for any sequence { l n } . Then, we achieve

(a) (Minimax lower bound) Assume that f satisfies assumption S.1 in Theorem 4.1. Then for any r < 1 and sequence { l n } , there exist two universal positive constants c 1 and c 2 such that

(b) (MLE rate) Let ̂ G n be the MLE defined in equation (3) , and the family { p G : G ∈ Ξ } satisfies condition A2. Then, for any sequence { l n } such that l n / log n →∞ ,

Proof of Theorem A.1 is in Appendix D.2. The results of part (b) are the generalization of those in Theorem 3.1 and Theorem 3.2 in [16] to the setting of strongly identifiable in the second-order kernel. The condition regarding the lower bound of λ in the formation of Ξ 1 ( l n ) is necessary to guarantee that ( ̂ µ n , ̂ Σ n ) and ̂ λ n are consistent estimators of ( µ ∗ , Σ ∗ ) and λ ∗ respectively. In particular, from the results in equation (34) of the proof of Theorem A.1, we have for any G ∗ ∈ Ξ that

Therefore, for any 1 ≤ i ≤ d 1 and 1 ≤ u, v ≤ d 2 we get

It indicates that

for the left-hand-side terms of the above display to go to 0 for all 1 ≤ i ≤ d 1 and 1 ≤ u, v ≤ d 2 .

The results of Theorem A.1 imply that as long as the kernel functions are strongly identifiable in the second order, the convergence rates of ̂ µ n to µ ∗ and ̂ Σ n to Σ ∗ are similar, which depend on the vanishing rate of ( λ ∗ ) 2 ∥ (∆ µ ∗ , ∆Σ ∗ ) ∥ 2 to 0. In our next result of location-covariance multivariate Gaussian distribution, we will demonstrate that such uniform convergence rates of different parameters no longer hold.

Theorem A.2. (Weakly identifiable and non-distinguishable settings) Assume that f is a family of location-covariance multivariate Gaussian distributions, and h 0 ( x ) = f ( x | µ 0 , Σ 0 ) for some ( µ 0 , Σ 0 ) ∈ Θ × Σ . We define

for any sequence { l n } . Then, the following holds:

(a) (Minimax lower bound) For any r < 1 and sequence { l n } , there exist two universal positive constants c 1 and c 2 such that

(b) (MLE rate) Let ̂ G n be the estimator defined in (3) . Then, for any sequence { l n } such that l n / log n →∞ the following holds

Proof of Theorem A.2 is in Appendix D.3. A few comments are in order:

(i) Similar to the argument after Theorem A.1, the condition regarding λ in the formulation of Ξ 2 ( l n ) is to guarantee that ( ̂ µ n , ̂ Σ n ) and ̂ λ n are consistent estimators of ( µ ∗ , Σ ∗ ) and λ ∗ , respectively.

(ii) The results of part (b) indicate that the convergence rate of estimating Σ ∗ is generally much faster than that of estimating µ ∗ regardless of the circumstance of ( λ ∗ ) 2 { ∥ ∆ µ ∗ ∥ 4 + ∥ ∆Σ ∗ ∥ 2 } . The non-uniformity of these convergence rates is mainly due to the structure of the heat partial differential equation, where the second-order derivative of the location parameter and the first-order derivative of covariance parameter correlate.

̸

(iii) From the results of part (b), it is clear that when ∥ ∆ µ ∗ ∥ + ∥ ∆Σ ∗ ∥ ̸→ 0 , i.e., ( µ ∗ , Σ ∗ ) → ( µ, Σ) = ( µ 0 , Σ 0 ) , and λ ∗ ̸→ 0 , the convergence rate of ̂ λ n to λ ∗ is n -1 / 2 . Furthermore, by using the result from part (a) of Proposition C.4 we can verify that

̸

where the supremum is taken over { G ∗ ∈ Ξ 2 ( l n ) : K ( G ∗ , G ) ≤ ϵ } , and G = ( λ, µ, Σ) , λ ∗ → λ , and ϵ is some sufficiently small positive constant. Since λ = 0 , we achieve the optimal convergence rate n -1 / 2 of estimating ( µ ∗ , Σ ∗ ) within a sufficiently small neighborhood of G under metric K . These results imply that even though the convergence rate of estimating G ∗ may be extremely slow

when G ∗ moves over the whole space Ξ 2 ( l n ) (global convergence), such convergence rate can be at standard rate n -1 / 2 when G ∗ moves within a sufficiently small neighborhood of some appropriate parameters G (local convergence).

As we have seen from the convergence rate results from location-covariance multivariate Gaussian distributions, the heat PDE structure plays a key role in the slow convergence rates of location and covariance parameters as well as the mismatch of orders of these rates.

B Convergence Rate of Density Estimation

B.1 General Theory and Proof of Theorem 2.3

We now describe the convergence rate of density estimation under the Hellinger distance in detail and give a general result for the multivariate deviated model. We recall some popular notions in Empirical Process theory as follows. An ϵ -net for a metric space ( P , d ) is a collection of balls with radius ϵ (with respect to metric d ) having union contains P . The minimal cardinality of such ϵ -nets is called the covering number and denoted by N ( ϵ, P , d ) . The logarithm of N ( ϵ, P , d ) is called the entropy number and is denoted by H ( ϵ, P , d ) . The bracketing number N B ( ϵ, P , d ) is the minimal number n such that there exists n pairs ( f i , f i ) n i =1 such that f i < f i , d ( f i , f i ) < ϵ , and their union covers P . The logarithm of N B ( ϵ, P , d ) is called the bracketing entropy number and is denoted by H B ( ϵ, P , d ) . In the following discussion, if P is a family of density and we omit d , we understand that d is the distance associated with L 2 ( m ) , where m is the Lebesgue measure.

Denote P (Ξ) = { p G : G ∈ Ξ } and P (Ξ) = { ( p G ∗ + p G ) / 2 : G ∈ Ξ } for the fixed true parameter G ∗ . The convergence rate can be deduced from the complexity of the set:

where for any G ∈ Ξ , we denote ¯ p G := ( p G + p G ∗ ) / 2 . We measure the complexity of this class through the bracketing entropy integral

where H B ( ϵ, P ) denotes the ϵ -bracketing entropy number of a metric space P . We recall assumption A2:

A2. Given a universal constant J > 0 , there exists N > 0 , possibly depending on Θ and k , such that for all n ≥ N and all ϵ > (log n/n ) 1 / 2 ,

Theorem B.1. Assume that Assumption A2 holds, and let k ≥ 1 . Then, there exists a constant C > 0 depending only on Θ and k such that for all n ≥ 1 ,

This result can be obtained by modifying the proof of Theorem 7.4 in [33]. Recall that we defined the function class

where for any G ∈ Ξ , we write ¯ p G = ( p G + p G ∗ ) / 2 , and measure the complexity of this class through the bracketing entropy integral

where N B ( ϵ, X, η ) denotes the ϵ -bracketing number of a metric space ( X,η ) and ν is the Lebesgue measure. We denote by P G the distribution corresponding to the density p G . The technique to prove this theorem is to bound the convergence rate by the increments of an empirical process:

where P n = 1 n ∑ n i =1 δ X i is the empirical measure ( X 1 , . . . , X n iid ∼ p G ∗ ). We first recall Theorem 5.11 in [33] with the notations adapted from our setting:

Theorem B.2. Let R > 0 , k ≥ 1 , and G be a subset of Ξ , which contains G ∗ . Given C 1 < ∞ , for all C sufficiently large, and for n ∈ N and t > 0 satisfying and

we have

Now we proceed to prove Theorem 2.3, the proof is divided into three parts: Bounding the tail probability of h ( p ˆ G n , p G ∗ ) by sums of empirical processes increments using the chaining technique, bounding the empirical processes increments using Theorem B.2, and bounding the expectation of h ( p ˆ G n , p G ∗ ) using its tail probability.

Step 1 (Bounding the tail probability h ( p ˆ G n , p G ∗ ) by sums of empirical processes increments): Firstly, by Lemma 4.1 and 4.2 of [33], we have

Hence, for any δ > δ n := (log n/n ) 1 / 2 , we have

where S is a smallest number such that 2 S δ/ 4 > 1 , as h ( p G , p G ∗ ) ≤ 1 . Now we will bound each term above using Theorem B.2.

Step 2 (Bounding the empirical processes increments using Theorem B.2): In Theorem B.2, choose R = 2 s +1 δ, C 1 = 15 and t = √ n 2 2 s ( δ/ 4) 2 , we can readily check that Condition (10) satisfies (because 2 s -1 δ/ 4 ≤ 1 for all s = 0 , . . . , S ). Condition (11) satisfies thanks to Assumption A3:

So the conclusion of Theorem B.2 gives us

where c is a large constants that does not depend on G ∗ .

Step 3 (Implying the bound on supremum of expectation): Thus, we have

for some ˜ c does not depend on λ ∗ , G ∗ . Hence, we finally proved that

As a consequence, we obtain the conclusion of the theorem.

B.2 Proof for Proposition 2.4

We further introduce some more notations that are required for the proof. Let N ( ϵ, P (Ξ) , ∥ · ∥ ∞ ) be the ϵ -covering number of ( P (Ξ) , ∥ · ∥ ∞ ) and N B ( ϵ, P (Ξ) , h ) be the bracketing number of P (Ξ) measured by Hellinger metric h . H B ( ϵ, P (Ξ) , h ) = log N B ( ϵ, P (Ξ) , h ) is called the bracketing entropy of P (Ξ) under metric h . We want to show that

for all n > N large enough and ϵ > (log n/n ) 1 / 2 . We proceed to show that claim (14) will be proved if

and then prove claim (15) and (16).

Proof of that claim (16) implies claim (14) Because P 1 / 2 (Ξ , δ ) ⊂ P 1 / 2 (Ξ) and from the definition of Hellinger distance,

Now use the fact that for densities f ∗ , f 1 , f 2 , we have h 2 (( f 1 + f ∗ ) / 2 , ( f 2 + f ∗ ) / 2) ≤ h 2 ( f 1 , f 2 ) / 2 , one can readily check that H B ( δ √ 2 , P (Ξ) , h ) ≤ H B ( δ, P (Ξ) , h ) . Hence, if claim (16) holds true, then

which implies that

for all ϵ > (log n/n ) 1 / 2 . Hence, claim (14) is proved.

Proof of claim (15) As λ ∈ [0 , 1] , we can choose an ϵ -net for it with the cardinality no more than 1 ϵ . Similarly, because Θ and Ω are compact, we can cover them by hypercube [ -a, a ] d 1 and

(

2

a

ϵ

[

-

b, b

]

. Hence, there exists

ϵ

-

nets for them with the cardinality no more than and

( 2 b ϵ ) d 2 2 . Let S be the Cartesian product of them. We have log |S| ≲ log(1 /ϵ ) and for every G = ( λ, µ, Σ) ∈ Ξ , there exists G ′ = ( λ ′ , µ ′ , Σ ′ ) ∈ S such that | λ -λ ′ | , ∥ µ -µ ′ ∥ , ∥ Σ -Σ ′ ∥ ≤ ϵ . By triangle inequalities,

thanks to the uniform bounded and Lipchitz assumptions. Hence,

)

1

d

2

×

d

2

d

Proof of claim (16) Now, from the entropy number, we are going to bound the bracketing number, we let η ≤ ϵ which will be chosen later. Let f 1 , . . . , f N be a η -net for P (Ξ) , where f i ( x ) = (1 -λ i ) h 0 ( x ) + λ i f ( x | µ i , Σ i ) . Let

is an envelop for f ( x | µ, Σ) . We can construct brackets [ p L i , p U i ] as follows.

Because for each f ∈ P (Ξ) , there is f i such that ∥ f -f i ∥ ∞ < η , we have p L i ≤ f ≤ p U i . Moreover, for any B ≥ B ,

where we use spherical coordinate to have

and

Hence, in (18), if we choose B = B (log(1 /η )) 1 /b 3 then

Therefore, there exists a positive constant c which does not depend on η such that

Let ϵ = cη (log(1 /η )) d/b 3 , we have log(1 /ϵ ) ≍ log(1 /η ) , which combines with inequality ∥·∥ 1 ≤ h 2 leads to

Thus, we have proved claim (16).

C Proofs for Geometric Inverse Bounds

C.1 Proof of Theorem 3.3

The second inequality in Theorem 3.3 is straightforward from the equivalent form of W 1 ( G,G ∗ ) in Lemma E.1 (see Appendix E). Therefore, we will only focus on establishing the first inequality in that theorem. We start with the following key result:

Proposition C.1. Given the assumptions in Theorem 3.5 and G = ( λ, µ, Σ) such that λ ∈ [0 , 1] and ( µ, Σ) can be equal to ( µ 0 , Σ 0 ) . Then, we have

Proof. The high level idea of the proof of Proposition C.3 is to utilize the Taylor expansion techniques previously employed in [9, 28, 22, 19]. Indeed, following Fatou's argument from Theorem 3.1 in [22], to obtain the conclusion of Proposition C.3 it suffices to demonstrate that

Assume that the above conclusion does not hold. It implies that we can find two sequences G n = ( λ n , µ n , Σ n ) and G ∗ ,n = ( λ ∗ n , µ ∗ n , Σ ∗ n ) such that K ( G n , G ) → 0 , K ( G ∗ ,n , G ) → 0 , and ∥ p G n -p G ∗ ,n ∥ ∞ / K ( G n , G ∗ ,n ) → 0 as n → ∞ . Now, we only consider the most challenging setting of ( µ n , Σ n ) and ( µ ∗ n , Σ ∗ n ) when they share the same limit point ( µ ′ , Σ ′ ) . The other settings of these two components can be argued in the same fashion. Here, ( µ ′ , Σ ′ ) is not necessarily equal to ( µ 0 , Σ 0 ) or ( µ, Σ) as λ n , λ ∗ n can go to 0 or 1 in the limit. Under that setting, by means of Taylor expansion up to the first order we obtain

where R 1 ( x ) is Taylor remainder and α = ( α 1 , α 2 ) in the summation of the second equality satisfies α 1 = ( α (1) 1 , . . . , α (1) d 1 ) ∈ N d 1 , α 2 = ( α (2) uv ) ∈ N d 2 × d 2 , | α | = d 1 ∑ i =1 α (1) i + ∑ 1 ≤ u,v ≤ d 2 α (2) uv , and

α ! = d 1 ∏ i =1 α (1) i ! ∏ 1 ≤ u,v ≤ d 2 α (2) uv ! . As f admits the first order uniform Lipschitz condition, we have R 1 ( x ) = O ( ∥ ( µ n , Σ n ) -( µ ∗ n , Σ ∗ n ) ∥ 1+ γ ) for some γ > 0 , which implies that

as n →∞ . Therefore, we can treat [ p G n ( x ) -p G ∗ ,n ( x )] / K ( G n , G ∗ ,n ) as the linear combination of h 0 ( x | θ 0 , Σ 0 ) and ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) when | α | ≤ 1 . Assume that the coefficients of these terms

go to 0. Then, by studying the coefficients of h 0 ( x | θ 0 , Σ 0 ) , ∂f ∂µ i ( x | µ 0 , Σ 0 ) , and ∂f ∂ Σ uv ( x | µ 0 , Σ 0 ) , we achieve

( λ ∗ n -λ n ) / K ( G n , G ∗ ,n ) → 0 , λ n ( µ n -µ ∗ n ) i / K ( G n , G ∗ ,n ) → 0 , λ n (Σ n -Σ ∗ n ) uv / K ( G n , G ∗ ,n ) → 0 for all 1 ≤ i ≤ d 1 and 1 ≤ u, v ≤ d 2 where ( a ) i denotes the i -th element of vector a and A uv denotes the ( u, v ) -th element of matrix A . It would imply that

Therefore, we achieve

a contradiction. Therefore, not all the coefficients of h 0 ( x | θ 0 , Σ 0 ) and ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) go to 0. If we denote m n to be the maximum of the absolute values of the coefficients of h 0 ( x | θ 0 , Σ 0 )

and ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) , then we get 1 /m n ̸→∞ as n →∞ , i.e., 1 /m n is uniformly bounded. Hence, we achieve for all x that

for some coefficients η and τ α such that they are not all 0. However, as f is distinguishable from h 0 up to the first order, the above equation indicates that η = τ α = 0 for all | α | ≤ 1 , a contradiction. As a consequence, we achieve the conclusion of the proposition.

Now, assume that the conclusion of Theorem (3.3) does not hold. It implies that we can find two sequences G ′ n and G ′ ∗ ,n such that A n = ∥ p G ′ n -p G ′ ∗ ,n ∥ 2 / K ( G ′ n , G ′ ∗ ,n ) → 0 as n →∞ . Since Θ and Ω are two bounded subsets, we can find subsequences of G ′ n and G ′ ∗ ,n such that K ( G ′ n , G 1 ) and K ( G ′ ∗ ,n , G 2 ) vanish to 0 as n →∞ where G 1 , G 2 are some discrete measures having one component to be ( µ 0 , Σ 0 ) . Because A n → 0 , we obtain V ( p G ′ n , p G ′ ∗ ,n ) → 0 as n →∞ . By means of Fatou's lemma, we have

Due to the fact that f is distinguishable from h 0 up to the first order, the above equation implies that G 1 ≡ G 2 . However, from the result of Proposition C.1, regardless of the value of G 1 we would have A n ̸→ 0 as n →∞ , which is a contradiction. Therefore, we obtain the conclusion of the theorem.

C.2 Proof of Theorem 3.5

Prior to presenting the proof of Theorem 3.5, we introduce the definition of second-order uniform Lipschitz:

Definition C.2 (Second-order Uniform Lipschitz) . We say that f is uniformly Lipschitz up to the second order if the following holds: there are positive constants δ 3 , δ 4 such that for any R 4 , R 5 , R 6 > 0 , γ 1 ∈ R d 1 , γ 2 ∈ R d 2 × d 2 , R 4 ≤ √ λ 1 (Σ) ≤ √ λ d 2 (Σ) ≤ R 5 , ∥ θ ∥ ≤ R 6 , θ 1 , θ 2 ∈ Θ , Σ 1 , Σ 2 ∈ Ω , there are positive constants C 1 depending on ( R 4 , R 5 ) and C 2 depending on R 6 such that for all x ∈ X ,

Now, we are back to the main proof. Utilizing the same Fatou's argument as that of Proposition C.1 , to achieve the conclusion of the first inequality in Theorem 3.5 it suffices to demonstrate the following result

Proposition C.3. Given the assumptions in Theorem 3.5 and G = ( λ, µ, Σ) such that λ ∈ [0 , 1] and ( µ, Σ) can be identical to ( µ 0 , Σ 0 ) . Then, the following holds

̸

̸

Proof. The proof of part (a) is essentially similar to that of Proposition C.1; therefore, we only provide the proof for the challenging settings of part (b). Here, we only consider the setting that ( µ 0 , Σ 0 ) ≡ ( µ, Σ) as the proof for other possibilities of ( µ 0 , Σ 0 ) can be argued in the similar fashion. Under this assumption, ( µ 0 , Σ 0 ) = ( µ 0 , Σ 0 ) , G = ( λ, θ 0 , Σ 0 ) , and h 0 ( x | θ 0 , Σ 0 ) = f ( x | θ 0 , Σ 0 ) for all x ∈ X . Assume that the conclusion of Proposition C.3 does not hold. It implies that we can find two sequences G n = ( λ n , µ n , Σ n ) and G ∗ ,n = ( λ ∗ n , µ ∗ n , Σ ∗ n ) such that D ( G n , G ) = λ n ∥ (∆ µ n , ∆Σ n ) ∥ 2 → 0 , D ( G ∗ ,n , G ) = λ ∗ n ∥ (∆ µ ∗ n , ∆Σ ∗ n ) ∥ 2 → 0 , and ∥ p G n -p G ∗ ,n ∥ ∞ / D ( G n , G ∗ ,n ) → 0 as n →∞ . For the transparency of presentation, we denote A n = ∥ (∆ µ n , ∆Σ n ) ∥ , B n = ∥ (∆ µ ∗ n , ∆Σ ∗ n ) ∥ , and C n = ∥ ( µ n , Σ n ) -( µ ∗ n , Σ ∗ n ) ∥ = ∥ (∆ µ n , ∆Σ n ) -(∆ µ ∗ n , ∆Σ ∗ n ) ∥ . Now, we have three main cases regarding the convergence behaviors of ( µ n , Σ n ) and ( µ ∗ n , Σ ∗ n )

Case 1: Both A n → 0 and B n → 0 , i.e., ( µ n , Σ n ) and ( µ ∗ n , Σ ∗ n ) vanish to ( µ 0 , Σ 0 ) as n → ∞ . Due to the symmetry between λ n and λ ∗ n , we assume without loss of generality that λ ∗ n ≥ λ n for infinite values of n . Without loss of generality, we replace these subsequences of G n , G ∗ ,n by the whole sequences of G n and G ∗ ,n . Now, the formulation of D ( G n , G ∗ ,n ) is

Now, by means of Taylor expansion up to the second order, we get

where R 1 ( x ) and R 2 ( x ) are Taylor remainders that satisfy R 1 ( x ) = O ( B 2+ γ n ) and R 2 ( x ) = O ( C 2+ γ n ) for some positive number γ due to the second order uniform Lipschitz condition of kernel density function f . From the formation of D ( G n , G ∗ ,n ) , since A n + B n ≥ C n (triangle inequality), as A n → 0 and B n → 0 it is clear that

as n →∞ for all x ∈ X . Therefore, we achieve for all x ∈ X that

Hence, we can treat [ p G n ( x ) -p G ∗ ,n ( x )] / D ( G n , G ∗ ,n ) as a linear combination of ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) for all x and α = ( α 1 , α 2 ) such that 1 ≤ | α | ≤ 2 . Assume that all the coefficients of these terms go to 0 as n →∞ . By studying the vanishing behaviors of the coefficients of ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) as | α | = 1 , we achieve the following limits

for all 1 ≤ i ≤ d 1 and 1 ≤ u, v ≤ d 2 where ( a ) i denotes the i -th element of vector a and A uv denotes the ( u, v ) -th element of matrix A . Furthermore, for any 1 ≤ i, j ≤ d ( i and j can be equal), the coefficient of ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) when ( α 1 ) i = ( α 1 ) j = 1 and α 2 = 0 leads to

When i = j , the above limits lead to

Therefore, we would have

Plugging the results from (22) into (20), we ultimately achieve for any 1 ≤ i, j ≤ d that

Using the results from (20) and (23), we would have

for any 1 ≤ i, j ≤ d . Therefore, it leads to

The above results mean that

By applying the above argument with the coefficients of ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) when α 1 = 0 and ( α 2 ) u 1 v 1 = ( α 2 ) u 2 v 2 = 1 for any two pairs ( u 1 , v 1 ) , ( u 2 , v 2 ) (not neccessarily distinct) such that 1 ≤ u 1 , u 2 , v 1 , v 2 ≤ d or ( α 1 ) i = 1 and ( α 2 ) uv = 1 for any 1 ≤ i ≤ d and 1 ≤ u, v ≤ d , we respectively obtain that

Combining the results from (21), (24), and (25) leads to

which is a contradiction. As a consequence, not all the coefficients of ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) go to 0 as 1 ≤ | α | ≤ 2 . Follow the argument of Proposition C.1, by denoting m n to be the maximum of the absolute values of the coefficients of ∂ | α | f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) we achieve for all x that

where τ α ∈ R are some coefficients such that not all of them are 0. Due to the second order identifiability condition of f , the above equation implies that τ α = 0 for all α such that | α | = 2 , which is a contradiction. As a consequence, Case 1 cannot happen.

̸

Case 2: Exactly one of A n and B n goes to 0, i.e., there exists at least one component among ( µ n , Σ n ) and ( µ ∗ n , Σ ∗ n ) that does not converge to ( µ 0 , Σ 0 ) as n →∞ . Due to the symmetry of A n and B n , we assume without loss of generality that A n ̸→ 0 and B n → 0 , which is equivalent to ( µ n , Σ n ) → ( µ ′ , Σ ′ ) = ( µ 0 , Σ 0 ) while ( µ ∗ n , Σ ∗ n ) → ( µ 0 , Σ 0 ) as n →∞ . We denote

Since [ p G n ( x ) -p G ∗ ,n ( x )] / D ( G n , G ∗ ,n ) → 0 , we achieve that [ p G n ( x ) -p G ∗ ,n ( x )] / D ′ ( G n , G ∗ ,n ) → 0 for all x as D ( G n , G ∗ ,n ) ≲ D ′ ( G n , G ∗ ,n ) . By means of Taylor expansion up to the first order, we have

,

where R ′ 1 ( x ) is Taylor remainder that satisfies ( λ ∗ n -λ n ) | R ′ 1 ( x ) | / D ′ ( G n , G ∗ ,n ) = O ( B γ ′ n ) → 0 for some positive number γ ′ > 0 . Since ( µ n , Σ n ) and ( µ ∗ n , Σ ∗ n ) do not have the same limit, they will be different when n is large enough, i.e., n ≥ M ′ for some value of M ′ . Now, as n ≥ M ′ , [ p G n ( x ) -p G ∗ ,n ( x )] / D ′ ( G n , G ∗ ,n ) becomes a linear combination of ∂f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) for all | α | ≤ 1 and f ( x | µ n , Σ n ) . If all of the coefficients of these terms go to 0, we would have λ n / D ′ ( G n , G ∗ ,n ) → 0 , ( λ ∗ n -λ n )( -∆ µ ∗ n ) i / D ′ ( G n , G ∗ ,n ) → 0 , and ( λ ∗ n -λ n )( -∆Σ ∗ n ) uv / D ′ ( G n , G ∗ ,n ) → 0 for all 1 ≤ i ≤ d 1 and 1 ≤ u, v ≤ d 2 . It would imply that ( λ ∗ n -λ n ) B n / D ′ ( G n , G ∗ ,n ) → 0 λ n A n / D ′ ( G n , G ∗ ,n ) → 0 , and λ n B n / D ′ ( G n , G ∗ ,n ) → 0 . These results lead to

a contradiction. Therefore, not all the coefficients of ∂f ∂µ α 1 ∂ Σ α 2 ( x | µ ∗ n , Σ ∗ n ) and f ( x | µ n , Σ n ) go to 0. By defining m ′ n to be the maximum of these coefficients, we achieve for all x that

where η ′ and τ ′ α are coefficients such that not all of them are 0, which is a contradiction to the first order identifiability of f . As a consequence, Case 2 cannot hold.

Case 3: Both A n and B n do not go to 0, i.e., ( µ n , Σ n ) and ( µ ∗ n , Σ ∗ n ) do not converge to ( µ 0 , Σ 0 ) as n → ∞ . Since D n ( G n , G ∗ ,n ) ≲ K ( G n , G ∗ ,n ) = | λ n -λ ∗ n | + ( λ n + λ ∗ n ) C n and [ p G n ( x ) -p G ∗ ,n ( x )] / D ( G n , G ∗ ,n ) → 0 , we achieve that [ p G n ( x ) -p G ∗ ,n ( x )] / K ( G n , G ∗ ,n ) → 0 for all x . From here, by using the same argument as that of the proof of Proposition C.1, we also reach the contradiction. Therefore, Case 3 cannot happen.

In sum, we achieve the conclusion of the proposition.

C.3 Proof of Theorem 3.6

For the simplicity of proof argument, we will only consider the univariate setting of Gaussian kernel, i.e., when both µ and Σ = σ 2 are scalars. The argument for the multivariate setting of Gaussian kernel can be argued in the rather similar fashion, which is omitted. Throughout this proof, we denote v := σ 2 . Now, according to the proof argument of Proposition C.1 and Proposition C.3, to achieve the conclusion of the theorem it suffices to demonstrate the following result:

Proposition C.4. Given G = ( λ, µ, v ) such that λ ∈ [0 , 1] and ( µ, v ) can be identical to ( µ 0 , v 0 ) . Then, the following holds

̸

̸

Proof. We will only provide the proof for part (b) since the proofs for part (a) can be argued in similar fashion as that of Proposition C.1. Assume that the conclusion of Proposition C.4 does not hold. It implies that we can find two sequences G n = ( λ n , µ n , v n ) and G ∗ ,n = ( λ ∗ n , µ ∗ n , v ∗ n ) such that Q ( G n , G ) → 0 , Q ( G ∗ ,n , G ) → 0 , and ∥ p G n -p G ∗ ,n ∥ ∞ / Q ( G n , G ∗ ,n ) → 0 as n →∞ . Due to the symmetry between λ n and λ ∗ n , we can assume without loss of generality that λ ∗ n ≥ λ n . Therefore, we achieve that

In this proof, we only consider the scenario when ∥ (∆ µ n , ∆ v n ) ∥ → 0 and ∥ (∆ µ ∗ n , ∆ v ∗ n ) ∥ → 0 since the arguments for other settings of these two terms are similar to those of Case 2 and Case 3 in the proof of Proposition C.3. As being indicated in Section 3.2.2, the univariate Gaussian kernel contains the partial differential equation structure ∂ 2 f ∂µ 2 ( x | µ, v ) = 2 ∂f ∂v ( x | µ, v ) for all µ ∈ Θ and

v ∈ Ω . Therefore, for any α = ( α 1 , α 2 ) we can check that

where β = α 1 +2 α 2 . Now, by means of Taylor expansion up to the fourth order, we obtain

where R 1 ( x ) , R 2 ( x ) are Taylor remainders and the range of α 1 , α 2 in the summation of the second equality satisfies β = α 1 +2 α 2 . As Gaussian kernel admits fourth-order uniform Lipschitz condition, it is clear that

as n → ∞ for some γ > 0 . Therefore, we can consider [ p G n ( x ) -p G ∗ ,n ( x )] / Q ( G n , G ∗ ,n ) as a β

linear combination of ∂ f ∂µ β ( x | µ ∗ n , v ∗ n ) for 1 ≤ β ≤ 8 . If all of the coefficients of these terms go to 0, then we obtain

for any 1 ≤ β ≤ 8 . Now, we divide our argument with L β into two key cases

Case 1: ( λ n ( | ∆ µ n | 2 + | ∆ v n | ) + λ ∗ n | (∆ µ ∗ n | 2 + | ∆ v ∗ n | ) ) / { λ n ( | µ n -µ ∗ n | 2 + | v n -v ∗ n | ) } ̸→∞ . It implies that as n is large enough, we would have

Combining the above result with L β → 0 for all 1 ≤ β ≤ 8 , we get

Note that, when the denominator of the above limits is ( λ ∗ n -λ n )( | ∆ µ ∗ n | 4 + | ∆ v ∗ n | 4 ) + λ n ( | ∆ µ n -∆ µ ∗ n | 4 + | ∆ v n -∆ v ∗ n | 4 ) , the technique for studying the above system of limits with this denominator has been considered in Proposition 2.3 in [21]. However, since the current denominator of H β strongly dominates by the previous denominator, we must develop a more sophisticated control of H β as 1 ≤ β ≤ 8 to obtain a concrete understanding of their limits. Due to the symmetry between λ ∗ n -λ n and λ n , we assume without loss of generality that λ ∗ n -λ n ≤ λ n for all n (by the subsequence argument). We have two possibilities regarding λ n and λ ∗ n

Case 1.1: ( λ ∗ n -λ n ) /λ n ̸→ 0 as n →∞ . Under that setting, we define p n = max { λ ∗ n -λ n , λ n } and

Additionally, we let ( λ ∗ n -λ n ) /p n → c 2 1 , λ n /p n → c 2 2 , ∆ µ ∗ n /M n →-a 1 , (∆ µ ∗ n -∆ µ n ) /M n → a 2 , ∆ v ∗ n /M 2 n →-2 b 1 , and (∆ v n -∆ v ∗ n ) /M 2 n → 2 b 2 . From here, at least one among a 1 , a 2 , b 1 , b 2 and both c 1 , c 2 are different from 0. Now, by dividing both the numerators and the denominators of H β as 1 ≤ β ≤ 4 by p n M β n , we achieve the following system of polynomial equations

As being indicated in Proposition 2.1 in [21], this system will only admits the trivial solution, i.e., a 1 = a 2 = b 1 = b 2 = 0 , which is a contradiction. Therefore, Case 1.1 cannot happen.

Case 1.2: ( λ ∗ n -λ n ) /λ n → 0 , i.e., λ ∗ n /λ n → 1 , as n → ∞ . Under that setting, if M n ∈ max { | ∆ µ n -∆ µ ∗ n | , | ∆ v n -∆ v ∗ n | 1 / 2 } , then we have

By dividing both the numerator and the denominator of H 1 by λ n M n , given that the new denominator of H 1 goes to 0, its new numerator also goes to 0, i.e., we obtain

Since ( λ ∗ n -λ n ) /λ n → 0 and | ∆ µ ∗ n | ≤ M n , we have ( λ ∗ n -λ n )( -∆ µ ∗ n ) / { λ n M n } → 0 . Therefore, we have (∆ µ n -∆ µ ∗ n ) /M n → 0 . With the previous results, by dividing both the numerator and the denominator of H 2 by λ n M 2 n and given that the new denominator goes to 0, we have

As ( λ ∗ n -λ n )( -∆ v ∗ n ) / { λ n M 2 n } → 0 (due to the assumption of M n ), we get (∆ v n -∆ v ∗ n ) /M 2 n → 0 . These results imply that

which is a contradiction. Therefore, we would only have M n ∈ max { | ∆ µ ∗ n | , | ∆ v ∗ n | 1 / 2 } . For the simplicity of the proof, we only consider the setting when M n = | ∆ µ ∗ n | for all n (by subsequence argument). The setting that M n = | ∆ v ∗ n | 1 / 2 for all n can be argued in the similar fashion. Now, if we have

then by dividing the numerator and denominator of H i with λ n ( max { | ∆ µ n -∆ µ ∗ n | , | ∆ v n -∆ v ∗ n | 1 / 2 }) i as 1 ≤ i ≤ 2 , we would achieve

a contradiction. Therefore, we must have

as n →∞ . Now, we further divide the argument under that setting of M n into two small cases

Case 1.2.1: ( λ ∗ n -λ n ) | ∆ µ ∗ n | 4 ≤ λ n | ∆ µ n -∆ µ ∗ n | 4 for all n (by subsequence argument). Since M n = | ∆ µ ∗ n | , we would have ( λ ∗ n -λ n ) | ∆ µ ∗ n | i ≤ λ n | ∆ µ n -∆ µ ∗ n | i for all n and 1 ≤ l ≤ 4 . From here, we obtain that

If | ∆ µ n -∆ µ ∗ n | / | ∆ v n -∆ v ∗ n | 1 / 2 ̸→ 0 , by diving both the numerator and the denominator of H 1 by λ n | ∆ µ n -∆ µ ∗ n | and given that the new denominator goes to 0, the new numerator must converge to 0, i.e. we have

However, since we have | (∆ µ n -∆ µ ∗ n ) / ∆ µ ∗ n → 0 , the above result would imply that

which is a contradiction to the assumption of Case 1.2.1.1. As a consequence, we must have | ∆ µ n -∆ µ ∗ n | / | ∆ v n -∆ v ∗ n | 1 / 2 → 0 . Now, we also have that

for all 1 ≤ i ≤ 3 . Without loss of generality, we assume that ∆ v n -∆ v ∗ n > 0 for all n . We denote ( -∆ µ ∗ n ) = q n 1 (∆ v n -∆ v ∗ n ) and ∆ v ∗ n = q n 2 (∆ v n -∆ v ∗ n ) for all n . From the result of (26), we would have | q n 1 |→∞ . Given the above results, by dividing the numerators and the denominators of H β by λ n (∆ v n -∆ v ∗ n ) β/ 2 for any 1 ≤ β ≤ 3 , we would have the new denominators go to 0. Therefore, all the new numerators of these H β also go to 0, i.e. we achieve the following system of limits

Since | q n 1 |→∞ , the last limit in the above system implies that ( λ ∗ n -λ n ) ( ( q n 1 ) 2 3 + q n 2 ) /λ n → 0 .

Combining this result with the second limit in the above system yields that ( λ ∗ n -λ n )( q n 1 ) 2 /λ n + 3 / 2 → 0 , which cannot happen. Therefore, Case 1.2.1 does not hold.

Case 1.2.2: ( λ ∗ n -λ n ) | ∆ µ ∗ n | 4 > λ n | ∆ µ n -∆ µ ∗ n | 4 for all n (by subsequence argument). If ( λ ∗ n -λ n ) | ∆ µ ∗ n | 4 ≤ λ n | ∆ v n -∆ v ∗ n | 2 for all n , the by using the same argument as that of Case 1.2.1, we quickly achieve the contradiction. Therefore, we must have ( λ ∗ n -λ n ) | ∆ µ ∗ n | 4 > λ n | ∆ v n -∆ v ∗ n | 2 . Denote (∆ µ n -∆ µ ∗ n ) = m n 1 ( -∆ µ ∗ n ) , ( -∆ v ∗ n ) = m n 2 (∆ µ ∗ n ) 2 , and (∆ v n -∆ v ∗ n ) = m n 3 (∆ µ ∗ n ) 2 . Since M n = | ∆ µ ∗ n | , we would have | m n i | ≤ 1 for all 1 ≤ i ≤ 3 . Denote m n i → m i for all 1 ≤ i ≤ 3 (by subsequence argument). The results of (26) lead to m 1 = m 3 = 0 . Now by dividing both the numerator and denominator of H β by ( λ ∗ n -λ n )( -∆ µ ∗ n ) β for any 1 ≤ β ≤ 4 , as the new denominators of H | β | do not go to ∞ , we would also achieve that the new numerators of H | β | go to 0, i.e. the following system of limits hold

Combining with m n 1 → 0 , the first and third limit of the above system of limits imply that m 2 = -1 / 3 . From here, the second and fourth limit yields that 1 / 6 + m 2 + m 2 2 / 2 = 0 , which is a contradiction. Therefore, Case 1.2.2 cannot hold.

We will demonstrate that Q ( G n , G ∗ ,n ) ≍ Q ( G n , G ∗ ,n ) . In fact, from the above formulation of Q ( G n , G ∗ ,n ) , we would have that

where the first inequality is due to the triangle inequality and basic inequality ( a + b ) 2 ≤ 2( a 2 + b 2 ) and the second inequality is due to the following result

On the other hand, we also have that

where the last inequality is due to triangle inequality and basic inequality ( a + b ) 2 ≤ 2( a 2 + b 2 ) . Therefore, we conclude that Q ( G n , G ∗ ,n ) ≍ Q ( G n , G ∗ ,n ) . Now, since H β → 0 for all 1 ≤ β ≤ 8 , we would have that

Similar to Case 1, under Case 2 we also consider two distincts setting of λ ∗ n /λ n

Case 2.1: λ ∗ n /λ n ̸→∞ . Under this case, we denote

From the assumption of Case 2, we would have

Due to the symmetry between ( | ∆ µ n | 2 , | ∆ v n | ) and ( | ∆ µ ∗ n | 2 , | ∆ v ∗ n | ) , we assume without loss of generality that M ′ n ∈ max { | ∆ µ n | 2 , | ∆ v n | } . Under that assumption, we have two distinct cases

Case 2.1.1: M ′ n = | ∆ µ n | 2 for all n (by the subsequence argument). From (27), we have | ∆ µ n -∆ µ ∗ n | / | ∆ µ n |→ 0 , i.e., ∆ µ n / ∆ µ ∗ n → 1 . To be able to utilize the assumptions of Case 2, we will need to study the formulations of F β more deeply. In fact, when β = 1 simple calculation yields

When β = 2 , we have

Combining with the result of A 1 , it is clear that

Combining the above result with F 2 → 0 , we would have

Now, we have two small cases

Case 2.1.1.1: ∆ v n / (∆ µ n ) 2 → 0 as n →∞ . From (27), since we have | ∆ v n -∆ v ∗ n | / (∆ µ n ) 2 → 0 , it implies that ∆ v ∗ n / (∆ µ n ) 2 → 0 . Since ∆ µ n / ∆ µ ∗ n → 1 , we also have that ∆ v ∗ n / (∆ µ ∗ n ) 2 → 0 . Now, from the formulations of Q ( G n , G ∗ ,n ) we have

From the result that A 2 → 0 , by multiplying A 2 with ∆ µ n ∆ µ ∗ n , we would also have that

As λ ∗ n /λ n ̸→∞ , we have two distinct settings of λ ∗ n /λ n

Case 2.1.1.1.1: λ ∗ n /λ n ̸→ 1 . Using the result from (28) and the fact that ∆ µ n / ∆ µ ∗ n → 1 , we would obtain that

Combining the above result with (29), it leads to

Combining (28) and (30), we would achieve that

From the formulation of Q ( G n , G ∗ ,n ) , the above limit implies that

Due to the previous assumptions, we obtain that

By combining the results from (28) and (30), we can verify that

Therefore, we achieve E → 0 , which is a contradiction. As a consequence, Case 2.1.1.1.1 cannot happen.

Case 2.1.1.1.2: λ ∗ n /λ n → 1 . Under this case, if we have

then we will achieve that

From here, by using the same argument as that of Case 2.1.1.1.1, we will obtain E → 0 , which is a contradiction. Therefore, we would have that

With that assumption, it leads to Q ( G n , G ∗ ,n ) ≍ ( λ ∗ n -λ n )(∆ µ ∗ n ) 2 (∆ µ n ) 2 ≍ ( λ ∗ n -λ n )(∆ µ ∗ n ) 4 as ∆ µ ∗ n / ∆ µ n → 1 . Now, we denote ∆ µ n -∆ µ ∗ n = τ n 1 ∆ µ ∗ n and ∆ v n -∆ v ∗ n = τ n 2 (∆ µ ∗ n ) 2 . From the assumption of Case 2.1.1.1, we would have that τ n 1 → 0 and τ n 2 → 0 . By dividing both the numerator and the denominator of F 3 by ( λ ∗ n -λ n )(∆ µ ∗ n ) 3 , as the new denominators of F 3 goes to 0, we also obtain the numerator of this term goes to 0, i.e., the following holds

From (31), we have that λ n ( τ n 1 ) 2 / ( λ ∗ n -λ n ) ̸→∞ and λ n τ n 2 / ( λ ∗ n -λ n ) ̸→∞ . Therefore, since τ n 1 → 0 and τ n 2 → 0 , we would achieve that λ n ( τ n 1 ) 3 / ( λ ∗ n -λ n ) → 0 and λ n τ n 1 τ n 2 / ( λ ∗ n -λ n ) → 0 . By plugging these results to the above limit, it implies that -1 / 6 = 0 , which is a contradiction. As a consequence, Case 2.1.1.1.2 cannot hold.

Case 2.1.1.2: ∆ v n / (∆ µ n ) 2 ̸→ 0 as n →∞ . Under that case, we will only consider the setting that λ ∗ n /λ n → 1 as the argument for other settings of that ratio can be argued in the similar fashion. Since we have | ∆ v n -∆ v ∗ n | / (∆ µ n ) 2 → 0 , it leads to ∆ v ∗ n / (∆ µ n ) 2 ̸→ 0 . Combining with ∆ µ n / ∆ µ ∗ n → 1 , it implies that as n is large enough we would have

According the formulation of Q ( G n , G ∗ ,n ) , we achieve

If we have B → 0 , we would get ( λ ∗ n -λ n ) | ∆ v n ∆ v ∗ n | Q ( G n , G ∗ ,n ) → 0 . Combining with (32), we can check that all the results in (28), (30), and (31) hold. With similar argument as Case 2.1.1.1, we achieve Q ( G n , G ∗ ,n ) / Q ( G n , G ∗ ,n ) → 0 , a contradiction. Therefore, we must have B ̸→ 0 . It implies that as n is large enough we must have

Furthermore, as ( λ ∗ n -λ n ) /λ n → 0 , we obtain | ∆ v ∗ n | / | ∆ v n -∆ v ∗ n | → ∞ and | ∆ v ∗ n | / | ∆ µ n -∆ µ ∗ n | 2 → ∞ , i.e., ∆ v n / ∆ v ∗ n → 1 . With all of these results, we can check that Q ( G n , G ∗ ,n ) ≲ ( λ ∗ n -λ n ) | ∆ v ∗ n | 2 . Denote (∆ v n -∆ v ∗ n ) = k n 1 | ∆ v ∗ n | , (∆ µ n -∆ µ ∗ n ) = k n 2 | ∆ v ∗ n | 1 / 2 , and ∆ µ ∗ n = k n 3 | ∆ v ∗ n | 1 / 2 for all n . From all the assumptions we have thus far, we get k n 1 → 0 , k n 2 → 0 , and | k n 3 | ̸→ ∞ . Additionally, as B ̸→ 0 , we further have λ n | k n 1 | / ( λ ∗ n -λ n ) ̸→∞ and λ n ( k n 2 ) 2 / ( λ ∗ n -λ n ) ̸→∞ . By dividing both the numerator and the denominator of F 3 and F 4 respectively by ( λ ∗ n -λ n ) | ∆ v ∗ n | 3 / 2 and ( λ ∗ n -λ n ) | ∆ v ∗ n | 2 , as the new denominators of F 3 , F 4 do not go to infinity, we obtain the new numerators of these terms go to 0, i.e., the following holds

With the assumptions with k n 1 , k n 2 , and k n 3 , we would have

for any 3 ≤ i ≤ 4 and 1 ≤ j ≤ 2 . If we denote k n 3 → k 3 , by combining all the above results we achieve the following system of equations

which does not admit a solution, a contradiction. Hence, Case 2.1.1.2 cannot hold.

Case 2.1.2: M ′ n = | ∆ v n | for all n (by the subsequence argument). From (27), we would have | ∆ v n -∆ v ∗ n | / | ∆ v n | → 0 , i.e., ∆ v n / ∆ v ∗ n → 1 , and | ∆ µ n -∆ µ ∗ n | 2 / | ∆ v n | → 0 . The argument under this case is rather similar to that of Case 2.1; therefore, we only sketch the key steps. By using the result that A 1 → 0 and A 2 → 0 , we would obtain that

As F 4 → 0 , we equivalently have

Under Case 2.1.2, we only consider the setting when (∆ µ n ) 2 / ∆ v n → 0 as other settings of this term can be argued in the similar fashion as that of Case 2.1.2. Since | ∆ µ n -∆ µ ∗ n | 2 / | ∆ v n |→ 0 ,

we have (∆ µ ∗ n ) / ∆ v n → 0 . As ∆ v n / ∆ v ∗ n → 1 , we also further have that (∆ µ ∗ n ) 2 / ∆ v ∗ n → 0 and (∆ µ n ) 2 / ∆ v n → 0 . Therefore, we have ∆ µ n ∆ µ ∗ n / ∆ v n → 0 and ∆ µ n ∆ µ ∗ n / ∆ v ∗ n → 0 . Now, from the formulation of Q ( G n , G ∗ ,n ) , we achieve

Combining these results with A 4 → 0 , we achieve ( λ ∗ n -λ n )∆ v n ∆ v ∗ n / Q ( G n , G ∗ ,n ) → 0 . From here, we can easily verify that all the results in (31) hold. Thus, by using the same argument as that of Case 2.1.1, we would get Q ( G n , G ∗ ,n ) / Q ( G n , G ∗ ,n ) → 0 , a contradiction. As a consequence, Case 2.1.2 cannot happen.

Case 2.2: λ ∗ n /λ n →∞ . Remind that M ′ n = max { | ∆ µ n | 2 , | ∆ v n | , | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } . We can verify that Q ( G n , G ∗ ,n ) ≲ λ ∗ n ( M ′ n ) 4 . By dividing both the numerator and the denominator of A 1 and A 2 respectively by λ ∗ n ( M ′ n ) 1 / 2 and λ ∗ n M ′ n , given that the new denominators go to 0 we would obtain the new numerators also go to 0, i.e., we have the following results

Since λ n /λ ∗ n → 0 , the first limit implies that ∆ µ ∗ n /M ′ n → 0 . Combining this result with the second limit, we obtain ∆ v ∗ n /M ′ n → 0 . Therefore, we would have M ′ n = max { | ∆ µ n | 2 , | ∆ v n | } . Without loss of generality, we assume that M ′ n = | ∆ µ n | 2 as the argument for other possibility of M ′ n can be argued in the similar fashion. With these assumptions, | ∆ v n -∆ v ∗ n | / | ∆ µ n | 2 ̸→∞ , i.e., as n is large enough we get | ∆ v n -∆ v ∗ n | ≲ | ∆ µ n | 2 . Now, we have two distinct cases

Case 2.2.1: λ ∗ n max { | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } / ( λ n | ∆ µ n | 2 ) → ∞ . Due to this assumption, we can check that as n is large enough, Q ( G n , G ∗ ,n ) ≍ λ ∗ n | ∆ µ n | 2 max { | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } . If max { | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } = | ∆ µ ∗ n | 2 for all n , then by dividing both the numerator and denominator of A 1 by λ ∗ n ∆ µ ∗ n , given that the new denominator of A 1 goes to 0, its new numerator must go to 0, i.e., we have

which cannot hold since λ | ∆ µ n | 2 / ( λ ∗ n | ∆ µ ∗ n | 2 ) → 0 (assumption of Case 2.2.1) and | ∆ µ n | / | ∆ µ ∗ n |→ ∞ . Therefore, we must have max { | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } = | ∆ v ∗ n | for all n . By dividing both the numerator and denominator of A 2 by λ ∗ n ∆ v ∗ n , as the new denominator of A 2 goes to 0, we would have

Since λ n | ∆ v n | λ ∗ n | ∆ v ∗ n | ≤ λ n | ∆ µ n | 2 λ ∗ n | ∆ v ∗ n | → 0 and ( λ ∗ n -λ n ) /λ ∗ n → 1 , the above limit shows that ∆ µ n ∆ µ ∗ n / ∆ v ∗ n → 1 . Since (∆ µ n ) 2 / | ∆ v ∗ n | → ∞ , it implies that (∆ µ ∗ n ) 2 / ∆ v ∗ n → 0 . Now, by combing the result that A 1 → 0 and A 2 → 0 , since F 3 → 0 , we can verify that it is equivalent to

By dividing both the numerator and the denominator of A 3 by λ ∗ n ∆ µ n ∆ v ∗ n , we obtain

As (∆ µ ∗ n ) 2 / ∆ v ∗ n → 0 and ∆ µ n ∆ µ ∗ n / ∆ v ∗ n → 1 , the above limit leads to ∆ µ ∗ n ∆ v n / (∆ µ n ∆ v ∗ n ) → -1 / 3 . Now, by studying A 4 → 0 with the assumption that Q ( G n , G ∗ ,n ) ≍ λ ∗ n | ∆ µ n | 2 | ∆ v ∗ n | , we eventually get the equation 1 / 24 -1 / 12 = 0 , which is a contradiction. Therefore, Case 2.2.1 cannot hold.

Case 2.2.2: λ ∗ n max { | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } /λ n | ∆ µ n | 2 ̸→ ∞ . Therefore, as n is large enough, we would have λ ∗ n max { | ∆ µ ∗ n | 2 , | ∆ v ∗ n | } ≲ ( λ n | ∆ µ n | 2 ) . Hence, we achieve under this case that Q ( G n , G ∗ ,n ) ≍ λ n | ∆ µ n | 4 . Denote ∆ µ ∗ n = l n 1 ∆ µ n , ∆ v n = l n 2 (∆ µ n ) 2 , and ∆ v ∗ n = l n 3 (∆ µ n ) 2 . From the assumptions of Case 2.2.2, we would have l n 1 → 0 and l n 3 → 0 while l n 2 ̸→∞ . Additionally, λ ∗ n max { ( l n 1 ) 2 , | l n 3 | } /λ n ̸→ 0 . By dividing the numerators and denominators of A i by λ n (∆ µ n ) i for 1 ≤ i ≤ 3 , we achieve the following system of limits

As l n 1 → 0 , the first limit in the above system implies that λ ∗ n ( l n 1 ) 2 /λ n → 0 . If we have max { ( l n 1 ) 2 , | l n 2 | } = | l n 1 | 2 for all n , the previous result would mean that λ ∗ n l n 3 /λ n → 0 . Therefore, the second limit in (33) demonstrates that l n 2 → -1 . However, plugging these results to the third limit in this system would yield 1 / 3 -1 = 0 , which is a contradiction. Hence, we must have max { ( l n 1 ) 2 , | l n 2 | } = | l n 3 | for all n . Under this setting, by denoting λ ∗ n l n 3 λ n → a as n →∞ , the first and second limit in (33) leads to l n 2 → a -1 . With this result, the third limit in this system shows that a = 2 / 3 . With these results, by dividing both the numerator and denominator of A 4 by λ n (∆ µ n ) 4 , we quickly achieve the equation 1 / 24 -5 / 72 = 0 , which is a contradiction. Therefore, Case 2.2.2 cannot hold.

In sum, not all the coefficients of ∂ | β | f ∂µ β ( x | µ ∗ n , v ∗ n ) as 1 ≤ | β | ≤ 8 go to 0. From here, by using the same argument as that of Proposition C.1 and Proposition C.3, we achieve the result of part (b) of the proposition. As a consequence, we reach the conclusion of the theorem.

D Proofs for Convergence Rates of Parameter Estimation and Minimax Lower Bounds

In this appendix, we provide the proofs for the convergence rates of the MLE as well as the corresponding minimax lower bounds introduced in Section D.

D.1 Proof of Theorem 4.1

(a) For any G = G ( λ , µ , Σ ) and G = G ( λ , µ , Σ ) , we denote the following distance

Even though d 2 ( G 1 , G 2 ) is a proper distance, it is clear that d ( G 1 , G 2 ) is not symmetric and only satisfies a weak triangle inequality, i.e. we have

Therefore, we will utilize the modification of Le Cam method for nonsymmetric loss in Lemma 6.1 of [16] to deal with such distance. We start with the following proposition

Proposition D.1. Given that f satisfies assumption (S.1) in Theorem 4.1, we achieve for any r < 1 that

Proof. (i) For any sequences G 1 ,n = ( λ n , µ 1 ,n , Σ 1 ,n ) and G 2 ,n = ( λ n , µ 2 ,n , Σ 2 ,n ) , we have

where the first inequality is due to √ p G 1 ,n ( x ) + √ p G 2 ,n ( x ) > √ λ n f ( x | µ 2 ,n , Σ 2 ,n ) . By Taylor expansion up to the first order, we have

Now, by choosing λ 1 -2 r n ∥ ( µ 1 ,n , Σ 1 ,n ) -( µ 2 ,n , Σ 2 ,n ) ∥ 2 -2 r → 0 , and ∥ ( µ 1 ,n , Σ 1 ,n ) -( µ 2 ,n , Σ 2 ,n ) ∥ → 0 and using condition (S.1), we can easily verify that h ( p G 1 ,n , p G 2 ,n ) /d r 1 ( G 1 ,n , G 2 ,n ) → 0 . Therefore, we achieve the conclusion of part (i).

(ii) The argument for this part is essentially similar to that in part (i). In fact, for any two sequences G ′ 1 ,n = ( λ 1 ,n , µ n , Σ n ) and G ′ 2 ,n = ( λ 2 ,n , µ n , Σ n ) , we also obtain

By choosing ( λ 1 ,n -λ 2 ,n ) 2 -2 r / { (1 -λ 1 ,n ) ∧ λ 1 ,n } → 0 , we also achieve the conclusion of part (ii).

Now, given G ∗ = ( λ ∗ , µ ∗ , Σ ∗ ) and r < 1 . Let C 0 be any fixed constant. According to part (i) of Proposition D.1, for any sufficiently small ϵ > 0 , there exists G ′ ∗ = ( λ ∗ , µ ∗ 1 , Σ ∗ 1 ) such that d 1 ( G ∗ , G ′ ∗ ) = d 1 ( G ′ ∗ , G ∗ ) = ϵ and h ( p G ∗ , p G ′ ∗ ) ≤ C 0 ϵ r . By means of Lemma 6.1 of [16], we achieve

where p n G ∗ denotes the density of the n -iid sample X 1 , . . . , X n . From there,

Hence, we obtain

By choosing ϵ 2 r = 1 C 2 0 n , we achieve

for any r < 1 where c 1 is some positive constant. Using the similar argument, with the result of (ii) in Proposition D.1 we also immediately obtain the result inf ̂ G n ∈ Ξ sup G ∈ Ξ E p G ( | ̂ λ n -λ | 2 ) ≥ c 2 n -1 /r . As a consequence, we reach the conclusion of part (a) of the theorem.

(b) The proof of this part is a direct consequence of Theorem 3.3 and Theorem 2.3. Indeed, for ̂ G n = ( ̂ λ n , ̂ µ n , ̂ Σ n ) being the MLE as in equation (3), we have

Because all inequalities are uniform in G ∗ , we achieve the conclusion of part (b) of the theorem.

D.2 Proof of Theorem A.1

(a) Similar to the proof argument of part (a) of Theorem 4.1, we define

for any G 1 = G 1 ( λ 1 , µ 1 , Σ 1 ) and G 2 = G 2 ( λ 2 , µ 2 , Σ 2 ) . It is clear that both d 3 ( G 1 , G 2 ) and d 4 ( G 1 , G 2 ) still satisfy weak triangle inequality. To achieve the conclusion of this part, it suffices to demonstrate the following results

(i) There exists two sequences G 1 ,n = ( λ n , µ 1 ,n , Σ 1 ,n ) ∈ Ξ 1 ( l n ) and G 2 ,n = ( λ n , µ 2 ,n , Σ 2 ,n ) ∈ Ξ 1 ( l n ) such that d 3 ( G 1 ,n , G 2 ,n ) → 0 and h ( p G 1 ,n , p G 2 ,n ) /d r 3 ( G 1 ,n , G 2 ,n ) as n →∞ .

(ii) There exists two sequences G ′ 1 ,n = ( λ 1 ,n , µ n , Σ n ) ∈ Ξ 1 ( l n ) and G ′ 2 ,n = ( λ 2 ,n , µ n , Σ n ) ∈ Ξ 1 ( l n ) such that d 4 ( G 1 ,n , G 2 ,n ) → 0 and h ( p G ′ 1 ,n , p G ′ 2 ,n ) /d r 4 ( G 1 ,n , G 2 ,n ) as n →∞ .

for any r < 1 . The proof argument for the above results can proceed in a similar fashion as that of Proposition D.1; therefore, it is omitted. We achieve the conclusion of part (a) of the theorem.

(b) Combining the result of Theorem 3.5 and the fact that D ( G,G ∗ ) ≍ D ( G,G ∗ ) for any G and G ∗ , we immediately achieve the following convergence rates

It is clear that the second result in (34) does not match with the second result in the conclusion of part (b) of the theorem. To circumvent this issue, we utilize the fact that G ∗ ∈ Ξ 1 ( l n ) . Indeed, notice that ( ̂ µ n , ̂ Σ n ) -( µ ∗ , Σ ∗ ) = (∆ ̂ µ n , ∆ ̂ Σ n ) -(∆ µ ∗ , ∆Σ ∗ ) , we have

Hence, by the AM-GM inequality, we have

uniformly in G ∗ , where in the last inequality we use (35) combining with the fact that | ̂ λ n -λ ∗ | is uniformly bounded by 2. Hence,

which is the conclusion of the theorem.

D.3 Proof of Theorem A.2

(a) Similar to the proof argument of part (a) of Theorem 4.1, we define

for any G 1 = G 1 ( λ 1 , µ 1 , Σ 1 ) and G 2 = G 2 ( λ 2 , µ 2 , Σ 2 ) . It is clear that d 6 ( G 1 , G 2 ) satisfies weak triangle inequality while d 5 ( G 1 , G 2 ) no longer satisfies weak triangle inequality. In particular, we have

A close investigation of Lemma 6.1 of [16] reveals that modified Le Cam method still works under this setting of d 5 metric. More specifically, for any ϵ > 0 the following holds

where G 1 , G 2 ∈ Ξ 2 ( l n ) such that d 5 ( G 1 , G 2 ) ∧ d 5 ( G 1 , G 2 ) ≥ ϵ/ 4 . From here, to achieve the conclusion of part (a), it suffices to demonstrate for any r < 1 that

(i) There exists two sequences G 1 ,n = ( λ n , µ 1 ,n , Σ 1 ,n ) ∈ Ξ 2 ( l n ) and G 2 ,n = ( λ n , µ 2 ,n , Σ 2 ,n ) ∈ Ξ 1 ( l n ) such that d 5 ( G 1 ,n , G 2 ,n ) → 0 and h ( p G 1 ,n , p G 2 ,n ) /d r 5 ( G 1 ,n , G 2 ,n ) as n →∞ .

(ii) There exists two sequences G ′ 1 ,n = ( λ 1 ,n , µ n , Σ n ) ∈ Ξ 2 ( l n ) and G ′ 2 ,n = ( λ 2 ,n , µ n , Σ n ) ∈ Ξ 1 ( l n ) such that d 6 ( G 1 ,n , G 2 ,n ) → 0 and h ( p G ′ 1 ,n , p G ′ 2 ,n ) /d r 6 ( G 1 ,n , G 2 ,n ) as n →∞ .

Following the proof argument of Proposition D.1, we can quickly verify the above results. As a consequence, we reach the conclusion of part (a) of the theorem.

(b) From the discussion after Theorem 3.5, we can show that:

Hence, from Theorem 3.6 combining with Theorem 2.3, we have

Similar to the proof of Theorem A.1 and with the definition of Ξ 2 ( l 2 ) , we have

uniformly in G ∗ ∈ Ξ 2 ( l 2 ) . Hence,

As a consequence, we obtain the conclusion of the theorem.

E Proofs for Auxiliary Results

Lemma E.1. For any r ≥ 1 , we define

for any G and G ∗ . Then, we have W r r ( G,G ∗ ) ≍ D r ( G,G ∗ ) for any r ≥ 1 where W r is the r -th order Wasserstein distance.

Proof. Without loss of generality, we assume throughout the lemma that λ < λ ∗ . Therefore, we obtain from the formulation of D r ( G,G ∗ ) that

Direct computation of W r r ( G,G ∗ ) yields three distinct cases:

Case 2: If || (∆ µ, ∆Σ) || r + || (∆ µ ∗ , ∆Σ ∗ ) || r < || ( µ, Σ) -( µ ∗ , Σ ∗ ) || r and λ + λ ∗ ≤ 1 , then

From Cauchy-Schartz's inequality, we have || (∆ µ, ∆Σ) || r + || (∆ µ ∗ , ∆Σ ∗ ) || r ≳ || ( µ, Σ) -( µ ∗ , Σ ∗ ) || r . Therefore, under Case 2 we have || (∆ µ, ∆Σ) || r + || (∆ µ ∗ , ∆Σ ∗ ) || r ≍ || ( µ, Σ) -( µ ∗ , Σ ∗ ) || r , which directly implies that W r r ( G,G ∗ ) ≍ D r ( G,G ∗ ) .

Therefore, we also have W r r ( G,G ∗ ) ≍ D r ( G,G ∗ ) under Case 3.

Combining the results from these cases, we reach the conclusion of the lemma.

F Discussion and Additional Experiments

F.1 Parameter Changes with the Sample Size

In statistics and machine learning, researchers often want to know how many samples are enough to achieve some pre-specified ϵ error for the estimation of parameter θ in the fitted model. In the language of probability, we want to find an inequality such as E ∥ ̂ θ n -θ ∥ < C ∗ rate ( n ) , where rate ( n ) is a decreasing sequence in n and C does not depend on n . Usually, in the parametric models, we have rate ( n ) = 1 / √ n , and therefore it takes C 2 /ϵ 2 samples to achieve average ϵ error in estimation. In complex models such as hierarchical models or the multivariate deviated model that we consider in this paper, difficulties arise because of the singularity and identifiability of the model. For example, in Eq. (2), if λ ∗ = 0 , any pair of ( µ ∗ , Σ ∗ ) yields the same model. Hence, when λ ∗ ≈ 0 , it should be harder to estimate ( µ ∗ , Σ ∗ ) , and researchers may need more samples to have an accurate estimation for them. Notably, we have shown, for example in Theorem 4.1, the precise dependence of the convergence rate of ( µ ∗ , Σ ∗ ) on the magnitude of λ ∗ . In particular, we have

where C is a constant that does not depend on λ ∗ , µ ∗ , Σ ∗ and n . Hence, one can have a good estimation (with error ϵ ) for λ ∗ with C 2 /ϵ 2 samples, while he needs C 2 / ( ϵ ∗ λ ∗ ) 2 samples to achieve such a good estimation for ( µ ∗ , Σ ∗ ) . The simulation studies in the next section will make this clearer.

(a) Rate of

̂

λ

n

(b) Rate of

̂

µ

n

(c) Rate of

̂

σ

2

n

(a) Rate of

̂

λ

n

(b) Rate of

µ

n

(c) Rate of

σ

2

n

-1.2

y =-0.51 x+-0.51

y =-0.15 x +-0.37

y=-0.1x+-0.38

1.4

-0.4

0.4

1.6

.-“yl)60

=

-1.8

(1,n )6o

0.6

(l(,0)-l)60

0.6

2.0

0.8

0.8

2.2

1.0

2.4

1.0

2.6

-1.2

2.8

1.2

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

1.25

y=-0.49x+-0.62

Z'0

y=-0.02x+-0.39

0.0

y=0.01x+-0.52

1.50

0.0

Z°0

=

1.75

1,)6

=

0.2

(1（0)-

“y1)60l

2.00

-0.4

0l)601

0.4

2.25

2.50

0.6

0.6

2.75

0.8

8°0

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

̂

F.2 Additional Experiments for the Distinguishable Settings

We have seen in the main text that in two cases where λ ∗ is either fixed or decreasing with rate n -1 / 4 , the convergence rate of λ ∗ is C × n -1 / 2 , where the constants C is the same for both cases. The convergence rate for ( µ ∗ , Σ ∗ ) is C × n -1 / 4 for the latter case, which is much slower than the parametric rate in the former case. This phenomenon is quite rare for parametric models. We want to further bring readers' attention to two more extreme cases:

λ ∗ = 0 . 5 /n 3 / 8 as n increases;

where we consider the same ( µ ∗ , Σ ∗ ) with the experiments in the main text. The convergence rate for ( λ, µ, σ 2 ) in both cases in the log domain can be seen in Figure 3 and Figure 4. Hence, in all cases, the rate of convergence for λ ∗ is always of order n -1 / 2 , meanwhile, the rate for ( µ ∗ , Σ ∗ ) becomes slower as λ ∗ tends to 0 faster. From the theoretical result, when λ ∗ = 0 . 5 /n 3 / 8 , we expect the rate for ( µ ∗ , Σ ∗ ) to be of order n -1 / 8 , which is demonstrated in Figure 3(b)&(c)). At the extreme case λ ∗ = 0 . 5 /n 1 / 2 , it is even impossible to recover ( µ ∗ , Σ ∗ ) as n → ∞ (cf. Figure 4(b)&(c)). This suggests practitioners collect more data when ˆ λ n is small to have a good estimate for ( µ ∗ , Σ ∗ ) . Finally, in the case ˆ λ n is extremely small (of order n -1 / 2 ), we suggest not to report the estimated values ( ̂ µ n , ̂ Σ n ) , as they are highly uncertain.

F.3 Non-distinguishable Settings

Finally, we consider the weakly identifiable and non-distinguishable setting here to demonstrate that the convergence rate for λ ∗ can be slower than the parametric rate when f is near h 0 . Let both h 0 and f belong to the location-scale Gaussian family. h 0 ( x ) = f ( x | 0 , 1) and consider two cases of ( λ ∗ , µ ∗ , Σ ∗ ) :

λ ∗ = 0 . 25 , µ ∗ = 0 . are fixed and σ ∗ = 1 + n -1 / 8 as n increases;

λ ∗ = 0 . 25 , σ ∗ = 1 . are fixed and µ ∗ = n -1 / 8 as n increases;

̂

(a) Rate of

̂

λ

n

(b) Rate of

̂

µ

n

(c) Rate of

̂

σ

2

n

(a) Rate of

̂

λ

n

(b) Rate of

̂

µ

n

(c) Rate of

̂

σ

2

n

y =-0.3 x +-0.17

0.4

y=-0.51x+0.34

y=-0.43x+0.77

0.4

0.0

0.6

.-yl)6o

0.6

0.8

(1l,r-)6o

0.8

1.0

(l<( ,0)-l)6o1

0.4

0.2

-1.0

1.2

0.6

-1.2

1.4

0.8

1.6

-1.4

1.8

1.0

1.6

2.0

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

(u)60)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

0.3

y=0.03x+-0.81

0.2

y =-0.15x+-0.27

y=-0.49x+0.3

0.4

0.3

0.4

9°0

0.5

0.4

(1(,0)-el)6ol

8°0

)-“y1)601

0.6

-“)6o|

0.5

1.0

0.7

0.6

-1.2

0.8

0.7

1.4

0.9

0.8

1.6

1.0

0.9

1.8

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

2.00

2.25

2.50

2.75

log(n)

3.00

3.25

3.50

3.75

4.00

Recall that we have proved that { ∥ ∆ µ ∗ ∥ 4 + ∥ ∆Σ ∗ ∥ 2 } | ̂ λ n -λ ∗ | = O ( n -1 / 2 ) and λ ∗ ( ∥ ∆ µ ∗ ∥ 2 + ∥ ∆ ̂ µ n ∥ 2 + ∥ ∆Σ ∗ ∥ + ∥ ∆ ̂ Σ n ∥ )( ∥ ̂ µ n -µ ∗ ∥ 2 + ∥ ̂ Σ n -Σ ∗ ∥ ) = O ( n -1 / 2 ) , where there is a mismatch in the orders of convergence rates of the location and scale parameter. Notably, the rate of convergence for λ ∗ also depends on the rate ∆ µ ∗ and ∆( σ ∗ ) 2 → 0 . The experiments do support this theoretical finding, where we have the rate for λ ∗ is ≈ n -1 / 4 is the first case (as ∥ ∥ ∆( σ ∗ ) 2 ∥ ∥ 2 = O ( n -1 / 4 ) ) and it does not convergence in the second case λ ∗ is ≈ n -1 / 4 is the first case (as ∥ ∆ µ ∗ ∥ 4 = O ( n -1 / 2 ) ). The mismatch rate for ∥ ̂ µ n -µ ∗ ∥ and ∥ ̂ σ 2 n -( σ ∗ ) 2 ∥ can also be seen clearly in the second case, where the rate for the scale parameter is still of the parametric rate, whereas it is slower for the mean.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the optimal rate for multivariate deviated models. Specifically, they consider the model $(1-\\lambda) h(x) + \\lambda f(x|\\mu,\\Sigma)$, where $h$ is known and the goal is to estimate the other parameters. The authors propose to use the notion of *distinguishability* and study the convergence rate of parameters using MLE under both distinguishable and non-distinguishable cases. The authors present three pairs of upper and lower bounds for distinguishable, non-distinguishable but $f$ is strongly-identifiable, and distinguishable but $f$ is a family of location-covariance multivariate Gaussian distributions. Experiments are also provided to corroborate their theoretical results.\n\nSTRENGTHS:\n- The paper is clear. The notations are very consistent for such a lengthy work.\n- There are plenty of explanations and discussions around each main result, making it an interesting paper to read.\n- To the best of my knowledge, the proofs are technically sound and highly sophisticated. The notion of distinguishability and identifiability seems very suitable and intuitive.\n- Lower bounds are also presented and match their convergence rate for all three cases considered.\n\nWEAKNESSES:\n- The idea of distinguishability is not wholly novel. Similar notions have been used in [1] for a different model, which in turn is derived from the notion of *identifiability* adopted in [2] and many other previous works.\n- The section for related works is very concise. \n- Given the classical parametric setting and the MLE estimation, I am not very sure if this paper would be a good match for neurips rather than a more statistically-focused journal. \n\n[1] Do, Dat, Nhat Ho, and XuanLong Nguyen. \"Beyond black box densities: Parameter learning for the deviated components.\" Advances in Neural Information Processing Systems 35 (2022): 28167-28178.\n[2] Nguyen, XuanLong. \"Convergence of latent mixing measures in finite and infinite mixture models.\" (2013): 370-400.\n\nQUESTIONS:\n- I totally understand the logic behind the organization of the sections but I would suggest shortening Section 3 and moving the results in Appendix A to the main content. Theorem 3.3/3.5/3.6 seem to be intermediate results for proving the convergence rates with artificial distances $\\mathcal{K}$, $\\mathcal{D}$, and $\\mathcal{G}$. So I don't quite understand why they should take up nearly three pages, forcing the main results to be postponed to the appendix.\n- Say we would like to adopt the model to fit some data in practice. How should we obtain $h_0$ which is claimed to be known in this paper? Or $h_0$ could just be good enough and then this model may take over? Would you please comment on how the results may provide possible guidance to the methodology?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors establish the rate for estimating true parameters in the multivariate deviated model by using the MLE method. They mainly try to address two challenges encountered in deriving the rate of convergence for MLE estimators, i.e. 1) the interaction between the null hypothesis density $h_0$ and the alternative density function $f$, 2) the likelihood of the deviated proportion $\\lambda$ vanishing to either extreme points of the interval [0, 1]. To this end, they develop the distinguishability condition to capture the linear independent relation between the function $h_0$ and the density function $f$, and derived the optimal convergence rate of the MLE under both distinguishable and non-distinguishable settings.\n\nSTRENGTHS:\nThe paper is well-structured and effectively presents the problem setup, theoretical framework, and main results. The definitions and explanations of key concepts are well presented. The authors address a fundamental statistical problem and provide insights into the behavior of the MLE in the multivariate deviated model. The derived convergence rates and minimax rates contribute to the understanding of parameter estimation and hypothesis testing in complex data scenarios.\n\nWEAKNESSES:\nI feel that in the paper the comparison to existing literature is a bit limited. Particularly, how does this paper compare with the current literature on heterogeneous mixture detection?\nIn the experiment section, it seems that the setup is a bit limited with f being Gaussian. It would be great if the authors could show some more numerical results with more expanded scenarios.\n\nQUESTIONS:\nHow does this paper compare with the current literature on heterogeneous mixture detection in terms of assumptions and results?\n\nCan the results in Section 3.2.2 be extended to non-Gaussian distributions for f?\n\nLIMITATIONS:\nNo potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
## Summary

The authors study the minimax rate for parameter recovery in deviated multivariate models.
In this setting, we observe samples from a mixture (of *unknown* weight \lambda) of a "null" distribution h_0 and a distribution from a parametric family f( | \mu, \Sigma).
The goal is to recover from n samples the (\lambda, \mu, \Sigma).

## Contribution

The authors study the MLE performance in various regimes (depending on how "far" is h_0 from the parametric family of distributions.
Their analysis is very tight, leading to obtaining the minimax rates.

STRENGTHS:
I like the result: it is mathematically clean and leads to tight results. It is always nice to see the minimax rates for new problems.

WEAKNESSES:
I am unfortunately failing to conclude the position of the paper in the literature. 
Are the authors the first to obtain results in this setting?
If yes, please explain (much) more why studying the model is interesting/important.
If not, please compare thoroughly the results with the existing ones.

Also, are the techniques used new in any way? Or are the results simply an application of known techniques?

All in all, I think the paper needs a much more thorough literature review.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of parameter recovery in the multivariate deviated model where the data is generated according to the following distribution:\n$$\n    (1 - \\lambda) h_0 (x) + \\lambda f(x | \\mu, \\sigma)\n$$\nwhere $f$ belongs to a mean-variance family and $h_0$ is known. One prominent example of such a family is the family of Gaussian distributions.\n\nThe paper studies the recovery problem under three settings. The first is the distinguishable setting where $h_0$ and the density $f$ are distinguishable (essentially $h_0$ cannot be written as a linear combination of two distributions from the family the family) where statistical recovery is guaranteed at a $\\sqrt{n}$ rate. In the second setting of strong identifiability, $f$ is assumed to belong to the mean-variance family and here the convergence rates depend on the closeness of $\\mu_0, \\Sigma_0$, the parameters of $h_0$, to $\\mu, \\Sigma$, the parameters of the unknown mixture component. Finally, in the third setting where the mean-variance family is the Gaussian distribution where different convergence behavior is observed from the strongly identifiable setting where a second order PDE guarantees improved performance.\n\nFrom a technical standpoint, the algorithm (MLE) is analyzed in the following way. First, they show that under some mild assumptions on the function class, the MLE solutions approximate the distribution in Hellinger-distance. Then, by noting the relationship between the Hellinger and TV distances, the paper then shows that the for this class of distributions, the distance between the parameters is upper bounded by a constant multiple of the TV distance. The first step follows by standard empirical process theory. The second, however, relies on some intricate recently developed machinery. Roughly speaking, one first shows that the TV and parameter-distances approximate each other locally where the limit of the neighborhood is taken to $0$. Subsequently, a short analytic argument leads to a global approximation guarantee. This technique, while inspired prior work, still takes significant care to execute. \n\nOverall, the results in the paper are interesting and the technical contributions seem strong. The fact that the statistical performance in this setting may be distinguished from algorithms that operate on mixture models where both components are unknown is also intriguing. However, Theorem 3.6 is only proved for the univariate setting (Appendix C3) while the rest of the paper focuses on the multivariate setting.\n\nSTRENGTHS:\nSee main review\n\nWEAKNESSES:\nSee main review\n\nQUESTIONS:\nSee main review\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper tackles the issue of parameter estimation in the deviated Gaussian mixture of experts problem using the Maximum Likelihood Estimation (MLE) method. The authors propose new distances and analyze the convergence of MLE under distinguishable and non-distinguishable conditions.

STRENGTHS:
This paper is in relatively good shape. The results seem to be solid.

WEAKNESSES:
The major weakness is the novelty. 

This paper basically considers a much simpler case than the paper https://huynm99.github.io/Deviated_MoE.pdf. 

They consider multiple $k$ while this paper considers a single $k$. 

The definitions, results,  organization, and even the notations are almost the same. For example, the hellinger distance and TV distance (quite strange to me but adopted by both papers interesting), although this paper changes the distance from $D$.

QUESTIONS:
1. what is the mini-max rate of this problem? how close is the current rate to the min-max rate? 

2. What is the difference and technical novelty compared to https://huynm99.github.io/Deviated_MoE.pdf

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07