SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dat Do*
University of Michigan at Ann Arbor
dodat@umich.edu
Huy Nguyen*
huynm@utexas.edu
We study the maximum likelihood estimation (MLE) in the multivariate deviated model where the data are generated from the density function  $(1 - \lambda^{*})h_{0}(x) + \lambda^{*}f(x|\mu^{*},\Sigma^{*})$  in which  $h_0$  is a known function,  $\lambda^{*}\in [0,1]$  and  $(\mu^{*},\Sigma^{*})$  are unknown parameters to estimate. The main challenges in deriving the convergence rate of the MLE mainly come from two issues: (1) The interaction between the function  $h_0$  and the density function  $f$ ; (2) The deviated proportion  $\lambda^{*}$  can go to the extreme points of [0, 1] as the sample size tends to infinity. To address these challenges, we develop the distinguishability condition to capture the linear independent relation between the function  $h_0$  and the density function  $f$ . We then provide comprehensive convergence rates of the MLE via the vanishing rate of  $\lambda^{*}$  to zero as well as the distinguishability of two functions  $h_0$  and  $f$ .
The goodness-of-fit test [11] is one of the foundational tools in statistics with several applications in data-driven scientific fields, namely kernel Stein discrepancy [27, 31], point processes [37] and Bayesian statistics [32], etc. Given a sample set of data and a pre-specified distribution with density function  $h_0$ , the test indicates whether the samples are reasonably distributed according to  $h_0$  (null hypothesis) or to another family of distributions  $\{p(\cdot |\theta):\theta \in \Theta \}$  (alternative hypothesis). It is worth noting that knowledge about the null hypothesis distribution can come from prior knowledge of scientists. A key to understanding the statistical efficiency of testing is via the likelihood ratio and the maximum likelihood estimation (MLE) methods. [6].
While traditional testing problems often assume the null distribution  $h_0 = p(\cdot |\theta_0)$  and the alternative one  $p(\cdot |\theta)$  are from a single simple family of distributions such as exponential families, there are also many problems in science require to test  $h_0$  against the alternative  $f(\cdot |\theta)$  that can be deviated from  $h_0$  by a distribution from a potentially different family. Specifically, in this paper, we consider the family of distributions named multivariate deviated model with density functions defined as follows:
$$
p _ {G} (x) := (1 - \lambda) h _ {0} (x) + \lambda f (x | \mu , \Sigma), \tag {1}
$$
where  $x \in \mathbb{R}^d$ ,  $G \coloneqq (\lambda, \mu, \Sigma)$  are the model's parameters with  $\lambda \in [0,1]$  being the deviated proportion (from  $h_0$ ) and  $(\mu, \Sigma) \in \Theta \times \Omega$  are parameters of a vector-matrix family of distributions  $f$ , where  $\Theta \subset \mathbb{R}^d$  and  $\Omega \subset \mathbb{R}^{d \times d}$  being compact. When  $\lambda = 0$ , this recovers the null hypothesis distribution  $h_0$ .
The deviated model can be motivated by many applications in science. For instance, in microarray data analysis, it can be used to detect differentially expressed genes under two or more conditions [1, 2], where  $h_0$  is the uniform distribution and  $f(\cdot |\mu ,\Sigma)$  is required to estimate. Many other applications can be seen in many contamination problems in astronomy and biology [29]. Besides, the deviated model can also be viewed as a low-rank adaptation model in the domain adaptation problem [23], where  $h_0$  is a pre-trained model on large data, and  $f$  is a simpler component to be estimated from the smaller data domain. Our goal in this paper is to study the parameter estimation rate of the deviated model.
Problem setup. Suppose that we observe  $n$  i.i.d. samples  $X_{1},\ldots ,X_{n}$  from the true multivariate deviated model:
$$
p _ {G _ {*}} (x) := \left(1 - \lambda^ {*}\right) h _ {0} (x) + \lambda^ {*} f \left(x \mid \mu^ {*}, \Sigma^ {*}\right), \tag {2}
$$
where  $G_{*} \coloneqq (\lambda^{*},\mu^{*},\Sigma^{*})$  are true but unknown parameters with  $\lambda^{*} \neq 0$ . Throughout the paper, we allow  $G_{*}$  to change with the sample size  $n$  (see Appendix F.1 for a discussion). To facilitate our presentation, we suppress the dependence of  $G_{*}$  on  $n$ , and then estimate  $G_{*}$  from the data. The main focus of this paper is to establish both a uniform convergence rate and minimax rate for parameter estimation via the MLE approach, which is given by:
$$
\widehat {G} _ {n} \in \underset {G \in \Xi} {\arg \max } \sum_ {i = 1} ^ {n} \log p _ {G} \left(X _ {i}\right), \tag {3}
$$
where  $\widehat{G}_n\coloneqq (\widehat{\lambda}_n,\widehat{\mu}_n,\widehat{\Sigma}_n)$  and  $\Xi \coloneqq [0,1]\times \Theta \times \Omega$
Contribution. There are two main challenges in studying the convergence rate of the MLE  $\widehat{G}_n$ : (1) The interaction between the function  $h_0$  and the density function  $f$ , e.g.,  $h_0$  belongs to the family of  $f$  and  $(\mu^*, \Sigma^*)$  approaches  $h_0$  as the sample size  $n$  goes to infinity; (2) The deviated proportion  $\lambda^*$  can go to the extreme points of [0, 1] as the sample size goes to infinity and make the estimation become more challenging, because when  $\lambda^* = 0$ , all the parameters  $(\mu^*, \Sigma^*)$  yield the same model. To address these singularity and identifiability issues, we first develop the distinguishability condition to capture the linear independent relation between the function  $h_0$  and the density function  $f$ . We then study the optimal convergence rate of parameters under both distinguishable and non-distinguishable settings of the multivariate deviated model. Our theoretical results can be summarized as follows:
1. Distinguishable settings: We demonstrate that as long as the function  $h_0$  and the density function  $f$  are distinguishable, the convergence rate of  $\widehat{\lambda}_n$  to  $\lambda^*$  is  $\mathcal{O}(n^{-1/2})$  while  $(\widehat{\mu}_n, \widehat{\Sigma}_n)$  converges to  $(\mu^*, \Sigma^*)$  at a rate determined by the vanishing rate of  $\lambda^*$  as follows:
$$
\lambda^ {*} \| (\widehat {\mu} _ {n}, \widehat {\Sigma} _ {n}) - (\mu^ {*}, \Sigma^ {*}) \| = \mathcal {O} (n ^ {- 1 / 2}).
$$
It indicates that if  $\lambda^{*}$  goes to 0, the convergence rate of estimating  $(\mu^{*},\Sigma^{*})$  is slower than the parametric rate.
2. Non-distinguishable settings: When  $h_0$  and  $f$  are not distinguishable, it becomes complicated to capture the convergence rate of the MLE. To shed light on the behaviors of the MLE under the non-distinguishable settings of multivariate deviated model, we specifically study the settings when  $h_0$  belongs to the same family as  $f$ , namely,  $h_0(.)=f(.|\mu_0,\Sigma_0)$  for some  $(\mu_0,\Sigma_0)$ . To precisely characterize the rates of the MLE under this setting, we consider the second-order strong identifiability of  $f$ , which requires the linear independence up to second-order derivatives of  $f$  with respect to its parameters. The second-order identifiability had also been considered in the literature to investigate the convergence rate of parameter estimation in finite mixtures [9, 28, 22, 21, 20, 19].
2.1. Strongly identifiable and non-distinguishable settings: When  $f$  is strongly identifiable in the second order, we demonstrate that  $\| (\Delta \mu^{*},\Delta \Sigma^{*})\|^{2}|\widehat{\lambda}_{n} - \lambda^{*}| = \mathcal{O}(n^{-1 / 2})$  and
$$
\lambda^ {*} (\| (\Delta \mu^ {*}, \Delta \Sigma^ {*}) \| + \| (\Delta \widehat {\mu} _ {n}, \Delta \widehat {\Sigma} _ {n} \|) \| (\widehat {\mu} _ {n}, \widehat {\Sigma} _ {n}) - (\mu^ {*}, \Sigma^ {*}) \| = \mathcal {O} (n ^ {- 1 / 2}),
$$
where  $\Delta \mu \coloneqq \mu -\mu_0$  and  $\Delta \Sigma \coloneqq \Sigma -\Sigma_0$ . It indicates that the convergence rate of  $\widehat{\lambda}_n$  to  $\lambda^*$  depends on that of  $(\mu^{*},\Sigma^{*})$  to  $(\mu_0,\Sigma_0)$  while the convergence rate of  $(\widehat{\mu}_n,\widehat{\Sigma}_n)$  to  $(\mu^{*},\Sigma^{*})$  depends on both the rate of  $\lambda^*$  to 0 and the rate of  $(\mu^{*},\Sigma^{*})$  to  $(\mu_0,\Sigma_0)$ . These results are strictly different from those in the distinguishable settings, which is mainly due to the non-distinguishability between  $h_0$  and  $f$ .
2.2. Weakly identifiable and non-distinguishable settings: When  $f$  is weakly identifiable, i.e., it is not strongly identifiable in the second order, we specifically consider the popular setting when  $f$  is the density of a multivariate Gaussian distribution. The loss of the strong identifiability of the Gaussian distribution is due to the following partial differential equation (PDE) between the location and scale parameters (the heat equation):
$$
\frac {\partial^ {2} f}{\partial \mu \partial \mu^ {\top}} (x | \mu , \Sigma) = 2 \frac {\partial f}{\partial \Sigma} (x | \mu , \Sigma).
$$
Due to the above PDE, the convergence rate of the MLE under this setting exhibits very different behaviors from those under the strongly identifiable setting. In particular, we prove that  $\left[\| \Delta \mu^{*}\|^{4} + \| \Delta \Sigma^{*}\|^{2}\right]\left|\widehat{\lambda}_{n} - \lambda^{*}\right| = \mathcal{O}(n^{-1 / 2})$  and
$$
\lambda^ {*} (\| \Delta \mu^ {*} \| ^ {2} + \| \Delta \widehat {\mu} _ {n} \| ^ {2} + \| \Delta \Sigma^ {*} \| + \| \Delta \widehat {\Sigma} _ {n} \|) (\| \widehat {\mu} _ {n} - \mu^ {*} \| ^ {2} + \| \widehat {\Sigma} _ {n} - \Sigma^ {*} \|) = \mathcal {O} (n ^ {- 1 / 2}).
$$
Notably, there is a mismatch in the orders of convergence rates of the location vector and covariance matrix. Furthermore, the rate of the deviated mixing proportion also depends on different orders of  $\mu^{*}$  to  $\mu_0$  and  $\Sigma^{*}$  to  $\Sigma_0$ . Such rich behaviors of the MLE are mainly due to the PDE between the location and scale parameters.
Comparing to moment methods. We would like to remark that the results for the MLE under the non-distinguishable settings in the paper are (much) tighter than those obtained from moment methods for a general mixture of two components in the literature. In particular, when  $f$  is multivariate Gaussian distribution with fixed covariance matrix, i.e.,  $f$  is strongly identifiable in the second order and we do not estimate  $\Sigma^{*}$ , an application of the results with moment methods from [36] to the deviated models leads to  $\| \Delta \mu^{*} \|^{3} |\lambda_{n}^{\text{moment}} - \lambda^{*}| = \mathcal{O}(n^{-1/2})$  and  $\lambda^{*} \| \mu_{n}^{\text{moment}} - \mu^{*} \|^{3} = \mathcal{O}(n^{-1/2})$ , which are much slower compared to the results for the MLE in the strongly identifiable and non-distinguishable settings, where  $(\lambda_{n}^{\text{moment}}, \mu_{n}^{\text{moment}})$  denote moment estimators of  $\lambda^{*}$  and  $\mu^{*}$ .
When  $f$  is a multivariate Gaussian density and we estimate both the location vector and covariance matrix, i.e.,  $f$  is weakly identifiable, an adaptation of the moment estimators from the seminal work [18] to the multivariate deviated models shows that  $\lambda^{*}||\|\tilde{\mu}_{n}^{\mathrm{moment}} - \mu^{*}\|^{6} = \mathcal{O}(n^{-1/2})$ ,  $\lambda^{*}||\|\tilde{\Sigma}_{n}^{\mathrm{moment}} - \Sigma^{*}\|^{3} = \mathcal{O}(n^{-1/2})$  and  $(\|\Delta \mu^{*}\|^{6} + \|\Delta \Sigma^{*}\|^{3})|\tilde{\lambda}_{n}^{\mathrm{moment}} - \lambda^{*}| = \mathcal{O}(n^{-1/2})$ , where  $(\tilde{\lambda}_{n}^{\mathrm{moment}}, \tilde{\mu}_{n}^{\mathrm{moment}}, \tilde{\Sigma}_{n}^{\mathrm{moment}})$  are moment estimators of  $(\lambda^{*}, \mu^{*}, \Sigma^{*})$ . These results are also much slower than those of the MLE in weakly identifiable settings.
Other related work. The hypothesis testing and MLE problem related to the multivariate deviated model had been considered in previous work, including the problem of detecting sparse homogeneous and heteroscedastic mixtures [14, 15, 4, 3, 5, 34], the problem of determining the number of components [8, 26, 10, 24, 25], and the problem of multiple testing [30, 12]. In particular, [4] considers testing problem for the deviated model with  $h_0 = N(0,1)$  and  $f = N(\mu^*,1)$  being one-dimensional Gaussian distributions. They show that no test can reliably detect  $\lambda^{*} = 0$  against  $\lambda^{*} > 0$  if  $\lambda^{*}\mu^{*} = o(n^{-1 / 2})$ , while the Likelihood Ratio test can consistently do it when  $\lambda^{*}\mu^{*} \gtrsim n^{-1 / 2 + \epsilon}$  for any  $\epsilon >0$ . However, no guarantee for estimation of  $\lambda^{*}$  and  $\mu^{*}$  is provided. In the same setting where  $f$  is the density of a location Gaussian distribution, the convergence rate of parameter estimation in the deviated model had been studied in the work of [16]. Since the location Gaussian distribution is a special case of the strongly identifiable distribution, our result in the strongly identifiable and nondistinguishable settings is a generalization of the results in [16], but with a different proof technique as their proof technique relies strictly on the properties of the location Gaussian distribution.
Organization. The paper is organized as follows. In Section 2, we provide background on the identifiability and density estimation rate of the multivariate deviated model. Then, we establish the lower bounds of the Total Variation distance between two densities in terms of loss functions among parameters under both the distinguishable and non-distinguishable settings in Section 3. Next, we characterize the convergence rates of parameter estimation as well as derive the corresponding minimax lower bounds in Section 4. In Section 5, we carry out a simulation study to empirically verify our theoretical results before concluding the paper in Section 6. Rigorous proofs and additional results are deferred to the supplementary material.
Notations. For any  $a, b \in \mathbb{R}$ , we denote  $a \vee b \coloneqq \max \{a, b\}$  and  $a \wedge b \coloneqq \min \{a, b\}$ . Next, we say that  $h_0$  is identical to  $f$  if  $h_0(x) = f(x|\mu_0, \Sigma_0)$  for some  $(\mu_0, \Sigma_0) \in \Theta \times \Sigma$ . For each parameter  $G \in \Xi$ , let  $\mathbb{E}_{p_G}$  be the expectation taken with respect to product measure with density  $p_G$ . Lastly, for any two density functions  $p$  and  $q$  (with respect to the Lebesgue measure  $m$ ), the Total Variation distance between them is given by  $V(p, q) \coloneqq \frac{1}{2} \int |p(x) - q(x)| dm(x)$ , while we define their squared Hellinger distance as  $h^2(p, q) \coloneqq \frac{1}{2} \int [\sqrt{p(x)} - \sqrt{q(x)}]^2 dm(x)$ .
2.1 Identifiability Condition
Our principal goal in this paper is to assess the statistical efficiency of parameter estimation from the MLE method. To do that, we should be able to guarantee the parameter identifiability of the deviated model (2), i.e., if  $p_{G}(x) = p_{G_{*}}(x)$  for almost surely  $x \in \mathcal{X}$  where  $G = (\lambda, \mu, \Sigma)$ , then  $G \equiv G_{*}$ . That identifiability condition leads to the following notion of distinguishability between the density function  $h_0(\cdot)$  and the family of density functions  $\{f(\cdot | \mu, \Sigma) : (\mu, \Sigma) \in \Theta \times \Omega\}$ .
Definition 2.1 (Distinguishability). We say that the family of density functions  $\{f(\cdot |\mu ,\Sigma),(\mu ,\Sigma)\in$ $\Theta \times \Omega \}$  (or in short,  $f$  ) is distinguishable from  $h_0$  if the following holds:
A1. For any two distinct components  $(\mu_1, \Sigma_1)$  and  $(\mu_2, \Sigma_2)$ , if we have real coefficients  $\eta_i$  for  $1 \leq i \leq 3$  such that  $\eta_1 \eta_2 \leq 0$  and  $\eta_1 f(x | \mu_1, \Sigma_1) + \eta_2 f(x | \mu_2, \Sigma_2) + \eta_3 h_0(x) = 0$ , for almost surely  $x \in \mathbb{R}^d$ , then  $\eta_1 = \eta_2 = \eta_3 = 0$ .
We can verify that as long as  $f$  is distinguishable from  $h_0$ , the parameter identifiability of our multivariate deviated model follows. In particular, assume that there exists  $G = (\lambda, \mu, \Sigma)$  such that
$$
\left(1 - \lambda^ {*}\right) h _ {0} (x) + \lambda^ {*} f \left(x \mid \mu^ {*}, \Sigma^ {*}\right) = \left(1 - \lambda\right) h _ {0} (x) + \lambda f (x \mid \mu , \Sigma), \tag {4}
$$
for almost surely  $x \in \mathcal{X}$ . The above equation is equivalent to  $(\lambda - \lambda^{*})h_{0}(x) + \lambda^{*}f(x|\mu^{*},\Sigma^{*}) - \lambda f(x|\mu ,\Sigma) = 0$ . Assume that  $f$  is distinguishable from  $h_0$ , then equation (4) indicates that if  $(\mu ,\Sigma)\neq (\mu^{*},\Sigma^{*})$ , we have  $\lambda = \lambda^{*} = 0$ . Since  $\lambda^{*}\neq 0$  from our assumption, we obtain that  $(\mu ,\Sigma) = (\mu^{*},\Sigma^{*})$ . As a result, equation (4) becomes  $(\lambda -\lambda^{*})h_{0}(x) + (\lambda^{*} - \lambda)f(x|\mu ,\Sigma) = 0$ . By applying the distinguishability condition again, we get  $\lambda = \lambda^{*}$ . Therefore, the multivariate deviated model (2) is identifiable.
In the following example, we will verify the distinguishability condition in Definition 2.1 given some specific choices of function  $h_0$  and density  $f$ .
Example 2.2. (a) Assume that  $f$  belongs to a location family of density functions, i.e.,  $f(x|\mu, \Sigma) = f_{\Sigma}(x - \mu)$  for all  $x$  where  $\Sigma$  is a fixed covariance matrix. If  $h_0(x) \neq f(x)$  for almost surely  $x \in \mathcal{X}$ , then  $f$  is distinguishable from  $h_0$ .
(b) When  $h_0$  is a finite mixture of multivariate Gaussian densities and  $f$  belongs to a class of multivariate Student's density functions with any fixed odd degree of freedom  $\nu > 1$ , we get that  $f$  is distinguishable from  $h_0$ .
(c) When  $f$  is identical to  $h_0$ , then  $f$  is not distinguishable from  $h_0$ .
2.2 Convergence Rate of Density Estimation
Our strategy to obtain the convergence rate of the MLE  $\widehat{G}_n$  is by first establishing the convergence rate of density  $p_{\widehat{G}_n}$  and then studying the geometric inequalities between the parameter space and density space. For the former, the standard method is to use the empirical process theory [17, 33], while for the latter step, we investigate those inequalities under various settings of distinguishability in Section 3. Due to space constraints and the popularity of empirical process theory, we choose to informally present a main result for yielding the parametric convergence rate for density estimation in this section. For full explanation and definition, readers are referred to Appendix B. The convergence rate for density estimation can be characterized by bounding the complexity of the parameter space  $\Xi$  via a function called bracketing entropy integral  $\mathcal{J}_B(\epsilon, \overline{\mathcal{P}}^{1/2}(\Xi, \epsilon))$  (cf. equation (8)).
Theorem 2.3. Assume the following assumption holds:
A2. Given a universal constant  $J > 0$ , there exists  $N > 0$ , possibly depending on  $\Xi$ , such that for all  $n \geq N$  and all  $\epsilon > (\log(n)/n)^{1/2}$ , we have  $\mathcal{J}_B(\epsilon, \overline{\mathcal{P}}^{1/2}(\Xi, \epsilon)) \leq J\sqrt{n}\epsilon^2$ .
Then, there exists a constant  $C > 0$  depending only on  $\Xi$  such that for all  $n\geq 1$
$$
\sup _ {G _ {*} \in \Xi} \mathbb {E} _ {p _ {G _ {*}}} h (p _ {\widehat {G} _ {n}}, p _ {G _ {*}}) \leq C \sqrt {\log n / n}.
$$
Therefore, in order to get the convergence rate for density estimators based on the MLE method, we only need to check Assumption A2, which holds true for several parametric models [33]. For our model, we give an example that it holds for a general class of  $f$  and  $h_0$ .
Proposition 2.4. Suppose that both  $\Theta$  and  $\Omega$  are compact, and  $\{f(x|\mu ,\Sigma):\mu \in \Theta ,\Sigma \in \Omega \}$  is a vector-matrix family of densities being uniformly bounded, Lipschitz, and light tail, i.e. there exists constants  $M,L,B,b_{1},b_{2},b_{3} > 0$  such that  $|f(x|\mu ,\Sigma)|\leq M,|f(x|\mu ,\Sigma) - f(x|\mu^{\prime},\Sigma^{\prime})|\leq$ $L(\| \mu -\mu^{\prime}\| +\| \Sigma -\Sigma^{\prime}\|)$  for all  $x\in \mathbb{R}^d$ , and
$$
| f (x | \mu , \Sigma) | \leq b _ {1} \exp (- b _ {2} \| x \| ^ {b _ {3}}) \quad \forall \| x \| > B,
$$
for all  $(\mu, \Sigma) \in \Theta \times \Omega$ . Additionally, if the density  $h_0$  is bounded, then the corresponding multivariate deviated model defined in equation (1) satisfies assumption A2.
Example 2.5. We can check that the location-scale Gaussian density  $f(x|\mu, \Sigma)$  with  $\Sigma \in \Omega$  having eigenvalues bounded below by a positive constant satisfies the condition of Proposition 2.4. This condition for  $h_0$  is mild and is satisfied by most distributions such as Gaussian and t-distribution.
3 From the Convergence Rate of Densities to Rate of Parameters
The objective of this section is to develop a general theory according to which a small distance between  $p_{G}$  and  $p_{G_{*}}$  under the Hellinger distance (or Total Variation distance) would imply that  $G$  and  $G_{*}$  are also close under appropriate distance where  $G = (\lambda, \mu, \Sigma)$  and  $G_{*} = (\lambda^{*}, \mu^{*}, \Sigma^{*})$ . By combining those results with Theorem 2.3, we can obtain the convergence rate for parameter estimation (cf. Section 4). The distinguishability condition between  $h_{0}$  and  $f$  implicitly requires that  $p_{G} = p_{G_{*}}$  would entail  $G = G_{*}$ ; however, to obtain quantitative bounds for their Total Variation distance, we need stronger notions of both distinguishability and classical parameter identifiability, ones which involve higher order derivatives of the densities  $h_{0}$  and  $f$ , taken with respect to mixture model parameters. Throughout the rest of this section, we denote  $G = (\lambda, \mu, \Sigma)$  and  $G_{*} = (\lambda^{*}, \mu^{*}, \Sigma^{*})$ .
3.1 Distinguishable Settings
Definition 3.1 (First-order Distinguishability). We say that  $f$  is distinguishable from  $h_0$  up to the first order if  $f$  is differentiable in  $(\mu, \Sigma)$ , and the following holds:
D1. For any component  $(\mu', \Sigma') \in \Theta \times \Omega$ , if we have real coefficients  $\eta, \tau_{\alpha}$  for all  $\alpha = (\alpha_1, \alpha_2) \in \mathbb{N}^{d_1} \times \mathbb{N}^{d_2 \times d_2}$ ,  $|\alpha| = |\alpha_1| + |\alpha_2| \leq 1$  such that
$$
\eta h _ {0} (x) + \sum_ {| \alpha | \leq 1} \tau_ {\alpha} \frac {\partial^ {| \alpha |} f}{\partial \mu^ {\alpha_ {1}} \partial \Sigma^ {\alpha_ {2}}} (x | \mu^ {\prime}, \Sigma^ {\prime}) = 0
$$
for all  $x\in \mathcal{X}$  , then  $\eta = \tau_{\alpha} = 0$  for all  $|\alpha |\leq 1$
We can verify that the examples from part (a) and part (b) of Example 2.2 satisfy the first-order distinguishability condition. Next, we introduce a notion of uniform Lipschitz condition in the following definition.
Definition 3.2 (Uniform Lipschitz). We say that  $f$  admits uniform Lipschitz condition up to the first order if the following holds: there are positive constants  $\delta_1, \delta_2$  such that for any  $R_1, R_2, R_3 > 0$ ,  $\gamma_1 \in \mathbb{R}^{d_1}$ ,  $\gamma_2 \in \mathbb{R}^{d_2 \times d_2}$ ,  $R_1 \leq \lambda_{\min}^{1/2}(\Sigma_1) \leq \lambda_{\max}^{1/2}(\Sigma_2) \leq R_2$ ,  $\| \mu_1 \|, \| \mu_2 \| \leq R_3$ ,  $\mu_1, \mu_2 \in \Theta, \Sigma_1, \Sigma_2 \in \Omega$ , we can find positive constants  $C(R_1, R_2)$  and  $C(R_3)$  such that for all  $x \in \mathcal{X}$ ,
$$
\begin{array}{l} \left| \gamma_ {1} ^ {\top} \left(\frac {\partial f}{\partial \mu} (x | \mu_ {1}, \Sigma) - \frac {\partial f}{\partial \mu} (x | \mu_ {2}, \Sigma)\right) \right| \leq C \left(R _ {1}, R _ {2}\right) \| \mu_ {1} - \mu_ {2} \| ^ {\delta_ {1}} \| \gamma_ {1} \|, \\ \left| \operatorname {t r} \left(\left(\frac {\partial f}{\partial \Sigma} (x | \mu , \Sigma_ {1}) - \frac {\partial f}{\partial \Sigma} (x | \mu , \Sigma_ {2})\right) ^ {\top} \gamma_ {2}\right) \right| \leq C (R _ {3}) \| \Sigma_ {1} - \Sigma_ {2} \| ^ {\delta_ {2}} \| \gamma_ {2} \|. \\ \end{array}
$$
Now, we have the following results characterizing the behavior of  $V(p_{G}, p_{G_{*}})$  regarding the variation of  $G$  and  $G_{*}$ .