Beyond black box densities: Parameter learning for the deviated components
Authors: Dat Do
Nhat Ho
XuanLong Nguyen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Mixture Model
Wasserstein Metric
Statistical Learning Theory

Abstract: As we collect additional samples from a data population for which a known density function estimate may have been previously obtained by a black box method, the increased complexity of the data set may result in the true density being deviated from the known estimate by a mixture distribution. To model this phenomenon, we consider the \emph{deviating mixture model} $(1-\lambda^{*})h_0 + \lambda^{*} (\sum_{i = 1}^{k} p_{i}^{*} f(x|\theta_{i}^{*}))$, where $h_0$ is a known density function, while the deviated proportion $\lambda^{*}$ and latent mixing measure $G_{*} = \sum_{i = 1}^{k} p_{i}^{*} \delta_{\theta_i^{*}}$ associated with the mixture distribution are unknown. Via a novel notion of distinguishability between the known density $h_{0}$ and the deviated mixture distribution, we establish rates of convergence for the maximum likelihood estimates of $\lambda^{*}$ and $G^{*}$ under Wasserstein metric. Simulation studies are carried out to illustrate the theory.

Beyond Black Box Densities: Parameter Learning for the Deviated Components
Dat Do
Nhat Ho* Department of Statistics and Data Sciences University of Texas Austin
Department of Statistics University of Michigan at Ann Arbor Ann Arbor, MI 48 18109 dodat@umich.. ch.
Austin, TX 78712
minhnhat@utexas.cem
XuanLong Nguyen
Department of Statistics University of Michigan at Ann Arbor Ann Arbor, MI 48109 xuanlong@umi eduedu
Abstract
As we collect additional samples from a data population for which a known density function estimate may have been previously obtained by black box method, the increased complexity of the data set may result in the true density being deviated from the known estimate by mixture distribution. To learn about this phenomenon, we consider the deviating mixture model > P. (x](1))) where ho is known density function, while the deviated proportion X and latent mixing unknown. measure G. = = P. ōg: associated with the mixture distribution Using a novel notion of distinguishability between the known density ho and the deviated mixture distribution, likelihood estimates of X" and G* under Wasserstein metrics. Simulation studies are carried illustrate the theory.
establish rates of convergence for the maximum
Introduction
Most data-driven learning processes typically consist of an iteration of steps that involve model training and fine-tuning, with more data in-take leading to further model re-training and refinement. As more samples become available and exhibit more complex patterns, the initial model may be obsolete, risks being discarded, or absorbed into richer class models that adapt better to the increased complexity. It takes considerable resources to train complex models on rich data population. Moreover, many successful models modern real-world applications have become so complex that make them hard to properly evaluate and interpret; aside from the predictive performance they may well be considered as black boxes. Nonetheless, as data populations evolve and so must the learning models, several desiderata remain worthy: the ability to adapt aspects of old "wise" model, and the ability to interpret the changes.
complexity while retaining
In this paper we will investigate a class of complex models for density estimation that are receptive to adaptation, reuse and interpretablity: we posit that there is an existing distribution ho which may have been obtained priori by some means for the data population of interest, e.g., via kernel density estimation (KDE) (22 or mixture models [20 or some modern black box methods, such generative adversarial networks (GANs) [3] I normalizing flows 19. Nonetheless, more
:: Dat Do and Nhat Ho contributed equally to this work.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

samples become available and/or as the data population changes, it is possible that the true density may deviate from hgg While hg is potentially difficult explicate, is the deviation from the known ho that we wish to learn and interpret. We will use a mixture distribution to represent this deviation, leading to what we call a deviating mixture model for the underlying data population:
Pxca. (x) == (1 - (=))o(x) + X Fxx
(1)
for x E Rd, where F(x, C(= [xx(0))) (()))) represents a mixture distribution for the density
components deviating from ha- Such deviating components are from a known family of density function f. The unknown parameters for this model are the mixing proportion X E [0, 1], and the
mixing measure G. = Ex= p:503 where k, 2 1 number of deviated components. The choice of mixture distribution F(a,, G.) allows us to express complex deviation from hog yet the overall model remains amenable to the interpretation of its parameters: X, represents the amount of deviation from the existing candidate ho. while the mixing measure G. represents heterogeneous patterns of the deviation. Because ho might be complex and trained with great computational resource estimate the density of prior data population, it is reasonable to assume ho be known in the model (T.. The primary contribution of this paper is a rigorous investigation into the rather challenging questions of identifiability and parameter learning rates that arise from a standard maximum likelihood estimation procedure.
Relations to existing works. This modeling framework owes its roots to several significant bodies of work in both statistics and machine learning literature. In classical statistics, dominant approach to address the increased complexity of data populations is via hypothesis testing: one can test an alternative (possibly composite) hypothesis represented by a class of distributions against the null hypothesis represented by ho. Due to the constraint for obtaining simple and theoretically valid test statistics in order to accept or reject the null hypothesis, the testing upproaches were mostly restricted to simple choices of distribution for the null and alternative hypotheses (6 IO 7 4 g. More similar to [T is the class of contaminated mixture models for density estimation: this framework, the data are assumed to be sampled from mixture of Po and Q where either Po or Q can be an unknown distribution that needs to be estimated. While this approach offers more flexibility in terms of modeling, it does not always guarantee the identifiability of the mixing weight or mixture components Po. Q (25 23 19) Without identifiability, it is virtually impossible to interpret the model parameters for the data domains. To avoid the identifiability issue, several researchers added the or parametric structures on Po and Q. such as Po and Q are mixture distributions 12 [2]. However, to the best of our knowledge, the convergence theory of these models remains poorly understood, except for some simple settings (see also (..)). The main distinction between our modeling framework of deviating mixture models and the existing research on contaminated mixtures lies in our assumption that one of the mixture components, namely hgg is known, allowing us to focus on the inference of the deviation from hg for which considerable learning theory for the parameters of interest can be established and will be presented in this paper. Finally, estimating parameters of mixture distributions is an essential problem in mixture models. The convergence properties have been studied using identifiability notions and Wasserstein distances [81]8]] [5]. Our technical approach requires generalization of the identifiability notion take account of structural property of the existing component he. which helps to shed light on considerably more complex convergence behavior of the deviated components.
Contributions. The primary contribution of this paper is a rich theory of identifiability and rates of convergence for parameters and density estimation that arise in the deviating mixture model ([) under various settings of the existing component fig, and that of the deviating components (via s and G.). Because the convergence of density estimation in Hellinger distance under the MLE procedure is well studied in 266), the bulk of our technical innovation lies in establishing a collection of inverse bounds which relate the Hellinger distance of densities in model ( in terms of that of their parameters. To do that, we introduce a novel notion of distinguishability between hg and family of density f. The inverse bounds will be characterized under such distinguishability conditions (or the lack thereof). Our proof technique allows us to characterize different convergence rates of parameters in the deviating mixture model under distinguishable settings. It also gives rise to several new types of inverse bounds in partially distinguishable settings, where we may not have identifiability in our model. To the best of our knowledge, this is the first work in which such bounds are obtained in mixture modeling literature. Moreover, we will provide many examples to demonstrate the broad applicability of our theory, including cases where the existing component ho is obtained by a black box method (e.g., deep learning model) and a more traditional method (c.g., via KDE's or mixture

---

models). By doing so, we are able to push the boundary of identifiability and learning theory of mixture models toward a larger class of modern machine learning models.
Organization. The remainder part of this paper is structured as follows. In Section 2. we review the MLE method and the identifiability conditions, where the notion of distinguishability is presented. In Section 3, the main results of inverse bounds and convergence rates for parameters estimation of model ! are shown. In Section 4, multiple simulation experiments are carried out to support the theory. Finally, Section 5 is used to discuss and conclude. Proofs of all the results in the main text are deferred to the Supplementary Material.
Notation. We denote by (+)
0,0, E 1 sisk) E'
P.J(z)),)) : = I. 2 0, 0, E @ s }} the family of mixtures with no more than K components. Ek (+)
the family of mixtures with exactly k components and (x(0)= -
E pxf())))) E.- 1 P: = L.P; 2 co, 02 E LI < i < k, - K) is the family of mixtures with exactly K components and mixing proportions being bounded below by co. and OK,eo (0)
EE p=)(2)),)) Σ P; p, 2 co, 0g E vI S i S K.W < Kj. |1-112 is the usual (2 norm for vectors in I& and matrices in Rdd We write g(x) Z M() gf) > cl(x)) for all x, where c is a constant does not depend on æ (similar for g(x) S h(x)). For any X E R and B C R. denote by 1 1p the function that takes value 1 if À E B, and 0 otherwise. For any two densities p and q, denote h(p 9) by the Hellinger distance and ((p, q) by the Total Variation distance between them.
2 Identifiability and distinguishability theory
The principal goal of the paper is to establish the efficiency of parameter learning for the deviating mixture model (I) via the standard maximum likelihood estimation (MLE) method. To achieve this goal, the parameters have to be identifiable to begin with. Thus, our theory builds and extends standard notion of identifiability of families of density {fx]0) : 0 E Q} that has been considered in previous work 21 15].
Definition 2.1. The family {f(=))) E θ) (or in short, f) is identifiable in the order r, for some 2 1, if far (0) is differentiable up to the order in 0 and the following holds:
(0 such that for A1. For any k 2 1, given k different elements 0 . O2 +, if we have an almost all æ
alvi aov
(zl0.) =
n
then Ciyy (1) 0 for all S i S l and |n) Sr.
Many commonly used families for mixture modeling satisfy the first order identifiability condition, including location-scale Gaussian distributions, e.g., [(x(0)) = N(rlu,a²) where and 6² represent the mean (location) and variance (scale) parameters, and location-scale Student's t-distributions. In model ( however, due to the presence of the existing component hg, the deviated mixture components need to be distinguishable from hg. This motivates a more general notion of identifiability. namely, distingnishability that we now define. This condition specifies property jointly for both the existing component ho and the family of density functions f that make up the deviated components. Definition 2.2. For any natural numbers k, 2 1, we say that the family of density functions { 0 }} with complexity level k (or in short, (f,k) is distinguishable up to the order from ho if the following holds:
A2. For any & distinct components OI...., such that
...,, Ok. if we have real coefficients on
( for 0 ssisi S k
(4) I (zl0.) = 0,
a (0) ho(z)
Σ
an 00
1=0
for almost surely E X,, then a (0) = an = 0 for 1 sis l and | S..
We observe that the identifiable condition is direct consequence of the corresponding distinguishable condition. A simple but non-trivial example of the distinguishability condition can derived directly from the definitions.
3

---

Example 2.3. (a) When ho(x) = Σi= (x)(0) for some given weights (pl...., pk.)) and param- eters (f9..... 0 where kn 2 1, then (f.k) is distinguishable in the order from ho long as < Kg and the family of density f is identifiable in the order r.
(b) Given the choice of hg in (a), (f, is not distinguishable in the order from ho when k 2 ko-
More significantly, we can establish broad class of ho and families f for which distinguishability holds. This is exemplifted by the following theorem, where f represents a family of location or location-scale Gaussian kernels, and ho is subject to relatively weak condition.
Theorem 2.4. (a) Suppose that log (z) 2 || " or or log (o( S ||=1|/² for all |||||| > 20. for some To > 0, 31 > 2, and $2 < 2. Then, for f being family of location-scall Gaussian and any k > O. (f. k) is distinguishable from he up to the first order, where the derivatives in Assumption A2 are taken with respect to both location and scale parameters, and (f. k) is also distinguishable from ho up to any order, where the derivatives in Assumption A2 are taken only with location parameters.
(b) Suppose that ho is the pdf of a pushforward measure of N00 Ia) by a piecewise linear function with finite and positive number of breakpoints. Then, the same conclusions as in part (a) hold.
The proof of Theorem? 2.4js is in Appendix C.I where the main proof technique is carefully examining the tail densities of ho and f at infinity. Note that in part (a), hg can be a pdf of any distribution possessing a lighter or heavier tail than Gaussian distributions, and in part (b). hg represents the pushforward of a Gaussian distribution by any piecewise linear function (recall that family of piecewise linear functions is dense in the Banach space of continuous functions with compact support). In the sequel we shall demonstrate several examples of interest that are applicable to Theorem 2.44where hgg may have been estimated by some popular "black box" methods.
Kernel based representation. Suppose that ho was obtained from m- sample YI....,,, Ym Rd by a classical kernel density estimation (KDE) method [22] or a RKHS-based method (24), so that
ho(z)
Ko (I, Ky) Vo E R
(2)
where ko is a kernel function with bandwidth a. Popular choices of kernels include the Gaus- sian kernel ho (2,,x')
d
|
2
exp
and the multivariate Student's kernel
V2TO
20
2td
ko (xt, x²)
(((x+d)/2) T(p/2)
+
12-x' 1
The corresponding distinguishability guar-
va
antee is follows.
Corollary 2.5. Suppose ho is defined by Eq. 2 where ho is Gaussian kernel and m > K, or ko the multivariate Student's 's kernel. Then, for being family of location-scale Gaussian, (J, K) is distinguishable from ha up to the first order, where the derivatives in Assumption A2 are taken with respect to both location and scale parameters, and (f, K) is also distinguishable from by up to any order, where the derivatives in Assumption A2 are taken only with location parameters.
In application, it is common that the condition m > K is satisfied. also matches with the scenario that we consider in the paper, where ho is already trained using a big data set, and there is small number of deviated components.
Neural networks. Deep neural networks represent a powerful, albeit black box, approximation device for constructing rich classes of distribution for generative models [1. Accordingly, hg is the pdf function of a Gaussian distribution being fush-forwarded by a map T, which is represented by a neural network (NN). Suppose that the NN representing has a positive and finite number of layers I., and so
T(x) = <WW a(W + b1)) + bL-1) ( bb),
(3)
where Wi...., Wr E Rddd are the weights and bi...., by. E Ie are the biases. The activation function a is chosen to be rectified linear unit (ReLU) function defined by a(x) = max (2r, 0), and applied elementwise to any vector in Rdd The corresponding guarantee on the distinguishability condition is as follows.
Corollary 2.6. Suppose that ho is the pdf a pushforward measure of ( Ia) by a map T defined by Eq. 3) Then, for being family of location-scal Gaussian and any k > 0, (f. k) is distinguishable

---

from ho up to the first order, where the derivatives in Assumption A2 are taken with respect to both location and scale parameters, and (f.b) is also distinguishable from hg up to any order, where the derivafives in Assumption A2 are taken only with location parameters.
3 Convergence rates of density estimation
In this section, we first establish the rate of density estimation for the deviating mixture models in Section 3.1 We then describe general procedure obtain the convergence rate of parameter estimation based on that density estimation via inverse bounds Section Finally, we provide comprehensive inverse bounds under several settings of the deviating mixture models in Section 3.3
3.1 MLE for deviating mixture model
Given n i..d. sample X2...... Xn from G.*G. as model (T, where G, has k, components, want to estimate X and G. from the data. We refer to the problem in exact-fitted setring if k. known, and we refer to it as in over-fitted setting k, is unknown but is known to be bounded by some number K. We denote the MLE for exact-fitted setting by
Any Go E
arg max Ae(0.1),EEE. (0)
log(Px((X:)),,
and for the over-fitted setting, we replace & (0) in the equation above by (8), where K 2 k.. In order to state rate of convergence for the density estimators PG, under the Hellinger distance h 126, we need condition on the complexity of the function class
//2 / ( () 1// G Ox(O), h(xc MPx..PX-G. sc)
(4)
where for any Ge Ox((O) (O), write paG (pxa+ px 1/2. The definition of (x originates from 26 We measure the complexity of this class through the bracketing entropy integral
JBle,P 1/2 (0,c),)) (0.,,)
Sea/218 Vlog P/ (0, v)du V
f
(5)
where NB x) denotes the c-bracketing number of metric space (X,1) and is the Lebesgue measure. We require the following assumption.
A3. Given a universal constant > 0, there exists N > 0, possibly depending on O and such that for all N and all > (log n 1/2
JB(C, P/ 1/2 (0, (0,c),v) S J
Theorem 3.1. Assume that Assumption A3 holds, and let l 1. There exists constant C depending only on θ, k such that for all n 2 1,
Ex-ac MPx, G. S Clognn/n.
sup (.)
Therefore, in order to get convergence rate for density functions based on MLE procedure, we only need check assumption A3. This assumption holds true for wide range class of parametric model 226. For our model, we give an example that it holds when ho has an exponential tail (satisfied for KDE's and Neural networks above) and is location-scale Gaussian distribution.
Proposition 3.2. Suppose f is location-scale Gaussian family and θ Iaa,, a]d x Ω, where Ω is subset of S++ whose eigenvalues are bounded in [2, 7,, a, X. X > 0, and ho is bounded with tail -log (x) 2 ||=]]2 for some 3 > Then, the family of densities (pxc A E [0, 1), (x())) satisfies assumption A3.
3.2 Parameter learning rates of deviated components
The core of this paper lies in establishing collection of inverse bounds, provided that some distin- guishability condition developed Section holds. The inverse bounds basically say that a small distance between PAG and G.*G. under the total variation distance entails that (X, G) and (X", G.)
5

---

are similar under appropriate distances, where G.) is fixed. To this end, we employ Wasserstein metries (27] and their extensions.
Wasserstein distances. Wasserstein distances are natural and useful for assessing the convergence
of latent mixing measures in mixture models [21 16 14]. Given two measures G = = psdo, and
G' = Eド on a space O endowed with metric the Wasserstein metric of order r 2 1 is:
W' = [inf] Σ 4:pp (0,, 8,)
where the infimum is taken over all joint distribution on [1....,k] k] x [l,..., k'] such that Σ q6 = PI,E, E, 9tj Ps. Note that if Gn is sequence of discrete measures that converges to G in Wasserstein distance, then for every atom of G, there is subset of atoms of G., converges to it. Therefore, the convergence Wasserstein metries implies convergence of parameters in mixture models. In this paper, space θ is often chosen to be compact subset of R. and is the usual 1 distance. In the case of location-scale Gaussian mixtures, space @ is a compact subset of R x St+ where S is the set of positive definite and symmetric matrices in Raxd and for every (R, E), (nL, E') O, the distance is defined by p (Ju, E) (Y) = || / ll2 + - From inverse bounds to parameter learning rates. Suppose that some distinguishablity condition satisfied, then we will establish an inverse bound providing a guarantee that small distance between px-c, and PAG entails small distance between À and X and between G and G.. More concretely, define a divergence between two measures XG and A= via
W, V7 == JA + (A (G,))
for all r 2 1, and the inverse bounds will have the form that V(PAG. (..c-PX....) Z W, GG..) for some that depends the level of distinguishable level of the model. Since total variational distance upper bounded by Hellinger distance, if Assumption A3. holds, then combining the aforementioned inverse bound with Theorem 3.1 we immediately obtain
Ex,.G W, (A, A.G.) S CV logn
This further implies that the convergence rate of Ån to X is of order (log(n)/n)1/² and the conver- gence rate of W, (G., G-) to 0 is of order (1)
3.3 Inverse bounds in distinguishable setting
We shall establish inverse bounds provided distinguishability condition for model ! holds under either exact-fitted and over-fitted settings regarding the true number of components k.,
Theorem 3.3. Assume that k. is known and (f. k.) is distinguishable the first order from hg- Then, for any G E &x. (Q), there exist positive constant C1 and C2 depending only X", G.,, Go, O such that the following holds:
(a) When X* 0, then V(pxa.,PXG) 2 CiA.
(b) When X E (0, 1}, then V(px-G. PPG) 2 C.W,(AG,
We now present proof sketch for Theorem
3.3 It is a combination of the Taylor expansion around the true parameters and the Fatou's lemma; the proof technique for the remaining results also shares similar spirit as that of Theorem 3.3 Detailed proof of Theorem 3.3 is deferred to the Appendix.
Proof sketch for part (b): Suppose that the bound not correct, so there exists sequence An (0. 1] and G, Ex, (9) such that V(px-G PA,, 3/W ////(WXCG G,, - 0. Because of the compactness of the parameter space, by extracting a subsequence if necessary, we can assume
An + X. G, W1, G¹. If (X, G') * (X*, G.), we have W (A* + W (A*G., XG') * 0. It indicates that V(px-G., PA,G.J + O. which leads to PXG.. = PxG. It contradicts to the distinguishable condition when (X,G") * (A),
Otherwise, we have An + x², Gn + G., and can present G, = -k- P. Öan and G. E p Sa; such that D. + p" 00 0,, Because of these limits and by Taylor expansion, we can ar- range the difference (px,G, (x) Pxc. (x))/W (A, X²G,) in terms of linear combination of
*

---

ho(a)) (x) 0 ( such that at least one coefficient is different from O. By Fatou's lemma, 00
0 lim inf V GPA, dx 2 f lim
in inf PA, - (x)..(t) dr, which equals to the Wi,,,Gn,X'G.
(A,
absolute integral of the linear combination above. Hence, there exists a non-trivial linear combination of f 2s [x]((]] that equals O, which contradict to the distinguishability condition. Therefore, complete the proof.
a
In application, the true number of components k, might not be known and we often fit the model [ with G E Ox (0) for some large K 2 kg, The next result shows that similar bounds can also be established in this case, where we require distinguishability of f and ho in a higher order.
Theorem 3.4. Assume that k. is unknown and strictly upper bounded by a given K. Assume additionally that (J, ) is distinguishable in second order from hy. Then, for any G (x(O), there exist positive constant C1 and C2 depending only X", G.,, ho, O such that the following holds:
(a) When X 0, then V(Px-.. PPG)) 2 C2A.
(b) When X 1), then V(px-G.,Pxc) 2 C2W2(AG,GG.)
Thanks to the distinguishability up to second order, no matter how large the number of over-fitted components K is, we always get the W2 lower bound for the total variation distances. Proof of this theorem shares the same spirit with what of Theorem 3.3 The difference here is when we overfit G, with some G, there are some atoms of & that converges to the same atom of G., which requires us do Taylor expansion up to second order and explain the higher order of Wasserstein distance here. Next, we relax the assumption of Theorem 3.4by working on the setting where f is not second order identifiable. This is an instance of the so-called weakly identifiable setting - One popular example of weakly identifiable f is location-scale Gaussian distribution, which admits the partial
differential equation (PDE) structure J² (1) Σ) = -y
0f (z|u, Σ), for all x E H where fz]|,, Σ)
0u²
stands for location-scale Gaussian density function with location fs and covariance Σ. In order to illustrate the result of our bound for that weak identifiability setting of f, we specifically consider f to be location-scale Gaussian distribution. In this case, the parameter space θ is a compact subset of Rd x Stt+, where S7 + is the set of positive definite and symmetric matrices in Rdxd equipped with the usual Frobenius norm. To put our result in context, we shall adopt a notion used in analyzing the convergence rate of parameter estimation in location-scale Gaussian mixtures in [[6]. For any k 2 1, let F(k) be the minimum value of r such that the following system of polynomial equations:
Gafl Dez my !ng!
Σ
Σ
0 for each a =1.-. I.....r;
(6)
does not have any nontrivial solution for the unknown variables (a,, by, e where the ranges of ny and n2 in the second sum consist of all natural pairs satisfying the equation n1 + 2ng = a. A solution to the above system is considered nontrivial if all of variables ej are non-zeroes, while at least one of the aj is non-zero. Some examples of known values of 7 are F(1) = 4 and F(2) = 6.
and F(k) > 7 for all k 2 3. Using this notion, we can characterize the convergence of parameters of model (I) for the location-scale Gaussian family via the following theorem for inverse bounds.
Theorem 3.5. Assume that G* Eo= (O), and k. is unknown and strictly upper bounded by given K. In addition, f is location-scale Gaussian distribution and (J, K) with varied location, fixed variance parameters distinguishable in any order from hgg Then, for any G E O OK (0), there exist positive constant C₁ and C2 depending only on X", G., ho, O such that the following holds:
(a) When X 0, then V(px 2 C...
(b) When X E (0, 1]. then V(P 2 CWF
The proof technique of this result involves doing Taylor expansion of both location and scale 0f
0²f
parameter up to order 7., then utilize the heat equation OΣ (zlp, Σ) 2 0μ² (z/H, Σ) to compress this expression into linear combination of ho and derivatives of f(>lu, Σ)) with respect to μ only. This allows us to use the condition in this theorem to imply contradiction, and gives rise to Eq. (6)
7

---

3.4 Inverse Bounds in Partially Distinguishable Setting
What happens if the distinguishability condition required by Def. 2.2 no longer holds generally? Recall in Example 2.3 (b) that this situation is not uncommon, specifically when
lgg
ho ho(ar) (a;; Go) Σ (((())))
where Go := Ehe Dg0 In some specific cases of this setting, in fact, we fail to attain distin- guishability, and the model may not even be identifiable in the classical sense, i.c. PXG = G. does not guarantee to have XG = G.. Since hg is the pdf of mixture distribution - a popular choice for modeling complex forms of probability densities given its amenability to interpretation compared to black box type models
is of interest to study the implication of parameter estimation for the deviated components in this setting, provided that the distinguisability condition may be at least partially achieved in some suitable sense. As we shall see, our theory demands more refined analysis. To facilitate the presentation, denote A : S i Sisk. 02 E (ff..... 0 }-- Also, set k : |A], which stands for the cardinal of the set A. Our results will be divided into three separate regimes of k and A (i) X O, (ii) k < Ko and X* E (0, 1], and (iii) k = ko and X E (0, 1]. We only choose present results of the second regime (ii) in the main text because of limited space and because of its representativeness as it shows all the intriguing behaviours of the model in this partially distinguishable setting. The first and third regime are deferred to Appendix A
3.4.1 Regime B: k ko and X E (0, 1]
First, we consider the exactly-specified setting of model ① namely, h, is known. When < Ko- we can check that we still have dishtinguishability of hg and linear combinations of {f(=)())) and its derivatives. Therefore, long as is first order identifiable, one can invoke the proof of Theorem 3.3 to establish the same lower bound V(PAc: PX-G.) in terms of W W,,AG, X² G.) for some r 2 1. Thus, our focus in this subsection is the settings when k is unknown.
Over-fitted setting with strongly identifiable f. Moving to the over-fitted settings of model setup (I). i.e., k. is unknown and strictly upper bounded by a given K, as long as K 2 ko- (f, K) is not distinguishable from ha- Therefore, the results of Theorem 3.3 are not always applicable to the setting when K 2 ku- Besides, in the over-fitted setting, the identifiability of model [ no longer holds. Indeed, for any A > X", if we take
G.(A) = (1 X1-X/A + (M/AA)..
then GX-G. = (X) We present this pathological behavior in the following result.
Theorem 3.6. Assume that ho takes the form 77 and k < ko. Besides that, K 2 ko and f is second order identifiable. Then, for any G E Og ()), there exist positive constants C1 and C2 depending only on X², G.,, ho, θ such that the following hold:
(a) If K < k. + ko ko 1, then "(Px-,G.,PAd 2 &WW2(A
(b) K 2 k. + tho - K. then
V(px G. PA. 2 C2 (LOSS)) + 1aAAA W3(G,,GA)))
(c) As special case, if K == ki, + kn k we have
G.: .P..) 2 Sf for all 8 >>0 0, where C3 depends on X", G., bog 0,8.
As can see, the magnitude of À compared to A will decide the solution of (A, G) to the identifiable equation PxG = a* therefore lead to different lower bounds such in part (b) of the theorem. In particular, if A < X" the solution is (X, G) = (X", ,.,), and for any X X", the solution is G = (A) given in Eq. (8) Specifically, when À is strictly larger than X by some amount ó > 0. then the latter case is well separated from the former, and we have an exact- fitted result when K Ko + K. - K.
8

---

4 Experiments
We now would like to demonstrate the convergence rates in Section via two synthetic experi- ments: one for distinguishable setting and one for partially distinguishable setting. For the partially distinguishable one, the experiments are in Appendix B
Distinguishable setting. We conduct an experiment where the original data distribution comes from an uniform distribution on curve (half circle) in R2 convoluted with Gaussian noises (red curve and blue points in Fig. [a), and train a Normalizing Flow neural network [TT] (Masked Autoregressive architecture) with 5 layers to get good density estimation ho for this dataset. Then assume that there are new data coming in, and the original distribution hn is deviated by a mixture of distributions in the location Gaussian family fa)(()) So the true generating density now is
Px-G. (x) (1 X ho)) (x) + A Σ p. (x]0)).
where \" = G. = p; o0;, where pi 0.3, p5 0.3, P3 = 0.4,0; 0.7
(0.1, (1.0, 1.5). Samples from the deviated component are green points in Fig. [a). It can be seen from Proposition 2.4Ja) a) that hy is distinguishable with family f. For each n, we simulate n data points from true model (9) estimate An G₂ by the EM algorithm (it is possible because Normalizing Flows provides exact density computation), and measure its convergence to the true X² G.G. We conduct 16 replications for each sample size. The average error estimations with a 75% error bar can be seen in Fig. ! The W1 error in the exact-fitted case is of order (log(n)/n) 1/2 and W2 error in the over-fitted case is of order (log(n)/n)u4 1/4 Meanwhile, thanks to the distinguishability. the estimation errors in both cases of À are all of the order (nggn))/)) 1/2, These simulation results matched with the theoretical results found in Theorem [.3 and Theorem 3.4 From the result, we see that the deviating mixture model successfully learns the deviated components and reuses the pre-trained black box model ho. which helps to reduce computational costs.
Wi Mu Ord Joec Moot aethgggeegg poggrreee
03
-0
200
00
(a) Synthetic data set
(b) Convergence rates of A,
(c) Convergence rales of ((G.. G.)
Figure I: Convergence rates for parameter estimation in the distinguishable case.
5 Discussion
In this work, we have presented the deviating mixture model and studied its parameter learning rates under MLE procedure. With a novel notion of distinguishability between distributions, we are able to prove inverse bounds for our model under several distinguishability settings, which allow us to deduce the parameter learning rates from the convergence rate of density functions. The distinguishability condition is shown to be satisfied for multiple families of distributions including those that come from black box models.
We now discuss practical implication of the theory. The deviating mixture model designed to capture the deviated mixture components, and learning its parameters can reveal meaningful information about subpopulations in the data. When there distinguishability in the model, our theory implies that we can learn the deviated proportion with the parametric rate and deviated components with a rate depending on the identifiablity of f. However, our theory does not support employing the deviating mixture model when the existing distribution ho itself is mixture distributions in family f and possesses parameters similar to deviated part, as the learning rate can slow, and the deviated proportion estimator may not converge to the true value. Asymptotically, when ho is estimated using

---

very complex model (eg. wide and deep neural network) and somehow approximates a mixture of f. and/or the signal from deviating components is low, then the provided learning rates in the paper, while still the same with respect to sample size may deteriorate from a large multiplicative constant that depends on hi, X", and G..
We believe that this work is the first attempt in the effort of understanding a broader class of mixture models combining with black box models, and interpreting the learned model parameters. There is room for future work going forward. From theoretical viewpoint, one may be interested in establishing minimax lower bounds for the learning behavior of the deviating mixture model, or show uniform inverse bounds for the model when X and G, are considered signals that will change with samples. From a modeling viewpoint, it worthwhile to explore mixtures of black box models and develop a suitable notion of identifiability and inverse bounds so that the learning process is efficient.
Acknowledgements
Nhat Ho acknowledges support from the NSF IF 2019844 and the NSF AI Institute for Foundations of Machine Learning. Long Nguyen is partially supported by NSF grant DMS-2015361.
References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein generative adversarial
networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[2] L.. Bordes, S. Mottelet, and P. Vandekerkhove. Semiparametric estimation of a two-component mixture model. Annals of Stafistics, 34, 2006.
[3] Cristina Butucea and Pierre Vandekerkhove. Semiparametric mixtures of symmetric distribu-
tions. Scandinavian Journal of Statisticss, 41(1):227-239, 2014.
[4] T. Cai, X. Jeng, and Jin. Optimal detection of heterogeneous and heteroscedastic mixtures. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73, 2011.
[5] Tony Cai, Jiashun Jin, and Mark Low. Estimation and confidence sets for sparse normal mixtures. The Annals of Statistics, 35(6):2421-244) 2007.
[6] H. Chen and J. Chen. Tests for homogeneity in normal mixtures in the presence of a structural parameter. Statistica Sinica, 13:351-365. 2003.
[7] J. Chen and P. Li. Hypothesis test for normal mixture models: the approach. Annals of Statistics,
2009.
[8] Chen, Li, and Y. Fu. Inference on the order of a normal mixture. Journal of the American Statistical Association, 107: 107:1096-1105, 2012.
[9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[10] D. Donoho and J. Jin. Higher criticism for detecting sparse heterogeneous mixtures. Annals of Statistics, 32, 2004.
[11] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. nflows: normalizing flows in PyTorch, November 2020.
[12] Sébastien Gadat, Jonas Kahn, Clément Marteau, and Cathy Mhuugs-Rabusseart Parameter recovery in two-component contamination mixtures: The 12 strategy. In Annales de I' I'Institur Henri Poincaré, Probabilités et Statistiques, volume 56, pages 1391-1418. Institut Henri Poincaré, 2020.
[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in newral information processing systems, 27, 2014.
[14] Aritra Guha, Nhat Ho, and XuanLong Nguyen. On posterior contraction of parameters and interpretability in bayesian mixture modeling. Bernoulli, 27(4):115--2188 2021.
[15] Philippe Heinrich and Jonas Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. Annals of Statistics, 46(6A))2844-2870, 2018.
10

---

[16] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. Annals of Statistics, 44:2726-2755 2755, 2016.
[17] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, (0:271-300) 2016
[18] Nhat Ho and I Long Nguyen. Singularity structures and impacts on parameter estimation in finite mixtures of distributions. SIAM Journal on Mathematics of Data Science, 144::300-75,, 2019.
[19] Katz-Samuels, G. Blanchard, and Scott. Decontamination of mutual contamination models. Journal of Machine Learning Research, 20, 2019.
[20] Geoffrey J. Mclaach and David Peel. Finite mixture models, volume 299 of Probability and Statistics - Applied Probability and Statistics Section. Wiley, New York, 2000.
[21] X. Nguyen. Convergence latent mixing measures in finite and infinite mixture models. Annals of Statistics, 4(1):370-400, 2013.
[22] Emanuel Parzen. On estimation of probability density function and mode. The annals of mathematical statistics, 3363),1065-1076, 1962.
[23] Rohit Kumar Patra and Bodhisattva Sen. Estimation a two-component mixture model with applications to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 8864))86. 2016.
[24] B. Schölkopf and A. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.
[25] C. Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, 2015.
[26] Sara van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
[27] C. Villani. Optimal Transport: Old and New. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathemtical Sciences]. Springer, Berlin, 2009.
[28] Sidney Yakowitz and John D Spragins. On the identifiability of finite mixtures. The Annals of Mathematical Statistics, (9(1):209-214 1968.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 1
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] It can be seen in the source code.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Figure 2, 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] The experiments are run on CPU's only.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...

---

(a) If your work uses existing assets, did you cite the creators? [No] We do not use any existing assets.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [No] We do not crowdsoureing/oonddttedd research with human subjects
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [No]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [No]
12


Reviews:
[{"review_id": "yKkYuECGf10", "reviewer": "Reviewer_sdP4", "summary": "The paper considers a deviating mixture model in which they try to establish rates for parameter estimation. The deviating mixture model is of the form $(1-\\lambda^*) h_0 + \\lambda^* \\sum_{i=1}^{k} p_i^* f(x|\\theta_i^*)$ where $h_0$ is known but $\\lambda^*, p^*$ and $theta^*$ are unknown. The authors analyze the maximum likelihood and provide rates under some assumptions on the complexity of the class of allowed distributions (I am not an expert on this but is related to covering numbers). The assumption is justified via a proposition that location scale Gaussian families satisfy the assumption. They provide analysis both when $k$ is known and unknown. Along the way, they define a notion of distinguishability between effectively $h_0$ and $f$.", "questions": "Does A.3 work for truncated sub-Gaussian distributions? Does it suffice that the covariance matrix is well-behaved (in terms of eigenvalues)?", "limitations": "This is theoretical work.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\nThe paper is studying an interesting setting. Technically looks non-trivial, though the reviewer is not familiar with the literature to argue about its novelty.\n\nCons: \nI tried to read the overview with the contributions and the section is not well-written, the writing needs to be improved and further discussion on the techniques should be added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LyhAyOEjve", "reviewer": "Reviewer_UbgM", "summary": "The paper presents a new model called the deviating mixture model to capture the phenomenon that a known probability density estimate in practice could shift with new data coming in. The shift or deviation is modeled as a mixture distribution in the following manner: $(1-\\lambda)h_0 + \\lambda \\sum_{i=1}^k p_i f(x | \\theta_i)$.\n$h_0$ is known already by some means, we care about the deviation from $h_0$. \nThe work studies conditions on $h_0$ and $f(.|\\theta)$ under which maximum likelihood estimation can recover the parameters of the deviation.\nA key property which is typically assumed in such estimation problems is that of identifiability: that is, two distributions with sufficiently different parameters produce statistically distinguishable distributions. \nTo identify parameters in the presence of $h_0$ we need a stronger condition which the authors call distinguishability. They follow this up with multiple examples of $h_0$ for which the deviation modeled as a mixture of Gaussians is distinguishable.\nThe authors then proceed to show bounds for recovering the parameters of the deviation under the assumptions of distinguishability. First they show that for Gaussian mixture models convergence in Wasserstein metric implies convergence in parameters.\nThen they show that the total variation distance between two deviation models is lower bounded by a quantity which closely relates to the Wasserstein distance (under the assumption of distinguishability).\nThey also provide results under partial distinguishability assumptions and supplement theory with some experiments on synthetic data to show that empirically their rate of convergence matches theoretical predictions.\n", "questions": "**Questions:**\n1. On the face of it, the deviating mixture setting seems closely related to the well studied setting of learning mixtures of Gaussians. Could you elaborate on what makes this problem harder and where the technical challenges you need to overcome are?\n\nOverall the paper is well written and well presented. Some minor suggestions below.\n**Suggestions:**\n1. Explicitly state what is meant by location and scale parameters of a Gaussian.\n2. Explicitly state that distinguishability implies identifiability.\n3. I believe it could help readers to see a high level overview of some of the salient proofs in the paper in the main body rather than a comprehensive set of Theorem statements. This will help the reader understand why this setting is more challenging than the vanilla mixture of Gaussians setting.\n", "limitations": "One limitation of the work which the authors could add a brief discussion on is the lack of strong non-asymptotic rates for high-dimensional settings. i.e. for a really large (but finite) neural network, the distinguishability assumption is still satisfied and hence MLE will give a convergence rate that scales as $1/\\sqrt{n}$ but as the neural net becomes larger it could become harder and harder to distinguish it from the deviation. How does the error in parameter recovery scale with the size of the neural network?\n\nThe authors have adequately addressed any potential negative societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well written and is clear. It could do with a bit more thorough preliminaries discussion in places but the flow and presentation are done well.\n2. As far as I can tell, the results of the paper are original and novel and the direction is one of relevance to NeurIPS audience.\n3. Understanding conditions under which we can recover parameters of a distribution from samples is an important problem and the current paper looks at a twist on the classical setting wherein we know one component of a mixture and are trying to learn the remaining components. The paper presents a significant set of results in this direction.\n\n**Weaknesses:**\n1. The bounds focus mainly on the convergence rates with respect to $n$ the number of samples and do not focus on the dimension of the distributions being considered. In practice, as dimensions can be huge, the approaches outlined in this paper might be impractical?\n2. A possible artifact of the above issue is that we can only run the MLE procedure on low-dimensional data. Is computation the reason why the synthetic experiments were only performed on low-dimensional settings? It would be interesting to see if the approach can be scaled up to settings where the target distribution has at least 10-20 dimensions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CkzWCAKmoaW", "reviewer": "Reviewer_iPiq", "summary": "\nThis paper studies a model known as ``deviating mixture model’’ which is a mixture of a single known distribution h0 and an mixture model whose parameters are unknown. The authors modify previous work to give a notion of distinguishability for when one can learn the parameters of the new mixture model. Furthermore, they use this notion, to give convergence estimates under wasserstein metrics. \n\n\nMore concretely,  the authors give a definition of identifiability (heavily based on prior work [15,21] ) for this new deviated model. This notion extends to the case when h0 is a distribution that is not too heavy tailed or light tailed or a distribution captured by Neural Networks. Using an Assumption A.3 on the complexity of the mixture model, they can upper bound the Hellinger distance between MLE estimate and true optimal bounds decay proportional to sqrt( log n/ n) given n samples. The main part of the paper then shows that these bounds also imply the same convergence for the parameters in the wasserstein metric. To do so, they show \"inverse bounds\" of the form that the total variation distance is bounded below by the Wasserstein distance. Since the hellinger distance upper bounds the TV distance, this completes the proof. \n\nThe main proof technique goes as follows:\n1)  First we show that if total variation distance is not bounded below by W-distance, one can look at the taylor expansion of the difference of the distributions and express it as a linear combination of the first few derivatives of the functions \n2) This linear combination must have non-zero coefficients and must also converge to 0 almost everywhere. \n3) However this contradicts the distinguishability assumption which states that the coefficients must be zero if the linear combination of these derivatives is 0 almost everywhere. \n\nThe authors also extend this setting to the case where the number of components of the mixture model is not known but only an upper bound is known. \n\n\n", "questions": "I noticed that Theorem 3.1 does not have any citation or proof. Could you please provide one?\n\nLine 118, should $i$ range from $1$ to $k$ instead of $k'$? \nLine 278, should \"limited spaces\" should be limited space. \n", "limitations": "I think the authors do a good job of describing the limitations of their work. ", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\nThe work shows a notion of identifiability which is valid deviating mixture model. Using this definition they show that small TV distance implies that the parameters are close in wasserstein distance. \n\nThe proofs seem to be complete, rigorous and thorough. The proofs are able to show this even when the known distribution h0 is a black box model such as a Neural network. This seems nice and very useful in practice. \n\n\nWeakness:\n\nThe main notion of identifiability that is used is largely derived from prior work [21,15]. Furthermore the bounds on hellinger distance, which already imply bounds on the TV distance follow from prior work (Thm 3.1). The main novelty is that this also implies bounds on the Wasserstein distance in the parameters, however this is not entirely surprising.  \n\nThe presentation is a bit terse and notation heavy with less intuition given about the proofs in the main body. One has to read the appendix to even get any intuition of the proofs. \n\nI gave this a weak reject only because I am not entirely convinced of the novelty of work but I am happy to be convinced otherwise. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-oRtt2UxVoI", "reviewer": "Reviewer_eLfh", "summary": "This paper considers the \"deviating mixture model\", which is defined by $(1-\\lambda^*) h_0 + \\lambda^* (\\sum_{i=1}^kp_i^* f(x \\mid \\theta_i^*))$, where $h_0$ is known and the distribution family $f$ is known. \nMain Contributions:\n- The authors propose a notion of distinguishability of this model.\n- The authors provide the converging rate of MLE for this model (in estimating $\\lambda*$ and $\\theta_i's$).\n- The authors provide simulations to verify the bounds they derived.\n", "questions": "See the weaknesses part.", "limitations": "NA.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper lie in the follow two parts:\n- The first part is on the novelty: for proposing the \"deviating mixture model\", and also the definition of the distinguishability.\n- The second part is on the rates of convergence part, where assuming the distinguishability conditions, the authors derived inverse bounds to upper bound the Wasserstein distance by the total variation distance (which is known to be upper bounded by Hellinger distance).\n\nThe weaknesses of paper are mainly on the significance and importance of this proposed \"deviating mixture model\". In what scenes/example we can use the \"deviating mixture model\" to model the data distribution? Some questions in my mind (that need more explanations/arguments) are: \n-  The authors mentioned that, from some black-box methods (NN or kernel method) we can get the density estimation $h_0$. But in those cases, $h_0$ itself is just an estimate (a maybe complex deviation) of the underlying true distribution. Why can one further assume that the new data is from a deviation of $h_0$, where the deviation is from a known distribution family?\n- The assumption that the distribution family $f$ is known, and the number of components in the mixture ($k^*$) is bounded, I think also needs motivations from practice or explanations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept