Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper studies a model known as ``deviating mixture model’’ which is a mixture of a single known distribution h0 and an mixture model whose parameters are unknown. The authors modify previous work to give a notion of distinguishability for when one can learn the parameters of the new mixture model. Furthermore, they use this notion, to give convergence estimates under wasserstein metrics. 


More concretely,  the authors give a definition of identifiability (heavily based on prior work [15,21] ) for this new deviated model. This notion extends to the case when h0 is a distribution that is not too heavy tailed or light tailed or a distribution captured by Neural Networks. Using an Assumption A.3 on the complexity of the mixture model, they can upper bound the Hellinger distance between MLE estimate and true optimal bounds decay proportional to sqrt( log n/ n) given n samples. The main part of the paper then shows that these bounds also imply the same convergence for the parameters in the wasserstein metric. To do so, they show "inverse bounds" of the form that the total variation distance is bounded below by the Wasserstein distance. Since the hellinger distance upper bounds the TV distance, this completes the proof. 

The main proof technique goes as follows:
1)  First we show that if total variation distance is not bounded below by W-distance, one can look at the taylor expansion of the difference of the distributions and express it as a linear combination of the first few derivatives of the functions 
2) This linear combination must have non-zero coefficients and must also converge to 0 almost everywhere. 
3) However this contradicts the distinguishability assumption which states that the coefficients must be zero if the linear combination of these derivatives is 0 almost everywhere. 

The authors also extend this setting to the case where the number of components of the mixture model is not known but only an upper bound is known. 



strengths_and_weaknesses: Strengths 

The work shows a notion of identifiability which is valid deviating mixture model. Using this definition they show that small TV distance implies that the parameters are close in wasserstein distance. 

The proofs seem to be complete, rigorous and thorough. The proofs are able to show this even when the known distribution h0 is a black box model such as a Neural network. This seems nice and very useful in practice. 


Weakness:

The main notion of identifiability that is used is largely derived from prior work [21,15]. Furthermore the bounds on hellinger distance, which already imply bounds on the TV distance follow from prior work (Thm 3.1). The main novelty is that this also implies bounds on the Wasserstein distance in the parameters, however this is not entirely surprising.  

The presentation is a bit terse and notation heavy with less intuition given about the proofs in the main body. One has to read the appendix to even get any intuition of the proofs. 

I gave this a weak reject only because I am not entirely convinced of the novelty of work but I am happy to be convinced otherwise. 


questions: I noticed that Theorem 3.1 does not have any citation or proof. Could you please provide one?

Line 118, should $i$ range from $1$ to $k$ instead of $k'$? 
Line 278, should "limited spaces" should be limited space. 

limitations: I think the authors do a good job of describing the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers a deviating mixture model in which they try to establish rates for parameter estimation. The deviating mixture model is of the form $(1-\lambda^*) h_0 + \lambda^* \sum_{i=1}^{k} p_i^* f(x|\theta_i^*)$ where $h_0$ is known but $\lambda^*, p^*$ and $theta^*$ are unknown. The authors analyze the maximum likelihood and provide rates under some assumptions on the complexity of the class of allowed distributions (I am not an expert on this but is related to covering numbers). The assumption is justified via a proposition that location scale Gaussian families satisfy the assumption. They provide analysis both when $k$ is known and unknown. Along the way, they define a notion of distinguishability between effectively $h_0$ and $f$.
strengths_and_weaknesses: Pros
The paper is studying an interesting setting. Technically looks non-trivial, though the reviewer is not familiar with the literature to argue about its novelty.

Cons: 
I tried to read the overview with the contributions and the section is not well-written, the writing needs to be improved and further discussion on the techniques should be added.
questions: Does A.3 work for truncated sub-Gaussian distributions? Does it suffice that the covariance matrix is well-behaved (in terms of eigenvalues)?
limitations: This is theoretical work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a new model called the deviating mixture model to capture the phenomenon that a known probability density estimate in practice could shift with new data coming in. The shift or deviation is modeled as a mixture distribution in the following manner: $(1-\lambda)h_0 + \lambda \sum_{i=1}^k p_i f(x | \theta_i)$.
$h_0$ is known already by some means, we care about the deviation from $h_0$. 
The work studies conditions on $h_0$ and $f(.|\theta)$ under which maximum likelihood estimation can recover the parameters of the deviation.
A key property which is typically assumed in such estimation problems is that of identifiability: that is, two distributions with sufficiently different parameters produce statistically distinguishable distributions. 
To identify parameters in the presence of $h_0$ we need a stronger condition which the authors call distinguishability. They follow this up with multiple examples of $h_0$ for which the deviation modeled as a mixture of Gaussians is distinguishable.
The authors then proceed to show bounds for recovering the parameters of the deviation under the assumptions of distinguishability. First they show that for Gaussian mixture models convergence in Wasserstein metric implies convergence in parameters.
Then they show that the total variation distance between two deviation models is lower bounded by a quantity which closely relates to the Wasserstein distance (under the assumption of distinguishability).
They also provide results under partial distinguishability assumptions and supplement theory with some experiments on synthetic data to show that empirically their rate of convergence matches theoretical predictions.

strengths_and_weaknesses: **Strengths:**
1. The paper is well written and is clear. It could do with a bit more thorough preliminaries discussion in places but the flow and presentation are done well.
2. As far as I can tell, the results of the paper are original and novel and the direction is one of relevance to NeurIPS audience.
3. Understanding conditions under which we can recover parameters of a distribution from samples is an important problem and the current paper looks at a twist on the classical setting wherein we know one component of a mixture and are trying to learn the remaining components. The paper presents a significant set of results in this direction.

**Weaknesses:**
1. The bounds focus mainly on the convergence rates with respect to $n$ the number of samples and do not focus on the dimension of the distributions being considered. In practice, as dimensions can be huge, the approaches outlined in this paper might be impractical?
2. A possible artifact of the above issue is that we can only run the MLE procedure on low-dimensional data. Is computation the reason why the synthetic experiments were only performed on low-dimensional settings? It would be interesting to see if the approach can be scaled up to settings where the target distribution has at least 10-20 dimensions.

questions: **Questions:**
1. On the face of it, the deviating mixture setting seems closely related to the well studied setting of learning mixtures of Gaussians. Could you elaborate on what makes this problem harder and where the technical challenges you need to overcome are?

Overall the paper is well written and well presented. Some minor suggestions below.
**Suggestions:**
1. Explicitly state what is meant by location and scale parameters of a Gaussian.
2. Explicitly state that distinguishability implies identifiability.
3. I believe it could help readers to see a high level overview of some of the salient proofs in the paper in the main body rather than a comprehensive set of Theorem statements. This will help the reader understand why this setting is more challenging than the vanilla mixture of Gaussians setting.

limitations: One limitation of the work which the authors could add a brief discussion on is the lack of strong non-asymptotic rates for high-dimensional settings. i.e. for a really large (but finite) neural network, the distinguishability assumption is still satisfied and hence MLE will give a convergence rate that scales as $1/\sqrt{n}$ but as the neural net becomes larger it could become harder and harder to distinguish it from the deviation. How does the error in parameter recovery scale with the size of the neural network?

The authors have adequately addressed any potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers the "deviating mixture model", which is defined by $(1-\lambda^*) h_0 + \lambda^* (\sum_{i=1}^kp_i^* f(x \mid \theta_i^*))$, where $h_0$ is known and the distribution family $f$ is known. 
Main Contributions:
- The authors propose a notion of distinguishability of this model.
- The authors provide the converging rate of MLE for this model (in estimating $\lambda*$ and $\theta_i's$).
- The authors provide simulations to verify the bounds they derived.

strengths_and_weaknesses: The strengths of the paper lie in the follow two parts:
- The first part is on the novelty: for proposing the "deviating mixture model", and also the definition of the distinguishability.
- The second part is on the rates of convergence part, where assuming the distinguishability conditions, the authors derived inverse bounds to upper bound the Wasserstein distance by the total variation distance (which is known to be upper bounded by Hellinger distance).

The weaknesses of paper are mainly on the significance and importance of this proposed "deviating mixture model". In what scenes/example we can use the "deviating mixture model" to model the data distribution? Some questions in my mind (that need more explanations/arguments) are: 
-  The authors mentioned that, from some black-box methods (NN or kernel method) we can get the density estimation $h_0$. But in those cases, $h_0$ itself is just an estimate (a maybe complex deviation) of the underlying true distribution. Why can one further assume that the new data is from a deviation of $h_0$, where the deviation is from a known distribution family?
- The assumption that the distribution family $f$ is known, and the number of components in the mixture ($k^*$) is bounded, I think also needs motivations from practice or explanations.
questions: See the weaknesses part.
limitations: NA.
review_title: null

Meta-review: metareview: The authors study a mixture of a single known distribution h0 and an mixture model whose parameters are unknown. They propose new notions of distinguishability and partial distinguishability that they use for characterizing convergence rates.

The reviewers had a mixed opinion about the paper and had several comments about the technical novelty of the work. The authors addressed these suitably in their rebuttal. In particular, their revised introduction is much clearer about the motivation and contributions. I am happy to recommend acceptance of the paper.
confidence: Less certain
Decision: Accept