Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the problem of assessing implicit generative models using kernelised Stein discrepancy (KSD).
The paper introduces a non-parametric (NP) Stein operator to allow implicit models to be used in KSD (which wasn't possible due to the fact KSD requires at least the unnormalised density of the model).
Results on both synthetic and real datasets compared to MMD (which also only needs samples to compute) show that the proposed NP-KSD method has a better test power than MMD.
strengths_and_weaknesses: Pros
- The paper proposes a novel method that extends KSD to implicit models.
- The paper is well written and technical. The paper is not hard to follow.
- The paper is a solid contribution to better access the generation quality of generative models. This is becoming more important as synthetic data generation is seen as an important tool to solve privacy.
- The results look promising. It would be interesting to include a neural sampler trained by MMD, which would be an even stronger indication of the stronger test power of NP-KSD.

Cons
- Discussion compared to MMD is not enough; see my Qs below.
questions: - In algorithm 1, are we effectively learning the score function so to estimate KSD?
  - Is this what the author(s) mean(s) in Line 232-234 when criticising MMD?
- In algorithm 2, do we effectively use more samples from G compared to MMD?
limitations: Limitations and potential negative societal impact are discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a kernel Stein discrepancy based test for black box generative models.
Since the normal KSD cannot be applied, as the true score function of the black box model is unknown, the authors proposed a KSD variant where this unknown score function is estimated from samples from the model.
The resulting hypothesis test then is whether a given dataset that was used to fit the generative model is distributed according to the estimated score function of that model.
The score estimation is achieved via estimating a component wise distribution conditioned on all other components, through of summary statistics.
Some consistency results are provided.
An experimental evaluation on toy data and and simple real datasets shows some benefits against the original KSD test where applicable, and one MMD based baseline otherwise.
strengths_and_weaknesses: A well written paper that addresses an important issue.
The proposed methodology is based around a few nice tricks on estimating the conditional distributions and carried our in a thorough manner.
In the current form of the paper, however, it is not clear whether the proposed methodology is really as useful as the authors claim. The experimental evaluation is not thorough, and lacks baselines and comparisons to alternative strategies.

A few concrete points:
* There is no like to like comparison with an MMD based method. Only MMDAgg as a very specific case; the point of MMDAgg is that it can select a kernel without having to sacrifice samples, but in the case of a generative model, we can generate many samples so it doesn't seem to be a well fitting baseline). What about e.g. a plain MMD test with the median heuristic, or with a learnt optimal kernel? What about e.g. the linear time tests that learn feature locations (E.g. https://arxiv.org/abs/1605.06796).
* It is actually not clear what is gained compared to an MMD based test. This surely depends on the number of samples drawn from the generator, as well as computational costs. See also question below. An empirical exploration of the performance of the various approaches as a function of drawn samples and compute would be very helpful. The authors only make some vague comments on this.
* It is not clear why we could not just use score matching to learn the score using a powerful neural network instead of the conditional distribution approach? This would allow using many samples from the generator as well.
* There are no empirical explorations of how the proposed score estimation approach scales in any direction. As that is crucial for this method to work, and since it is also a kind of harsh approximation, that weakens the paper. Related: what is the impact of the NN architecture for the score estimation in this case.

Missing paper references that are highly relevant, as the problem of assessing performance of generative models was first discussed here (among others)
* Statistical Model Criticism using Kernel Two Sample Tests by LLoyd et al
* Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy by Sutherland et al

Minor:
* l97 The KSD was originally proposed in Chwialkowski et al 2016 and Liu at al 2016, not Gorham Mackey 2017
* l109 The wild bootstrap is only really used if there is correlation to break (e.g. for time series data). Otherwise plain permutation test is sufficient and has no free parameters
* l198 the normality result of Song is in a different context. Could you at least empirically explore this?
questions: A high level question is: Does it make sense to asses the quality of generative models via a hypothesis test? What do we really learn if we e.g. fail to reject the null hypothesis? What do we learn if we can reject it? It shouldn't actually be a surprise that it is easy for most statistical tests to reject the null. Some of the references I put above go further and explain what features in the data lead to the rejection. Something like this could be used to improve a generative model, but that is actually not done in this paper.

limitations:
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an adaptation of kernelized Stein discrepancy, called non-parametric KSD (NP-KSD), for assessing the implicit generative models. In particular, the author considered a scenario that (1) we do not have access to the (unnormalized) densities of the generative model; (2) the data dimension is large; (3) fixed number of true observations. To enable the assessment of this type of model, the author proposed a non-parametric Stein operator based (1) conditional distributions with summary statistics; (2) resampling the conditional dimensions; (3) score matching for gradient estimation. Together with the kernel trick and Monte Carlo based goodness-of-fit test, it results in the NP-KSD method. 

 
strengths_and_weaknesses: **Strength**:

The paper is clearly written and easy to follow. Although the motivation of the proposed method can be improved. 
This paper aims to tackle an important problem: the evaluation of implicit models, which can have significant impacts to the current deep generative models. To sidestep the computational challenges, the author also provides several tricks to enable the NP-KSD. Theoretically, the author shows the consistency and convergence results of NP-KSD. 

**Weakness**:

First, the motivation of the proposed method can be improved. For example, two-sample test is a typical method for evaluating implicit models. Why do we need goodness-of-fit test instead? Goodness-of-fit test is typically used when we have access to the density function, that is where its name comes from. Why do we want to use this for the implicit model with the gradient estimation? To me, it seems that we achieve what the two-sample test is designed for but with an alternative method. So a better motivation for goodness-of-fit test is needed. 

There are several tricks used in deriving NP-KSD. The author has shown that NP-KSD satisfied Stein identity. However, a more interesting (or more important) aspect is to show when NP-KSD =0, it means p=q. Currently, I don't think it is true, especially with summary statistics and score matching. If not, then the goodness-of-fit test procedure will not be valid. The author mentioned equivalence class, but it is unclear how this will impact the evaluation of the implicit model. E.g. with a summary statistics, the distribution inside an equivalence class has similar visual quality?

Another concern is about its novelty. To me, the Stein operator \Tau is defined similarly as the Langevin-Stein operator but with conditional distributions. This is equivalent to the original Langevin-Stein discrepancy. This idea has also been explored in paper [1] and the author should consider citing it. If so, the main novelty of NP-KSD is score-matching, resampling and summary statistics. However, they seem to be more like tricks rather than novel methodologies.  Although I agree the theoretical consistency and convergence results give some guarantees of using the resampling and score matching, which seems to be novel. 

[1] Singhal, R., Han, X., Lahlou, S., & Ranganath, R. (2019). Kernelized complete conditional Stein discrepancy. arXiv preprint arXiv:1904.04478.

questions: The questions are mainly the summary of what I have described in Weakness.

**Major**:
1. The motivation for using the goodness-of-fit test should be clearer.  E.g. advantages compared to two-sample test? Also, the implicit model has two application scenarios: (1) sampler, where we want to generate samples from a target distribution and (2) data synthesis, where we want to generate observations that mimic the real data samples, like GAN. From the context of this paper, it seems that it focuses on (2) instead of (1). It would be better to clarify this. 
2. Do we have theoretical guarantees when NP-KSD =0 $\Rightarrow$ p=q? Does it only hold for equivalence class? What is the impact of equivalence class on the evaluation of the implicit model?
3. The concern about the novelty, as detailed in the weakness section. Particularly, if I understood correctly, the main framework of the proposed method is based on goodness-of-fit test but with estimated gradients from score matching. Those ideas are not new. So maybe consider elaborating more on the novel contributon?
4. For synthetic distribution experiments, why the proposed method is better than KSD? From my understanding, NP-KSD is an approximation of KSD. 

**Minor**:
1. $\mathbb{R}^m$ in line 86 should be $\Omega_q$?
2. In the related work section, maybe consider adding citations regarding various types of Stein discrepancies and their usage in generative models. 
3. I wonder why the proposed method is better than two-sample tests. Since the gradient estimation used in NP-KSD is based on the generated samples, it means NP-KSD does not have access to more information compared to the two-sample test. Where does this performance increase come from?
limitations: The author mentioned NP-KSD can distinguish distributions in equivalence classes. But the potential impact of the equivalence class to model evaluation can be discussed in more detail.  It would be great to consider some simple cases with simple summary statics like mean, and show some properties of the equivalence class.
review_title: null

Meta-review: metareview: Decision: Accept

This paper introduces a non-parametric (NP) Stein operator to allow implicit models to be used in KSD. So this enable the use of KSD for evaluating the performance of implicit models, and the new test statistic shows better test power compared to MMD test.

Reviewers commended that the paper writing is clear, and the contribution is solid and novel. There were a few technical concerns regarding the proposed KSD as well as comparisons to MMD, which were mostly addressed in author-reviewer discussions. 

In revision for camera ready, I'd encourage the authors to include the additional experiments & discussions provided in the author feedback. Perhaps adding more MMD-based test baselines would strengthen the paper even further.
confidence: Certain
Decision: Accept