A Kernelised Stein Statistic for Assessing Implicit Generative Models
Authors: Wenkai Xu
Gesine Reinert
Venue: NeurIPS 2022 Accept (2022)
Keywords: Stein's method
kernel method
model assessment
generative models

Abstract: Synthetic data generation has become a key ingredient for training machine learning procedures,  addressing tasks such as data augmentation, analysing privacy-sensitive data, or visualising representative samples. Assessing the quality of such synthetic data generators hence has to be addressed. As (deep) generative models for synthetic data often do not admit explicit probability distributions, classical statistical procedures for assessing model goodness-of-fit may not be applicable. In this paper, we propose a principled procedure to assess the quality of a synthetic data generator. The procedure is a Kernelised Stein Discrepancy-type test which is based on a non-parametric Stein operator for the synthetic data generator of interest. This operator is estimated from samples which are obtained from the synthetic data generator and hence can be applied even when the model is only implicit. In contrast to classical testing, the sample size from the synthetic data generator can be as large as desired, while the size of the observed data that the generator aims to emulate is fixed. Experimental results on synthetic distributions and trained generative models on synthetic and real datasets illustrate that the method shows improved power performance compared to existing approaches.

A Kernelised Stein Statistic for Assessing Implicit Generative Models
Wenkai Xu Department of Statistics University of Oxford wenkai. xuustats. ac uk
Gesine Reinert Department of Statistics University of Oxford and Alan Turing Institute reinert@stats ox ac
Abstract
Synthetic data generation has become key ingredient for training machine learning procedures, addressing tasks such data augmentation, analysing privacy-sensitive data, or visualising representative samples. Assessing the quality such synthetic data generators hence has be addressed. As (deep) generative models for syn- thetic data often do not admit explicit probability distributions, classical statistical procedures for assessing model goodness-of-fi may not be applicable. In this paper, we propose a principled procedure to assess the quality synthetic data generator. The procedure is a kernelised Stein discrepancy (KSD)-type test which is based on a non-parametric Stein operator for the synthetic data generator of interest. This operator is estimated from samples which are obtained from the synthetic data generator and hence can be applied even when the model is only implicit. In contrast classical testing, the sample size from the synthetic data generator can be as large as desired, while the size of the observed data which the generator aims to emulate is fixed. Experimental results on synthetic distributions and trained generative models on synthetic and real datasets illustrate that the method shows improved power performance compared to existing approaches.
Introduction
Synthetic data capturing main features of the original dataset are of particular interest for machine learning methods. The use of original dataset for machine learning tasks can be problematic or even prohibitive certain scenarios, e.g. under authority regularisation on privacy-sensitive information, training models small-sample dataset, or calibrating models with imbalanced groups. High quality synthetic data generation procedures surpass some of these challenges by creating de-identified data to preserve privacy and to augment small or imbalanced datasets. Training deep generative models has been widely studied in recent years [Kingma and Welling, 2013 Radford et al., 2015 Song and Kingmal | and methods such those based Generative Ad Networks (GANs) Goodiellow et al. 2014] provide powerful approaches that learn to generate synthetic data which resemble the original data distributions. However, these deep generative models usually do not provide theoretical guarantees on the goodness-of-f fft to the original data [Creswell et al. 2018).
To the best of our knowledge, existing mainstream developments for deep generative models (Song and Ermon 2020 Li et al.. 2017) do not provide a systematic approach to assess the quality of synthetic samples. Instead, heuristic methods are applied; e.g. for image data, the quality of samples are generally decided via visual comparisons. The study of the training quality has been relying largely on the specific choice of training loss, which does not directly translate into measure of sample quality; in the case of the log-likelihood see for example Theis et al. [2015]. Common quality assessment measures for implicit generative models, images for example, include Inception Scores
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(IS) [Salimans et al. 2016] and Fréchet Inception Distance (FID) Heusel et al. 2017 which are motivated by human inception systems in the visual cortex and pooling Wang al. 2004 Binkowski al. (0188) pointed out issues for IS and FID and developed the Kernel Inception Distance (KID) for more general datasets. Although these scores can be used for for comparisons, they do not provide a statistical significance test which would assess whether a deemed good generative model is "good enough". A key stumbling block is that the distribution from which a synthetic method generates samples is not available; one only ever observes samples from it.
For models in which the density known explicitly, least up to normalising constant, some assessment methods are available. Gorham and Mackey [2015] proposed to assess sample quality using discrepancy measures based on Stein's 's method; [Chwialkowski al. [2016] and al. [2016]
created a kernelised test statistic called kernelised Stein discrepancy (KSD) based on Stein operators
and used the unit ball of an RKHS as test functions. [Schrab et al. [2022] assesses the quality of
generative models on the MNIST image dataset from [LeCun et al. [1995] using an aggregated kernel Stein discrepancy (KS test; still an explicit density is required. The only available implicit goodness-of fit test, AgraSSt [u and Reinert 2022]. applies only to generators of finite graphs; it is also of KSD form and makes extensive use of the discrete and finite nature of the problem. To date, quality assessment procedures of implicit deep generative models for continuous data remains unresolved. This paper addresses this gap.
The underlying idea can be sketched as follows. Traditionally, given set of n observations, each in RT, one would estimate the distribution of these observations from the data and then check whether the synthetic data can be viewed as coming from the data distribution. Here instead we characterise the distribution which is generated possibly implicitly from the synthetic data generator, and then test whether the observed data can be viewed as coming from the synthetic data distribution. The advantage of this approach is that while the observed sample size n may be fairly small, the synthetic data distribution can be estimated to any desirable level of accuracy by generating large number of samples from it. Similarly to the works mentioned in the previous paragraph for goodness-of-fn tests, we use KSD approach, based on a Stein operator which characterises the synthetic data distribution. As the synthetic data generator is usually implicit, this Stein operator is not available. We show however that under mild assumptions it can be estimated from synthetic data samples to any desired level of accuracy.
Our contributions We introduce method to assess (deep) generative models, which are often black-box approaches, when the underlying probability distribution is continuous, usually in high- dimensions. To this purpose, we develop a non-parametric Stein operator and the corresponding non-parametric kernel Stein discrepancies (NP-KSD), based on estimating conditional score functions. Moreover, we give theoretical guarantees for NP-KSD.
This paper is structured as follows. We start with review of Stein's 's method and KSD goodness-of-fi tests for explicit models in Section[ before in Section 3 we introduce NP-KSD and analyse the model assessment procedures. We show results of experiments in Section 4 and conclude with future directions in Section 5 Theoretical underpinnings. and additional results are provided in the supplementary material. The code is available at //ttps://gii
com wton//nnaaxx//nppseee git.
2 Stein's method and kernel Stein discrepancy tests
provides an elegant tool to characterise distributions via Stein operators, which can be used to assess distances between BaampleBarbbuur probability distributions, see for example Barbour and Chen [2005] Mijoule et al.) [2021). Given distribution g. an operator Ag is called a Stein operator w.r.t. q and Stein class F if the following Stein identity holds for any test function fe F: [, = 0. For test function h one then aims to find a function f = f&E F which solves the Stein equation
Stein identities, equations, and operators Stein's 's method Steiin 1972
Agf(x) = h(x) - [,((())).
(I)
Then for any distribution p. taking expectations E, in Eq. I assesses the distance| |E,hh - -E,h- through [E,,A,]),, an expression in which randomness enters only through the distribution p.
When the density function q is given explicitly, with smooth support 22C RT is differentiable and vanishes at the boundary of Sgy a common choice of Stein operator in the literature is based on the

---

score- foreeffnctionn 4 V log (with the convention that S, = 0 outside of Sg), see for example Mijoule Here, denotes the gradient operator and taken to be column vector. The score-Stein operator is the vector-valued operator acting on (vector-valued) function f.
al. 1.
A,((x) - f(x)" v log g( + V f(æ),,
(2)
and the Stein identity Eg [g holds for functions f which belong to the so-called canonical Stein class defined in Mijoulc et al. (2021). Definition 3.2. As it requires knowledge of the density q only via its score function, this Stein operator is particularly useful for unnormalised densities
[Hyvärinen, 2005). appearing e.g. in energy based models (EBM) [LeCun et al. 2006].
Kernel Stein discrepancy To assess discrepancies between two probability distributions p and q. the Stein discrepancy (w.rt. class B C FF) is defined as [Gorham and Mackey 2015
SD(pllq B) sup E EplA,f]] sup Ey1A, SEB
(3)
JEB
As the sup f over general class B can be difficult to compute, taking B as the unit ball of reproducing kernel Hilbert space (RKHS) has been suggested in Chwialkowski et al. [2016). Liu et al. (016), resulting in the kernel Stein discrepancy (KSI) defined as
KSD(pl1,, H)
sup SEB.(M)
|Ep[Ag]].
(4)
Denoting by l the reproducing kernel associated with the RKHS H over set X, the reproducing property ensures that ff E H. f(x) = (f, k(z,k( ,),, Vx E X. Algebraic manipulations yield
KSD2 (ullp) Fa, (2,((,,( (ar, })
where (2, .)) = (A, (2x -), (
(0, ))H,, which takes the exact sup without approximation and
does not involve the (sample) distribution p. Then, KSD can be estimated through empirical means, over samples from p. For example, V-statistics (Van der Vaart 2000) and U-statistics [Lee 1990 estimates are
((lp))
Σ u. (x,,,,)) 2
(glP))) m(m 1)
Σ b
(x, 2z).
(6)
m
KSD has been proposed as discrepancy measure between distributions for testing model goodness-of- fit in Chwialkowski al. [2016) and Liu et al. [2016].
., F. from the unknown distri- bution p. To test the null hypothesis Ho p q against the (broad class of) alternative hypothesis H1 : p * q, KSD can empirically estimated via Eq. The null distribution usually simulated via the wild-bootstrap procedure [Chwialkowski et al. 2014). Then if the empirical quantile, i.e. the proportion of wild bootstrap samples that are larger than (glp), is smaller than the pre-defined test level (or significance level) a, the null hypothesis is rejected; otherwise the null hypothesis is not rejected. In this way, systematic poramaratttric goodness-of-itt testing procedure is obtained, which is applicable to unnormalised models.
KSD testing procedure Suppose we have observed samples æ æi,...,
Non-Parametric kernel Stein discrepancies
The construction of a KSD relies on the knowledge of the density model, up to normalisation. How- ever, for deep generative models where the density function is not explicitly known, the computation of the Stein operator in Eq.2 which is based
explicit parametric density, is no longer feasible.
While in principle one could estimate the multivariate density function from synthetic data, density estimation in high dimensions is known to be problematic, see for example Scott and Sain (005) Instead, Stein's 's method allows to use a two-step approach: For data in RT we first pick coordinate i E [m] == {1,.... m. and then we characterize the univariate conditional distribution of that coordinate, given the values of the other coordinates. Using score Stein operators from Ley al.) [0077). this approach only requires knowledge estimation of univariate conditional score functions.
also referred to as Langevin Stein operator Barp et al. 2019

---

First we introduce some notation. Observed data 2 Zn are taken to be column vectors, (2) .......; ( Rm. For the generative model G we write X G to denote random Rm valued element from the (often only given implicitly) distribution which is underlying G. Using G, we generate N samples denoted by y/. ................................ UN In our case, n is fixed and n < N, allowing N + oo in theoretical results. The kernel of an RKHS is denoted by l and is assumed to be bounded.
IR and g(æ) : RT IR, we write 92(-1) (x:) : IR - R for the univariate function which acts only on the coordinate and fixes the other coordinates equal x j # i, so that Ix(-o (x: g(x(1) 2lf-1) I, x((+1) ,...,, ").....(!)) (m))..
For x E RTM,
For [m]] let (+) denote a Stein operator for the conditional distribution Q(+) Qa(-o with E Eq - 9et- (x) = Elgyt= (X))x() = y j + i]. The proposed Stein operator A acting on functions g: R + IR underlying the non-parametric Stein operator is
Ag(x (m))
T gg()((((())))
We note that for X ~ q, the Stein identity TAg(X): = 0 holds and thus A is a Stein operator. The domain of the operator will depend the conditional distribution in question. Instead of using the weights w;
for the T(O), other positive weights which sum to would be possible, but for simplicity we use equal weights. A more detailed theoretical justification of Eq. is given in Appendix A
In what follows we use Stein operator for a differentiable univariate density q the score operator from Eq 2 given by
f(x) - f() + f(x) (x(
((x)
In Proposition D.I of Appendix D we shall see that the operator in 7 equals the score-Stein operator in Eq in Appendix [D] an example is also given. For the development in this paper, Eq. is more convenient as it relates directly to conditional distributions. Other choices of Stein operators are
discussed for example in Ley et al. 2017]. Mijoule et al. 2022 Xu [0022
Re-sampling Stein operators The Stein operator Eq. depends on all coordinates i E [m]. When m is large we can estimate this operator via re-sampling with replacement, as follows. We draw B samples fi1..... iB} } with replacement from [m] such that (iz,.... dn) } ~ Multinom (B, { he(())
The re-sampled Stein operator acting on f Rm ii
1
= -
Σ
A (6) f(z).
B
b=1
Then we have
R (()
f(x) 0. So A is again a Stein operator.
In practice, when m is large, the stochastic operator in Eq Eq. creates a computationally efficient way for comparing distributions. A similar re-sampling strategy for constructing stochastic operators has been suggested in the context of Bayesian inference [Gorham et al. 2020 where conditional score functions, which are given in parametric form, are re-sampled derive score-based (or Langevin) Stein operators for posterior distributions. The conditional distribution in Stein operators has been considered for example in (Singhal et al. [2019] and Wang al. (018 : see also Zhuo et al. (2018) and[in and Wang (2016) in the context of graphical models. In graphical models, the conditional distribution is simplified to conditioning
the Markov blanket (Wang al. 2018). which is a subset of the full coordinate; however, no random re-sampling is used. Conditional distributions also apply in message passing, but there, the sequence of updates is ordered.
Estimating Stein operators via score matching Usually the score function 9'/9 in Eq. is not available but needs to be estimated. An efficient way of estimating the score function is through score-matching, see for example Hyvärinen] (2055) [Song and Kingmal [2021] Wenliang et al. (019
Score matching relies on the following acore-matching (SM) objective [Hyvärinen] 2005).
J(pl Ep,
[| ||Vlog p(æ) - V log q(x)
(10)

---

Algorithm Estimating the conditional probability via summary statistics
Input:
Generator G; summary statistics t(-); number of samples N from G; re-sample size B
Procedure:
Generate samples fy...... yr from G.
2: Generate coordinate index sample fi......i)
(yy-"))))e HE(N via the score- 3: For is E [m]] I E [M] estimate q(2(1)) ((2(6)))((=-1)) from samples {3/ (6)) matching objective in Eq. [0]
Output: (, (t000((((((0)))) Vi [m].
which is particularly useful for unnormalised models such as EBMs. Additional details are included in Appendix E Often score matching estimators can be shown to be consistent, see for example Song et al. 2020 Proposition 3.1 proven in Appendix B gives theoretical guarantees for the consistency of general form of Stein operator estimation, as tollows.
Proposition 3.1. Suppose that for i [m], is a consistent estimator of the univariate score function s(9) T() be Stein operator for the univariate differentiable probability distribution Q of the generalised density operator form Eq. 8 Let
T - g +
and
Āg
Σ T I
m
Then (1)) is consistent estimator for T1), and À is consistent estimator of A.
Non-parametris Stein operators with summary statisties In practice, the data y(-1) RTT-1 can be high dimensional, e.g. image pixels, and the observations can be sparse. Thus, estimation of the conditional distribution can be unstable exponentially large sample size is required. Motivated by Xu and Reinert [2021] and Xu and Reinert [2022]. we use a low-dimensional measurable non-trivial summary statistic and the conditional distribution of the data given i. as new target distributions. Heuristically, if two distributions match, then so do their conditional distributions.
Thus, the conditional distribution (A) is replaced by Q!") (A) ( Al)(=(-))) Setting t(x(-0) = z(-²) replicates the actual conditional distribution. We denote the univariate score function of (z]t(x(( ()) by ( (x](((<<(-))) or by st (x) when the context clear. The summary statistics t(x(-0) can be univariate or multivariate, and they may attempt to capture useful distributional features. Here we consider univariate summary statistics such as the sample mean.
Taking these component-dependen conditional Stein operators results in the Stein operator in general to be no longer equal to the Langevin Stein operator of the corresponding conditional multivariate distribution. Thus, Proposition D.I (presented in the Appendix) no longer applies.
The non-parametric Stein operator enables the construction of Stein-based statistics based on Eq. 7wth (i) estimated score functions $2 using generated samples from the model G, as shown in Algorithm The re-sampled non-parametric Stein operator is
1 Σ B
1 Σ B
(BC-To + +
A =
IN -
(11)
Non- parmmmrttt kernel Stein discrepancy Using the non -on-mmmmetrii Stein operator in Eq I define the corresponding non-parametric Stein discrepancy (NP-KSD) employing the Stein discrepancy notion in Eq.3land choosing as set of test functions the unit ball of the RKHS. Similarly to Eq. 4 define the NP-KSD with summary statistic t
NP-KSD,
sup (NB,(N)
E
(12)
A similar quadratic form as in Eq. 5 applies to give
BB (P-K)D;(GlIp) = Le [4 (x, x)],
(13)
5

---

where 0 (x((0,)) =
)), ( 1),, 1
The empirical estimate is
NP-KSD, (GIl)) n
R ( mz}], [JC(m]
Σ
(14)
where S = z,, ~ p. Thus, NP- allows the computation between set of samples and a generative model, enabling the quality assessment of synthetic data generators for implicit models.
The relationship between NP-KSD and KSD is clarified in the following result; we use the notation $r.v = (z). i E [m]). Here we set
1 m
m
KSDI( Ez. (AAktt ), A,k(z,))
with Aig(z)
T 21
921
(z10)
(15)
(x]((x((11)) More details in Eq. and following Eq. 8 Tye O2(-0 = 22 (1) -z( (1) (
(8)
about the interpretation of this quantity are given in Appendix B.
Theorem 3.2. Assume that the score function estimator vector $ ($I.N: 1....,m) mm) is
asymptotically normal with mean O and covariance matrix N-15.g. Then NP-KSD converges in probability to KSD (4e llp) at rate at least min(B- ..N )).
The proof of Theorem 3.2 which is found in Appendix B also shows that the distribution
NP-KSD;
(G pG - KSD (p) involves a mixture of normal variables. The assumption of asymp- totic normality for score matching estimators is often satisfied, see for example (Song al. [2220).
Model assessment with NP-KSD Given an implicit generative model G and set of observed samples S [Zi..... ). we aim to assess whether the observed samples S are consistent with the hypothesis that they were generated from G. This test assumes that samples generated from follow some (unknown) distribution y and that the samples S are generated according to some (unknown) distribution p. The null hypothesis is Ho p = q while the alternative is H1 p f q. We note that the observed sample size is fixed.
NP-KSD testing procedures NP-KSD can be applied for testing the above hypothesis using the testing procedure outlined in Algorithm 2 In contrast to the KSD testing procedure in Section 2
the NP-KSD test in Algorithm i Monte Carlo based test [Xu and Reinert 2021 2022 Schrab
it al. 2022] for which the null distribution is approximated via samples generated from G instead
of the wild bootstrap procedure as in Chwialkowski et al.) [2014). The reasons for employing the
Monte Carlo testing strategy instead of the wild-bootstrap are 1). The non parammaetttiic Stein operator depends on the random function S so that classical results for V-statistics convergence which assume that the sole source of randomness is the bootstrap may not apply 2). While the wild-bootstrap is asymptotically consistent as observed sample size n + oo, it may not necessarily control the type-I error in non-asymptotic regime where n is fixed. More details can be found in Appendix F.
Here we note that any test which is based on the summary statistic will only be able to test for a distribution up to equivalence of their distributions with respect to the summary statistic t; two distributions P and Q are equivalent w.r.t. the summary statistics if P(X]+(X)) Thus the null hypothesis for the NP-KSD test is that the distribution is equivalent to P with respect to Hence, the null hypothesis specifies the conditional distribution, not the unconditional distribution.
Related works To assess whether an implicit generative models can generate samples that are significantly consistent with the desired data model, several hypothesis testing procedures have been studied. Jitkrittum et al. [2018) has proposed two kernel-based tests, called Relative Unbiased Mean Embedding (Rel-UME) test and Relative Finite-Set Stein Discrepancy (Rel-FSSD) test, for testing relative model goodness-of-fit that is, testing whether model S is a better fit to the observations than model R. While Rel-UME is applicable to implicit generative models, Rel-FSSD still requires explicit knowledge of the unnormalised density.
The idea for assessing sample quality for implicit generative models could be viewed very imbalanced two-sample problem, where samples generated from the implicit model are compared
AA KSD with random Stein kernel has been briefly discussed in Fernández et al. 2020 when the h, function requires estimation from relevant survival functions.

---

Algorithm 2 Assessment procedures for implicit generative models
Input:
Observed sample set {z 1......, Zn the generator G and generated sample size N;
estimation statistics t; RKHS kernel K; re-sampling size B; bootstrap sample size b, confidence
level a:
1: Estimate (( (t2(")(((())) based on Algorithm
1
2: Uniformly generate re- re-sampling index
i , bB) from [m], with replacement.
3: Compute r = NP-KSD (3e; 5) in Eq. (14.
4: Simulate S, (y',.... yn for 2 E (6] from G.
5: Compute 7. NP-KSD St) in again with index re-sampling.
6: Estimate the empirical (1- a) quantile 71 via (T...... }.
Output: Reject the null hypothesis if r > 71-oi otherwise do not reject.
with the observed data. For general two-sample problems,
cattiuuu---ma---isepppeaaee (MMD)
could in principle also be applied for assessing sample qualities for implicit models. With an efficient choice of (deep) kernel. [iu et al. [2020] applied MMD tests to assess the distributional difference for image data, c.g. MNIST [LeCun al. 1998) v.s. digits image trained via deep convolutional GAN (DCGAN) [Rdford et al.) 2015|; CIFARIO (Krz 2009] v.s. CIFARIO.1 [Recht et all, 2019 However, we note that the MMD-based two-sample approach for model assessment suffers from two major limitations. Firstly, MMD tests [Gretton et al.) 2012a] require a balanced sample size for two sets of samples; if nı and n2 are the sizes for each sample set, then MMD tests have well-controlled type-I error when n R T/2. In the model assessment setting, we may observe a small fixed sample set n while being able generate N >> n samples from the generator. The second limitation concerns the idea to use matched sample size in order to enjoy the theoretical guarantees which are available for MMD tests, in which case may only generate (n)) samples from the generator to balance the observed sample size. As such, the empirical estimator for MMD becomes less accurate, which would significantly reduce the test power for small n. This approach would not exploit the fact that the data generator can generate very large amount of samples if desired. Detailed discussions on MMD two-sample testing procedures can be found in Appendix F.T including Table showing uncontrolled type-I errors for imbalanced sample size.
4 Experiments
4.1 Baseline and competing approaches
We illustrate the proposed NP-KSD testing procedure with different choice of summary statistics. We denote by NP-KSD the version which uses the estimation of the conditional score, i.c. t(x(-1))-
K-nKDmeaann x (1; by NP-KSD leaammmann the version which uses conditioning on the mean statistics, i.e. ((x(-1))
Ejet 2651; and by NP-KSD G the version which fits a Gaussian model as conditional density
Two-sample testing methods can be useful for model assessment, where the observed sample set is tested against sample set generated from the model. In our setting where < N, we consider a consistent non-asymptotic MMD-based test, MMDAgg (Schrab et al., 2021], as our competing approach; see Appendix [ for details on computing MMDAgg statisties and discussions on its
comparison with the MMD tests [Gretton et al. 2012a
For synthetic distributions where the null models have explicit densities, we include the KSD goodness-of-fe testing procedure in Section 2 as the baseline. Gaussian kernels are used and the median heuristic [Gretton et al., 2007 is applied for bandwidth selection. As caveat, in view of [Gorham and Mackey 2015) when the kernel decays more rapidly than the score function grows,
$ 'NP-KSD for non-Gaussian densities generally mis-specified. We deliberately check this case to assess the robustness of the NP-KSD procedure under model mis-specification.
*We remark another desirable property for MMDAgg. MMDAgg able (implicitly) select the optimal kernel parameter to achieve better test power without data- vutaasplitiing, where data-splitting results into smaller sample size available for performing the test, thus reducing data efficiency. However, this property is not the reason we choose MMDAgg as our baseline here; we choose it because MMDAgg is non con-tsympt MMD-based test which is capable of handling the imbalanced sample situation, iee N.
7

---

NP-KSD
NP-KSD
NP-KSD
KSD
MMDAgg
1.0
Rejection rate
1.0
Rejection rate
1.0
Rejection rate
0.6
0.8
0.5-
Relection
0.6
0.4
0.4
04
0.3
0.0
0.0
Varlance perturbation
Generated sample size
Covariance perturbation
NP-KSD re-sample
(a) GVD: n 100
(b) GVD:o 0.4
(c) MoG: n = 200
(d) MoG: m = 40
Figure 1: Rejection rates of the synthetic distributions: test level 0.05; 100 trials per round of experiment; 10 rounds of experiment are taken for average and standard deviation; bootstrap sample size 6 = 500; m for (a) and (b); 6 for (c); n 100, per 0.5 for (d).
then identifiability of a: through a KSD method may be guaranteed. We note that MMD tests do not have controlled type-I error when n < N, thus are not suitable in this setting. While MMD-base methods compare and criticise two generative models have been explored [Lloyd and Ghahramani 2015, (Sutherland et al., 2017]. MMD tests [Gretton et al. 2012a] are therefore not included in the comparison list. Detailed discussions and illustrations are found in Appendix F.1
4.2 Experiments on synthetic distributions
Gaussian Variance Difference (GVD) We first consider a standard synthetic setting, studied in Jitkrittum et al. [2077) in which the null distribution is multivariate Gaussian with mean zero and identity covariance matrix. The alternative is set to perturb the the diagonal terms of the covariance matrix, iec the variances, all by the same amount.
The rejection rate against the variances perturbation is shown Figure I(a) From the result, we that all the tests presented have controlled type-I error. For all the tests the power increases with increased perturbation. NP-KSD and NP-KSD mean outperform the MMDAgg approach. Using the mean statisties, NP-KSD mean is having slightly higher power than KSD. The mis-specified NP-KSD G has lower power, but is still competitive to MMDAgg
The test power against the sample size N generated from the null model is shown in Figure I(b) generated samples are used as another sample set for the MMDAgg two-sample procedure, while used for estimating the conditional score for NP- JP-KSD-basse increases, the power of MMDAgg increases more slowly than that of the NP-KSD-based methods, which achieve maximum test power in the presented setting. The OP-KSD-based tests tend to have lower variability of the test power, indicating more reliable testing procedures than MMDAgg.
The
methods. As the generated sample size
Mixture of Gaussian (MoG) Next, consider as difficult problem that the null model is two-component mixture of two independent Gaussians. Both Gaussian components have identity covariance matrix. The alternative is set to perturb the covariance between adjacent coordinates.
The rejection rates against this perturbation of covariance terms are presented Figure [(c] The results show controlled type-I error. The NP-KSD and NP-KSD mean tests have better test power compared to KSD and MMDAgg, although NP-KSD has slightly higher variance. Among the NP-KSD tests, the smallest variability is achieved by NP-KSD _eaan. For the test with = 40, we also vary the re-sample size As shown Figure T(d) the variability of the average test power increased slightly. From the result, also that for B 20 m/2 the test power is already competive compared to B- 40. Additional experimental results including computational runtime and training generative models for synthetic distributions are included in Appendix C
4.3 Applications to deep generative models
For real-world applications, we assess models trained from well-studied generative modelling proce- dures, including Generative Adversarial Network (GAN) [Goodfellow et al. 2014] with multilayer perceptron (MLP), Deep Convolutional Generative Adversarial Network (DCGAN) [Radford et al.) 2015). and Variational Autoencoder (VAE) [Kingma and Welling 2013). We also consider Noise

---

GAN MLP
DCGAN
VAE
NCSN
Real
NP-KSD NP-KSD _
1.00
0.92
1.00
1.00
0.03
1.00
1.00
1.00
1.00
0.01
MMDAgg
1.00
0.73
0.93
1.00
0.06
Table 1: Rejection rate for MNIST generative models.
DCGAN
NCSN
CIFARTO. 10.1
Real
NP-KSD NP-KSD m
0.68
0.73
0.92
0.06
0.74
0.81
0.96
0.02
MMDAgg
0.48
0.57
0.83
0.07
Table 2: Rejection rate for CIFARI0 generative models.
Conditional Score Network (NCSN) [Song and Ermon 2020 which is a score-based generative modelling approach, where the score functions are learned [Song and Ermon 2019] to performed annealed Langevin dynamics for sample generation. We also denote Real as the scheme that generates samples randomly from the training data, by using a held-out set of samples from the real data, and which hence can be viewed as an implicit generator of the null distribution.
MNIST Dataset This dataset contains 28 x 28 grey-scale images of handwritten digits [LeCun al. 1998 It consist of 60, 000 training samples and 10, 000 test samples. Deep generative models in Table are trained using the training samples. We assess the quality of these trained generative models by testing against the true observed MNIST samples (from the test set). Samples from both distributions are visually illustrated in Figure 3 in Appendix C For our experiments, 600 samples are generated from the generative models and 100 samples are used for the test;the samples are then down-sampled into 7 x images, as in /Schrab et al. [2021]. 'he test level is 0.05. From Table [ we see that all the deep generative models have high rejection rates. Testing with the Real scheme has controlled type-I error. Thus, NP-KSD detects that the "real" data are a true sample set from the underlying dataset, but often rejects the synthetic data generators.
CIFARIO Dataset This dataset contains 32 x 32 RGB coloured images (Krizhessky 2009 It consist of 50, 000 training samples and 10, 000 test samples. Deep generative models in Table 2 are trained using the training samples, and test samples are randomly drawn from the test set. Samples are illustrated in Figure in Appendix C We also compare with the CIFARIO.1 dataset [Recht et al., 2018 which is created to differ from CIFARIO to investigate generalisation power of training classifiers For our experiments, 800 samples are generated from the generative models and 200 samples are used for the test; the samples are then down-sampled into 8 x 8 images, in similar fashion as in Schrab et al. [202]] The test level is a = 0.05. Table shows higher rejection rates for NP-KSD tests compared to MMDAgg, echoing the results for synthetic distributions. The trained DCGAN generates samples with lower rejection rate in the CIFARIO dataset than in the CIFAR10. dataset. We also see that the score-based NCSN has higher rejection than the non-score-bate based DCGAN, despite NP-KSD being a score-based test. The distribution difference between CIFARIO and CIFAR10.1 can be well-distinguished from the tests. Testing with the Real scheme again has controlled type-I error.
5 Conclusion and future directions
Synthetic data are in high demand, for example for training ML procedures; quality is important. Synthetic data which miss important features in the data can lead to erroneous conclusions, which for example in medical applications could be fatal, and loan applications could be detrimental to personal or business development. NP-KSD provides a method for assessing synthetic data generators
http:///wwaca.......ataa/a///tta datasets. MNIST .html
CIFAR10 html

---

which comes with theoretical guarantees. Our experiments on synthetic data have shown that NP-KSD achieves good test power and controlled type-I error. On real data, NP-KSD detects samples from the true dataset. That none of the classical deep learning methods used in this paper has a satisfactory rejection rate indicates scope for further developments synthetic data generation.
Future research will assess alternatives to the computer-intensive Monte Carlo method for estimating the null distribution, for example through adapting wild-bootstrap procedures. It will explore alternative choices of score estimation as well as of kernel functions.
Finally, some caution is advised. The choice of summary statistic may have strong influence on the results and a classification based on NP-KSD may still miss some features. Erroneous decisions could be reached when training classifiers. Without scrutiny this could lead to severe consequences for example in health science applications. Future work will devote more attention on analysing the choice of summary statistic. Yet NP-KSID is important step towards understanding black-box data generating methods and thus understanding their potential shortcomings.
Acknowledgements
The authors thank the anonymous reviewers for their constructive comments and suggestions to improve the paper. G.R. and W.X. acknowledge the support from EPSRC grant EP/T01 84451. G.R also supported in part by EPSRC grants EP/W037211/1, EP/V056883/1, and EP/R018472/L
References
AD Barbour and LHY Chen. An Introduction to Stein's method. Lecture Notes Series. Institute for Mathematical Sciences. National University of Singapore, 4, 2005.
Alessandro Barp, Francois-Xavice Briol, Andrew Duncan, Mark Girolami, and Lester Mackey. Minimum Stein discrepancy estimators. In Advances in Neural Information Processing Systems, pages 12964-12976, 2019.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference Learning Representations, 2018.
Christopher M Bishop and Nasser M Nasrabadi. Pattern Recognition and Machine Learning. Springer, 2006.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In International Conference on Machine Learning, pages 2606-2615. PMLR, 2016.
Kacper P Chwialkowski, Dino Sejdinovic, and Arthur Gretton. A wild bootstrap for degenerate kernel tests. Advances in Neural Information Processing Systems, pages 3608-3616, 2014.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1): $3-65,22 2018.
Tamara Fernández, Wenkai Xu, Mare Ditzhaus, and Arthur Gretton. A kernel test for quasi- independence. Advances in Neural Information Processing Systems, 33:15326-15337 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jackson Gorham and Lester Mackey. Measuring sample quality with Stein's method. In Advances in Neural Information Processing Systems, pages 226-234, 2015.
Jackson Gorham and Lester Mackey. Measuring sample quality with kernels. In International Conference on Machine Learning, pages 1292-1301. PMLR, 2017.
Jackson Gorham, Anant Raj, and Lester Mackey. Stochastic Stein discrepancies. Advances Neural Information Processing Systems, 13:17931-17942, 2020.
10

---

A. Gretton, K. Fukumizu, C.H. Teo, L. Song, B. Schölkopf, and A.J. Smola. A kernel statistical test of independence. In Advances in Neural Information Processing Systems, pages 585-592, 2008.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the wwssssapllpppbbeeee In Advances in Neural Information Processing Systems, pages 513-520, 2007.
Arthur Gretton, Kenji Fukumizu, Zaid Harchaoui, and Bharath K Sriperumbudur. A fast, consistent kernel two-sample test. In Advances in Neural Information Processing Systems, pages 673-681, 2009.
kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012a. Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems, pages 1205-1213, 2012b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two icme-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Susan Holmes and Gesine Reinert. Stein's method for the bootstrap. In Stein's 's Method, volume 46, pages 93-133. Institute of Mathematical Statistics, 2004.
Aapo Hyvärinen. Estimation of normaazme statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709 2005.
Wittawat Jitkrittum, Zoltán Szabó, Kacper P Chwialkowski, and Arthur Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems, pages 181-189, 2016.
Wittawat Jitkrittum, Wenkai Xu, Zoltán Szabó, Kenji Fukumizu, and Arthur Gretton. A linear-time time kernel goodness-of-fit test. Advances in Neural Information Processing Systems, pages 262-271, 2017.
Wittawat Jitkrittum, Heishiro Kanagawa, Patsorn Sangkloy, James Hays, Bernhard Schölkopf, and Arthur Gretton. Informative features for model comparison. In Advances in Neural Information Processing Systems, pages 808-819, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141226990 2014.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv preprint 11X. 2013.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
Yann LeCun, Lawrence D Jackel, Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Urs A Müller, Eduard Säckinger, Patrice Simard, and Vladimir Vapnik. Learning algorithms for classification: comparison handwritten digit recognition. Neural networks: the statistical mechanics perspective, 261(27)) 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228-23324 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based based learning. Predicting Structured Data, 1(0), 2006.
A. J. Lee. U-Statistics: Theory and Practice. CRC Press, 1990.
Christophe Ley, Gesine Reinert, and Yvik Swan. Stein's method for comparison of univariate distributions. Probability Surveys, 14:1-52, 2017.

---

Chun-Liang Li, Wei- Wei-Cheng Chang, Yu Cheng, Yiming Yang. and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. Advances in Neural Information Processing Systems, 30, 2017.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J Sutherland. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning, pages 6316 6326. PMLR, 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. In Advances In Neural Information Processing Systems, pages 2378-2386, 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kemnllized Stein discrepancy for goodness-of-fh tests.
In International Conference on Machine Learning, pages 276-284, 2016.
James Robert Lloyd and Zoubin Ghahramani. Statistical model criticism using kernel two sample tests. In NeurlPS, pages 829-837, 2015.
Guillaume Mijoule, Gesine Reinert, and Yvik Swan. Stein's 's density method for multivariate continu- ous distributions. arXiv preprint arXiv: wwt.::2:01000000 10/09507, 2021.
Frédéric Ouimet. General formulas for the central and non-central moments of the multinomial 2021.
distribution. Stats,
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 151.0044,, 2015.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 10 classifiers
generalize to cifar-10? arXiv preprint arXiv: 1r000:1800000451,, 2018
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference Machine Learning, pages 5389-5400.
PMLR, 2019.
Gesine Reinert. Three general approaches to Stein's method. An introduction to Stein's 's method, 4: 183-221, 2005.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in Neural Information Processing Systems, 29, 2016.
Antonin Schrab, Ilmun Kim, Mélisande Albert, Béatrice Laurent, Benjamin Guedj, and Arthur Gretton. MMD aggregated two-sample test. arXiv preprint arXiv:2110.1503,, 2021.
Antonin Schrab, Benjamin Guedj, and Arthur Gretton. KSD aggregated goodness-of-te test. arXiv preprint ar arXiv:2202.0088 2022.
David W Scott and Stephan R Sain. Multidimensional density estimation. Handbook of statistics, 24: 229-261, 2005.
Raghav Singhal, Xintian Han, Saad Lahlou, and Rajesh Ranganath. Kernelized complete conditional stein discrepancy. arXiv preprint UrXivv/904.000 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pages 11895-11907 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems, 13:12-2812 2020.
Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021.
Yang Song. Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
12

---

Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12(Jul): 2389-24100 2011.
Charles Stein. A bound for the error in the normal approximation to the distribution of sum of dependent random variables. In Proceedings of the Sixth Berkeley Symposium on Mathemati- cal Statisties and Probability, Volume 2: Probability Theory. The Regents of the University of California, 1972.
Danica Sutherland, Hsiao-Yi Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alexander! J Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In ICLR (Poster), 2017.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844 2015.
Iya Tolstikhin, Bharath Sriperumbudur, and Krikamol Muandet. Minimax estimation of kernel mean embeddings. The Journal of Machine Learning Research, 18(1):3002-3048, 2017.
Aad W Van der Vaart. Asymptotic Statisties, volume 3. Cambridge University Press, 2000.
Dilin Wang, Zhe Zeng, and Qiang Liu. Stein variational message passing for continuous graphical models. In International Conference on Machine Learning, pages 5219-5227. PMLR, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4):600-6) 612, 2004.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pages 68 1-6-8. Citeseer, 2011.
Li Wenliang, Danica J Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kernels for exponential family densities. In International Conference on Machine Learning, pages 6737-6746.
PMLR, 2019.
Wenkai Xu. Standardisation-funeee kernel Stein discrepancy (Sf-KSD): A unifying approach for non-parametric goodness-of of-fit testing. In International Conference on Artificial Intelligence and Statistics, 2022.
Wenkai Xu and Gesine Reinert. A Stein goodness-offtest for exponential random graph models. In International Conference Artificial Intelligence and Statistics, pages 415-423. PMLR, 2021.
Wenkai Xu and Gesine Reinert. AgraSSt: Approximate graph Stein statistics for interpretable assessment of implicit graph generators. arXiv preprint arXiv:2203.03677, 2022.
Yuhao Zhou, Jiaxin Shi, and Jun Zhu. Nonparametric score estimators. In International Conference Machine Learning, pages 11513-11522 PMLR, 2020.
Jingwei Zhuo, Chang Liu, Jiaxin Shi. Jun Zhu, Ning Chen, and Bo Zhang. Message passing Stein variational gradient descent. In International Conference on Machine Learning, pages 6018-6027. PMLR, 2018
13


Reviews:
[{"review_id": "rCZy6idlYox", "reviewer": "Reviewer_qNmx", "summary": "The paper presents a kernel Stein discrepancy based test for black box generative models.\nSince the normal KSD cannot be applied, as the true score function of the black box model is unknown, the authors proposed a KSD variant where this unknown score function is estimated from samples from the model.\nThe resulting hypothesis test then is whether a given dataset that was used to fit the generative model is distributed according to the estimated score function of that model.\nThe score estimation is achieved via estimating a component wise distribution conditioned on all other components, through of summary statistics.\nSome consistency results are provided.\nAn experimental evaluation on toy data and and simple real datasets shows some benefits against the original KSD test where applicable, and one MMD based baseline otherwise.", "questions": "A high level question is: Does it make sense to asses the quality of generative models via a hypothesis test? What do we really learn if we e.g. fail to reject the null hypothesis? What do we learn if we can reject it? It shouldn't actually be a surprise that it is easy for most statistical tests to reject the null. Some of the references I put above go further and explain what features in the data lead to the rejection. Something like this could be used to improve a generative model, but that is actually not done in this paper.\n", "limitations": " ", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A well written paper that addresses an important issue.\nThe proposed methodology is based around a few nice tricks on estimating the conditional distributions and carried our in a thorough manner.\nIn the current form of the paper, however, it is not clear whether the proposed methodology is really as useful as the authors claim. The experimental evaluation is not thorough, and lacks baselines and comparisons to alternative strategies.\n\nA few concrete points:\n* There is no like to like comparison with an MMD based method. Only MMDAgg as a very specific case; the point of MMDAgg is that it can select a kernel without having to sacrifice samples, but in the case of a generative model, we can generate many samples so it doesn't seem to be a well fitting baseline). What about e.g. a plain MMD test with the median heuristic, or with a learnt optimal kernel? What about e.g. the linear time tests that learn feature locations (E.g. https://arxiv.org/abs/1605.06796).\n* It is actually not clear what is gained compared to an MMD based test. This surely depends on the number of samples drawn from the generator, as well as computational costs. See also question below. An empirical exploration of the performance of the various approaches as a function of drawn samples and compute would be very helpful. The authors only make some vague comments on this.\n* It is not clear why we could not just use score matching to learn the score using a powerful neural network instead of the conditional distribution approach? This would allow using many samples from the generator as well.\n* There are no empirical explorations of how the proposed score estimation approach scales in any direction. As that is crucial for this method to work, and since it is also a kind of harsh approximation, that weakens the paper. Related: what is the impact of the NN architecture for the score estimation in this case.\n\nMissing paper references that are highly relevant, as the problem of assessing performance of generative models was first discussed here (among others)\n* Statistical Model Criticism using Kernel Two Sample Tests by LLoyd et al\n* Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy by Sutherland et al\n\nMinor:\n* l97 The KSD was originally proposed in Chwialkowski et al 2016 and Liu at al 2016, not Gorham Mackey 2017\n* l109 The wild bootstrap is only really used if there is correlation to break (e.g. for time series data). Otherwise plain permutation test is sufficient and has no free parameters\n* l198 the normality result of Song is in a different context. Could you at least empirically explore this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eRdRs3IXR5A", "reviewer": "Reviewer_P9SF", "summary": "The paper considers the problem of assessing implicit generative models using kernelised Stein discrepancy (KSD).\nThe paper introduces a non-parametric (NP) Stein operator to allow implicit models to be used in KSD (which wasn't possible due to the fact KSD requires at least the unnormalised density of the model).\nResults on both synthetic and real datasets compared to MMD (which also only needs samples to compute) show that the proposed NP-KSD method has a better test power than MMD.", "questions": "- In algorithm 1, are we effectively learning the score function so to estimate KSD?\n  - Is this what the author(s) mean(s) in Line 232-234 when criticising MMD?\n- In algorithm 2, do we effectively use more samples from G compared to MMD?", "limitations": "Limitations and potential negative societal impact are discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- The paper proposes a novel method that extends KSD to implicit models.\n- The paper is well written and technical. The paper is not hard to follow.\n- The paper is a solid contribution to better access the generation quality of generative models. This is becoming more important as synthetic data generation is seen as an important tool to solve privacy.\n- The results look promising. It would be interesting to include a neural sampler trained by MMD, which would be an even stronger indication of the stronger test power of NP-KSD.\n\nCons\n- Discussion compared to MMD is not enough; see my Qs below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "82E1jiFV4b", "reviewer": "Reviewer_MBab", "summary": "This paper proposes an adaptation of kernelized Stein discrepancy, called non-parametric KSD (NP-KSD), for assessing the implicit generative models. In particular, the author considered a scenario that (1) we do not have access to the (unnormalized) densities of the generative model; (2) the data dimension is large; (3) fixed number of true observations. To enable the assessment of this type of model, the author proposed a non-parametric Stein operator based (1) conditional distributions with summary statistics; (2) resampling the conditional dimensions; (3) score matching for gradient estimation. Together with the kernel trick and Monte Carlo based goodness-of-fit test, it results in the NP-KSD method. \n\n ", "questions": "The questions are mainly the summary of what I have described in Weakness.\n\n**Major**:\n1. The motivation for using the goodness-of-fit test should be clearer.  E.g. advantages compared to two-sample test? Also, the implicit model has two application scenarios: (1) sampler, where we want to generate samples from a target distribution and (2) data synthesis, where we want to generate observations that mimic the real data samples, like GAN. From the context of this paper, it seems that it focuses on (2) instead of (1). It would be better to clarify this. \n2. Do we have theoretical guarantees when NP-KSD =0 $\\Rightarrow$ p=q? Does it only hold for equivalence class? What is the impact of equivalence class on the evaluation of the implicit model?\n3. The concern about the novelty, as detailed in the weakness section. Particularly, if I understood correctly, the main framework of the proposed method is based on goodness-of-fit test but with estimated gradients from score matching. Those ideas are not new. So maybe consider elaborating more on the novel contributon?\n4. For synthetic distribution experiments, why the proposed method is better than KSD? From my understanding, NP-KSD is an approximation of KSD. \n\n**Minor**:\n1. $\\mathbb{R}^m$ in line 86 should be $\\Omega_q$?\n2. In the related work section, maybe consider adding citations regarding various types of Stein discrepancies and their usage in generative models. \n3. I wonder why the proposed method is better than two-sample tests. Since the gradient estimation used in NP-KSD is based on the generated samples, it means NP-KSD does not have access to more information compared to the two-sample test. Where does this performance increase come from?", "limitations": "The author mentioned NP-KSD can distinguish distributions in equivalence classes. But the potential impact of the equivalence class to model evaluation can be discussed in more detail.  It would be great to consider some simple cases with simple summary statics like mean, and show some properties of the equivalence class. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n\nThe paper is clearly written and easy to follow. Although the motivation of the proposed method can be improved. \nThis paper aims to tackle an important problem: the evaluation of implicit models, which can have significant impacts to the current deep generative models. To sidestep the computational challenges, the author also provides several tricks to enable the NP-KSD. Theoretically, the author shows the consistency and convergence results of NP-KSD. \n\n**Weakness**:\n\nFirst, the motivation of the proposed method can be improved. For example, two-sample test is a typical method for evaluating implicit models. Why do we need goodness-of-fit test instead? Goodness-of-fit test is typically used when we have access to the density function, that is where its name comes from. Why do we want to use this for the implicit model with the gradient estimation? To me, it seems that we achieve what the two-sample test is designed for but with an alternative method. So a better motivation for goodness-of-fit test is needed. \n\nThere are several tricks used in deriving NP-KSD. The author has shown that NP-KSD satisfied Stein identity. However, a more interesting (or more important) aspect is to show when NP-KSD =0, it means p=q. Currently, I don't think it is true, especially with summary statistics and score matching. If not, then the goodness-of-fit test procedure will not be valid. The author mentioned equivalence class, but it is unclear how this will impact the evaluation of the implicit model. E.g. with a summary statistics, the distribution inside an equivalence class has similar visual quality?\n\nAnother concern is about its novelty. To me, the Stein operator \\Tau is defined similarly as the Langevin-Stein operator but with conditional distributions. This is equivalent to the original Langevin-Stein discrepancy. This idea has also been explored in paper [1] and the author should consider citing it. If so, the main novelty of NP-KSD is score-matching, resampling and summary statistics. However, they seem to be more like tricks rather than novel methodologies.  Although I agree the theoretical consistency and convergence results give some guarantees of using the resampling and score matching, which seems to be novel. \n\n[1] Singhal, R., Han, X., Lahlou, S., & Ranganath, R. (2019). Kernelized complete conditional Stein discrepancy. arXiv preprint arXiv:1904.04478.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept