Sliced Kernelized Stein Discrepancy
Authors: Wenbo Gong
Yingzhen Li
José Miguel Hernández-Lobato
Venue: ICLR 2021 Poster (2021)
Keywords: kernel methods
variational inference
particle inference

Abstract: Kernelized Stein discrepancy (KSD), though being extensively used in goodness-of-fit tests and model learning, suffers from the curse-of-dimensionality. We address this issue by proposing the sliced Stein discrepancy and its scalable and kernelized variants, which employs kernel-based test functions defined on the optimal one-dimensional projections. When applied to goodness-of-fit tests, extensive experiments show the proposed discrepancy significantly outperforms KSD and various baselines in high dimensions. For model learning, we show its advantages by training an independent component analysis when compared with existing Stein discrepancy baselines. We further propose a novel particle inference method called sliced Stein variational gradient descent (S-SVGD) which alleviates the mode-collapse issue of SVGD in training variational autoencoders.

Published as a conference paper at ICLR 2021
SLICED KERNELIZED STEIN DISCREPANCY
Wenbo Gong
Yingzhen Li
University of Cambridge wg2 wg2420cam. www.ccaaaaccuu...omo.o. ac.uk
Imperial College London y yingzhen. 1i@ imperialll...mmmmm . acc uk
José Miguel Herbández-Lobato University of Cambridge The Alan Turing Institute
jmh23300 cam.ccm..... ac. uk
ABSTRACT
Kernelized Stein discrepancy (KSD), though being extensively used in goodness-of- fit tests and model learning, suffers from the durre-of-iimensionaaity We address this issue by proposing the sliced Stein discrepancy and its scalable and kernel- ized variants, which employ kernel-based test functions defined on the optimal one-dimensional projections. When applied to goodness-of-fit tests, extensive experiments show the proposed discrepancy significantly outperforms KSD and various baselines in high dimensions. For model learning, show its advan- tages over existing Stein discrepancy baselines by training independent component analysis models with different discrepancies. We further propose novel particle inference method called sliced Stein variational gradient descent (S-SVGD) which alleviates the mode-collapse issue of SVGD in training variational autoencoders.
INTRODUCTION
Discrepancy measures for quantifying differences between two probability distributions play key roles in statisties and machine learning. Among many existing discrepancy measures, Stein discrepancy (SD) is unique in that it only requires samples from one distribution and the score function (i.e. the gradient up multiplicative constant) from the other (Gorham & Mackey, 2015). SD, a special case of integral probability metric (IPM) (Sriperumbudur et al., 2009), requires finding an optimal test function within given function family. This optimum is analytic when a reproducing kernel Hilbert space (RKHS) is used as the test function family. and the corresponding SD is named kernelized Stein discrepancy (KSD) (Liu et al., 2016; Chwialkowski et al., 2016). Variants of SDs have been widely used in both Goodness-of-fit (GOF) tests (Liu et al., 2016; Chwialkowski et al., 2016) and model learning (Liu& & Feng, 2016; Grathwohl et al., 2020; Hu et al., 2018; Liu & Wang, 2016).
Although theoretically elegant, KSD, especially with RBF kernel, suffers from the "curse- of-dimensionality" issue, which leads to significant deterioration of test power in GOF tests (Chwialkowski et al., 2016; Huggins & Mackey, 2018) and mode collapse in particle inference (Zhuo et al., 2017; Wang et al., 2018). A few attempts have been made to address this problem, however, they either are limited to specific applications with strong assumptions (Zhuo al., 2017; Chen & Ghattas, 2020; Wang et al., 2018) 8) or require significant approximations (Singhal et al., 2019). As an alternative, in this work we present our solution to this issue by adopting the idea of "slicing". Here the key idea is to project the score function and test inputs onto multiple one dimensional slicing directions, resulting
variant of SD that only requires to work with one-dimensional inputs for the test functions. Specifically, our contributions are as follows.
We propose a novel theoretically validated family of discrepancies called sliced Stein discrep- ancy (SSD), along with its scalable variant called max sliced kernelized Stein discrepancy (MaxSKSD) using kernel tricks and the optimal test directions.
A GOF test is derived based on an unbiased estimator of maxSKSD with optimal test directions. MaxSKSD achieves superior performance on benchmark problems and restricted Boltzmann machine models (Liu et al., 2016; Huggins & Mackey, 201 8).
WWor done Microsoft Research Cambridge

---

Published as a conference paper at ICLR 2021
We evaluate the maxSKSD in model learning by two schemes. First, train an independent component analysis (ICA) model in high dimensions by directly minimising maxSKSD, which results in faster convergence compared to baselines (Grathwohl et al., 2020). Further, we propose a particle inference algorithm based on maxSKSD) called the sliced Stein variational gradient descent (S-SVGD)
novel variant of the original SVGD (Liuu & Wang, 2016). It alleviates the posterior collapse of SVGD when applied to training variational autoencoders (Kingma & Welling, 2013; Rezende al., 2014).
BACKGROUND
2.1 KERNELIZED STEIN DISCREPANCY
For two probability distributions p and q supported on X S R with continuous differentiable densities p(x) and q(z), we define the score sp(x) V log p(x) and sa(x) accordingly. For test
function f X + R2D the Stein operator is defined as
Apf(z) sp(x)" f(x) + f()).
(1)
For a function fo : RDD + R. the Stein class F, of q is defined as the set of functions satisfying Stein's identity (Stein et al., 1972): [s, fz) f = O. This can be generalized to by letting Sa belongs the Stein class of for each i E D. Then the Stein discrepancy (Liu et al., 2016; Gorham & Mackey, 2015) is defined as
vector function IRD RD where f = (xx) (nn)
P) = sup Eg(Apf(x)] = sup (,)(s sy(x))) f(z)]. feJ,
(2)
Se..
When Fg is sufficiently rich, and vanishes at the boundary of X, the supremum obtained at f" (ax) sp(x:) 8x(2) with some mild regularity conditions on f (Hu et al., 2018). Thus, the Stein discrepancy focuses on the score difference of p and q. Kernelized Stein discrepancy (KSD) (Liu
al., 2016; Chwialkowski et al., 2016) restricts the test functions be in D-dimensional RKHS HD with kernel k to obtain an analytic form. By defining up(2, x') = (2)) 8p(x') (x'))(( x') 8p sp(x)"VV "are"VVakiie aitk x') "aa"""""" x') + Tr(V m.a: x²)) the analytic form of KSD is:
2
D²(q,p)=
sup /EHo,Ilffl EyLApf(x))]
E =otte)terr((y((022))
2.2 STEIN VARIATIONAL GRADIENT DESCENT
Although SD and KSD can be directly minimized for variational inference (VI) (Ranganath et al., 2016; Liu & Feng, 2016; Feng et al., 2017), Liu & Wang (2016) alternatively proposed novel particle inference algorithm called Stein variational gradient descen! (SVGD). It applies sequence of deterministic transformations to a set of points such that each of mappings maximally decreases the Kullback-Leibler (KL) divergence from the particles' underlying distribution q the target p.
To be specific, we define the mapping T'xx) : IR + RD as T(x) = x x+<p(x)) where & characterises the perturbations. The result from Liu & Wang (2016) shows that the optimal perturbation inside the RKHS is exactly the optimal test function in KSD.
+ eΦ(()) and (z)(z)) be the density of z T(x) when a ~ q(x). If the perturbation & is in the RKHS HD and 1| D(g,p), then the steepest descent directions ht is
Lemma 1. (Liu & Wang, 2016) Let T(x:)
2i and V V.K. [KL -D
Vv log (gg(())((z..) + Vak(a,))]
The first term in Eq. is called drift, which drives the particles towards mode of p. The second term controls the repulsive force, which spreads the particles around the mode. When particles stop moving, the KL decrease magnitude e DD2 p) is O, which means the KSD is zero and p a.c.
3 SLICED KERNELIZED STEIN DISCREPANCY
We propose the sliced Stein discrepancy (SSD) and kernelized version named maxSKSD. Theoreti- cally, we prove their correctness as discrepancy measures. Methodology-wise, we apply maxSKSD to GOF tests, and develop two ways for model learning.
2

---

Published as a conference paper at ICLR 2021
f I:R" -R" =5,4/680
oOT
SD
KSD
Equineenca Invert the comnation
[Fa3)
That statadic
gh
c ghhh E O.O
Dimeraiona
1:3" R F
b4 R
F No by F
Jripu for stasstes
SD/SSD
SSD
ass F ennily ()) (ou woofe)
Stop Redaas Dimoniton
SSD
SKSD 10
(Eq.)
reu
score
< 4g
Vi
ng th
go-1
e optimal
opeinel g
Step Hedaase Limenasnn
maxSSD (4
Tup
MaxSKSD (1)
1:2º 2
J D R
1.
g0-1
Rodon trandform
Figure (Left) The connections between SD, KSD and the proposed SSD family. (Right) The intuition of SSD. The rectangular boxes indicate what statistics the discrepancy wants to test. The circle represents the dimension of the test function required for such test. The double arrow means equivalence relations or invertable operations.
3.1 SLICED STEIN DISCREPANCY
Before moving to the details, we give a brief overview of the intuition on how to tackle the curse-of- fimensionality issue of SD (The right figure of Figure 1). For detailed explanation, refer to appendix B.1. This issue of Stein discrepancy (Eq.2) comes from two sources: the score function sp(z) and the test function f(x) defined on X' C RRP First, we notice that comparing sp and 8, is equivalent to comparing projected score 5 = 8 r and bg for all r E SD1 on an hyper-sphere (Green square in Figure 1 (Right)). This operation reduces the test function's 's output from R2 to IR (Green circle in Figure 1 (Right)). However, its input dimension not affected. Reducing the input dimension of test functions is non-trivial, as directly removing input dimensions results in the test power decrease. This is because less information is accessed by the test function (see examples in appendix B.1). Our solution to this problem uses Radon transform which is inspired by CT-scans. It projects the original test function (x) in Stein discrepancy (Bq. 2) (as an D - I mapping) to a group of - I functions along set of directions (g E SD-1). Then, this group of functions are used as the new test functions to define the proposed discrepancy. The invertibility of Radon transform ensures that testing with input in the original space Rº is equivalent to the test using group of low dimensional functions with input in RR. Thus, the above two steps not only reduce the dimensions of the test function's 's output and input, but also maintain the validity of the resulting discrepancy each step is either equivalent or invertible.
In detail, assume ((x),, and define the test functions [(; rg RDD - IR such that s(æ; r = feg o hy(x) = Srg (xy(xT g). where (.).) is the inner product with g and frg R - R. One should note that the r and g f(;r, g) should not just be treated as parameters in test function f. In fact, they like the index to indicate that for each pair of 9. we need a new f(:;r, g). i.e. new frg- which completely independent
distributions p and q supported RDD with differentiable densities p(æ)) and
more
other test functions. The proposed sliced Stein discrepancy (SSD), defined using two uniform distributions pr (r) and (g(g) over the hypersphere g n-1 is given by the following, with frg Fg meaning f(: (g))) Fgi
S(g.p) p) = -Epr
sup E,(s,(x))n) (x² g) r gV (x g)] Er
(5)
We verify the proposed SSD is valid discrepancy measure, namely, S(q.r) = iff. q = p a.e.
Theorem 1. (SSD Validity) If assumptions 1-4 in appendix are satisfied, then for two probability distributions and S(g. p) 2 0, and S(g. p) = 0 if and only if = 9 a.e.
Despite this attractive theoretical result, SSD is difficult to compute in practice. Specifically, the expectations and g can be approximated by Monte Carlo but this typically requires a very
3

---

Published as a conference paper at ICLR 2021
large number of samples in high dimensions (Deshpande et al., 2019). We propose relax such limitations by using only a finite number of slicing directions r from an orthogonal basis O, of IR,, e.g. the standard basis of me-hoot vectors, and the corresponding optimal test direction gr for each We call this variant maxSSD, which is defined as follows and validated in Corollary 1.1:
Sman (g.p) = Σ sup
(2) (2" gr) + r" V.Ve g. (x' g.)]. (6)
rEO, J.s, E.
Corollary 1.1. (maxSSD) Assume the conditions in Theorem 1, then Smaz (g. 0 iff q a.e.
3.2 CLOSED FORM SOLUTION WITH THE KERNEL TRICK
The optimal test function given and is intractable without further assumptions on the test function families. This introduces another scalability issue as optimizing these test functions explicitly can be time consuming. Fortunately, Assume for each test function frn E Hrgs where Hrg is a scalar-valued RKHS equipped with kernel (ax,, x'; r, r. = (a² 9; x² g) that satisfies assumption 5 in appendix and frg (ag( g) = (frgs kry(x' g, H). We define the following quantities:
can apply the kernel trick to obtain its analytic form.
Sprig( (r. ) (p)))) (x g. + r" gV oT (arg 9,
h (x, y) -5) TT g. g.v"" g) g)) + T V (x g. gn g)+
r" (x) "² (rg g. g) + (r" V g.v (an
g. y. 00 g).
(8)
The following theorem describes the optimal test function inside SSD (5). and maxSSD (E).
Theorem 2. (Closed form solution) Ey Ehphh (2, x)] < Jc, then
Dig (q.P) =
sup
(2 + +r gV (x g)]l
(9)
= [| (x) (+),, = (an
Next, we propose the kernelized version of SSD with orthogonal basis O,, called SKSD. Theorem 3. (SKSD discrepancy) For two probability distributions p and q. given assumptions 1,2 and in appendix A and E.G Dippin (2., x)] < do for all r and g, we define SKSD as
SKo(g.P) = Σ /
Po D2 P2 (..
(10)
rEO, JDD
which is equal to 0 if and only if == a.e.
Following the same idea of maxSSD (Fq.6), suffices to use optimal slice direction 9r for each O,, resulting slicing matrix G @Dx(D-1), We name this discrepancy as maxSKSD, or maxSKSD-g g when we need to distinguish it from another variant described later.
Corollary 3.1. (maxSKSD) Assume the conditions in Theorem 3 are satisfied. Then
SK (g.P)
Σ reO,-
sup D2 rgr (g.P)
(11)
equal if and only p = q a.e.
Figure (Left) clarifies the connections between the mentioned discrepancies. We emphasise that using a single projection g in maxSKSD may be insufficient when no single projected feature x" g is informative enough describe the difference between p and q. Instead, in maxSKSD, for each score projection r E O,. we have a corresponding gr- One can also use the optimal r to replace the summation over O,,, which provides additional benefits in certain GOF tests. We call this discrepancy maxSKSD-rg, and its validity can be proved accordingly. Interestingly, in appendix G, we show under certain scenarios maxSKSD-g can have inferior performance due to the noisy information provided by the redundant dimensions. Further, we show that such limitation can be efficiently addressed by using masSKSD-rg.
4

---

Published as a conference paper at ICLR 2021
Kernel choice and optimal G RBF kernel with median heuristics is common choice. However, better kernels, c.g. deep kernels which evaluate a given kernel on the transformed input Φ(x), might be preferred. It is non-trie trivial to directly use such kernel on SKSD maxSKSD. We propose an adapted form of (q.(1)) to incorporate such kernel and maintain its validity. We include the details in appendix D and leave the experiments for future work.
The quality of sliced direction G is crucial for the performance of both maxSKSD-g or maxSKSD-rg. Indeed, it represents the projection directions that two distributions differ the most. The closed-form solutions of G is not analytic in general, in practice, finding the optimal G involves solving other difficult optimizations as well (projection and test function frg). For the scope this work, we obtained G by optimizing maxSKSD-g g or maxSKSD-rg using standard gradient optimization, e.g. Adam, with random initialization. Still in some special cases (e.g. p. are full full-factorized), analytic solutions of optimal G exists, which is further discussed in appendix E.
3.3 APPLICATION OF MAXSKSD
Goodness-of-it Test Assume the optimal test directions gr E G' are available, maxSKSD (I)) can then be estimated using U-statistics (Hoeffding, 1992; Serflngg 2009). Given i.i.d. samples {æ; 1==1 q, have unbiased minimum variance estimator:
Σ SK mar (g.P) = N(N - rEO, ISIIJEN
Σ
hp.rgg (xi, T.))-
(12)
The asympiotic behavior of the estimator is analyzed in appendix F.I. We use bootstrap (Liu et al., 2016; Huskova Janssen, 1993; Arcones & Gine, 1992) to determine the threshold for rejecting the null hypothesis as indicated in algorithm 1. The bootstrap samples can be calculated by
SK
Σ ISIJIN
(w.T
1 Man
Σ rEO,
hypraa (2 xz)
(13)
where (wi Multi(N,
W m=1
are random weights
drawn
from
multinomial
distributions
Algorithm 1: GOF Test with maxSKSD U-statistics
Input
:Samples { {x, AN ~ q(x), score function (x), Orthogonal basis Or, optimal test direction 9, for each O,, kernel function krgs significant level a, and bootstrap
sample size M.
Hypothesis Ho: p q v.s. H1: 9 F
Compute SK manaa( p) using U-statistic (12(
Generate M bootstrap samples (SK 1M M using Eq.(13):
Reject null hypothesis Ho if the proportion SK SK maz (q, is less than a.
Model Learning The proposed maxSKSD can be applied to model learning in two ways. First, it can be directly used as a training objective, in such case q is the data distribu- tion and p is the model to be learned, and the learning algo- rithm performs min, 5KmakkKa (). The second model learn- ing scheme is to leverage the particle inference for latent vari- ables and train the model parameters using an EM-like (Demp- ster al., 1977) algorithm. Similar to the relation between SVGD and KSD, we can derive a corresponding particle in- ference algorithm based on maxSKSD, called sliced-SVGD (S-SVGD). In short, we define a specific form of the pertur- bation as (bx) = L (z""g.),.. ........., ., Φpp (x" gD)]" and modify the proofs of Lemma accordingly. The resulting S-SVGD algorithm uses kernels defined on one dimensional projected samples, which sidesteps the vanishing repulsive force problem of SVGD in high dimensions (Zhuo et al., 2017; Wang et al., 2018). We illustrate this in Figure 2 by estimating the variance
Variance Estimation
10
0.8
Varience
SVGC- 1:00-100
PaudivI1--110 120
10: SVCC-200 230
Dimersions
Figure Estimating the average variance of p(æ)) = A I) across dimensions using SVGD particles. SVGD-50 means the variance estimated using 50 samples.
5

---

Published as a conference paper at ICLR 2021
Cauchy RFF Gouss
FS5D Off
L1 IMQ
MMD
FS5D
KSD
Null
Laclace
Mutic
Difusson
0.25
0.75
8.10
0.50
0.50
8.06
0.23
0.23
0.25
100
0.00
20
40
80
60
0.00
100
0.00
40
60
50
200
Divergence Valte
Divergence Vitue
Civergence Valuan
Divergence Varue
1.5
0.5
Demppeae Vake
0.002
0.4
1.0
0.2
0.000
0.5
0.2
-0.022
60
100
0.0
29
40
62
0.0
100
0.0
40
60
30
100
Ormrrskoss
menolass
Dimins ons
Dimendoors
Figure 3: Each column reports GOF test results for a different alternative hypothesis, with the upper panel showing the rejection rate of the Null hypothesis and the lower panel showing the discrepancy value averaged over all trials. Both quantities are plotted w.r.t. the number of dimensions.
of a standard Gaussian with the particles obtained by SVGD or S-SVGD (see appendix J.1). We see that as the dimension increases, SVGD severely under-estimates the variance of p. while the S-SVGD remains robust. Furthermore, its validity is justified since in such case the KL gradient equals to maxSKSD which a valid discrepancy. Readers are referred to appendix F.2 for the derivations. We also give an analysis of their memory and computational cost for both GOF and model learning in appendix H.
EXPERIMENTS
4.1 GOODNESS OF FIT TEST
We evaluate maxSKSD (11)(11)) for GOF tests in high dimensional problems. First, we demonstrate its robustness to the increasing dimensionality using the Gaussian GOF benchmarks (Jitkrittum et al., 2017; Huggins& & Mackey, 2018; Chwialkowski et al., 2016). Next, we show the advantage of our method for GOF tests 50-dim Restricted Boltzmann Machine (RBM) (Liu et al., 2016; Huggins & Mackey, 2018: Jitkrittum et al., 2017). We included in comparison extensive baseline test statitics for GOF test: Gaussian Cauchy random Fourier features (RFF) (Rahimi & Recht, 2008), KSD with RBF kernel (Liu et al., 2016; Chwialkowski et al., 2016), finite set Stein discrepancy (FSSD) with random or optimized test locations (Jitkrittum et al., 2017), random feature Stein discrepancy (RFSD) with 1.2 SechExp and L1 IMQ kernels (Huggins & Mackey, 2018), and maximum mean discrepancy (MMD) (Gretton et al., 2012) with RBF kernel. Notice that we use gradient descent obtain the test directions gr (and potentially the slicing directions r) for Eq.(1
4.1.1 GOF TESTS WITH HIGH DIMENSIONAL GAUSSIAN BENCHMARKS
We conduct 4 different benchmark tests with p = _N I): (1) Null test: = (2) Laplace: q(æ) = I[ La factorized multivariate-t with degrees of freedom, mean and scale 1. In order to match the variance of and q, we change the variance of to 5 5-2: (4) Diffusion: g(x) = N NO, Σn) where the variance of 11- is 0.3 and the rest is the same in I. For the testing setup, we set the significance level a = 0.05. For FFSD and RFSD, we use the open- aeen-soureed code from the original publications. We only consider maxSKSD-g here as it already performs nearly optimally. We refer to appendix I.1 for details.
00, 1/ 1/v2) with mcan/variance matched to p. (3) Multivariate-t: is fully
Figure shows the GOF test performances and the corresponding discrepancy values. In summary, the proposed maxSKSD outperforms the baselines in all tests, where the result is robust to the increasing dimensions and the discrepancy values match the expected behaviours.
Null The left-most column in Figure shows that all methods behave as expected as the rejection rate is closed to the significance level, except for RFSD with L2 SechExp kernel. All the discrepancy values oscillate around 0, with the KSD being less stable.
Laplace and Multivariate-t The two middle columns of Figure 3 show that maxSKSD-g achieves nearly perfect rejection rate consistently as the dimension increases, while the test power for all

---

Published as a conference paper at ICLR 2021
Table 1: Test NLL for different dimensional ICA with different objective functions. The above results are averaged over 5 independent runs of each methods.
Method
Dimension
D =
= 40
= 60
D 80
D = 100
200
KSD LSD
-10.23
-15.98
-34.50
-56.87
-86.09
-116.51
-329.49
-10.42
-14.54
-17.16
-15.05
-12.39
-5.49
46.63
maxSKSD
-10.45
-14.50
-17.28
-15.70
-11.91
-4.21
47.72
baselines decreases significantly. For the discrepancy values, similar to the KL divergence between q and p. maxSKSD-g linearly increases with dimensions due to the independence assumptions..
Diffusion This is a more challenging setting since and q only differ in one of their marginal distributions, which can be easily buried in high dimensions. As shown in the rightmost column of Figure 3, all methods failed in high dimensions except maxSKSD-g, which still consistently achieves optimal performance. For the discrepancy values, we expect positive constant due the one marginal difference between and q. Only maxSKSD-g behaves as expected as the problem dimension increases. The decreasing value at the beginning is probably due the difficulty in finding the optimal direction g in high dimensions when the training set is small.
4.1.2 RBM GOF TEST
We demonstrate the power maxSKSD for GOF tests on RBMs, but we now also include results for maxSKSD-rg. We follow the test setups in Liu et al. (2016); Jitkrittum et al. (2017); Huggins& & Mackey (2018) where different amounts of noise are injected into the weights form the alternative hypothesis The samples are drawn using block Gibbs sam- plers. Refer to appendix I.2 for details. Figure 4 shows that maxSKSD based methods dominate the baselines, especially with maxSKSD-rg significantly outperforming the others. At perturbation level 0.01, maxSKSD-rr achieves 0.96 rejection rate, while oth- ers are all below 0.5. This result shows the advan- tages of optimizing the slicing directions r.
Test Power for RBM
1.0
Caudty
0.8
Spe
FSSD Rand
9.6
L1 MQ
L7 SeehExp
0.4
MWD
KSD
mac3K3DDrg
0.2
mac3(500g
0.005
0.015
0.030
0.035
Figure 4: RBM GOF Test with different levels of perturbation noise. The black vertical line indicates the perturbation level at 0.01.
4.2 MODEL LEARNING
We evaluate the efficiency of maxSKSD-basee algorithms in training machine learning models. First, independent component analysis (ICA) which is often used benchmark for evaluating training methods for -baeed model (Gutmann & Hyvärinen, 2010; Hyvärinen, 2005; Ceylan & Gutmann, 2018). Our approach trains the ICA model by directly minimizing maxSKSD. Next, we evaluate the proposed S-SVGD particle inference algorithm, when combined with amortization (Feng et al., 2017; Pu al., 2017), in the training of variational autoencoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) on binarized MNIST. Appendix J.5 also shows superior results for S-VVGD when training a Bayesian neural network (BNN) on UCI datasets (Dua & Graff, 2017).
4.2.1 ICA
ICA consists of a simple generative process ~ Lap(0. 1) and x = Wz, where the model parameters are a non-singular matrix W E DDD The log density for x is p(z) = log Pz (W-1 + C, where the normalization constant C' can be ignored when training with Stein discrepancies. We train the models on data sampled from a randomly initialized ICA model and evaluate the corresponding test log likelihoods. We compare maxSKSD with KSD and the state-of-the-e the-art LSD (Grathwohl et al., 2020). For more details on the setup, we refer the reader to appendix J.2.
Table shows that both maxSKSD and LSD robust to increasing dimensions, with maxSKSD being better when D is very large. Also at = 200, maxSKSD converges significantly faster than LSDD) (see Figure 10 in appendix J.3). This faster convergence is due to the closed-form solution for the optimal test functions, whereas LSD requires adversarial training. While KSD is also kerne-bbased, suffers from the sarre-of-ddimesioone the proposed maxSKSD can successfully avoid the problems of KSD with high dimensional data.
and fails train the model properly for D) 20. Instead
7

---

Published as a conference paper at ICLR 2021
Table 2: Average log likelihood on first 5, 000 test images for different D of latent dimensions.
Table 3: Label entropy and accuracy for imputed images.
Method
Latent Dim
Method
Entropy
Accuracy
D=16
D=32
D-48
D=64
Vanilla VAE
-91.50
-90.39
-90.58
-91.50
Vanilla VAE 0.297
0.718
SVGD VAE
-88.58
-90.43
-93.47
-94,88
SVGD VAE
0.538
0.691
S-SVGD VAE
-89.17
-87.55
-87.74
-87.78
S-SVGD VAE
0.542
0.728
4.2.2 AMORTIZED SVGD
Finally, we consider training VAEs with implicit encoders on dynamically binarized MNIST. The decoder is trained as in vanilla VAEs, but the encoder is trained by amortization (Feng et al., 2017; Pu al., 2017), which minimizes the mean square error between the initial samples from the encoder, and the modified samples driven by the SVGD/S-SVGD dynamics (Algorithm 3 in appendix J.4).
We report performance in terms of test log- likelihood (LL). Furthermore consider an imputation task, by removing the pixels in the lower half of the image and imputing the missing values using (approximate) posterior sampling from the VAE models. The performance is measured in terms of imputation diversity and correctness, using label entropy and accuracy. For fair comparisons, we do not tune the coefficient of the repulsive force. We refer to appendix J.4 for details.
Table 2 reports the average test LL. We observe that S-SVGD is much more robust to the increasing latent dimensions compared SVGD. To be specifie, with D 16, SVGD performs the best where S-SVGD performs slightly worse than SVGD. However, when the dimension starts to increase, LL of SVGD drops significantly. For D = 64, a common choice for latent space, performs even significantly worse than vanilla VAE. On the other hand, S-SVGD is much more robust. Notice that the purpose of this experiment to show compare their robustness instead of achieving the state-of- the-art performance. Still the performance can be easily boosted, e.g. running longer S-SVGD steps before encoder update, we leave it for the future work.
For the imputation task, we compute the label entropy and accuracy for the imputed images (Table 3). We observe S-SVGD has higher label entropy compared to vanilla VAE and better accuracy compared to SVGD. This means both S-SVGD and SVGD capture the muli- muli-modality nature of the posterior compared to uni-modal Gaussian distribution. However, high label entropy itself may not be a good indicator for the quality of the learned posterior. One can think of counter-example that the imputed images are diverse but does not look like any digits. This may also gives a high label entropy but the quality of the posterior poor. Thus, use the accuracy indicate the "correctness" of the imputed images, with higher label accuracy meaning the imputed images are closed to the original image. Together, good model should give higher label entropy along with the high label accuracy. We observe S-SVGD has more diverse imputed images with high imputation accuracy.
4.3 SUMMARY OF THE EXPERIMENTS IN APPENDIX
We present further empirical results on GOF tests and model learning in the appendix to demonstrate the advantages of the proposed maxSKSD. As summary glance of the results:
In appendix G, we analyse the potential limitations of maxSKSD-g and show that they can be mitigated by maxSKSD-rg, i.e. optimising the slicing direction r;;
In appendix I.3, successfully apply maxSKSD to selecting the step size for stochastic gradient Hamiltonian Monte Carlo (SGHMC) (Chen et al., 2014);
In appendix J.5, we show that the proposed S-SVGD approach out-performs the original SVGD on Bayesian neural network regression tasks.
5 RELATED WORK
Stein Discrepancy SD (Gorham & Mackey, 2015) and KSD (Liu et al., 2016; Chwialkowski et al., 2016) are originally proposed for GOF tests. Since then research progress has been made to improve these two discrepancies. For SD, LSD (Grathwohl et al., 2020; Hu et al., 2018) is proposed to increase the capacity of test functions using neural networks with L2 regularization. On the other hand, FSSD (Jitkrittum al., 2017) and RFSD (Huggins & Mackey, 2018) aim reduce
8

---

Published as a conference paper at ICLR 2021
the computation cost of KSD from O(n²) to O(n) where is the number of samples. Still the curse-of dimeessoimmm issue remains to be addressed in KSD, and the only attempt so far (to the best of our knowledge) is the kernelized complete conditional Stein discrepancy (KCC-SD (Singhal et al., 2019)), which share our idea of avoiding kernel evaluations on high dimensional inputs but through comparing conditional distributions. KCC-SD requires the sampling from g (oraaaar-in ad=d), which often needs sigmificant approximations in practice due to its intractability. This makes KCC-SD less suited for GOF test due to estimation quality in high dimensions. On the other hand, our approach does not require this approximation, and the corresponding estimator is well- behaved asymptotically.
Wasserstein Distance and Score matching Sliced Wasserstein distance (SWD) (Kolouri et al., 2016) and sliced score matching (SSM) (Song et al., 2019) also uses the "slicing" idea. However, their motivation is to address the computational issues rather than statistical difficulties in high dimensions. SWD leveraged the closed-form solution of 1D Wasserstein distance by projecting distributions onto ID slices. SSM uses Hutchson's 's trick (Hutchinson, 1990) to approximate the trace of Hessian.
Particle Inference Zhuo al. (2017); Wang et al. (2018) proposed message passing SVGD to tackle the well-known mode collapse problem of SVGD using local kernels in the graphical model. However, our work differs significantly both theory and applications. Theoretically, the discrepancy behind their work is only valid if p and g have the same Markov blanket structure (refer to Section 3 in Wang et al. (2018) for detailed discussion). Thus, unlike our method, no GOF test and practical inference algorithm can be derived for generic cases. Empirically, the Markov blanket structure information is often unavailable, whereas our method only requires projections that can be easily obtained using optimizations. Projected SVGD (pSVGD) is very recent attempt (Chen & Ghattas, 2020) which updates the particles in an adaptively constructed low dimensional space, resulting in biased inference algorithm. The major difference compared to S-SVGD is that our work still updates the particles in the original space with kernel being evaluated in 1D projections. Furthermore, S-SVGD can theoretically recover the correct target distribution. There is no real-worls experiments provided in (Chen & Ghattas, 2020), and a stable implementation of PSVGD is non-trivial, so we did not consider PSVGD when selecting the baselines.
CONCLUSION
We proposed sliced Stein discrepancy (SSD), as well as its scalable and kernelized version maxSKSD, to address the surse-f-ddimensionee
issues in Stein discrepancy. The key idea is to project the score function on one-dimensional slices and define (kernel-based) test functions on one-dimensional projections. We also theoretically prove their validity as discrepancy measure. We conduct extensive experiments including GOF tests and model lcarning to show maxSKSD's s improved performance and robustness in high dimensions. There are three exciting avenues of future research. First, although validated by our theoretical study in appendix D, practical approaches to incorporate deep kernels into SSD remains an open question. Second, the performance of maxSKSD crucially depends on the optimal projection direction, so better optimization methods to efficiently construct this direction needed. Lastly, we believe "slicing" is a promising direction for kernel design increase the robustness to high dimensional problems in general. For example, MMD can be easily extended to high dimensional two-sample tests using this kernel design trick.
REFERENCES
Miguel A Arcones and Evarist Gine. On the bootstrap of and v statisties. The Annals of Statistics, pp. 655-674, 1992.
Adi Bsn-Isr The Sfaang-oof-varabble formula using matrix volume. SIAM Journal on Matrix Analysis and Applications, 2101)3300-31,, 1999.
Ronald N Bracewell. Strip integration in radio astronomy. Australian Journal of Physics, 9(2): 198-217, 1956.
Claudio Carmeli, Ernesto De Vito, Alessandro Toigo, and Veronica Umanitá. Vector valued repro- ducing kernel hilbert spaces and universality. Analysis and Applications, 1901): 2010.
Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. arXiv preprint arXiv: /0td://8000000 036603 2018

---

Published as a conference paper at ICLR 2021
Peng Chen and Omar Ghattas. Projected Stein variational gradient descent. arXiv:2002.03469, 2020.
arXiv preprint
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691, 2014.
Andreas Christmann and Ingo Steinwart. Support vector machines.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. JMLR: Workshop and Conference Proceedings, 2016.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological)) 39(1): 1-22, 1977.
Ishan Deshpande, Ziyu Zhang, and Alexander Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3483-3491, 2018.
Ishan Deshpande, Yuan-Time Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander Schwing. Max-sliced Wasserstein distance and its use for gans. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10648-10656, 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://ccc/t.
ics. eec.ieedu/
Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized Stein variational gradient descent. arXiv preprint 17Xiv:1707.06620 2017.
Jackson Gorham and Lester Mackey. Measuring sample quality with Stein's method. In Advances in Neural Information Processing Systems, pp. 226 234,,44,22 2015.
Will Grathwohl, Kuan-Chieh Wang, Cutting out the middle-man: Training and evaluating energy-based models without sampling. arXiv preprint arXiv:2002.05616 2020.
Jacobsen, David Duvenaud, and Richard Zemel.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-777 2012.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Wassily Hoeffding. A class of statistics with asymptotically normal distribution. In Breakthroughs in Statistics, pp. 308-334. Springer, 1992.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. arXiv preprint arXiv: 100.v08810.03545, 2018.
Jonathan Huggins and Lester Mackey. Random feature Stein discrepancies. In Advances in Neural Information Processing Systems, pp. 1899-1909, 2018.
Marie Huskova and Paul Janssen. Consistency of the generalized bootstrap for degenerate u-statistics. The Annals of Statistics, pp. 1811-1823, 1993.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Statistics-Simulation and Computation, 19(2):433 450, 1990.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709, 2005.
10

---

Published as a conference paper at ICLR 2021
Wittawat Jitkrittum, Wenkai Xu, Zoltán Szabó, Kenji Fukumizu, and Arthur Gretton. A linear-time kernel goodness-off of-fit test. In Advances in Neural Information Processing Systems, pp. 262-271, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14tiv:/412.6900 2014.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 14, 2013.
Soheil Kolouri, Yang Zou, and Gustavo K Rohde. Sliced Wasserstein kernels for probability distributions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5258-5267, 2016.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced Wasserstein distances. In Advances in Neural Information Processing Systems, pp. 261-272, 2019.
Qiang Liu and Yihao Feng. Two methods for wild variational inference. arXiv preprint 1rxiv:1612.00081. 2016.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances in neural information processing systems, pp. 2378-2386, 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized Stein discrepancy for goodness-of-fit tests. In International conference on machine learning, pp. 276-284, 2016.
Yuchen Pu, Zhe Gan, Ricardo Henao, Chunyuan Li, Shaobo Han, and Lawrence Carin. VAE learning via Stein variational gradient descent. In Advances in Neural Information Processing Systems, pp. 4236-4245, 2017.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, pp. 1177-1184, 2008.
Rajesh Ranganath, Dustin Tran, Jaan Altosaar, and David Blei. Operator variational inference. In Advances in Neural Information Processing Systems, pp. 496-504, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and
approximate inference deep generative models. arXiv preprint 4rXiv:1401.4082. 2014.
Robert J Serfling. Approximation theorems of mathematical statistics, volume 162. John Wiley & Sons, 2009.
Raghav Singhal, Xintian Han, Saad Lahlou, and Rajesh Ranganath. Kernelized complete conditional Stein discrepancy. arXiv preprint arXiv: /90404440 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. arXiv preprint arXiv:1905.07000 07088, 2019.
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics, \phi-divergeenes and binary classification. arXiv preprint arXiv:0901 2698,, 2009.
Charles Stein, Persi Diaconis, Susan Holmes, Gesine Reinert, et al. Use of exchangeable pairs in the analysis of simulations. Stein's Method. pp. 1-25. Institute of Mathematical Statistics, 2004.
Charles Stein et al. A bound for the error in the normal approximation to the distribution of sum of dependent random variables. In Proceedings of the Sixth Berkeley Symposium Mathematical Statistics and Probability, Volume 2: Probability Theory. The Regents of the University of California, 1972.
Dilin Wang. Zhe Zeng, and Qiang Liu. Stein variational message passing for continuous graphical models. In International Conference Machine Learning, pp. 5219-5227. PMLR, 2018.

---

Published as a conference paper at ICLR 2021
Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the quantitative analysis of becoder-based generative models. arXiv preprint arXi::611104273, 2016.
Jingwei Zhuo, Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang. Message passing Stein variational gradient descent. arXiv preprint arXiv:1711.04425, 2017.
12


Reviews:
[{"review_id": "U-TQ5IP6o6P", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to solve the curse-of-dimensionality problem of KSD and corresponding mode-collapse problem of SVGD by projecting both the input and output of test function onto 1D slices. By doing so, the paper proposes the new discrepancies called SSD and maxSKSD, and a new variant of SVGD called S-SVGD. Experiments on goodness-of-fit test (synthetic high-dim Gaussian & RBM) and model learning (ICA on synthetic data & amortized SVGD on MNIST) are reported in the main body of the paper.\n\nThis paper is well motivated and the writing is good. Curse-of-dimensionality problem is common in kernel based methods like KSD and corresponding SVGD. To the best of my knowledge, the idea of 'slicing' is novel to solve the high dimensional problem of KSD and SVGD. However, the derivation and analysis (e.g., computational complexity, whether closed form or not, how to get it in each experiment, etc...) of the optimal test direction $g_r$ are missed in the main body of the paper, which is critical to evaluate the quality of the proposed maxSKSD and S-SVGD according to my understanding.\n\nPros:\n1. The 'slicing' idea is novel to KSD and SVGD;\n2. The paper is easy to follow in general, though some minor points need further improvement;\n3. Experiments are conducted on various tasks and datasets, which provides a thorough comparison.\n\nCons:\n1. Lack of analysis of the optimal test direction $g_r$. According to Eq. (6), it seems that $g_r \\in R^D$ has no closed-form solution and corresponding $G \\in R^{D\\times D}$ in general.This raises several questions:\n\n     1.1 In the high dimensional case (D >> 1),  $D\\times D$ seems unacceptable. Is it possible to provide the empirical storage consumption of both S-SVGD and SVGD in experiments, especially for neural networks?\n\n     1.2 Getting $g_r$ requires solving a maximization problem (see Algorithm 2 in the appendix), which introduces an additional inner loop for each iteration of S-SVGD. So what is the time complexity? Is it possible to provide the comparison of wall clock time of S-SVGD and SVGD in experiments?\n\n     1.3 Does $g_r$ have closed-form solution in some special case, e.g., Gaussian?\n\n2. The notation $f$ is used for denoting both $R^D\\to R^D$ and $R^D \\to R$ mapping, which is misleading. $f(\\cdot;r,g):R^D \\to R$ and $f_{rg}:R\\to R$ are even more confusing. Besides, it is a little hard to understand how Eq. (5) can be derived based on Eq. (2).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q0cRD4WgEtE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##  Summary of the paper\nThe authors proposed the `sliced version of the kernelized stein discrepancy and solved the collapsed problem for high dimensional GOF and particle based model learning.\n\n## Strong and weak points of the paper\n### Strong points\n- Provided a novel slicing technique for KSD and provided its statistical estimator based on U-static.\n- Experimental results supports that the proposed sliced KSD is very promising approach for high dimensional models.\n\n### Weak points (Questions)\n- I think the presentation should be modified so that the reader can easily understand the proposed methods. At least, the main algorithms should be presented in the main paper. For example, the algorithm for GOF test is not shown in the main paper but included in Appendix E, although it is the main algorithm of this work.\n- I could not understand when I should use MAXSKSD-G, not MAXSKSD-RG although detailed discussion is shown in Appendix F. To me, based on the discussion of Appendix F, MAXSKSD-RG seems always be better than MAXSKSD-G. So don't we need MAXSKSD-G?\n- (I might overlooked, but) the strategy on how to choose optimal G for GOF test is not shown in the main paper although it is expained in Appendix G. The explanation of how to tune G should be included in the main paper.\n\n## Rating\n- Clarity: For me, the main paper is not enough to understand the paper and the proposed method clearly. I needed to read the Appendix carefully. \n- Correctness: I did not check the proofs of each lemma in Appendix B.\n- Novelty: The idea seems very interesting and important in the community.\n\n## Comments and Questions\n- Comments) I think, instead of Figure 1,  Appendix B.1 should be included in the main paper, which is very helpful to understand the overall strategy of the proposed discrepancy.  \n\n- Minor comments ) \nI think Eq numbers in Figure 1 Left seems wrong, especially for SSD and max SSD. \nOn page 17, appendix  B.2, the sentence above Eq.29 says, ``\"~~~ as in Theorem 3~~~\", I think this is a typo and should be replaced with \"Lemma 3\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "HWeMKPXxSB0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nThe paper proposes a new discrepancy between probability distributions called the sliced Kernelized Stein Discrepancy.  It is based on the idea of computing the standard KSD on random 1 dimensional projections and then average those. Some other proposed variants are based on optimizing over the projections.\n\nThe motivation behind this new discrepancy is to overcome the 'curse of dimensionality' that standard KSD suffers from. Thus the main selling point is that the sliced version has a much better behavior as the dimension increases.\n\nThe authors then propose to use the new divergence for two applications: goodness of fit testing and for models learning.\n\nThe robustness to increasing dimensions is illustrated through multiple experiments on both tasks and yields convincing results.\n\nStrength:\n\t- The proposed framework is neat and the experiments are thorough and convincing with many additional discussions and results in the appendix. \n\t- The proposed method is relatively easy to implement and seems to address several known limitations of KSG and in SVGD: the scaling with dimensions.\n\t\n\t\n\nWeaknesses:\n The paper is a bit dense with many references to the appendix. However the main idea is clearly explained  and the advantage is clear both in terms of theory and throughout the experiments.\n \n \nQuestions:\n\t- In 4.1.1, the distributions p and q although high dimensional, they often have independent components. This might be very advantageous for the sliced version of the algorithm, especially when using a set of orthogonal projections for the projections $r$. What happens to the Null rejection rate when more dependence between the dimensions is introduced ? What is the exact parameter choice for the multivariate-t distribution, I couldn't find this in the appendix?\n\n Minor remarks.\n - A discussion on the limitations of existing methods KSD and SVGD could be useful as a transition from section 2 to 3 to motivate the slicing.\n- Figure 1 is a bit dense especially with all the equations\n- The subscript notation $f_{rg}$ is sometimes confusing as f depends on $r$ and $g$ only implicitly after optimizing the objective in 5. It might be worth either mentioning where this dependence comes from or even remove it.\n- In the paragraph right after corollary 3.1. The authors mention a limitation of a particular version of Max Sliced KSD over the other but then refer to appendix F without really saying what this limitation is. It might be worth saying a little bit more about those limitations.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propses a slice method for approximaing the Kernel Stein Discrepancy, which has been popularly used for learning and inference with unnormalized density models.  The proposed method uses a finite set from the orthogonal bases for the slice to approximate the Stein Discrepancy.  The experimental results show that they outperform exsiting methods in high-dimensional cases in the applications of goodness-of-fit tests and learning of energy-based models.  

The proposed slice idea is novel and significant.  Especially, unlike sliced Wasserstein, the slices are taken from the limited number of vectors, which should be an advantageous feature of the method.  Experiments demonstrate clear advantages in high-dimensional cases, as expected.  The paper is worth accepting in ICLR. 
Decision: Accept (Poster)