Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##  Summary of the paper
The authors proposed the `sliced version of the kernelized stein discrepancy and solved the collapsed problem for high dimensional GOF and particle based model learning.

## Strong and weak points of the paper
### Strong points
- Provided a novel slicing technique for KSD and provided its statistical estimator based on U-static.
- Experimental results supports that the proposed sliced KSD is very promising approach for high dimensional models.

### Weak points (Questions)
- I think the presentation should be modified so that the reader can easily understand the proposed methods. At least, the main algorithms should be presented in the main paper. For example, the algorithm for GOF test is not shown in the main paper but included in Appendix E, although it is the main algorithm of this work.
- I could not understand when I should use MAXSKSD-G, not MAXSKSD-RG although detailed discussion is shown in Appendix F. To me, based on the discussion of Appendix F, MAXSKSD-RG seems always be better than MAXSKSD-G. So don't we need MAXSKSD-G?
- (I might overlooked, but) the strategy on how to choose optimal G for GOF test is not shown in the main paper although it is expained in Appendix G. The explanation of how to tune G should be included in the main paper.

## Rating
- Clarity: For me, the main paper is not enough to understand the paper and the proposed method clearly. I needed to read the Appendix carefully. 
- Correctness: I did not check the proofs of each lemma in Appendix B.
- Novelty: The idea seems very interesting and important in the community.

## Comments and Questions
- Comments) I think, instead of Figure 1,  Appendix B.1 should be included in the main paper, which is very helpful to understand the overall strategy of the proposed discrepancy.  

- Minor comments ) 
I think Eq numbers in Figure 1 Left seems wrong, especially for SSD and max SSD. 
On page 17, appendix  B.2, the sentence above Eq.29 says, ``"~~~ as in Theorem 3~~~", I think this is a typo and should be replaced with "Lemma 3".
review_title: Proposed sliced kernelized stein discrepancy, which is useful for high dimensional models.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tries to solve the curse-of-dimensionality problem of KSD and corresponding mode-collapse problem of SVGD by projecting both the input and output of test function onto 1D slices. By doing so, the paper proposes the new discrepancies called SSD and maxSKSD, and a new variant of SVGD called S-SVGD. Experiments on goodness-of-fit test (synthetic high-dim Gaussian & RBM) and model learning (ICA on synthetic data & amortized SVGD on MNIST) are reported in the main body of the paper.

This paper is well motivated and the writing is good. Curse-of-dimensionality problem is common in kernel based methods like KSD and corresponding SVGD. To the best of my knowledge, the idea of 'slicing' is novel to solve the high dimensional problem of KSD and SVGD. However, the derivation and analysis (e.g., computational complexity, whether closed form or not, how to get it in each experiment, etc...) of the optimal test direction $g_r$ are missed in the main body of the paper, which is critical to evaluate the quality of the proposed maxSKSD and S-SVGD according to my understanding.

Pros:
1. The 'slicing' idea is novel to KSD and SVGD;
2. The paper is easy to follow in general, though some minor points need further improvement;
3. Experiments are conducted on various tasks and datasets, which provides a thorough comparison.

Cons:
1. Lack of analysis of the optimal test direction $g_r$. According to Eq. (6), it seems that $g_r \in R^D$ has no closed-form solution and corresponding $G \in R^{D\times D}$ in general.This raises several questions:

     1.1 In the high dimensional case (D >> 1),  $D\times D$ seems unacceptable. Is it possible to provide the empirical storage consumption of both S-SVGD and SVGD in experiments, especially for neural networks?

     1.2 Getting $g_r$ requires solving a maximization problem (see Algorithm 2 in the appendix), which introduces an additional inner loop for each iteration of S-SVGD. So what is the time complexity? Is it possible to provide the comparison of wall clock time of S-SVGD and SVGD in experiments?

     1.3 Does $g_r$ have closed-form solution in some special case, e.g., Gaussian?

2. The notation $f$ is used for denoting both $R^D\to R^D$ and $R^D \to R$ mapping, which is misleading. $f(\cdot;r,g):R^D \to R$ and $f_{rg}:R\to R$ are even more confusing. Besides, it is a little hard to understand how Eq. (5) can be derived based on Eq. (2).

review_title: Details of the optimal test direction are needed

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 

The paper proposes a new discrepancy between probability distributions called the sliced Kernelized Stein Discrepancy.  It is based on the idea of computing the standard KSD on random 1 dimensional projections and then average those. Some other proposed variants are based on optimizing over the projections.

The motivation behind this new discrepancy is to overcome the 'curse of dimensionality' that standard KSD suffers from. Thus the main selling point is that the sliced version has a much better behavior as the dimension increases.

The authors then propose to use the new divergence for two applications: goodness of fit testing and for models learning.

The robustness to increasing dimensions is illustrated through multiple experiments on both tasks and yields convincing results.

Strength:
	- The proposed framework is neat and the experiments are thorough and convincing with many additional discussions and results in the appendix. 
	- The proposed method is relatively easy to implement and seems to address several known limitations of KSG and in SVGD: the scaling with dimensions.
	
	

Weaknesses:
 The paper is a bit dense with many references to the appendix. However the main idea is clearly explained  and the advantage is clear both in terms of theory and throughout the experiments.
 
 
Questions:
	- In 4.1.1, the distributions p and q although high dimensional, they often have independent components. This might be very advantageous for the sliced version of the algorithm, especially when using a set of orthogonal projections for the projections $r$. What happens to the Null rejection rate when more dependence between the dimensions is introduced ? What is the exact parameter choice for the multivariate-t distribution, I couldn't find this in the appendix?

 Minor remarks.
 - A discussion on the limitations of existing methods KSD and SVGD could be useful as a transition from section 2 to 3 to motivate the slicing.
- Figure 1 is a bit dense especially with all the equations
- The subscript notation $f_{rg}$ is sometimes confusing as f depends on $r$ and $g$ only implicitly after optimizing the objective in 5. It might be worth either mentioning where this dependence comes from or even remove it.
- In the paragraph right after corollary 3.1. The authors mention a limitation of a particular version of Max Sliced KSD over the other but then refer to appendix F without really saying what this limitation is. It might be worth saying a little bit more about those limitations.
review_title: Interesting work, the proposed method seems to address several known limitations of the KSD and SVGD

Meta-review: This paper propses a slice method for approximaing the Kernel Stein Discrepancy, which has been popularly used for learning and inference with unnormalized density models.  The proposed method uses a finite set from the orthogonal bases for the slice to approximate the Stein Discrepancy.  The experimental results show that they outperform exsiting methods in high-dimensional cases in the applications of goodness-of-fit tests and learning of energy-based models.  

The proposed slice idea is novel and significant.  Especially, unlike sliced Wasserstein, the slices are taken from the limited number of vectors, which should be an advantageous feature of the method.  Experiments demonstrate clear advantages in high-dimensional cases, as expected.  The paper is worth accepting in ICLR. 
Decision: Accept (Poster)