Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work provides guarantees for goodness-of-fit tests (whether a given set of samples comes from a given null distribution) based on kernel stein discrepancies (KSD). They construct a test that allows for kernel selection to maximize power by using all data rather than using data split while maintaining the type I error level. In particular, the work extends the idea of aggregated tests already used for the two-sample problems via MMDs, and the independence problem via HSIC, to the goodness-of-fit setting via KSD. To estimate the test thresholds, they make use of a Monte Carlo strategy based on (a) parametric bootstrap (which requires samples from the null distribution p) and provides a non-asymptotic level, and power; and (b) wild bootstrap that provides the same guarantees asymptotically. 
strengths_and_weaknesses: + Their Theorem 3.1 characterizes the uniform separation rate between the null and the alternative densities so as to assert a certain power for a given test, and Theorem 3.3 extends it to multiple aggregated tests. The authors provide several numerical experiments that showcase that KSDAGG performs better than some other benchmarks.

+ Writing of the paper is very good, and the authors cover a fair amount of related work! 

- This work applies only to those settings where model density is known. 

For more comments, see questions.
questions: 1. Can the authors clarify what the constant C depends on in Theorem 3.1?

2. Corollary 3.4 has a logarithmic inflation factor in front of the sample size N (in the second term). If one uses an equal sample split to do kernel selection first and then test, isn't that factor just 2? If yes, then is KSDAGG better than sample split only if log(#kernels) <=2? If not, it would help to clarify what the rate for sample split is known to be.

3.  Given the non-transparent nature of the Stein operator, can the authors provide some examples of q's that satisfy theorem 3.1 / 3.3 for a given simple distribution p, say Gaussian distribution? The authors make a comment in l 175--176, but it would be helpful to provide concrete examples (also relevant for comment 2).

4. In Proposition 3.2, is there no requirement on B1, B2, and B3?

5. Can the authors state a mathematically precise asymptotic result for wild bootstrap? (Like how is the limit for N to infinity taken?) 

6. Is there no setting, where the KSD median would be competitive? It would be helpful to comment on the limitations of KSDAGG, namely when does KSDAGG either perform poorly or not better than the other sensible benchmarks. 

7. Do we really need to know the model density? Or just the gradient of log density, and a sampler of p for parametric bootstrap? (OPTIONAL: What if the sampler is approximate?)

8. Given the unknown nature of constants in the uniform separation rate, can the authors comment on how the power degrades when the separation deteriorates from the mentioned lower bound?

Minor comments: 

- To my understanding from the proof, Theorem 3.3 is effectively a corollary of Theorem 3.1 and a union bound; it would be helpful to provide that comment if true?

limitations: See questions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors propose a natural extension to the existing KSD goodness of fit tests, but allowing aggregate testing over multiple kernels (in particular, multiple kernel hyper-parameters) without requiring strategies such as data-splitting etc.    They achieve this by rescaling the critical value of each individual test by a positive weight, summing up to <= 1.  When the weights are equal to 1/N (where N is the number of tests), this is essentially a Bonferroni-type correction for aggregating finitely many independent tests.   

The authors provide an algorithm for performing this multiple testing protocol, based on wild-bootstrap or parametric bootstrap.  They then establish control over type II error through a uniform separation rate (USR) argument. 

Many of the arguments are similar to what is proposed in the preprint [Schrab, A., Kim, I., Albert, M., Laurent, B., Guedj, B., and Gretton, A. (2021). MMD aggregated two-sample test], which considers the similar case of MMD testing.  But in that situation things are more straightforward since terms within the USR argument can be bounded explicitly in terms of best approximation rates for densities with a given regularity, yielding minimax optimal rates, etc.     In this setting it is far less straightforward to obtain similar bounds.
strengths_and_weaknesses: Strengths:
* (Quality + Clarity) It is very well written.  The arguments are presented cleanly which made following the proofs quite easy.   
* (Significance) It tackles an important challenge which provides a principled alternative to the median heuristic or data-splitting approaches to addressing this problem.

Weaknesses:
 * (Originality) It is a bit incremental, and builds very obviously on previous work which tackled aggregate tests for MMD in the same light.
 * (Significance) While the uniform separation rates established in this paper are of theoretical interest, they aren't actionable in any manner due to the complexity of the Stein kernel.  
questions: 1. Do the authors have any intuition on the behaviour of the terms in the uniform separate rates established for the aggregate test in terms of dimension?  Even just heuristics or a case study on a Gaussian Stein kernel would be of interest.  
2. Do the authors have any recommendations on the choice of the weights?  This seems arbitrary and not really addressed anywhere in this work?  Is there any scenario where taking the sum to be < 1 make sense?   
3.  One often wishes to select a kernel over an infinite continuum family of parameters, and it's sometimes less obvious how to discretise this.  Are the authors aware of any strategies which would enable them to generalise to this setting.
limitations: I am satisfied with how the authors addressed the limitations of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new goodness-of-fit (GOF) testing method based on KSD, which sidesteps the challenge of selecting a single kernel for the test. Instead, the proposed method, KSDAGG, can aggregate tests with a collection of kernels so that it maximises the test power over these kernels. This is achieved by performing a GOF test with each kernel in the collection and rejecting the null hypothesis if any one of the tests rejects it. To ensure that the KSDAGG can still control the user-defined type-I error, the author proposed a bisection algorithm to select the proper test interval for each kernel in the collection. Theoretically, the author showed a condition for a uniform separation rate so that the KSDAGG can also control the type-II error. This condition shows that KSDAGG achieves the smallest uniform separation rate of the collection.

Empirically, the author applied KSDAGG to the kernel bandwidth selection problem and compared it with kernel selection baselines including data splitting, median heuristic and kernel selection with extra data. It shows that it achieves similar test power as the kernel selection with extra data on synthetic data and the best performance for MNIST normalising flow. 
strengths_and_weaknesses: **Strength**:
This paper takes an alternative point of view on the kernel selection problem, where it extends the recently proposed two-sample test aggregation method to goodness-of-fit with Stein discrepancy. 
The presentation clarity is reasonable but can be much improved. In terms of significance, I think the targeted community is generally limited but it is a nice addition to it. 

**Weakness**:
One of my confusions is its contribution. The author mentioned that the aggregation trick is not novel, which has already been proposed in the two-sample test. Thus, I fail to see the main contributions of KSDAGG. I saw the contribution section but I suggest the author summarise what are the key points and why they are novel. For example, why the extension to the GOF test is non-trivial (expand a bit on the challenge mentioned in related work?), what method do I use to solve this non-trivial extension, etc. So that the contribution and novelty are clearer.  

In terms of clarity, the background material to KSD is a bit dense. Since I have worked with the Stein method before, it is clear to me, but I can see that for a more general audience, it can be a bit difficult to understand. I suggest considering adding a more detailed background section in the appendix?
questions: 1. Like I have mentioned, the author should consider elaborating more on their novel contributions: why the extension to GOF is not trivial? Is it because of the derivation of the bound for uniform separation rate?

2. Another concern is the computational cost. It seems that with the aggregation trick, it performs a single test for each kernel in the collection. This means with a large collection (or the number of bootstrap samples), the computational cost can be much larger than a single test (with kernel selection). Can the author provide the performances with wall clock time? I am curious to see the trade-off. 

3. For a fixed set of weight $w_k$ and level $\alpha$, do we have a "fixed" choice of $u_\alpha$? Clearly, from the bisection algorithm, we won't have exactly the same $u_\alpha$, but they are closed to each other. So here "fixed" means a very distinct set of $u_\alpha$. Another question is can we select the best kernel using the KSDAGG. For example, if one kernel in the collection rejects the null hypothesis, does it mean this is the best kernel?

4. The introduction of KSD can be improved. For example, instead of directly introducing KSD and the complicated Stein kernel and Stein identity. Maybe consider adding a bit of intuition on Stein discrepancy. In line 87, consider adding one sentence to explain what is the consistency of the Stein test? Also for corollary 3.4, it seems that this is identical to theorem 3.3, which doesn't need to be re-introduced. Saving this space allows the author to expand a bit more on the background. 

5. For the synthetic experiment, why KSDAGG is better than KSD split extra data in RBM? It will be interesting to see the optimised bandwidth and the bandwidth of one that rejects the null in the collection. 
limitations: This paper does not have any negative societal impact. But this paper does not seem to discuss its limitations enough. For example, the trade-off between high computational cost and test power. It seems that this method cannot be applied to kernels that require training like the deep kernel.
review_title: null

Meta-review: metareview: The paper proposes a novel method of statistical tests with Kernel Stein Discrepancy, aggregating multiple tests with different kernels.  The method can avoid data splitting, which is commonly used to choose a kernel aiming at better power but may not be effective with a smaller sample size.  The paper gives theoretical analysis, and also experimental results outperforming other relevant methods.  The work gives solid theoretical and methodological advances in the field of kernel-based tests.  We think the work is worth being presented in NeurIPS. 
confidence: Certain
Decision: Accept