Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This papers studies the problem of estimating KL divergence from samples via variational methods with neural network discriminators. They propose a novel way to construct the discriminator in some RKHS and prove the consistency of the resulting estimator. By penalizing the RKHS norm of the discriminator, they reduce the variance of their estimator and stabilize the training.
main_review: **Originality**

Although their idea of constructing a function in the RKHS is not new, the usage of this construction in GAN-type objective for KL estimation seems to be novel. Related work is adequately cited and it is clear how their method differs from previous ones.

**Quality**

The submission is technically sound and most of the claims are well supported. Theoretical results are solid and informative. Experimental results are reasonable and show the improvements of the proposed method over previous ones. Below are several comments/questions I have:

1. In Abstract and line 40, they claim that methods with a neural-net based discriminator suffers from unreliability or instability. However, they do not include the two methods, VDM and MINE, cited in the paper in the experiments. I would like to see results that can support the claim.
2. In Theorem 2, the quantities $C_p$, $S_p$, $\mathscr{L}_p$ are not discussed. How are they related to the architecture of $\phi_\theta$ and $\gamma$? Are they always finite?
3. The meaning of $\text{sup}_{K_\theta}$ is unclear (e.g., in Theorem 2). Does that include all possible architectures of $\phi_\theta$ and all values of $\gamma$? If you maximize over $\gamma$, then the finiteness in the above comment may be an issue.
4. Since what you are advertising is the regularized version of the estimator, it would be better to include theoretical analysis for the consistency of this regularized estimator.
5. Spectral normalization is used in the proposed discriminator but not in the baseline. This seems to be unfair. Do you have an explanation about it?

**Clarity**

The paper is clearly written and easy to follow.

**Significance**

The results are likely interesting and useful to the community. The idea of constructing a function in RKHS and controlling its complexity could be used in GANs in general.
limitations_and_societal_impact: See comments above for suggestions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors consider the problem of KL divergence estimation. Modern machine learning methods for KL divergence estimation make use of neural networks as function approximators. The authors assert that many of the challenges with using such methods is the uncontrolled complexity of the function space. They propose to resolve this by constructing their neural network functions to be RKHS functions, and control the RKHS norm. They prove learning-theoretic results about their method, showing that it is consistent and exhibits controllable fluctuations around its mean (deviation-from-mean error).
main_review: 
The paper is nicely written and introduces the problem well. I haven't read the proofs in detail but I believe the authors make a solid technical contribution. They are honest about the limitations of their method (discussion of assumptions and paragraph at end) and do not overstate their results, which is refreshing. They provide sketches of proofs in the main text to provide intuition / high level understanding of the results. 

I can't comment on the novelty of the idea; I am unaware of work that tries to use RKHS norms to control neural network function complexity in this way, but I am not an expert in this field and am not up to date with the latest literature. If this is a novel idea then I find it quite exciting and is worthy of merit in its own right (regardless of the rest of the paper). If it is not novel, appropriate citations should be added.

I'm generally happy with it but have a few points I'd like to be addressed. I've given "weak accept" but would be willing to raise this rating if the authors can address them: 

- the results are all cast in terms of the L2(\tau) norm of g. How does this relate to the RKHS norm of f? At the beginning of the paper, I thought you would control the RKHS norm of f -- I think this is implicitly being done, but would be great if you could make this more explicit / comment on this.
- variance analysis: I understand that theorem 3 only applies under the strong assumption of the deviation-from-mean being Gaussian. Is it not possible to make some statements about how the variance decays under some other weaker assumptions? If not, are there (perhaps pathological) cases where the variance does not decay?
- at the beginning you claim that the existing methods suffer due to having uncontrolled function complexity. Do you have any evidence to support this claim? Do these other methods not use regularisation in practice?


Non-technical issues:
- Figures: I think you're converting your figures to PNG or JPG before embedding them into the document. Much better is to save them as PDFs -- that way they are not blurry when you zoom in on a computer.
- I think you are maybe just writing KL in math mode, but if you use use add this latex command: \newcommand{\KL}{\mathrm{KL}} and then use \KL it may look better depending on your taste (feel free to ignore this if you don't like it though).


 
Other detailed comments:
- line 54: gap before (RKHS)
- line 138: "w ~ N(O, \gamma I)" -> is this saying that \tau is the normal distribution? Could you state that explicitly if so?
- line 140: could you maybe make it explicit here that \phi (and I guess also g?) will be implemented as neural networks?
- line 144: "both the RKHS kernel and its norm" -> "both the RKHS kernel and the norm of f"
- You don't really discuss the RKHS norm of f directly. Could you at least briefly mention it here, in particular how it relates to \phi and g? 
- line 151: both \tau and h have already been used, but I don't think the occurrences in this line are referring to either of them?
- line 166: what does M-bounded mean?
- line 194: approximately what kind of order of magnitude do you expect the positive part inside the exp to be? (The practical usefulness of the results does depend on this.)
- equation (14): I guess the \sigma on the left hand side is supposed to be the standard deviation of  the estimator? I
- line 264: typo "Guaussian"


[Edit: raised score by 1 point after authors addressed comments in review]
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: An improved method for estimating the KL divergence between two distributions based on samples. To estimate the log density ratio, the authors proposed to train a discriminator while constraining the family of the discriminator function. It constructs the function using the defined in F. Bach (2017), which is a convolution between two network functions. The authors did a careful (though no very thorough) performance analysis of this method, and demonstrated the performance in simple datasets for three different tasks involving the KL divergence. 
main_review: The idea of constructing a constrained RHKS function for this density ratio estimation appears to be new. It provides a different formulation of the deep kernels used in supervised and unsupervised settings. The analysis method uses standard theoretical methods, such as concentration inequalities and covering numbers. Though some analyses involved much deeper derivations/proofs than others, and the experiments were not yet strong enough, I think the method and theoretical contributions are clear.

The written quality needs some improvement. For example, in eqn (5), $\psi$ is only defined later, and $\tau$ is never defined. 

Here are some detailed comments:
I cannot fully follow the logic in the proof of Lemma 1, and there appear to be some (minor) inconsistencies. It also appears that what would be useful is the contrapositive of Eqn (17), i.e. $\ell_z(f_j)\le \epsilon \Longrightarrow \dots le 2\epsilon$, and then we can proceed by writing $\text{Prob}(\dots \le 2\epsilon)$ rather than a  $\ge$ statement. Also, is $\eta$ set to $\frac{\epsilon}{2\sqrt{S_K}}$ (in the proof) or $\frac{\epsilon}{4\sqrt{S_K}}$ (in the main text)?

Line 219. How's the norm of g related to R in Theorem 2?

There are a few questions regarding the experimental results.
1. Figure 1: top row shows a small bias for the blue clusters. Does this never go away? The bias in the bottom row also doesn't go away. 
2. Figure 3: the neural network decoding seems bad at the end of training, but before the loss blows up, there seems to be a relatively stable region. How does the model at that stage perform in terms of decoding? Also the y axis are not equal, making the results a little hard to compare. Also, although the authors reminded the readers that the point is on stability, but is the proposed method reliability and effective, in terms of standard metrics like the log-likelihood or sample quality (FID, KID)?
limitations_and_societal_impact: While the theory might be clear, the experiments were less than sufficient. They were mostly simple Gaussian distributions, and MNIST only appears to demonstrate stability.
review_title: null

Meta-review: Converging reviews regarding the acceptance of the paper after a fruitful discussion between reviewers and authors, where the latter took time to clarify all the questions raised.
The recommendation for this paper is to accept it, counting on the authors to take into account the outcomes of the discussions with all the reviewers in the final version of the work (preferably in the main text rather than in the supplementary material).


Decision: Accept (Spotlight)