Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes to train two models jointly, one explicit density estimator, and one implicit generative model. The idea is that the explicit (energy) model specifies a valid and easy-to-evaluate likelihood function, while the implicit model is flexible enough to draw high-quality samples. 
Because the explicit model is an energy model, we don't have the normalizing constant, meaning that we cannot evaluate how close the explicit model is to the implicit model. The authors propose to minimize the stein discrepancy between the implicit model and the energy model, and the Wasserstein distance between the implicit model and the data. 

The contributions are proposed as follows: 
* A joint learning framework from training an explicit energy model and an implicit model to learn to sample from a target distribution where we only have access to samples. 
* A training algorithm that trains the implicit model to sample via minimizing the Wasserstein distance then trains the energy model to learn an explicit density by minimizing the stein discrepancy between the energy model and the implicit model. 
main_review: I found this work enjoyable. The idea of learning a deep energy model via the stein discrepancy has been proposed before, but the addition of an implicit generative model really seems to help distribution fit. This is possibly due to the increased size of the generative model function space. [1, 2] consider the space of L2 functions, which need to vanish at the boundaries -- necessitating potentially harmful regularization on the generator function. This method avoids this issue by training an implicit model, that is known to perform well for sample generation. 
Figure 4 and figure 5 were quite convincing to me. They show that both the implicit model and the energy model are able to capture the data distribution well, while their components trained independently more or less fail. 
On a practical note, I believe it's clear from these toy examples that the learned distribution is more faithful with this joint training algorithm. But what is less clear is the advantage in high(er) dimensional inference tasks. The MNIST and CIFAR-10 results do not look much better than WGAN-GP. I see that the inception score is better for Joint-W, but the 2X training time cost is high given this margin. 

As an aside, I found the fact that minimizing the stein discrepancy between the two models acts as a gradient penalty on the implicit model interesting. This might make sense since the energy model is "slower" than WGAN. 


[1] Grathwohl, Will, et al. "Learning the stein discrepancy for training and evaluating energy-based models without sampling." International Conference on Machine Learning. PMLR, 2020.

[2] Hu, Tianyang, et al. "Stein neural sampler." arXiv preprint arXiv:1810.03545 (2018).

-----------------------
### Update After Rebuttal
After reading the other reviews/discussions, I agree that higher-dimensional inference tasks are important to evaluate on for this method. I do not intend to change my score, but I hope that the authors can show the applicability of their approach to tasks other than MNIST and CIFAR-10.

limitations_and_societal_impact: The authors mention that deep generative models can be used for generating false information, and therefore pose some risk.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Explicit and implicit model training have their own advantages and limitations. The author proposes a joint training framework, which aims to combine the benefits from both worlds. Specifically, the author proposes a training framework called **Stein bridging** that allow joint learning of explicit and implicit models.  Particularly, the explicit model considered in this paper is the energy-based model (EBM) and the implicit model is the GAN-like generator. The implicit model is trained based on GAN framework, the explicit model is learned via Stein discrepancy, and the bridge between them is also the Stein discrepancy between the generated samples and the explicit model. 

Theoretically, the author proves that the Stein bridging can (1) act as a dual Sobolev norm regularization for the discriminator that helps the GAN training; (2) act as a Moreau-Yoshida regularization for the Stein discrepancy to smooth the energy function; (3) also help to stabilize the training dynamics. 

Empirically, the author conducts extensive experiments to confirm the effectiveness of the proposed bridging, in terms of generating quality and OOD detection. 
main_review: ### Originality
The main idea of briding is simple and straightforward. However, the theoretical analysis is the main contribution of the paper and seems to be non-trivial. It provides some insights into the regularization effect of the Stein bridging. The author also addresses the differences between this work and the previous ones in the appendix. So in summary, I think this work is interesting and novel. 

### Quality and Clarity
The main idea is easy to follow and clearly written. Some of the simple toy examples (from Theorem 1 and Theorem 2) help to visualize the effect of the regularization. However, I have several concerns regarding the Stein discrepancy and proof. I think there is still a large room for improvements of the quality. I will elaborate on them in the question section. 

### Significance
This proposed framework seems to have many desirable advantages in terms of implicit model training stability and explicit model training quality. In terms of theory, the author shed some light on the effect of joint learning of these two types of models. So it should have some impacts in this community. 

### Questions
1. The first question regards the usage of Stein discrepancy. Using Stein discrepancy to train models can be quite problematic. For example, the Stein discrepancy can be highly non-convex and does not agrees with commonly used divergence (e.g. KL divergence). In other words, when use Stein discrepancy as the training objective (in you case, the learning of explicit model and Stein bridging), a lower discrepancy value does not necessarily mean a good model (the two densities $P_E$ and $P_G$ can be arbitrarily far away in KL divergence sense) unless the global optimum is achieved [1]. This is particularly problematic for gradient-based optimization. So does this pathology of Stein discrepancy affect the Stein bridging? Or the Stein bridging only act as regularization and has no direct impact to model learning. 
2. One quick check for question 1 could be replacing the components in Two-Circle example by Student-t distributions. The EBM is initialized as a mixture of Student-t distribution with the same scale and degree-of-freedom but different means (need to be far-away from the ground truth).  I wonder about the behaviours of the explicit and implicit models, and its training dynamics. 
3. For the definition of Stein discrepancy, when $d'\neq d$, is the corresponding Stein discrepancy a valid one? Does the max-pooing matrix norm guarantee a valid discrepancy?
4. I am a bit confused on why Sobolev dual norm penalizes the large variation of critic. Maybe consider adding some explanation on the intuition behind this norm?
5. Maybe consider adding some intuitive explanations on Moreau-Yoshida regularization?
6. What is $p_r$ in line 615, is it $P_{real}$?
7. For KSD, the form mentioned below line 619 is different from the one you introduced in the main text. I understand this comes from integration by parts, but maybe explicitly mention this before line 619? Also, should it be $\nabla_{\pmb{x}}\log d\mathbb{P}_E/d\mathbb{P}$?
8. In line 629, why it is linear w.r.t. $h$? $h$ appears inside the $\log$.
9. I am confused about the derivation below the line 634. Why you can introduce the auxiliary variable $r$ and treat it as a decoupled parameter? $r$ should also depend on $h$ as well. 
10. For line 653, why $P$ is unconstrained? I thought that $P$ should be absolutely continuous to $P_E$, but $\gamma$ only cares about the agreement of the marginal $P$ and $P_{real}$. So it is possible that $P$ is not absolutely continuous to $P_E$. 
11. Why it is equivalent to the set of point masses in line 660? 


### Reference
[1] Barp, Alessandro, et al. "Minimum stein discrepancy estimators." arXiv preprint arXiv:1906.08283 (2019)

---
### Upates after rebuttal
I appreciated the response. After some discussions with the author, they managed to address my concerns. Therefore, I will raise my score accordingly. However, I still hope more intuitions and explanations about the kernel Sobolev and Moreau-Yoshida regularization can be added in the revised version.  

limitations_and_societal_impact: This paper is a methodology paper. So there shouldn't be any negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: ===== After Rebuttal =====

I have bumped up my score for the enhanced empirical results author(s) have provided in the discussion round. I remain conservative in my final recommendation because I am not fully convinced the gains are actually from the Stein energy penalty (i.e., some other regularization may also offer similar results) and the cost in modeling complexity may not be justified by the gains. 

=====================

This paper proposes to improve generative modeling via joining the strength of unnormalized density estimation (based on Stein discrepancy) and Wasserstein GAN. Numerical experiments verify the proposed solution enjoys better convergence on both synthetic and real-world datasets. Some theories are developed to justify the use of training loss. 
main_review: This work seeks to combine energy modeling and implicit generative modeling to improve the learning of probabilistic models. This direction has been extensively explored by various researchers in the past two years, with varying degrees of success. While the insight that the proposed formulation indeed implements Moreau-Yosida regularization, my major criticism is that there are too many moving parts in this model and thus making it overly complicated. It feels like gluing the components together, and such bridging is not particularly meaningful. More discussion is needed to explain why this work is significant compared to prior arts.

Also, while the author(s) have claimed that the integrated solution overcomes the individual limits of each component, it can totally go to the other extreme, with the interacting components amplifying each other's deficiencies. No experiments are designed to verify this would not be the case (i.e., intentionally testing for the robustness). And there is an established consensus that working with Stein discrepancy is quite tricky. It is computationally intensive and scales poorly to data dimensionality and complexity (due to its kernel formulation and dependency on gradient computation). Since the author(s) are quite vague on the actual implementation of Stein discrepancy in this paper (for general Stein discrepancy, what is $\Gamma$, and for kernel Stein discrepancy, what kernel is used? ), I am unable to properly evaluate the effectiveness of the proposed Stein Bridging. 

Originality. The techniques used in this work are very well known. The main novelty is to promote the consensus of model distribution and data distribution via minimizing the Stein discrepancy relative to the same energy model. I am not sure this is the right thing to do, as the consensus can be a bad one. 

Quality & Clarity: Writing needs to be improved, many statements are not proper English. For clarity, details on Stein discrepancy, its implementation, and the reasoning why the proposed methods should function as expected are missing in the current presentation. I am not sure about the technical correctness of the theories. 

Significance: This work is unlikely to impact the community in a significant way. The solution is overly complicated and the theories are neither appealing to theorists nor practitioners. 
limitations_and_societal_impact: - Missing citations in paragraph 2 in Introduction
- Line 34: "the generated samples from implicit models could augment the dataset and help to alleviate mode collapse". This is a false statement. This sounds more like one goal of generative modeling. 
- Please explain why the Moreau-Yosida encourages the model to seek more modes. It is not obvious. 
- Line 118, requiring the statistical distance to be computationally efficient seems unnecessary because the main bottleneck is the computation of Stein discrepancy, where gradient evaluations are involved. 
- Line 140, "considers three discrepancies as a triangle", what does this line mean?
- Figure 4, all these competing solutions should work here, probably the author(s) should do a little bit of parameter tuning. Also, in Figure 5, the author(s) should compare the learned distribution from models in Figure 4.
- The statement in Theorem 1 seems to be different from the problem (3), which also minimizes for G.
- Experiments should include more complicated datasets. MNIST and Cifar10 do not adequately represent the complexity of real-world data, and that's what challenges Stein discrepancy based metrics. 
- Examples of bad English:
"mutual compensation -> synergy"
"contaminated -> corrupted"
"power->strength"
"vivid->compelling"
"arguably->\"
"illuminate-> illustrates"
"oscillatory behavior->dynamics"
"centrality->solution"
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose to jointly train implicit (like GANs) and explicit (EBMs) density models by proposing a new regularization technique that connects the training of the impicit and explicit model via Stein bridges that is based on stein discrepancy. The authors show that such a learning objective with a Stein bridge helps to overcome the drawbacks encountered while training an implicit or explicit model individually. The authors provide sufficient theoretical details in support of the method under the Wasserstein distance and perform experiments on toy examples and image datasets.
main_review: **Pros:** The idea of using a combination of implicit and explicit models for joint training has been explored before in several papers recently (as the authors discuss in the detailed literature review section)  and is an interesting direction of research. Furthermore, the proposal to use mutual regularization of the two models to train such models is intriguing and novel. Additionally, the experiments presented in the paper provide promising results and improvements.

**Cons:** 
- My major concern (which spans the previous works on jointly training implicit and explicit models as well) is the motivation behind the joint training of implicit and explicit model itself. It seems that this strategy introduces an expensive overhead of training two rather costly models to overcome the the drawbacks of the individual models. Furthermore, the solution itself does not directly address the drawbacks present in the individual models itself but indirectly makes things better for the models. While I liked the idea of incorporating the loss term between the implicit and the explicit model (D_3 in Table 1), I am still not clear what additional information this term provides during training apart from the data and the other two terms. 

- Secondly, the authors mentioned in the paper that any metric can be easily used for training the models (apart from Wasserstein or Jenson-Shanon etc). However, if I am not mistaken, the theoretical analysis presented by the authors in Section 4 is based on the Wasserstein metric. I wonder if these results will hold for other metrics as well? What impact do the authors foresee of different metrics on their analysis?

- (*minor*): The authors train an implicit and explicit model together. Which of these models is finally used after training? Is it the case that the implicit model is used for sample generation and the explicit model for inference based tasks? 

-  The methodology presented here comes across as rather too costly for training an energy based model specially given the fairly light-weight methods present for training EBMs. I believe the paper will benefit from having a comparison between the training using the proposed methodology and these already present methods for training EBMs.

- Finally, while the experiments comprehensively show that the proposed model achieves good results, I would have liked to see experiments on more complicated datasets than MNIST and CIFAR 10. This stems from my worry of the costly procedure of training two models jointly and how well this will scale to higher-dimensions and more complicated datasets. 
limitations_and_societal_impact: The authors have discussed limitations briefly towards the end but I believe the authors can include a more thorough discussion on the implications of their results in general, the limitations of theoretical analysis (restricted to only Wasserstein metric) and cost of training two models jointly and scalability to higher dimensions. The authors have alluded to some of the issues in the paper but I think detailed comments are warranted here.
review_title: null

Meta-review: This paper proposes Stein bridge as a generic framework to jointly train an explicit generative model as well as an implicit model. Some theoretical analyses are provided. Experiments are conducted on synthetic data as well as image generation with MNIST data.

Reviewers agree that the proposed method is new. However, as cited by the authors, there are previous works on the joint training view of EBMs and implicit models. In this sense, the paper focuses on a known research topic but with a new methodology.

Reviewers agree that the theoretical analysis result is non-trivial so indeed a contribution. However, some clarity concerns remain, which needs to be improved in revision.

The major concern from the reviewers is that the experimental results are not strong, and certainly in GAN literature, MNIST is already a toy-ish dataset. They are not convinced that there is strong evidence that the demonstrated advantage on modelling corrupted data comes from the proposed Stein bridging method.
Decision: Accept (Poster)