Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
Authors: Qitian Wu
Rui Gao
Hongyuan Zha
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep generative models
generative adversarial networks
energy models
stein's methods

Abstract: There are two types of deep generative models: explicit and implicit. The former defines an explicit density form that allows likelihood inference; while the latter targets a flexible transformation from random noise to generated samples.  While the two classes of generative models have shown great power in many applications, both of them, when used alone, suffer from respective limitations and drawbacks. To take full advantages of both models and enable mutual compensation, we propose a novel joint training framework that bridges an explicit (unnormalized) density estimator and an implicit sample generator via Stein discrepancy. We show that our method 1) induces novel mutual regularization via kernel Sobolev norm penalization and Moreau-Yosida regularization, and 2) stabilizes the training dynamics. Empirically, we demonstrate that proposed method can facilitate the density estimator to more accurately identify data modes and guide the generator to output higher-quality samples, comparing with training a single counterpart. The new approach also shows promising results when the training samples are contaminated or limited.

Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
Qitian Wu' Rui Gao
Hongyuan Zha 1:3
Department of Computer Science and Engineering.
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University University of Texas Austin $ School of Data Science, Shenzhen Institute of Artificial Intelligence and Robotics for Society, The Chinese University of Hong Kong. Shenzhen echo7400sjtu, edw.estttuuueeeeoom cn, rui www.gggmmmoooooo utexas edu, zhahy@cuhk, edu. cn
Abstract
There are two types of deep generative models: explicit and implicit. The former defines an explicit density form that allows likelihood inference; while the latter targets a flexible transformation from random noise to generated samples. While the two classes of generative models have shown great power in many applications, both of them, when used alone, suffer from respective limitations and drawbacks. To take full advantages of both models and enable mutual compensation, we propose a novel joint training framework that bridges an explicit (unnormalized) density estimator and implicit sample generator via Stein discrepancy. We show that our method 1) induces novel mutual regularization via kernel Sobolev norm penalization and Moreau- Yosida regularization, and 2) stabilizes the training dynamics. Empirically, we demonstrate that proposed method can facilitate the density estimator more accurately identify data modes and guide the generator to output higher-quality samples, comparing with training a single counterpart. The new approach also shows promising results when the training samples are contaminated or limited.
Introduction
Deep generative model, a powerful unsupervised framework for learning the distribution of high- dimensional multi-modas data, has been extensively studied in recent literature. Typically, there are two types of generative models: explicit and implicit. Explicit models define density function of the distribution 35 51 42]. while implicit models learn mapping that generates samples by
transforming an masy-to-sampll random variable 115
Both models have their own power and limitations. The density form in explicit models endows them with convenience characterize data distribution and infer the sample likelihood. However, the unknown normalizing constant often causes computational intractability. On the other hand, implicit models including generative adversarial networks (GANs) directly generate vivid samples in various application domains including images, natural languages, graphs, etc. Nevertheless, one important challenge to design training algorithm that do not suffer from instability and mode collapse. In view of this, it is natural to build unified framework that takes full advantages of the two models and encourages them compensate for each other.
Intuitively, an explicit density estimator and flexible implicit sampler could help each other's training given effective information sharing. On the one hand, the density estimation given by explicit
Part the work was done when the two authors were visiting The Chinese University of Hong Kong, Shenzhen.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

models can be good metric that measures quality of samples [ and thus can be used for scoring generated samples given by implicit model or detecting outliers as well noises in imput true samples [53] On the other hand, the generated samples from implicit models could augment the dataset and help to alleviate mode collapse especially when true samples are insufficient that would possibly make explicit model fail to capture an accurate distribution. We refer to Appendix A for a more comprehensive literature review.
Motivated by the discussions above, in this paper, we propose a joint learning framework that enables mutual calibration between explicit and implicit generative models. In our framework, an explicit model is used to estimate the unnormalized density; in the meantime, an implicit generator model is exploited minimize certain statistical distance (such as the Wasserstein metric or censen-Shannon divergence) between the distributions of the true and the generated samples. On top of these two models, a Stein discrepancy, acting as bridge between generated samples and estimated densities, introduced to push the two models to achieve a consensus. Unlike flow- how-based models [3 26]. our formulation does not impose invertibility constraints on the generative models and thus flexible in utilizing general neural network architectures. Our main contributions are follows:
Theoretically, prove that our method allows the two generative models impose novel mutual regularization on each other. Specifically, our formulation penalizes large kernel Sobolev norm of the critic in the implicit (WGAN) model, which ensures the critic not to change suddenly on the high-density regions and thus preventing the critic of the implicit model being too strong during training. In the mean time, our formulation also smooths the function given by the Stein discrepancy through Moreau- Yosida regularization, which encourages the explicit model to seek more modes in the data distribution and thus alleviates mode collapse.
ii) In addition, we show that our joint training helps to stabilize the training dynamics. Compared with other common regularization approaches for GAN models that may shift original optimum, our method can facilitate convergence to unbiased model distribution.
iii) We conduct comprehensive experiments justify our theoretical findings and demonstrate that joint training can help two models achieve better performance. On the one hand, the energy model can detect complicated modes in data more accurately and distinguish -uttof-distribution samples. On the other hand, the implicit model can generate higher-quality samples.
2 Background
Energy Model. The energy model assigns each data x Rd with a scalar energy value Ex)... where (-) is called energy function and parameterized by Φ. The model is expected to assign low energy to true samples according a Gibbs distribution P& (x) cxpf (x) normalizing constant dependent of Φb The term Zo is often hard to compute, making optimization intractable, and various methods are proposed detour such term (see Appendix A
where Zg is a
Stein Discrepancy. Stein discrepancy
R is a measure of closeness between two probability distributions and does not require knowledge for the normalizing constant of one of the compared distributions. Let and Q be two probability distributions on X' C IR:d, and assume Q has (unnormalized) density q. The Stein discrepancy S(I, Q) defined as
I6
30
38
S(P, Q == sup Exxr(Agf(x) = sup I(Ex~e Vx log (xx) T + [xf(x)), FEF
(1)
FEF
where F is often chosen to be a Stein class (see, e.g., Definition 2.1 in 30 f Rd Rd is a valued function called Stein critic and I' is operator that transforms d d' matrix into scalar value. One common choice of is trace operator when d' = d. One can also use other forms for I, like matrix norm when d' * 30 If F is a unit ball in some reproducing kernel Hilbert space (RKHS) with a positive definite kernel k, it induces Kernel Stein Discrepancy (KSD) [17] More details are in Appendix B
Wasserstein Metric, Wasserstein metric is suitable for measuring distances between two distributions with non-overlapping supports 2 The Wasserstein-1 metric between IP and Q is
W(P. Q == min [xxx)>> xll,,

---

Implicit Model Pg
Explicit Model
Table 1: Comparison of objectives
(x
Peol
(x)
Po(x) True Density
between different generative models, where D1 DD real, PG), D2
Wasserston
Sample
Slein Crilic
D
D2(Preaa, P Pg) and D3 == D3 (eG,,EE PE)
Density Estimator
denote general statistical distances be- tween Iwo distributions.
Po
Ga(z)
X
Sample
Fake Sample
Slein
D
Fake Dencity
Model R
Objective
Noise
Ganerator
(x)
Po(
GAN
D D2
Energy Model
Stein Bridge
Energy saay--e-eeddd GAN
55
Di
Data Flow With BP
Update & Impact
Contrastive Divergenee
D2
D. ((Praat Po)
D2 S PE)
Da S(Pa.Fe)
Cooperative Learning
3
Da
Two Generator Game
Dg + Dg
Figure I: Model framework for Stein Bridging.
Stein Bridging (ours)
Dj +
where the minimization with respect to y over all joint distributions with marginals IP and Q By Betilrovvic-Ruubnsttee duality, W(P, Q) has a dual representation
((P, Q) = max {Ex~r [(x) - Ey~q Ey-alD(y)]),
(2)
where the maximization is over all I-Lipschitz continuous functions.
Sobolev space and Sobolev dual norm. Let (P) be the Hilbert space Rd equipped with an inner product (u, L) := Se= ux)F(xx. The (weighted) Sobolev space H defined as the closure of CØ set of smooth functions on Rd with compact support, with respect to norm where IP has a density, For E L2, its Sobolev dual norm ||l]| == (uga + || [[llx defined by
/ u(x))d(x) =o.
= sup WEHi
(u. u)
V dF(x) 1 JRd
The constraint Ja (xx)xx= 0 is necessary guarantee the finiteness of the supremum, and the supermum can be equivalently taken over Co.
3 Proposed Model: Stein Bridging
In this section, we formulate our model Stein Bridging. scheme of our framework is illustrated in Figure [ Denote by Preat the underlying real distribution from which the data {x) are sampled. The formulation simultaneously learns two generative models one explicit and one implicit that represent estimates of IP Preell The explicit generative model has distribution Pg on X' with explicit probability density proportional to exp(-E(x)),
X, where E is referred to as an energy function. We focus on emergy-based explicit model in model formulation as it does not enforce any constraints or assume specific density forms. For specifications, one can also consider other explicit models, like autoregressive models or directly using some density forms such as Gaussian distribution with given domain knowledge. The implicit model transforms an masy-to-sample random noise with distribution Po via a generator G' to sample ă G(z) with distribution Pc- Note that for distribution PE, we have its explicit density without normalizing term, while for PG and Prest- use the Stein discrepancy (that does
have samples from two distributions. Hence, require the normalizing constant) as a measure of closeness between the explicit distribution PE and the real distribution IP reeal and use the Wasserstein metric (that only requires only samples from two distributions) as a measure of closeness between the implicit distribution Pa and the real data distribution PP Preat-
To jointly learn the two generative models PG and Pe, arguably a straightforward way to minimize the sum of the Stein discrepancy and the Wasserstein metric:
min W(P Pa) + AS(PP IP)) E.G.
where X 2 However, this approach appears no different than learning the two generative models separately. To achieve information sharing between two models, we incorporate another term
3

---

S(FG.PE) called the Stein bridge that measures the closeness between the explicit distribution FF and the implicit distribution Fa:
min E,G
W(Preat, Pc) A1 PE) A2S((Pc,
(3)
where X1, 22 2 O. The Stein bridge term in pushes the two models to achieve consensus.
Remark 1. Our formulation is flexible in choosing both the implicit and explicit models. In 3 we can choose statistical distances other than the Wasserstein metric (erreat, Pa) to measure closeness between Preal and PG, such as Sonsen-Shannon divergence, as long as computation requires only samples from the involved two distributions. Hence, one can use GAN architectures other than WGAN to parametrize the implicit model. addition, one can replace the first Stein discrepancy term S(Pra
real, Pæ) in 3 by other statistical distances long its computation is efficient and hence other explicit models can be used. For instance, if the normalizing constant of F is known or easy to calculate, one can use Kullback-Leeibler (KI.) divergence.
The choice of the Stein discrepancy for the bridging term S( crucial and cannot be replaced by other statistical distances such as KL divergence, since the data- gentragnnn distribution does not have an explicit density form (not even up to a normalizing constant). This exactly one important reason why Stein bridging was proposed, which requires only samples from the data distribution and only the log-density of the explicit model without the knowledge of normalizing constant as estimated in MCMC or other methods.
Remark 2.
In our implementation, we parametrize the generator in implicit model and the density estimator in explicit model as (z( and (x).. respectively. The Wasserstein term in ( is implemented using its equivalent dual representation in (2) with a parametrized critic (x). The two Stein terms in (3 can be implemented using (I) with either a Stein critic (parametrized as a neural network, i.c., t. (x)), or the con-parametric Kernel Stein Discrepancy. Our implementation iteratively updates the explicit and implicit models. Details for model specifications and optimization are in Appendix E.2
Comparison with Existing Works. There several studies that attempt combine explicit and implicit generative models from different ways, e.g. by energy- baaeed GAN [55], contrastive divergence 251 3 cooperative learning (50) two generator game [7] Here we provide high-level comparison in Table D where we note that the formulations of existing works only consider one-side discrepancy most two discrepancy terms. Such formulations cannot address the respective issues for both models and, even worse, the training for two models would constrain rather than exactly compensate each other (more discussions are in Appendix A..3) Differently, our model considers three discrepancies simultancously as triangle to jointly optimize two generative models. In the following, we will show that such new simple formulation enables models compensate each other via mutual regularization effects and stabilize the training dynamics.
4 Theoretical Analysis
In this section, we provide theoretical insights on proposed scheme, which illuminate its mutual regularization effects justification of our joint training and further show its merit for stabilizing the training dynamics. The proofs for all the results in this section are in Appendix D
4.1 Mutual Regularization Effects.
We first show the regularization effect of the Stein bridge on the Wasserstein critic. Define the kernel Sobolev dual norm as
|| Dl 1(P:6) == sup (D, Ex,x'< (u((x) T 1( x Vx)')( < Ex(u) 0}. MEC
can be viewed kernel generalization of the Sobolev dual norm defined in Section 2 which reduces to the Sobolev dual norm when x(x,x') [[x x') and IP is the Lebesgue measure.
Theorem 1. Assume that (TP&) ) exhausts all continuous probability distributions and is chosen as kernel Stein discrepancy. Then problem 3 is equivalent to
min max [Ey-Ps [EyP)!D(y)]] Ex 7 [(x)) The
SIIS(Preat, PP)
E

---

The kernel Sobolev norm regularization penalizes large variation of the Wasserstein critic D. Particu- larly, observe that 45 if k(xx, x') = I(x x') and Epp [D] and then
Wa((1 + E.
[ (P lim
where W2 denotes the 2- 2-Wasserstein metric. Hence, the Sobolev dual norm regularization ensures D change suddenly on high-density region of I' E.g and thus reinforces the learning of the Wasserstein critic. Stein bridge penalizes large variation of the Wasserstein critic, in the same spirit but of different form comparing critic from being too strong during training and thus encourages mode exploration of sample generator. To illustrate this, we conduct case study where we train a generator over the data sampled from a mixture of Gaussian (/41 [-1, -1], P2 1 and Σ 0.21). In Fig. 3 aa we compare gradient norms of the Wasserstein critic when training the generator with and without the Stein bridge. As we can see, Stein bridge can help to reduce gradient norms, with similar effect as WGAN-GP.
mudieent-based penalty (e.g.. [9 50)) ). It prevents Wasserstein
Moreover, the Stein bridge also plays a part smoothing the output from Stein discrepancy and we show the result in the following theorem.
Theorem 2. Assume {PG exhausts all continuous probability distributions, and the Stein class defining the Stein discrepancy is compact (in some linear topological space). Then problem 3 is equivalent to
min { (P, real, Pε) 22 max Ex ((Ap, (xi])
where (Ar, ()) (.) denotes the (generalized) Moreau- Yosida regularization of function Ar f with parameter X2 i.e., (Ap f)xx)xxxxx))) minyex (Ap (fy)(()) + x2 - -yll).
Note that (Aa f) Lipschitz continu- ous with constant 1 1/A2. Hence, the Stein bridge, together with the Wasserstein met- ric W(P real, Pa), plays as Lipschitz reg- ularization on the output of the Stein op- erator Ap f via Morcau- Yosida regulariza- tion. This suggests a novel regularization scheme for Stein-based GAN. By smooth- ing the Stein critic, the Stein bridge encour- ages the energy model to seek more modes in data instead of focusing on some dominated modes, thus alleviating mode-collapse issue. To illustrate this, we consider a case where we have an energy model initialized with one mode center and data sampled from distribution of another mode, depicted in Fig. 2 Fig. b and (e) compare the Stein critics when using Stein bridge and not, respectively. The Stein bridge helps to smooth the Stein critic, as indicated by less rapidly changing contour in Fig. [bb) compared to Fig. c)), learned from the data and model distributions plotted in Fig. bb).
(a)
(b)
Figure 2: (a) Contour of energy model with one mode and empirical data from distribution with different mode (blue dots); (b) & (c) Contours of the Stein critics between the two distributions learned w/ and w/o the Stein bridge, respectively.
4.2 Stability of training dynamics.
We further show that Stein Bridging could stabilize adversarial training between generator and Wasserstein critic with a local convergence guarantee. As known, the training for minimax game in GAN is difficult. When using traditional gradient methods, the training would suffer from some oscillatory behaviors 14 29 54 In order to better understand the optimization behaviors, we first compare the behaviors of WGAN, likelihood- and catropy-regularing WGAN, and our Stein Bridging under SGD via an easy to comprehend toy example -imeemnnsio case. Such a toy example (or a similar one) is also utilized by I3 x
shed lights on the instability of WGAN
training Consider linear critic (x(x)) = रhअ and generator Gz(z) Or. Then the Wasserstein GAN objective be written constrained bilinear problem: ming max s1 [xr (0 which could be further simplified as an unconstrained version (the behaviors can be generalized to multi-dimensional cases
min max &p +
(4)
Our theoretical discussions focus WGAN, and also compare with original GAN in the experiments.
5

---

Sittout Stam dri
With Stein bridge
CyEzam Pozi
-03
+20
(b)
(c)
Figure 3: (a) The gradient norm of Wasserstein critic with (bluc) and without (red) the Stein bridge when data are sampled from mixture of Gaussian. (b) Numerical SGD updates of Stein Bridging, WGAN and its variants with different regularizations.
Unfortunately, such simple objective cannot guarantee convergence by traditional gradient methods like SGD with alternate updating 0x+1 02 + mp thx+1 = e/x + 7 1all-6xx 0x+1). Such optimization would suffer from an oscillatory behavior, i.e., the updated parameters go around the optimum point (10 0-] [0. 1I) forming a circle without converging to the centrality, which shown in Fig. 3()). A recent study in 29 theoretically show that such oscillation is due to the interaction term in
One solution to the instability of GAN training to add (likelihood) regularization, which has been widely studied by recent literatures 46
28 With regularization term, the objective changes into ming wa vb|E[(]] +0 AE y (0z)), where y(()) denotes the likelihood function and À is hyperparameter. A recent study 44 proves that when À < 0
the extra term equivalent maximizing sample evidence, helping to stabilize GAN training; when À > 0
the extra term maximizes sample entropy, which encourages diversity of generator. Here we consider a Gaussian likelihood function for generated sample x' u(x')= exp( 2 (a' b)?) which is up to constant, and the objective becomes (see Appendix D.1
for details):
min max 1/ - 1b 0 X(02 1 -))-
(5)
#
The above system would converge with À < O and diverge with À > 0 in gradient-based optimization, is that the extra term changes the biased distribution, as proved by 44 In this case,
shown in Fig. Bb). Another issue of Akelihood-regullrrizati optimum point and makes the model converge one can verify that the optimum point becomes [vb* , 0*] = [-X, 1J. resulting in a bias. To avoid this issue, ! proposes to temporally decrease |\] through training. However, such method would also be stuck in oscillation when |\| gets close to zero as is shown in Fig.B]b
Finally, consider proposed model. We also simplify the density estimator Po(z) = exp(- (xr) whose score function V. log P²(x) = -r Ø. Then if we specify the two Stein discrepancies in KSD, the objective is (see Appendix D.T for details).
basic energy model
A2 (0 ()+
min max min 1ir - sb 0+
2 (1+ 2)2
(6)
Interestingly, for Fr1, A2, the optimum remains the same [10*, 0", -01 = [ -1]. Then we show that the optimization guarantees convergence to [3b", , ,6
Proposition 1. Using alternate SGD for 6 geometrically decreases the square norm N₂ |155122 | - 1|2 + + 1 any 0 < 1 < with X1 = x2
No+1 (1 21 N)
Fig.B As shown in Fig. b)). Stein Bridging achieves a good convergence to the right optimum. Compared with 4 the objective (6) adds a new bilinear term & 0, which acts like connection between the generator and estimator, and two other quadratic terms, which help to penalize the increasing of values through training. The added terms and original terms in cooperate to guarantee convergence to unique optimum. (More discussions in Appendix D.I. Moreover, Fig. ((c) presents the training dynamics w.r.t. different X and >₂ 's. As we can see, the convergence can be achieved with different trading-off parameters which in essence have impact on the convergence speed.
'Here, we adopt the most widely used alternate updating strategy. The simultancous updating, i.e., Ok-1 0% + m/v and dh+1 th n11 Oz), would diverge in this case.

---

We further generalize the analysis to multi cuulti-dmeesionall bilinear system F( 0)
0
A
b
c ab which is extensively used by researches for analysis of GAN stability 14 12
29
13 For
any bilinear system, with added term H 0) 2 (0 + Φ) ( + Φ) where B - (AA ) to the objective, we can prove that i) the optimum point remains the same as the original system (Proposition
3) and ii) using alternate SGD algorithm for the new objective can guarantee convergence (Theorem
More discussions are given in Appendix D.2
5 Experiments
In this section, we conduct experiments to verify the effectiveness of proposed method from mul- tifaceted views. The implementation codes are available at lattps://gitan com/ hammqittanwull SteinBridging
5.1 Setup
We mainly consider evaluation with two tasks: density estimation and sample generation. For density estimation, we expect the model to output estimated density values for input samples and the estimation is supposed match the ground- truth one. For sample generation, the model aims at generating samples that are akin to the real observed ones.
We consider two synthetic datasets with mixtures of Gaussian distributions: Two-Circle and Two- Spiral. The first one is composed of 24 Gaussian mixtures that lie in two circles. The second dataset consists of 100 Gaussian mixtures densely arranged on two centrally symmetrical spiral-shaped curves. The ground-trute distributions are shown in Fig. 4 a). Details for synthetic datasets are in Appendix E. Furthermore, apply the method to MNIST and CIFAR datasets which require the model to deal with high-dimensional image data.
In each dataset, we use true observed samples as input of the model and leverage them to train our model. In synthetic datasets, we sample N1 2000 and N₂ 5000 points from the ground-trute distributions as true samples for -io-CCerlee and Two-Spiral datasets, respectively. The true samples are shown in Fig. (a). In MNIST and CIFAR, we directly use pictures in the training sets as true samples. The details for each dataset are reported in Appendix
We term model Joint W if using Wasserstein metric in and Joint-JS if using JS divergence in this section. As we mentioned, our model is capable for 1) yielding estimated (unnormalized) density values (by the explicit energy model) for input samples and 2) generating samples (by the implicit generative model) from a noise distribution. We consider several competitors for performance comparison. For sample generation, we mainly compare our model with implicit generative models. Specifically, we basically consider the counterparts without joint training with energy model, which are equivalently valina GAN and WGAN with gradient penalty 19, for ablation study. Also, as comparison to the new regularization effects by Stein Bridging, we consider a recently proposed variational annealing regularization 4E! for GANs (short as GAN+VA/WGAN+VA, with denoising auto-encoder [1] to estimate the gradient for regularization penalty. For density estimation, we mainly compare with explicit models. Specifically, we also consider the counterparts without joint training with generator model, i.e., Deep Energy Model (DEM) using Stein discrepancy [ Besides we compare with energy calibrated GAN (EGAN) [6] and Deep Directed Generative (DGM) Model [25] which adopt contrastive divergence to train a sample generator with energy estimator. The hyper- parameters are tuned according to quantitative metrics (will be discussed later) used for different tasks. See Appendix E.3 for implementation details.
5.2 Density Estimation of Explicit Model
As shown in Two-Circle case in Fig both Joint-JS and Joint-W manage capture all Gaussian components while other methods miss some of modes. In Two-Spiral case in Fig Joint-JS and Joint-W exactly fit the ground-true -tuth distribution. Nevertheless, DEM misses one spiral while EGAN degrades to a uniform-lit to regions that have low densities in the groung-truth distribution. As quantitative comparison, we study three evaluation metrics: KI. & JS divergence and Area Under the Curve (AUC). Detailed information and results are in Appendix F.4 and Table X respectively. The values show that Joint- W and Joint-JS provide better density estimation than all the competitors over large margin.
distribution. DGM manages to fit two spirals but allocate high densities
7

---

S
S
S
(a) True
(b) DEM (c) ECAN
(d) DCM
(e) Joint-IS (f) Joint-W
Figure 4: Results for density estimation. (a) Densities of real distribution. (b)~((f) Estimated densities
given by the estimators of different methods on Two-Circle (upper) and Two-Spiral (bottom) datasets.
S
(a) True
(b) GAN
(c) WGAN (d) DGM
(e) Joint-JS (f) Joint-W
Figure 5: Comparison for sample quality. (a) Samples from real distribution. (b)~(f) Generated
samples produced by the generators of different methods on Two-Cirele (upper) and Two-Spiral (bottom) datasets.
We rank generated digits (and true digits) on MNIST w.r. densities given by the energy model in Fig. 13 Fig. 14 Fig. 15 As depicted in the figures, the digits with high densities (or low densities) given by Joint-JS possess enough diversity (the thickness, the inclination angles as well as the shapes of digits diverses). By constrast, all the digits with high densities given by DGM tend to be thin and digits with low densities are very thick. Also, as for EGAN, digits with high (or low) densities appear to have the same inclination angle (for high densities, '1' keeps straight and "9" 'leans' to the left while for low densities, just the opposite), which indicates that DGM and EGAN tend to allocate high (or low) densities to data with certain modes and miss some modes that possess high densities in ground truth distributions. By contrast, our method manages to capture these complicated features in data distributions.
We further study model performance on detection for cittof-ddistrbbttoo (OOD) samples. We consider CIFAR- 10 images as positive samples and construct negative samples by (I) flip images, (II) add random noise, (III) overlay two images and (IV) use images from LSUN dataset, respectively. A good density models trained on CIFAR-10 10 are expected to give high densities to positive samples and low densities to negative samples, with exception for case (I) (flipping images are not exactly negative samples and the model should give high densities). We use the density values rank samples and calculate AUC of false positive rate v.s. true positive rate, reported in Table g Our model Joint-V W manages to distinguish samples for (II), (III), (IV) and is not fooled by flipping images, while DEM and EGAN fail to detect out-of- nut-of-distribution samples and DGM gives wrong results, recognizing flipping images negative samples.
5.3 Sample Quality of Implicit Model
In Fig. we show the results of different generators in synthetic datasets. For Two-Circle, there are a large number of generated samples given by GAN, WGAN-GP and DGM locating between two Gaussian components, and the boundary for each component is not distinguishable. Since the ground- truth densities of regions between two components are very low, such generated samples possess low-quality, which depicts that these models capture the combinations of two dominated features (i.e., modes) in data but such combination makes no sense in practice. By contrast, Joint-JS and Joint-W could alleviate such issue, reduce the low-quality samples and produce more distinguishable

---

Table 2: Inception Scores (IS) and
Table 3: Area Under the Curve (AUC) for OOD detection in CIFAR- 10 datasets.
Fréchet Inception Distance (FID) on CIFAR-10 datasets.
WAIL
Method
IS
FID
Method
1
I1
III
IV
WGAN-GP
6.74::0.04
42.240.572
DEM
0.50
0.52
0.51
0.56
Energy GAN
6.89.10.081
45.6110.375
Peration
DGM
1.00
1.00
1,00
0.82
WGAN+VA
6.90:10:058
45.3410.307
DGM
6.51=00.44
08.820.492
EGAN
0,50
0.42
0.30
0.52
Figure 6:
Learning
7.12.10.101
41.010.546
Joint-W
0.50
0.92
0.95
0.85
curves in Two-Spiral.
Epoch 410
4 3
5
8
0
7
6
4
3
4
0
7
Epoch
7
3
5
8
9
7
6
3
4
0
7
5
Epoch
4
5
8
D
7
6
4
7
4
7
AUC
Epoch 440
7
2
8
8
9
7
6
3
4
0
7
5
Joint-W
Epoch 450
4
3
L
5
8
D
9
2
4
0
7 5
WGAN-GP
WGAN+VA
Ours
Noise Rabio 1%)
True Sample Sizw
(b)
Figure 7: Generated digits given by the same noise 2 in Figure 8: Impact of (a) noise in data and (b)
adjacent training epochs on MNIST.
insufficient data model performance.
boundaries. In Two-Spiral, similarly, the generated samples given by GAN and WGAN-GP form circle instead of two spirals while the samples of DGM 'link' two spirals. Joint-JS manages focus more on true high densities compared to GAN and Joint-W provides the best results. To quantitatively measure sample quality, adopt Maximum Mean Discrepancy (MMD) and High-quality Sample Rate (HSR). Details are in Appendix E.4 and results are in Table g where our models can outperform the competitors over large margin.
We report the Inception Score (IS) and Fréchet Inception Distance (FID) to measure the sample quality on CIFAR-10. As shown in Table 2 Joint-W W outperforms other competitors by 0.2 and achieves 5.6% improvement over WGAN-GP w.r.t IS. As for FID, Joint-W slightly outperforms WGAN-GP and beats energy-based GAN and variational annealing regularized WGAN over a large margin. One possible reason is that these methods both consider entropy regularization which encourages diversity of generated samples but will have a negative effect on sample quality. Stein Bridging can overcome this issue via joint training with explicit model. In practice, DGM is hard for convergence during training and gives much worse performance than others.
5.4 Further Discussions
Enhancing the Stability of GAN. In Fig. 6 we present the learning curve of Joint-W compared with WGAN and likelihood- and reguopy-r WGAN. The curves depict that joint training could reduce the variance of metric values especially during the second half of training. Furthermore, we visualize generated digits given by the same noise z in adjacent epochs Fig. 7 The results show that Joint-W gives more stable generation in adjacent epochs while generated samples given by WGAN-GP and WGAN+VA exhibit an obvious variation. Especially, some digits generated by WGAN-GP and WGAN+VA change from one class to another, which is quite similar to the oscillation without convergence discussed in Section 3.2. To quantify the evaluation of bias in model distributions, we calculate distances between the means of 50000 generated digits (resp. images) and 50000 true digits (resp. images) MNIST (reps. CIFAR-10) The results are reported in Table [ We can see that the model distributions of other methods are more seriously biased from true distribution, compared with Joint-W. W.
Contaminated or Limited Data. In Fig. g a) we compare Joint W with WGAN-GP for sample generation on noisy input in Two-Circle dataset. Details are in Appendix E. We can see that the noise ratio in data impacts the performance of WGAN-GP and Joint- W. but comparatively, the performance decline of Joint-W W is less significant, which indicates better robustness of joint training w.r.t. noised data. Moreover, in Fig. B), we compare Joint-W with DEM for density estimation with insufficient true data Two-Spiral dataset. When sample size decreases from 2000 to 100, the AUC value of DEM declines dramatically. By contrast, the AUC of Joint-W W exhibits small decline when the sample size is more than 500. The results demonstrate that Stein Bridging has promising power in some extreme cases where the training sample are contaminated limited.

---

Conclusions and Discussions
This paper aims jointly training implicit generative model and explicit generative model via bridging term of Stein discrepancy. Theoretically, we show that joint training could i) enforce dual regularization effects on both models and thus encourage mode exploration, and ii) help to facilitate the convergence of minimax training dynamics. Extensive experiments on various tasks show that our method can achieve better performance on both sample generation and density estimation.
Limitations. We mainly focus on GAN/WGAN as instantiations of implicit generative models and energy-based models instantiations of explicit models for theoretical analysis and empirical evaluation. In fact, our formulation can also be extended to other models like VAF, flow model, etc. combine the best of two worlds. Furthermore, since propose new learning paradigm piorneering endeavor on unifying the training of two generative models via concurrently optimizing three loss terms, our experiments mainly focus on synthetic datasets, MNIST and CIFAR-IO. We believe our method can be applied to more complicated high-dimensional datasets given the promising results in this paper.
Potential Societal Impacts. One merit of our method is that the joint training model makes it easier to add inductive bias to the generative models, discussed in Section 1 and 4.4. Such inductive bias enforced manually can used to control the distribution of output samples with some desirable properties that accord with ethical considerations, e.g., fairness. Admittedly, such inductive bias would also possibly be used by some speculators for generating false works for commercial purposes. More studies needed in the future to detect such works generated by machines in more intelligent way and further protect intellectual property of individuals.
References
[1] Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data- generating distribution. J. Mach. Learn. Res., 15(1): 15(1):3563-359) 2014.
[2] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, pages 214-223, 2017.
[3] Alessandro Barp, François-Xavii Xavier Briol, Andrew B. Duncan, Mark A. Girolami, and Lester W. Mackey. Minimum stein discrepancy estimators. In Advances in Neural Information Processing Systems, pages 12964 12964-12976, 2019.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[5] Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In ICML, pages 2606-2613 2016.
[6] Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard H. Hovy, and Aaron C. Courville. Calibrating energy-based generative adversarial networks. In ICLR, 2017.
[7] Lizhong Ding, Mengyang Yu, Li Liu, Fan Zhu, Yong Liu, Yu Li, and Ling Shao. Two generator game: Learning to sample via linear goodness-offfr test. In NeurIPS, pages 11257-1 1266,, 2019.
[8] Chao Du Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Learning implicit generative models by teaching explicit ones. CoRR, 0/t//887703877 2018
[9] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. CoRR,
2019.
[10] L.C. Evans. Partial Differential Equations. Graduate studies in mathematics. American Mathematical Society, 2010.
[11] Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell., 6(6):711-741, 1984.
[12] Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of gans using variational inequalities. CoRR, abs/1808.01531, 2018.
10

---

[13] Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. In ICLR, 2019.
[14] Ian J. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. CoRR, abs/1701.00160, 2017.
[15] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014.
[16] Jackson Gorham and Lester Mackey. Measuring sample quality with stein's method. In Advances in Neural Information Processing Systems, pages 226-234, 2015.
[17] Jackson Gorham and Lester Mackey. Measuring sample quality with kernels. In Proceedings of the 34th International Conference on Machine Learning- org, 2017.
Learning-Volume Volume 70, pages 1292-1301. JMLR.
[18] Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard S. Zemel. Cutting out the middle-man: Training and evaluating energy-based models without sampling. CoRR, abs/2002.05616, 2020.
[19] Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NIPS, pages 5767-5777, 2017.
[20] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, pages 1352 1361, 2017.
[21] Geoffrey E. Hinton. Product of experts. In ICANN Artificial Neural Networks, 1999.
[22] Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527-1525) 2006.
[23] Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. CoRR,
2018
[24] Aapo Hyvärinen. Estimation of non- non-normalized statistical models by score matching. J. Mach. Learn. Res., 6:695-709, 2005.
[25] Taesup Kim and Yoshua Bengio. Deep directed generative models with energy- based probability estimation. In ICLR, 2017.
[26] Durk P Kingma and Prafulla Dhariwal. Glow; Generative flow with invertible 1xl convolutions. In Advances in neural information processing systems, pages 10215-10224, 2018
[27] Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu Jie Huang. tutorial on energy- mbsed learning. Predicting Structured Data, MIT Press, 2006.
[28) Yingzhen Li and Richard F. Turner. Gradient estimators for implicit models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
[29] Tengyuan Liang and James Stokes. Interaction matters: A note non-asymptotic local convergence of generative adversarial networks. In AISTATS, pages 907-915, 2019.
[30] Qiang Liu, Jason D. Lee, and Michael I. Jordan. A kernelized stein discrepancy for goodness- of-fit tests. In ICML., pages 276-284, 2016.
[31] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In NIPS, pages 2370-2378, 2016.
[32] Qiang Liu and Dilin Wang. Learning deep energy models: Contrastive divergence vs. amortized MLE. CORR, abs/1707.00 abs/11077.0079, 2017.
[33] Vaishnavh Nagarajan and J. Zico Kolter. Gradient descent GAN optimization is locally stable. In NIPS, pages 5585-5595, 2017.

---

[34] Radford M. Neal. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. Handbook of Markov Chain Monte Carlo. 2. 2011.
[35] Jiquan Ngiam, Zhenghao Chen, Pang Wei Koh, and Andrew Ng. Learning deep energy models. In ICML, pages 1105-1112, 2011.
[36] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In CVPR, pages 5510-3520, 2017.
[37] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Yingnian Wu. On learning non-convergent non-persistent short-run meme toward energy-based model. CoRR, abs/ (bs//194.09970, 2019.
[38] Chris J Oates. Mark Girolami, and Nicolas Chopin. Control functionals for monte carlo integration. Journal of the Royal Statistical Society, Series B, 2017.
[39] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
[40] Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information processing systems, pages 2018-2028, 2017.
[41] Ruslan Salakhutdinov and Geoffrey E. Hinton. Deep boltzmann machines. In AISTATS, pages 448 455, 2009.
[42] Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. CoRR, abs/1805.08806, 2018.
[43] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyúiski. Lectures on stochastic programming: modeling and theory. SIAM, 2009.
[44] Chenyang Tao, Shuyang Dai, Liqun Chen, Ke Bai, Junya Chen, Chang Liu, Ruiyi Zhang, Georgiy V. Bobashev, and Lawrence Carin. Variational annealing gans: A langevin perspec- tive. In ICML, pages 61766618 6185, 2019.
[45] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008
[46] David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017
[47] Nik Weaver. Lipschitz algebras. World Scientific, 1999.
[48] Ying Nian Wu, Song Chun Zhu, and Xiuwen Liu. Equivalence of julesz ensembles and FRAME models, International Journal of Computer Vision, 38(3):247-28 2000.
[49] Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via MCMC teaching. In AAAI, pages 4292-4301, 2018.
[50] Jianwen Xie, Yang Lu, Song- Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. CoRR, abs/ (60009908, 2016.
[51] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu, A theory of generative convnet. In ICML., pages 2635-2644, 2016.
[52] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. A theory of generative convnet. In ICML, pages 2635-2644, 2016.
[53] Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In ICML, pages 1100-1109, 2016.
[54] Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In ICLR, 2020.
12

---

[55] Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial networks. In ICLR, 2017.
[56] Song Chun Zhu, Ying Nian Wu, and David Mumford. Minimax entropy principle and its application to texture modeling. Neural Compulation, 98):1627-1660, 1997.
13


Reviews:
[{"review_id": "QB9IlEESCNZ", "reviewer": "Reviewer_c3DX", "summary": "Explicit and implicit model training have their own advantages and limitations. The author proposes a joint training framework, which aims to combine the benefits from both worlds. Specifically, the author proposes a training framework called **Stein bridging** that allow joint learning of explicit and implicit models.  Particularly, the explicit model considered in this paper is the energy-based model (EBM) and the implicit model is the GAN-like generator. The implicit model is trained based on GAN framework, the explicit model is learned via Stein discrepancy, and the bridge between them is also the Stein discrepancy between the generated samples and the explicit model. \n\nTheoretically, the author proves that the Stein bridging can (1) act as a dual Sobolev norm regularization for the discriminator that helps the GAN training; (2) act as a Moreau-Yoshida regularization for the Stein discrepancy to smooth the energy function; (3) also help to stabilize the training dynamics. \n\nEmpirically, the author conducts extensive experiments to confirm the effectiveness of the proposed bridging, in terms of generating quality and OOD detection. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe main idea of briding is simple and straightforward. However, the theoretical analysis is the main contribution of the paper and seems to be non-trivial. It provides some insights into the regularization effect of the Stein bridging. The author also addresses the differences between this work and the previous ones in the appendix. So in summary, I think this work is interesting and novel. \n\n### Quality and Clarity\nThe main idea is easy to follow and clearly written. Some of the simple toy examples (from Theorem 1 and Theorem 2) help to visualize the effect of the regularization. However, I have several concerns regarding the Stein discrepancy and proof. I think there is still a large room for improvements of the quality. I will elaborate on them in the question section. \n\n### Significance\nThis proposed framework seems to have many desirable advantages in terms of implicit model training stability and explicit model training quality. In terms of theory, the author shed some light on the effect of joint learning of these two types of models. So it should have some impacts in this community. \n\n### Questions\n1. The first question regards the usage of Stein discrepancy. Using Stein discrepancy to train models can be quite problematic. For example, the Stein discrepancy can be highly non-convex and does not agrees with commonly used divergence (e.g. KL divergence). In other words, when use Stein discrepancy as the training objective (in you case, the learning of explicit model and Stein bridging), a lower discrepancy value does not necessarily mean a good model (the two densities $P_E$ and $P_G$ can be arbitrarily far away in KL divergence sense) unless the global optimum is achieved [1]. This is particularly problematic for gradient-based optimization. So does this pathology of Stein discrepancy affect the Stein bridging? Or the Stein bridging only act as regularization and has no direct impact to model learning. \n2. One quick check for question 1 could be replacing the components in Two-Circle example by Student-t distributions. The EBM is initialized as a mixture of Student-t distribution with the same scale and degree-of-freedom but different means (need to be far-away from the ground truth).  I wonder about the behaviours of the explicit and implicit models, and its training dynamics. \n3. For the definition of Stein discrepancy, when $d'\\neq d$, is the corresponding Stein discrepancy a valid one? Does the max-pooing matrix norm guarantee a valid discrepancy?\n4. I am a bit confused on why Sobolev dual norm penalizes the large variation of critic. Maybe consider adding some explanation on the intuition behind this norm?\n5. Maybe consider adding some intuitive explanations on Moreau-Yoshida regularization?\n6. What is $p_r$ in line 615, is it $P_{real}$?\n7. For KSD, the form mentioned below line 619 is different from the one you introduced in the main text. I understand this comes from integration by parts, but maybe explicitly mention this before line 619? Also, should it be $\\nabla_{\\pmb{x}}\\log d\\mathbb{P}_E/d\\mathbb{P}$?\n8. In line 629, why it is linear w.r.t. $h$? $h$ appears inside the $\\log$.\n9. I am confused about the derivation below the line 634. Why you can introduce the auxiliary variable $r$ and treat it as a decoupled parameter? $r$ should also depend on $h$ as well. \n10. For line 653, why $P$ is unconstrained? I thought that $P$ should be absolutely continuous to $P_E$, but $\\gamma$ only cares about the agreement of the marginal $P$ and $P_{real}$. So it is possible that $P$ is not absolutely continuous to $P_E$. \n11. Why it is equivalent to the set of point masses in line 660? \n\n\n### Reference\n[1] Barp, Alessandro, et al. \"Minimum stein discrepancy estimators.\" arXiv preprint arXiv:1906.08283 (2019)\n\n---\n### Upates after rebuttal\nI appreciated the response. After some discussions with the author, they managed to address my concerns. Therefore, I will raise my score accordingly. However, I still hope more intuitions and explanations about the kernel Sobolev and Moreau-Yoshida regularization can be added in the revised version.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper is a methodology paper. So there shouldn't be any negative societal impacts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "IUQTiRdMIA4", "reviewer": "Reviewer_cpPa", "summary": "===== After Rebuttal =====\n\nI have bumped up my score for the enhanced empirical results author(s) have provided in the discussion round. I remain conservative in my final recommendation because I am not fully convinced the gains are actually from the Stein energy penalty (i.e., some other regularization may also offer similar results) and the cost in modeling complexity may not be justified by the gains. \n\n=====================\n\nThis paper proposes to improve generative modeling via joining the strength of unnormalized density estimation (based on Stein discrepancy) and Wasserstein GAN. Numerical experiments verify the proposed solution enjoys better convergence on both synthetic and real-world datasets. Some theories are developed to justify the use of training loss. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work seeks to combine energy modeling and implicit generative modeling to improve the learning of probabilistic models. This direction has been extensively explored by various researchers in the past two years, with varying degrees of success. While the insight that the proposed formulation indeed implements Moreau-Yosida regularization, my major criticism is that there are too many moving parts in this model and thus making it overly complicated. It feels like gluing the components together, and such bridging is not particularly meaningful. More discussion is needed to explain why this work is significant compared to prior arts.\n\nAlso, while the author(s) have claimed that the integrated solution overcomes the individual limits of each component, it can totally go to the other extreme, with the interacting components amplifying each other's deficiencies. No experiments are designed to verify this would not be the case (i.e., intentionally testing for the robustness). And there is an established consensus that working with Stein discrepancy is quite tricky. It is computationally intensive and scales poorly to data dimensionality and complexity (due to its kernel formulation and dependency on gradient computation). Since the author(s) are quite vague on the actual implementation of Stein discrepancy in this paper (for general Stein discrepancy, what is $\\Gamma$, and for kernel Stein discrepancy, what kernel is used? ), I am unable to properly evaluate the effectiveness of the proposed Stein Bridging. \n\nOriginality. The techniques used in this work are very well known. The main novelty is to promote the consensus of model distribution and data distribution via minimizing the Stein discrepancy relative to the same energy model. I am not sure this is the right thing to do, as the consensus can be a bad one. \n\nQuality & Clarity: Writing needs to be improved, many statements are not proper English. For clarity, details on Stein discrepancy, its implementation, and the reasoning why the proposed methods should function as expected are missing in the current presentation. I am not sure about the technical correctness of the theories. \n\nSignificance: This work is unlikely to impact the community in a significant way. The solution is overly complicated and the theories are neither appealing to theorists nor practitioners. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Missing citations in paragraph 2 in Introduction\n- Line 34: \"the generated samples from implicit models could augment the dataset and help to alleviate mode collapse\". This is a false statement. This sounds more like one goal of generative modeling. \n- Please explain why the Moreau-Yosida encourages the model to seek more modes. It is not obvious. \n- Line 118, requiring the statistical distance to be computationally efficient seems unnecessary because the main bottleneck is the computation of Stein discrepancy, where gradient evaluations are involved. \n- Line 140, \"considers three discrepancies as a triangle\", what does this line mean?\n- Figure 4, all these competing solutions should work here, probably the author(s) should do a little bit of parameter tuning. Also, in Figure 5, the author(s) should compare the learned distribution from models in Figure 4.\n- The statement in Theorem 1 seems to be different from the problem (3), which also minimizes for G.\n- Experiments should include more complicated datasets. MNIST and Cifar10 do not adequately represent the complexity of real-world data, and that's what challenges Stein discrepancy based metrics. \n- Examples of bad English:\n\"mutual compensation -> synergy\"\n\"contaminated -> corrupted\"\n\"power->strength\"\n\"vivid->compelling\"\n\"arguably->\\\"\n\"illuminate-> illustrates\"\n\"oscillatory behavior->dynamics\"\n\"centrality->solution\"", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["Inadequate Data and Algorithm Evaluation"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "G9wFNO5ryAh", "reviewer": "Reviewer_p1LW", "summary": "The authors propose to jointly train implicit (like GANs) and explicit (EBMs) density models by proposing a new regularization technique that connects the training of the impicit and explicit model via Stein bridges that is based on stein discrepancy. The authors show that such a learning objective with a Stein bridge helps to overcome the drawbacks encountered while training an implicit or explicit model individually. The authors provide sufficient theoretical details in support of the method under the Wasserstein distance and perform experiments on toy examples and image datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:** The idea of using a combination of implicit and explicit models for joint training has been explored before in several papers recently (as the authors discuss in the detailed literature review section)  and is an interesting direction of research. Furthermore, the proposal to use mutual regularization of the two models to train such models is intriguing and novel. Additionally, the experiments presented in the paper provide promising results and improvements.\n\n**Cons:** \n- My major concern (which spans the previous works on jointly training implicit and explicit models as well) is the motivation behind the joint training of implicit and explicit model itself. It seems that this strategy introduces an expensive overhead of training two rather costly models to overcome the the drawbacks of the individual models. Furthermore, the solution itself does not directly address the drawbacks present in the individual models itself but indirectly makes things better for the models. While I liked the idea of incorporating the loss term between the implicit and the explicit model (D_3 in Table 1), I am still not clear what additional information this term provides during training apart from the data and the other two terms. \n\n- Secondly, the authors mentioned in the paper that any metric can be easily used for training the models (apart from Wasserstein or Jenson-Shanon etc). However, if I am not mistaken, the theoretical analysis presented by the authors in Section 4 is based on the Wasserstein metric. I wonder if these results will hold for other metrics as well? What impact do the authors foresee of different metrics on their analysis?\n\n- (*minor*): The authors train an implicit and explicit model together. Which of these models is finally used after training? Is it the case that the implicit model is used for sample generation and the explicit model for inference based tasks? \n\n-  The methodology presented here comes across as rather too costly for training an energy based model specially given the fairly light-weight methods present for training EBMs. I believe the paper will benefit from having a comparison between the training using the proposed methodology and these already present methods for training EBMs.\n\n- Finally, while the experiments comprehensively show that the proposed model achieves good results, I would have liked to see experiments on more complicated datasets than MNIST and CIFAR 10. This stems from my worry of the costly procedure of training two models jointly and how well this will scale to higher-dimensions and more complicated datasets. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed limitations briefly towards the end but I believe the authors can include a more thorough discussion on the implications of their results in general, the limitations of theoretical analysis (restricted to only Wasserstein metric) and cost of training two models jointly and scalability to higher dimensions. The authors have alluded to some of the issues in the paper but I think detailed comments are warranted here. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~6"}, "extra_text": {}}, {"review_id": "295HlGJu5s", "reviewer": "Reviewer_eewM", "summary": "This work proposes to train two models jointly, one explicit density estimator, and one implicit generative model. The idea is that the explicit (energy) model specifies a valid and easy-to-evaluate likelihood function, while the implicit model is flexible enough to draw high-quality samples. \nBecause the explicit model is an energy model, we don't have the normalizing constant, meaning that we cannot evaluate how close the explicit model is to the implicit model. The authors propose to minimize the stein discrepancy between the implicit model and the energy model, and the Wasserstein distance between the implicit model and the data. \n\nThe contributions are proposed as follows: \n* A joint learning framework from training an explicit energy model and an implicit model to learn to sample from a target distribution where we only have access to samples. \n* A training algorithm that trains the implicit model to sample via minimizing the Wasserstein distance then trains the energy model to learn an explicit density by minimizing the stein discrepancy between the energy model and the implicit model. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this work enjoyable. The idea of learning a deep energy model via the stein discrepancy has been proposed before, but the addition of an implicit generative model really seems to help distribution fit. This is possibly due to the increased size of the generative model function space. [1, 2] consider the space of L2 functions, which need to vanish at the boundaries -- necessitating potentially harmful regularization on the generator function. This method avoids this issue by training an implicit model, that is known to perform well for sample generation. \nFigure 4 and figure 5 were quite convincing to me. They show that both the implicit model and the energy model are able to capture the data distribution well, while their components trained independently more or less fail. \nOn a practical note, I believe it's clear from these toy examples that the learned distribution is more faithful with this joint training algorithm. But what is less clear is the advantage in high(er) dimensional inference tasks. The MNIST and CIFAR-10 results do not look much better than WGAN-GP. I see that the inception score is better for Joint-W, but the 2X training time cost is high given this margin. \n\nAs an aside, I found the fact that minimizing the stein discrepancy between the two models acts as a gradient penalty on the implicit model interesting. This might make sense since the energy model is \"slower\" than WGAN. \n\n\n[1] Grathwohl, Will, et al. \"Learning the stein discrepancy for training and evaluating energy-based models without sampling.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Hu, Tianyang, et al. \"Stein neural sampler.\" arXiv preprint arXiv:1810.03545 (2018).\n\n-----------------------\n### Update After Rebuttal\nAfter reading the other reviews/discussions, I agree that higher-dimensional inference tasks are important to evaluate on for this method. I do not intend to change my score, but I hope that the authors can show the applicability of their approach to tasks other than MNIST and CIFAR-10.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention that deep generative models can be used for generating false information, and therefore pose some risk. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Stein bridge as a generic framework to jointly train an explicit generative model as well as an implicit model. Some theoretical analyses are provided. Experiments are conducted on synthetic data as well as image generation with MNIST data.

Reviewers agree that the proposed method is new. However, as cited by the authors, there are previous works on the joint training view of EBMs and implicit models. In this sense, the paper focuses on a known research topic but with a new methodology.

Reviewers agree that the theoretical analysis result is non-trivial so indeed a contribution. However, some clarity concerns remain, which needs to be improved in revision.

The major concern from the reviewers is that the experimental results are not strong, and certainly in GAN literature, MNIST is already a toy-ish dataset. They are not convinced that there is strong evidence that the demonstrated advantage on modelling corrupted data comes from the proposed Stein bridging method.
Decision: Accept (Poster)