Mutual Calibration between Explicit and Implicit Deep Generative Models
Authors: Qitian Wu
Rui Gao
Hongyuan Zha
Venue:  (2021)
Keywords: deep generative models
generative adversarial networks
density estimation

Abstract: Deep generative models are generally categorized into explicit models and implicit models. The former defines an explicit density form that allows likelihood inference; while the latter targets a flexible transformation from random noise to generated samples. To take full advantages of both models, we propose Stein Bridging, a novel joint training framework that connects an explicit (unnormalized) density estimator and an implicit sample generator via Stein discrepancy. We show that the Stein bridge 1) induces novel mutual regularization via kernel Sobolev norm penalization and Moreau-Yosida regularization, and 2) stabilizes the training dynamics. Empirically, we demonstrate that Stein Bridging can facilitate the density estimator to accurately identify data modes and guide the sample generator to output more high-quality samples especially when the training samples are contaminated or limited.

Under review as a conference paper at ICLR 2021
MUTUAL CALIBRATION BETWEEN EXPLICIT AND IM- PLICIT DEEP GENERATIVE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep generative models are generally categorized into explicit models and im- plicit models. The former defines an explicit density form that allows likelihood inference; while the latter targets flexible transformation from random noise generated samples. To take full advantages of both models, we propose Stein Bridging. a novel joint training framework that connects an explicit (unnormal- ized) density estimator and an implicit sample generator via Stein discrepancy. We show that the Stein bridge 1) induces novel mutual regularization via kernel Sobolev norm penalization and Moseau-Yosida regularization, and 2) stabilizes the training dynamics. Empirically, we demonstrate that Stein Bridging can facil- itate the density estimator to accurately identify data modes and guide the sample generator to output more high-quality samples especially when the training sam- ples are contaminated or limited.
INTRODUCTION
Deep generative model, as a powerful unsupervised framework for learning the distribution of high- dimensional multi-modal data, has been extensively studied in recent literature. Typically, there are two types of generative models: explicit and implicit (Goodfellow et al. (014)). Explicit models define a density function of the distribution, while implicit models learn mapping that generates samples by transforming an easy-to- ta-sll random variable.
Both models have their own power and limitations. The density form in explicit models endows them with convenience to characterize data distribution and infer the sample likelihood. However, the unknown normalizing constant often causes computational intractability. On the other hand, implicit models including generative adversarial networks (GANs) can directly generate vivid samples in various application domains including images, natural languages, graphs, etc. (Goodfellow et al.) 2014 Radford et al. 2016 Arjovsky al. 2017; Brock al. 2019). Nevertheless, one important challenge is to design a training algorithm that do not suffer from instability and mode collapse. In view of this, it is natural to build unified framework that takes full advantages of the two models and encourages them to compensate for each other.
Intuitively, explicit density estimator and flexible implicit sampler could help each other's train- ing given effective information sharing. On the one hand, the density estimation given by explicit models can be a good metric that measures quality of samples (Dai et al. (0017). and thus can be used for scoring generated samples given by implicit model or detecting outliers as well as noises in input true samples (Zhai al., 2016). On the other hand, the generated samples from implicit models could augment the dataset and help to alleviate mode collapse especially when true samples are insufficient that would possibly make explicit model fail to capture an accurate distribution. We refer to Appendix A for more comprehensive literature review.
Motivated by the discussions above, in this paper, we propose a joint learning framework that enables mutual calibration between explicit and implicit generative models. In our framework, an explicit model is used to estimate the unnormalized density; in the meantime, an implicit generator model exploited to minimize certain statistical distance (such as the Wasserstein metric Jensen-Shannon divergence) between the distributions of the true and the generated samples. On top of these two models, a Stein discrepancy, acting as bridge between generated samples and estimated densities, is introduced to push the two models to achieve a consensus. Unlike flow-based models (Nguyen et al. 2017 Kingma & Dhariwal 2018 Papamakarios al. 2017), our formulation does not impose

---

Under review as a conference paper at ICLR 2021
invertibility constraints the generative models and thus is flexible in utilizing general neural network architectures. Our main contribution are as follows.
Theoretically, we prove that our method allows the two generative models to impose novel mutual regularization on each other. Specifically, of the critic in the implicit (WGAN) model. which ensures the critic not to change suddenly the high-density regions and thus preventing the critic of the implicit model being to strong during training. In the mean time, our formulation also smooths the function given by the Stein discrepancy through Moreau- Yosida regularization, which encourages the explicit model seek more modes in the data distribution and thus alleviates mode collapse.
formulation penalizes large kernel Sobolev norm
In addition, we also show that the joint training helps to stabilize the training dynamics. Compared with other common regularization approaches for GAN models that may shift original optimum, our method can facilitate convergence to unbiased model distribution.
Extensive experiments on synthetic and image datasets justify our theoretical findings and demon- strate that joint training can help two models achieve better performance. On the one hand, the energy model can detect complicated modes in data more accurately and distinguish out-of- distribution samples. On the other hand, the implicit model can generate hugher-quality samples, especially when the training samples are contaminated or limited.
BACKGROUND
We briefly provide some technical background related to our model.
Energy Model. The energy model assigns each data E IRd with a scalar energy value (x), where E-(-) is called energy function and is parameterized by o. The model expected to assign low energy true samples according to Gibbs distribution (x)
where Ze a normalizing constant dependent of cb The normalizing term Z is often hard to compute, making the training intractable, and various methods proposed to detour such term (see Appendix AA).
(x)
Stein Discrepancy. Stein discrepancy (Gorham & Mackey] 2015 Liu et al. 2016 CChiialkossii al. 2016 Oates al.) 2017] Grathwohl et al. (2020) is measure of closeness between two probability distributions and does not require knowledge for the normalizing constant of one of the compared distributions. Let P and Q be two probability distributions on X' C Rd. and assume Q has a (unnormalized) density The Stein discrepancy S S(I,, Q) is defined as
S(P, Q) := sup Ex-r(Agf(x) == sup (T(Ex-P [Vx log (xx) + vxf(x)))), fej
FEF
where F is often chosen to be a Stein class (see, e.g., Definition 2.1 in (Liu et al. 2016
f Rdd + Rdd is a vector-valued function called Stein critic and T is an operator that transforms a d x d' matrix into scalar value. One common choice of T is trace operator when d' = d. One can also use other forms for I, like matrix norm when d' * d (Liu et al., (016)). If F is unit ball in some reproducing kernel Hilbert space (RKHS) with positive definite kernel k, it induces Kernel Stein Discrepancy (KSD). More details are provided in Appendix B
Wasserstein Metric. Wasserstein metric is suitable for measuring distances between two distri- butions with non-overlapping supports (Arjovsky et al. 2017). The Wasserstein-1 metric between distributions TP and Q is defined as
W(P, Q) = min [xxx)>>Ilxx - yll,,
where the minimization with respect to 7 is over all joint distributions with marginals IP and Q. By Kantorovich-Rubine duality, W(P, Q) has a dual representation
W(P, Q) = max (Ex~p(D(x)) - Ey~al D
(2)
where the maximization is over all I-Lipschitz continuous functions.
Sobolev space and Sobolev dual norm. Let L (P) be the Hilbert space on Rd equipped with an inner product (u, ())) := S&x wxdP(x). The (weighted) Sobolev space H² is defined as the closure of Cg set of smooth functions on Rd with compact support, with respect to norm ||u|| by := (Jge (u² 2 +|Vu|[2))P()) 1/2 where P has a density. For E T,2, its Sobolev dual norm
2

---

does not enforce any constraints or assume specific density forms. For specifications, one can also consider other explicit models, like autoregressive models or directly using some density forms such as Gaussian distribution with given domain knowledge. The implicit model transforms easy-to- sample random noise z with distribution Po via generator G to a sample = G(z) with distribution Fo. Note that for distribution Pg. have its explicit density without normalizing term, while for Pa and IP real. we have samples from two distributions. Hence, we can use the Stein discrepancy (that does not require the normalizing constant) tribution Pg and the real distribution Preat, and use the Wasserstein metric (that only requires only measure of closeness between the implicit distribution Pa and Under review as a conference paper at ICLR 2021
||l| is defined by (Evans 2010
|| -
sup NEH H1
(u, u) JRd
|V dP(x)) 1, u(x)dP(x) - =0]
The constraint Jiz u(x)dx = 0 is necessary to guarantee the finiteness of the supremum, and the supermum can be equivalently taken over Co.
PROPOSED MODEL: STEIN BRIDGING
In this section, we formulate our model Stein Bridging. A scheme of our frame- work is illustrated in Figure Denote by Preat the underlying real distribution from which the data {x} are sampled. The for- mulation simultaneously learns two gener- ative models one explicit and one im- plicit that represent estimates of Preal- The explicit generative model has a dis- tribution PE on X' with explicit probabil- ity density proportional to exp(-F(xx)) E X. where E' is referred as an en- engy function. We focus on energy-based explicit model in model formulation it
Implicit Model Pc
Explicit Model
Dz(x)
[
Po(*)
Wasserstan Critc
True Sample
Critc
Stain
True Denstyy
Pol-1
Density Estimattr
Gg(x)
Generator
Sumple
Faloo Sample
Crttcc
Danaity
Naise
(u())
(X)
Stein Bridge
BP
Update Impact
P(P
D2 S(P Pp)
14g S(Po.Pg)
Figure I: Model framework for Stein Bridging.
measure of closeness between the explicit dis-
samples from two distributions) the real data distribution Peett-
To jointly learn the two generative models P& and Igg, arguably the most straightforward way is to minimize the sum of the Stein discrepancy and the Wasserstein metric:
min W(Preal, Pa) + AS IPE), E.G
where À 0. However, this approach appears no different than learning the two generative mod- els separately. To achieve information sharing between two models, we incorporate another term S(Pa, Px) called the Stein bridge that measures the closeness between the explicit distribution Pg and the implicit distribution Pa:
min ((Preat, Po) +,S(( reat, PF) + (PP E.G
where \1, A2 2 The Stein bridge term in (3) pushes the two models to achieve consensus.
Remark 1. Our formulation is flexible in choosing both the implicit and explicit models. In ( we can choose statistical distances other than the Wasserstein metric (Prrt,,, to measure closeness between Preat and PG. such as Jensen-Shannon divergence, as long as its computation requires only samples from the involved two distributions. Hence, one can use GAN architectures other than WGAN to parametrize the implicit model. In addition, term S(P PE)) in (3) by other statistical distances as long as its computation is efficient and hence other explicit models can be used. For instance, if the normalizing constant of Pg is known easy to calculate, one can use Kullback-Leibler (KL) divergence.
can replace the first Stein discrepancy
The choice of the Stein discrepancy for the bridging term (F, is erucial and cannot be replaced by other statistical distances such as KL divergence, since the data- mata-generating distribution does not have an explicit density form (not even up to normalizing constant). This
Remark 2.
3

---

Under review as a conference paper at ICLR 2021
exactly important reason why Stein bridging was proposed, which requires only samples from the data distribution and only the log-density of the explicit model without the knowledge of normalizing constant estimated in MCMC other methods.
In our implementation, we parametrize the generator in implicit model and the density estimator in explicit model as Gz(z) and Po (x),,, respectively. The Wasserstein term in is implemented using its equivalent dual representation in (2) with a parametrized critic D (x). The two Stein terms in neural network, i.e., to (x)), or the non-parametric Kernel Stein Discrepancy. Our implementation iteratively updates the
3) can be implemented using [T) with either Stein critic (parametrized
explicit and implicit models. Details for model specifications and optimization are Appendix [E.2]
We also compare with some related works that attempt to combine both of the worlds (such as energy-based GAN, contrastive learning and cooperative learning) in Appendix A.3
THEORETICAL ANALYSIS
In this section, we theoretically show that the Stein bridge allows the two models facilitate each other's training by imposing certain regularizations on both the implicit and the explicit models, as well as stabilizing the training dynamics.
4.1 REGULARIZATION VIA STEIN BRIDGE
We first show the regularization effect of the Stein bridge on the Wasserstein critic. Define the kernel Sobolev dual norn as
== sup (DD,)) wEC
Ex.x<P/Vuu(X l(xx x Vu(')) S 1, [x[u] = 0}.
which can be viewed as kernel generalization of the Sobolev dual norm defined in Section 2 which
reduces to the Sobolev dual norm when x(x,x)== = [[x = x') and IP is the Lebesgue measure.
Theorem 1. Assume that {Fata exhausts all continuous probability distributions and is chosen as kernel Stein discrepancy. Then problem 3 is equivalent to
min max Ey Ey [(y)] - Ex-P [D(x]
I%, [1D1 + A1 P
The kernel Sobolev norm regularization penalizes large variation of the Wasserstein critic D. Par- ticularly, observe that (Villani 2008) if k(x,xx) T[xx x') and Epg [D) 0, and then
Wa((1 + (D)P
D ( lim
where W2 denotes the 2-sserrsteiin metric. Hence, the Sobolev dual norm regularization ensures D not to change suddenly on high-density region of Pr. and thus reinforces the learning of Wasserstein critic. Stein bridge penalizes large variation of the Wasserstein critic, in the same spirit but of different form comparing to gradicnt-based penalty (c.g.. (Gulrajani et al.] 2017: Roth et al.) 2017 It prevents Wasserstein critic from being too strong during training and thus encourages mode exploration of sample generator. To illustrate this, we conduct case study where we train generator over the data sampled from mixture of Gaussian (/1 [-1, -1,, /22 [1, 1] and Σ = 0.2I). In Fig. 2(a) we compare gradient norms of the Wasserstein critic when training the generator with and without the Stein bridge. As we can see, Stein bridge can help reduce gradient
norms through training, with a similar effect as WGAN-GP.
Moreover, the Stein bridge also plays a part smoothing the output from Stein discrepancy and show the result in the following theorem.
Theorem 2. Assume {PatG exhausts all continuous probability distributions, and the Stein class defining the Stein discrepancy is compact (in some linear topological space). Then problem 3 is equivalent to
min A1 PE) + X2 max Ex.R [(Ap (x)1
where (Ap ()s(() denotes the (generalized) Moreau Yosida regularization of the function Arg with parameter X2, Le. (Arg f)x, (x) = m (Ar f(y) 2/2 x-yl)..
4

---

Under review as a conference paper at ICLR 2021
(a)
(h)
(c)
(d)
Figure 2: (a) The gradient norm Wasserstein critic with (blue) and without (red) the Stein bridge when data
are sampled from mixture of Gaussian. (b) Contour of an energy model with one mode and empirical data
from distribution with a different mode (blue dots); (c) & (d) Contours of the Stein critics between the two distributions in (b) learned with and without the Stein bridge, respectively.
Note that (Arg. is Lipschitz continuous with constant 1/Ag. Hence, the Stein bridge, together with the Wasserstein metric ((Freat: PG). plays as a Lipschitz regularization on the output of the Stein operator Ar f via Morcau- for Stein-based GAN. By smoothing the Stein critic, the Stein bridge encourages the energy model to seek more modes in data instead of focusing some dominated modes, thus alleviating mode- collapse issue. To illustrate this, we consider a case where we have an energy model initialized with one mode center and data sampled from distribution of another mode, depicted in Fig. b Fig. 2[c) and d)) compare the Stein critics when using Stein bridge and not, respectively. The Stein bridge helps smooth the Stein critic, as indicated by less rapidly changing contour in Fig. (() compared to Fig. d), learned from the data and model distributions plotted in Fig. (b).
regularization. This suggests a novel regularization scheme
4.2 TRAINING STABILITY
In this subsection, we further show that Stein Bridging could help stabilize adversarial training between genera- tor and Wasserstein critic with local convergence guar- antee. As is known, the training for minimax game in GAN is difficult. When using traditional gradient meth- ods, the training would suffer from some oscillatory be- haviors (Goodfellow 2017| Liang & Stokes] 2019 Zhang Yu 2020). In order to better understand opti- mization behaviors,
Stele Beilging WIANN
WGAN LR-VA
WGAN- ER
WSAN
Optimum Pont
first compare the behaviors of WGAN, and Stein Bridging under SGD via easy to compre- hend toy example one-dimensional case. Fig. 3 shows numerical results that compare the optimization behav- iors of above methods. As we can see, Stein Bridging achieves good convergence the optimum point, while
WGAN, likelihood- and entropy-regularize
Figure 3: Numerical SGD updates Stein Bridging, WGAN and its variants with dif- ferent regularizations.
WGAN suffers from an oscillation instead of converging. Entropy regularization (ER) can encour- age the generator to seek more modes but would make the model diverge in this case. By contrast, likelihood regularization (LR) can help for training stability but it changes the converging point to a biased distribution. A recently proposed variational annealing strategy (VA) (Tao al. 2019) for regularized GAN introduces trade-off between convergence and unbiased result. The detailed discussions and proofs are presented in Appendix D.I We also generalize the convergence results to multi-dimensional bilinear system F(sb, 0) O Axb b 0 c so in Appendix D.2 Our theoretical results indicate that Stein Bridging could stabilize the minimax training of GAN without changing its optimum. In the experiments, will empirically validate our analysis.
EXPERIMENTS
In this section, conduct experiments to verify the effectiveness of proposed method from mul- tifaceted views. We consider two synthetic datasets with mixtures of Gaussian distributions: Two- Circle and Two-Spiral. The first one composed of 24 Gaussian mixtures that lie in two circles. Such dataset is extended from the (Gaussian-mixtum scenario widely used in previous papers, so
The experiment codes will be relcased.
5

---

Under review as a conference paper at ICLR 2021
Figure 4: Comparison for density estimation. (a) True densities of real distribution and (b)~(f) estimated densi- ties given by the estimators of different methods on Two- Circle (upper line) and Two- Spiral (bottom line) datasets.
S
Comparison for generated sample quality. (a) True samples from real dis- tribution and (b)~((f) gener- ated samples produced by the generators of different meth- ods Two-Circle (upper line) and Two-Spiral (bottom line) datasets.
Figure 5:
S
(a) True
(b) GAN
(c) WGAN
(d) DGM
(e) Joint-JS
(f) Joint-W
that we use it to test the quality of generated samples and mode coverage of learned energy. The second dataset consists of 100 Gaussian mixtures whose centers are densely arranged on two centrally symmetrical spiral-shaped curves. This dataset be used examine the power of gen- erative model on complicated data distributions. The ground-truth distributions and samples are shown in Fig. a) and Fig. 5 a() Furthermore, we also apply the method to MNIST and CIFAR datasets which require the model to deal with high-dimensional image data. In each dataset, we use observed samples as input of the model and leverage them train our model. The details for each dataset are reported in Appendix E.
We term the model Joint- W if using Wasserstein metric 3 and Joint-JS if using JS divergence in this section. We consider several competitors. For implicit generative models, we basically consider the counterparts without joint training with energy model, which are equivalently valina GAN and WGAN with gradient penalty (Gulrajani et al. 2017). for ablation study. Also, as comparison to the new regularization effects by Stein Bridging, we consider a recently proposed variational annealing regularization (Tao et al. 2019) for GANs (short
We employ denois- ing antoder to estimate the gradient for regularization penalty, which is proposed by (Alain & Bengio 2014). For explicit models, we also consider the counterparts without joint training with generator model, i.e., directly training Deep Energy Model (DEM) using Stein discrepancy (Grath- wohl et al. 2020). Besides compare with energy calibrated GAN (EGAN) (Dai et al., 2017) and Deep Directed Generative (DGM) Model (Kim & Bengio 2017) which adopt contrastive divergence train a sample generator with an energy estimator. See Appendix A for brief introduction of these methods and Appendix E.3 for implementation details.
5.1 DENSITY ESTIMATION OF EXPLICIT MODEL.
One advantage of joint learning is that the gener- ator could help the density estimator capture more accurate distribution. As shown in Two-Cirele case in Fig 55] both Joint-JS and Joint-W manage to capture all Gaussian components while other methods miss some of modes. In Two-Spiral case in Fig Joint-JS and Joint-W exactly fit the ground-truth distribution. Nevertheless, DEM misses one spiral while EGAN degrades to uniform- like distribution. DGM manages to fit two spirals but allocate high densities to regions that have low densities in the groung-truth distribution. As quantitative comparison, we study three evaluation metries: KI. & IS divergence and Area Under the Curve (AUC). The detailed information and re- sults are given in Appendix E.4 and Table Jaspeejjespecctively The values show that Joint- W and Joint-JS provide better density estimation than all competitors over a large margin.
Mode Coverage for Complicated Distributions.
Density Rankings for High- Biigh-Dimmsioonll Digits. We also rank generated digits (and true digits) on MNIST w.r.t densities given by the energy model in Fig. [1] Fig. 12 and Fig. 13 As depicted in the figures, the digits with high densities (or low densities) given by Joint-JS possess enough diversity (the thickness, the inclination angles as well as the shapes of digits diverses). By constrast, all the digits with high densities given by DGM tend to be thin and digits with low densities are very thick.

---

Under review as a conference paper at ICLR 2021
Also, for EGAN, digits with high (or low) densities appear to have the same inclination angle (for high densities, 1I' keeps straight and '9' 'leans' to the left while for low densities. just the opposite), which indicates that DGM and EGAN tend to allocate high (or low) densities to data with certain modes and miss some modes that possess high densities in ground-truth distributions. By contrast, our method manages capture these complicated features in data distributions.
Detection for Qut-of-distribution Samples. We further study model performance on detection for out-of-distribution samples. We consider CIFAR-I0 images as positive samples and construct negative samples by (I) flip images, (II) add random noise, (III) overlay two images and (IV) use images from LSUN dataset, respectively. A good density models trained on CIFAR- 10 are expected to give high densities to positive samples and low densities to negative samples, with exception for case (I) (fipping images are not exactly negative samples and the model should give high densities). We use the density values rank samples and calculate AUC of false positive rate v.s. true positive rate, reported in [ Our model Joint-W W manages to distinguish samples for (II), (III, (IV) and is not fooled by flipping images, while DEM and EGAN fail to detect out-of-distribution samples and DGM recognizes flipping images as negative samples.
5.2 SAMPLE QUALITY OF IMPLICIT MODEL
Generated Samples over Synthetic Datasets.
Table I: Inception Scores (IS) and Fréchet Incep- tion Distance (FID) on CIFAR-10.
Calibrating explicit (unnormalized) density model with implicit generator is expected to improve the quality of generated samples. In Fig. [ we show the results of different genera- tors in Two-Circle and Two-Spiral datasets. In Two-Cirele, there are large number of gener- ated samples given by GAN, WGAN-GP and DGM locating between two Gaussian compo- nents, and the boundary for each component is not distinguishable. Since the ground-truth densities of regions between two components are very low, such generated samples possess low- quality, which depicts that these models capture the combinations of two dominated features (i.e.. modes) in data but such combination makes no sense in practice. By contrast, Joint-JS and Joint- W could alleviate such issue, reduce the low-quality samples and produce more distinguishable bound- aries. In Two-Spiral, similarly, the generated samples given by GAN and WGAN-GP form circle instead of two spirals while the samples of DGM 'link' two spirals. Joint-IS manages to focus more on true high densities compared to GAN and W provides the best results. To quantitatively measure the sample quality, we adopt Maximum Mean Discrepancy (MMD) and High-quality Sam- ple Rate (HSR). The details are in Appendix B.4] and we report results in Table 5 where our models significantly outperform the competitors over a large margin.
Method
IS
FID
WGAN-GP
6.7410.041
42.2:10.572
Energy GAN
6.89=00081
45.6:10.375
WGAN+VA
6,90++0.058
45.3++0.307
DGM Joint-W(ours)
6.51++0.041
08.8++0.422
7.12+t0.101
41.0++0.546
Sample Quality for Generated Images. We calculate the Inception Score (IS) and Fréchet Incep- tion Distance (FID) to measure the sample quality on CIFAR-10. As shown in Table [ Joint- W outperforms other competitors by 0.2 and achieves 5.6% improvement over WGAN-GP w.r.t IS. As for FID, Joint-W slightly outperforms WGAN-GP and beats energy-based GAN and variational annealing regularized WGAN over large margin. One possible reason that these methods both consider entropy regularization which encourages diversity of generated samples but will have a negative effect on sample quality. Stein Bridging can overcome this issue via joint training with explicit model. The performance of DGM tends to be much worse than others. In practice, DGM is hard for convergence and suffers from severe instability in training.
Model Performance in Contaminated or Limited Data. As further discussions, we highlight that Stein Bridging has promising power in some extreme cases where the training sample are con- taminated or limited. We consider noised data scenario and randomly add n noise points sam- pled from Gaussian distribution (00 ool) where og = 2 to the original true samples in Two- Circle dataset. The results on noised dataset are presented in Fig. B(a) where we set noise ratio n = [40, 100, 160, 300, 400, 600, 800, 1000] and report the HSRs of Joint-W and WGAN-GP. The noise ratio in data impacts the performance of WGAN-GP and Joint- W. but comparatively, the per- formance decline of Joint- W is less insignificant than WGAN- which indicates better robustness of joint training w.r.t. noised data.
7

---

Under review as a conference paper at ICLR 2021
AUCs for out-of-distriutiin sample detection on CIFAR-10. We use negative samples from (I) flip images, (II) add random noise, (III) overlay two im- ages and (IV) use images from LSUN dataset.
Table 2:
AUC
111
IV
Ieeration
Eoration
Joint-W W
0,50
0.92
0.95
0.85
(a) Two-Circle
(b) Two-Spiral
DEM
0.50
0.52
0.51
0.56
Figure 6: Learning curves of Joint-W (resp. Joint-IS) compared
DGM
1.00
1.00
1.00
0.82
with WGAN (resp. GAN) and its regularization-ban variants on Two-Cirele and Two-Spiral datasets.
EGAN
0.50
0.42
0.30
0.52
Epoch 410
4
3
[
3
8
7
4
3
4
0
7
5
Epoch 420
7
5
9
9
7
6
1
1
4
7
5
Epoch 430
2
1
5
8
7
6
4 {
1
4
0
7
5
Joint WGGN
Joine
Epoch 440
2
1
0
8
9
7
8
3
4
0
7
DEM
Epoch 450
4
3
5
8
D
9
6
1
2
4
0
5
Notoe Rado 180
True Sample Size
WGAN-GP
WGAN+VA
Ours
(a)
(b)
Figure 7: Generated digits given by the same noise a in adja- Figure 8: Impact of (a) noise in data and (b) cent training epochs on MNIST.
insufficient data on model performance.
To study the impact of insufficient data, in Fig. B),, we consider sample size N₂ as [100, 200, 300, 500, 700, 1000, 2000) in Two-Spiral dataset and report the AUC of Joint-W and DEM. When sample size decreases from 2000 to 100, the AUC value of DEM declines dramat- ically, showing its dependency
sufficient training samples. By contrast, the AUC of Joint W exhibits a small decline when the sample size is more than 500 and suffers from an obvious decline when it is less than 300. Such phenomenon demonstrates its lower sensitivity to data size.
5.3 ENHANCING THE STABILITY OF GAN
Joint training also helps stabilize training dynamics. In Fig. 6 we present the learning curves of Joint-W W (resp. Joint-JS) compared with WGAN (resp. GAN) and likelihood- and entropy- regularized WGAN (resp. GAN). The curves depict that joint training could reduce the variance of metric values especially during the second half of training. Furthermore, we visualize gener- ated digits given by the same noise adjacent epochs Fig. 7 The results show that Joint- W gives more stable generation in adjacent epochs while generated samples given by WGAN-GP and WGAN+VA exhibit an obvious variation. Especially, some digits generated by WGAN-GP and WGAN+VA change from one class to another, which is quite similar to the oscillation without con- vergence discussed in Section 3.2. To quantify the evaluation of bias in model distributions, we calculate distances between the means of 50000 generated digits (resp. images) and 50000 true dig- its (resp. images) in MNIST (reps. CIFAR-10). The results are reported in Table We can see that the model distributions of other competitors more biased from true data distribution, compared with Joint-W.
6 CONCLUSIONS
In this paper, we aim at uniting the training for implicit generative model (represented by GAN or WGAN) and explicit generative model (represented by deep energy-based model) via an bridging term of Stein discrepancy between the generator and the energy-based density estimator. Theoret- ically, we show that joint training could i) enforce dual regularization effects on both models and thus encourage mode exploration, and ii) help to facilitate the convergence of minimax training dy- namics. We also conduct extensive experiments on different tasks and applications to verify our theoretical findings as well demonstrate the superiority of our method compared with training generator models or energy-based models alone. Our formulation is flexible in handling various implicit explicit models. As such, for future works, one try other generative models such as VAE or flowed-be based model as replacement for our GAN and energy-basse based models. would also be interesting to exploit our formulation in the context of few-shot learning in generative models.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res., 15(1):3563-3593, 2014.
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, pp. 214-223, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In ICML,, pp. 1606-2615, 2016.
Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard H. Hovy, and Aaron C. Courville. Calibrat- ing energy-based generative adversarial networks. In ICLR, 2017.
Chao Du, Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Learning implicit generative models by teaching explicit ones. CoRR, abs/1807.00 807.03870, 2018.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-basee models. CoRR, abs/1903.08688 2019.
L.C. Evans. Partial Differential Equations. Graduate studies in mathematics. American Mathemat- ical Society, 2010. ISBN 978147041 14422 URL http:::t/t.sssss...ttoo/ google e. www/bookks/
Ad=CI.9Wt.EAAAAJ
Stuart Geman and Donald Geman Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. JEEE Trans. Pattern Anal. Mach. Intell., 66)):721-74,, 1984.
Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of gans using variational inequalities. CoRR, abs/1808.01534 2018.
Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A varia- tional inequality perspective on generative adversarial networks. In ICLR, 2019.
lan Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. CoRR, abs/1 700.00016, 2017.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Jackson Gorham and Lester Mackey. Measuring sample quality with stein's method. In Advances in Neural Information Processing Systems, pp. 226-234, 2015.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard S. Zemel. Cutting out the middle-man: Training and evaluating energy-based models without sampling. CoRR, abs/2002.05616, 2020.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein gans. In NIPS, pp. 5767-5777, 2017.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, pp. 1352-1361, 2017.
Geoffrey F. Hinton. Product of experts. In ICANN'99 99 Artificial Neural Networks, 1999.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 1877)1157-11554 2006.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. CoRR, abs/18 10000180,33555, 2018.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6:695-709, 2005.

---

Under review as a conference paper at ICLR 2021
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy- based probability
estimation. In ICLR, 2017.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc' Aurelio Ranzato, and Fu Jie Huang. A tutorial on seergy-baseed learning. Predicting Structured Data, MIT Press, 2006.
Yingzhen Li and Richard E. Turner. Gradient estimators for implicit models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings, 2018.
Tengyuan Liang and James Stokes. Interaction matters: A note non-asymptotic local convergence of generative adversarial networks. In AISTATS, pp. 907-915, 2019.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. In NIPS, pp. 2370-2378, 2016.
Qiang Liu and Dilin Wang- Learning deep energy models: Contrastive divergence amortized
MLE. CoRR, abs/ 0ss/17077007977 2017.
Qiang Liu, Jason D. Lee, and Michael I. Jordan. A kernelized stein discrepancy for goodness-of-ie 2016.
tests. ICML, pp.
Vaishnavh Nagarajan and Zico Kolter. Gradient descent GAN optimization is locally stable. In NIPS, pp. 5585-5595, 2017.
Radford M. Neal. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. Handbook of Markov Chain Monte Carlo, 2, 2011.
Jiquan Ngiam, Zhenghao Chen, Pang Wei Koh, and Andrew Y. Ng. Learning deep energy models. In ICMI., pp. 1105-1112 2011.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug play generative networks: Conditional iterative generation of images in latent space. In CVPR, pp. 3510-3520, 2017.
Erik Nijkamp, Mitch Hill, Song-Chin Long-Chun Zhu, and Yingnian Wu. On learning con-convergent non- persistent short-run meme toward energy- based model. CoRR, abs/1904.099770, 2019.
Chris J Oates, Mark Girolami, and Nicolas Chopin. Control functionals for monte carlo integration. Journal of the Royal Statistical Society, Series B, 2017.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information pro- cessing systems, pp. 2018-2028, 2017.
Ruslan Salakhutdinov and Geoffrey E. Hinton. Deep boltzmann machines. In AISTATS, pp. 448-
455, 2009.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezyúski. Lectures on stochastic program- ming: modeling and theory. SIAM, 2009.
Chenyang Tao, Shuyang Dai. Liqun Chen, Ke Bai. Junya Chen, Chang Liu, Ruiyi Zhang, Georgiy V. Bobashev, and Lawrence Carin. Variational annealing of gans: A langevin perspective. In ICML, pp. 6176-6185, 2019.
10

---

Under review as a conference paper at ICLR 2021
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
Nik Weaver. Lipschitz algebras. World Scientific, 1999.
Ying Nian Wu, Song Chun Zhu, and Xiuwen Liu. Equivalence of julesz ensembles and FRAME models. International Journal of Computer Vision, 38(3):247-265, 2000.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. CoRR, abs// 609.09408, 2016a.
Jianwen Xie, Yang Lu, Song- Chuu Zhu, and Ying Nian Wu. theory of generative convnet. In 2016b.
ICML, pp.
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via MCMC teaching. In AAAI, pp. 4292-4301, 2018.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In ICML. pp. 1100-1109 1009, 2016.
Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In ICLR, 2020.
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial net- works. In ICLR, 2017.
Song Chun Zhu, Ying Nian Wu, and David Mumford. Minimax entropy principle and its application to texture modeling. Neural Computation, 9(8):1677-166,, 1997.
11

---

Under review as a conference paper at ICLR 2021
A LITERATURE REVIEWS
We discuss some of related literature and shed lights the relationship between work with others.
A.1 EXPLICIT GENERATIVE MODELS
Explicit generative models are interested in fitting each instance with a scalar (unnormalized) density expected to explicitly capture the distribution behind data. Such densities are often up to a constant and called as energy functions which are common in undirected graphical models (LeCun et al,) 2006). Hence, explicit generative models are also termed as energy- baaee models. An early version of energy-based models is the FRAME (Filters, Random field, And Maximum Entropy) model (Zhu et al.
1997, Wu et 2000). Later some works leverage deep neural networks to model the en- ergy function (ggiam et al. 2011 Xie et al. 2016b) and pave the way for researches on deep energy model (DEM) (c.g.. (Liu & Wang 2017 Kim & Bengio 2017 Zhai et al.) 2016 HHarnoja al.)
2017 Du & Mordatch| 2019 Nijkamp et al.) 2019)). Apart from DEM, there are also some other forms of deep explicit models based on restricted Boltzmann machines like deep belief networks
(Hinton et al. 2006) and deep Boltzmann machines (Salakhutdinov & Hinton 2009).
The normalized constant under the energy function requires an intractable integral over all possi- ble instances, which makes the model hard to learn via Maximum Likelihood Estimation (MLE). To solve this issue, some works propose to approximate the constant by MCMC methods (Geman & Geman 1984 Neal 2011). However, MCMC requires an inner-loop samples in each training, which induces high computational costs. Another solution to optimize an alternate surrogate loss function. For example, contrastive divergence (CD) (Liu & Wang. 2017) is proposed to measure how much KI. divergence can be improved by running small numbers of Markov chain steps to- wards the intractable likelihood, while score matching (SM) (Hyvärinen] 2005) detours the constant by minimizing the distance for gradients of log-likelihoods. A recent study (Grathwohl et al. 2020) uses Stein discrepancy to train unnormalized model. The Stein discrepancy does not require the nor- malizing constant and makes the training tractable. Moreover, the intractable normalized constant makes it hard sample from. To obtain accurate samples from unnormalized densities, many studies propose to approximate the generation by diffusion-basee processes, like generative flow (Nguyen et al., 2017) and variational gradient descent ((Liu & Wang. 2016)). Also, recent work (uu et al. (2018) leverages Stein discrepancy to design a neural sampler from unnormalized densi- ties. The fundamental disadvantage of explicit model is that the energy-based learning is difficult to accurately capture the distribution of true samples due to the low manifold of real-world instances (Liu & Wang 2017).
A.2 IMPLICIT GENERATIVE MODELS
Implicit generative models focus on generation mapping from random noises to generated sam- plcs. Such mapping function is often called as generator and possesses better flexibility compared with explicit models. Two typical implicit models are Variational Auto-Encoder (VAE) (Kingmal & Welling 2014) and Generative Adversarial Networks (GAN) (Goodfellow et al. 2014). VAE introduces latent variable and attempts maximize the variational lower bound for likelihood of joint distribution of latent variable and observable variable, while GAN targets an adversarial game between the generator and a discriminator (or critic in WGAN) that aims discriminating the generated and true samples. In this paper, we focus on GAN and its variants (e.g., WGAN (Ar-) jovsky et al. 2017). WGAN-GP (Gulrajani al. 2017). DCGAN (Radford et al., (016)). etc.) as the implicit generative model and we leave the discussions VAE as future work.
Two important issues concerning GAN and its variants are instability of training and local optima. The typical local optima for GAN can be divided into two categories: mode-collapse (the model fails to capture all the modes in data) and mode-redundance (the model generates modes that do not exist in data). Recently there are many attempts to solve these issues from various perspectives. One perspective is from regularization. Two typical regularization methods are likelihood based and entropy-based regularization with the prominent examples Warde-Farley & Bengio 2017) and (Li & Turner, 2018) that respectively leverage denoising feature matching and implicit gradient approx- imation enforce the regularization constraints. The likelihood and entropy regularizations could
12

---

Under review as a conference paper at ICLR 2021
respectively help the generator to focus data distribution and encourage more diverse samples, and a recent work (Tao et al. 2019) uses Langevin dynamics to indicate that i) the entropy and like- lihood regularizations are equivalent and share opposite relationship in mathematics, and ii) both regularizations would make the model converge to a surrogate point with a bias from original data distribution. Then (Tao et al.] 2019) proposes a variational annealing strategy to empirically unite two regularizations and tackle the biased distributions.
To deal with the instability issue, there are also some recent literatures from optimization perspec- tives and proposes different algorithms to address the non-convergence of minimax game optimiza- tion (for instance, [Gemp & Mahadevan 2018 Liang & Stokes 2 al. 2019). Moreover, the disadvantage of implicit models is the lack of explicit densities over instances, which disables the black-box generator to characterize the distributions behind data.
A.3 ATTEMPTS TO COMBINE BOTH OF THE WORLDS
Recently, there are several studies that attempt to combine explicit and implicit generative models from different ways. For instance, (Zhao et al. 2017) proposes energy-based GAN that leverages energy model as discriminator to distinguish the generated and true samples. The similar idea is also used by (Kim & Bengio 2017) and (Dai et al. 2017) which let the discriminator estimate a scaler energy value for each sample. Such discriminator is optimized to give high energy to generated samples and low energy true samples while the generator aims generating samples with low energy. The fundamental difference is that (Zhao et al., (017) and (Dai et al.) (017) both aim minimizing the discrepancy between distributions of generated and true samples while the motivation of (Kim & Bengio| 2017) is to minimize the KL divergence between estimated densities and true samples. (Kim & Bengio (017) adopts contrastive divergence (CD) to link MLE for energy model over true data with the adversarial training of energy-based GAN. However, both CD-based method and energy-based GAN have limited power for both generator and discriminator. Firstly, if the generated samples resemble true samples, then the gradients for discriminator given by true and generated samples are just the opposite and will counteract each other, and the training will stop before the discriminitor captures accurate data distribution. Second, since the objective boils down to minimizing the KL divergence (for (Kim & Bengio (017)) or Wasserstein distance (for (Dai et al. 2017)) between model and true distributions, the issues concerning GAN (or WGAN) like training instability and mode-collapse would also bother these methods.
Another way for combination is by cooperative training. (Xie et al. (01 (and its improved version (Xie et al. 2018) leverages the samples of generator as the MCMC initialization for energy-based model. The synthesized samples produced from finite-step MCMC are closer the energy model and the generator is optimized to make the finite-step MCMC revise its initial samples. Also, a recent work (Du et al. 2018) proposes to regard the explicit model as a teacher net who guides the training of implicit generator as a student net to produce samples that could overcome the mode-collapse issue. The main drawback of cooperative training is that they indirectly optimize the discrepancy between the generator and data distribution via the energy model as a 'mediator', which leads to a fact that once the energy model gets stuck in a local optimum (e.g., mode-collapse or mode- redundance) the training for the generator would be affected. In other words, the training for two models would constrain rather than exactly compensate each other. Different from existing methods, our model considers three discrepancies simultaneously triangle to jointly train the generator and the estimator, enabling them to compensate and reinforce each other.
B BACKGROUND FOR STEIN DISCREPANCY
Assume qxx) to be continuously differentiable density supported on X C Rd and f Rd + Rd smooth vector function. Define [x(]) = Vx log fx) + fxf(x) as a Stein operator. If is Stein class (satisfying some mild boundary conditions) then we have the following Stein identity property:
Ex-q [f)]( = Ex Vx (x)) + Vxf(x)] = 0.
Such property induces Stein discrepancy between distributions PP p(x) and Q q(x), x X:
S(Q. P) = feF
(Ex [xx]]]]]]] = supp[T(Bx.,
+ vxf(x))) (4)
feF
13

---

Under review as a conference paper at ICLR 2021
where f is what we call Stein critic that exploits over function space FF and if F is large enough then (Q), = if and only if Q = P. Note that in (I), we do not need the normalized constant for p(x) which enables Stein discrepancy to deal with unnormalized density.
F a unit ball in a Reproducing Kernel Hilbert Space (RKHS) with a positive definite kernel func- tion k(...), then the supremum in (1) would have close form (see (Liu et al. 2016 Chwialkowski et al. 2016 Oates et al. 2017) for more details):
(5)
where "x(x,x')
Vx lolog((x) x(x,x)VV logp(x') + log (x,((x +
Vak(x, x') Vx logp(x)) + (KSD).
The 5 gives the Kernel Stein Discrepancy
C PROOFS OF RESULTS IN SECTION 4.1
C.I PROOF OF THEOREM I
Proof. Applying Kantorovich's duality on W(Pa, P,) and using the exhaustiveness assumption on the generator, rewrite the problem as
min max E.P D [Er[D] Er [D] S(Praa,, Pr) + A2S(P,
(6)
where the minimization with respect to E is over all energy functions, the minimization with respect to IP is all probability distributions with continuous density, and the maximization with respect to D is over all I- I-Lipschitz continuous functions. Recall the definition of kernel Stein discrepancy
S(P, P8) = Ex.x'= (VV log dIP / k(x, x') V, log dIP ddPPp(xx (xx),,
where (P//dPP v is the Radon-Nikodym derivative. Observe that S(P, PF.) is infinite if P is not abso- lutely continuous with respect to Pg. Hence, to minimize the objective of 66) it suffices to consider those P's that are absolutely continuous with respect to Pg. Introducing variable replacement h(x) == dP /LP/dl dP (x) 1, then problem (6) becomes
{ [Fel(L h]D] 1E [D] + S(P((Pr
min max E.h D
+ A2 Ex.x/~ V log(1 + ((x)) k(x, x') V log(1 + ((x)))
where the minimization with respect is over all T.a (TP)) functions with Pre-expccatiion zero.
Fixing E, we claim that we can swap min; and maxp- Indeed, without loss of generality, we can restrict D to be such that (xx)) = 0 for some element Xg, constant shift does not change the value of Epg [11 + h) Ee, Ee,.a [D]. The set of Lipschitz functions that vanish at Xp is a Banach space, and the of 1-Lipschitz functions is compact (Weaver 1999). Moreover, L ((Prr)
) is also Banach space and the objective function is linear in both h and D. The above verifies the condition of Sion's 's minimax theorem, and thus the claim proved.
Swapping ming and max in (7) and fixing E and D, we consider
min (Brg [hD] + A2 Bx /v log(1 + f(x))) T k(x, V log(1 + ((x'))))
h:Ee, [0]=0
V [x
Vah(x')
min
Erg [DD] X2 E
+ 11(x) k(x, x²) + 1(xx)
h:Egg [0]=0
= min hrEn [M]=
EE [hD] + A2
Ex
Vzh(x)
x (Vxxx,x')V,,xxxx')
where the first equality follows from the chain rule of the derivative, and the second equality follows from change of measure dP + h)dPg. Introducing an auxiliary variable so that 2 is an
14

---

Under review as a conference paper at ICLR 2021
upper bound of Ex [zh(x) xxxx x') V~h(x'), we have that
min (Epg [hD] + A2 Ex.x'~Pa PP [V Vx)(x) (xxx
x(xxx')Vahxx)]
A:E5g [0]=0
min
min
{E [hD] + Agr Exxx Vzh(x) V2h(() (xx 2
r20 k [k]=0
-min
min
{rEp [hD] + 221² Exx [v,((x) xxx Mxlk)s) s
r00 h:lee [k]=0
=min
A22r - r H -((
r20
4A2
(PE:))
where the first equality holds because Exxx~PE (rh))((x) A x') (rh)(x))
r Bx [Va((x) x(xxx') for all r 2 0 and by introducing auxiliary vari- v')x)' the second equality follows from change of variable from to rh; and the third equality follows from the definition of the kernel Sobolev dual norm. Plugging back in (7) yields the ideal result.
able p² xx [v=(x) k(x, x') xxx,)x'xxVVVXVVX
C.2 PROOF FOR THEOREM 2
Proof. Applying the definition of Stein discrepancy S(PE, Pc) and under the exhaustiveness assumption of G, we rewrite the problem
min max E.
S (IPral) IPE + A2E + ([Preea,, PP);
where the minimization with respect to E is over the set of all engergy functions; the minimization with respect to over all continuous distributions; and the maximization with respect to is over the Stein class for Pg. Let us fix E. Using similar argument in the proof of Theorem it suffices to restrict IP on the set of distributions that are absolutely continuous with respect to E, which can be identified the set of T. T.'(PP) functions with IP Pg-mean
and is thus Banach.
Together with the compactness assumption of the Stein class, using Sion's 's minimax theorem, we can swap the minimization over IP and the maximization over f. Now, fixing further f, consider
min f ( f(y)] + W(P Pe)l P)}.
(8)
Recall the definition of Wasserstein metric
W(Preal: P) min Exx,)>>>l1* [||x yll..
where the minimization is over all joint distributions of (x, y) with x-marginal Preal and y-marginal P. We rewrite problem ( as
min m (x.))~> Do Ar fly) + |x -yll),
where y has marginals PP real and IP. Since IP is unconstrained, the above problem is further equivalent to
min (T A22 f(y) + --x->[l])
where the minimization is over all joint distributions of (x,y) with x-marginal being Preat- the law of total expectation, the problem above is equivalent to
Using
min toe heapppplll
ExwP...
BByrz D2 f(y) + ||x --yl /
min Baynn DeAAs f(y) + ||1- }
ExxPioa
Tx
Ex
min min (2AP f(y) + ||x-
YEX
where the minimization the first line of the equation is over 7x. the set of all conditional distri- butions of y given x where is over the support supp Preea of Preat the exchanging of min and E
15

---

Under review as a conference paper at ICLR 2021
in the first equality follows from the interchangcbabiiity principle (Shapiro et al. 2009); the second equality holds because the infimum can be restricted to the set of point masses. Finally, the original problem is equivalent to
min max S real: PE) Ex-Pr
(2 f(y) + ||x 1|X-|-XX/ >lI ||xx>|1]]] yEE
Therefore, the proof is completed using the definition of Moreau- Moreau-Yosida regularization.
D DETAILS AND PROOFS IN SECTION 4.2
D.1 DISCUSSIONS ON ONE-DIMENSIONAL CASE
The training for minimax game in GAN is difficult. When using traditional gradient methods, the training would suffer from some oscillatory behaviors (Goodfellow 2017 Liang & Stokes 2019). In order to better understand the optimization behaviors, we first study a one-dimension linear system that provides some insights on this problem. Such toy example (or similar one) is also utilized by Gidel et al. 2019 Nagarajan & Kolter 2017) to shed lights on the instability of WGAN training
Consider a linear critic (x) your and generator Gø(z) Oz. Then the Wasserstein GAN objective can be written as constrained bilinear problem: ming |ax [[[r] [2 which could be further simplified as an unconstrained version (the behaviors can be generalized multi-
dimensional cases (Gidel et al. 2019)):
min max V -.
Unfortunately, such simple objective cannot guarantee convergence by traditional gradient methods like SGD with alternate updating [3] 0x+1 = 0₂ + mpbk. th+11 t + t (=<00x=+)) (0e+1.Suuee Such optimization would suffer from an oscillatory behavior, i.e., the updated parameters go around the optimum point ([v'", 0*] = [0, 1)) forming a circle without converging the centrality, which is shown in Fig. 3a). A recent study in (Liang & Stokes 2019) theoretically show that such oscillation is due to the interaction term in (9).
One solution to the instability of GAN training is to add (likelihood) regularization, which has been widely studied by recent literatures Farley & Bengio 2017 7 Li & Turner, 2018). With regu- larization term, the objective changes into ming max [/
<6 AE log ((02)]. where (())) denotes the likelihood function and À is a hyperparameter. A recent study (Tao et al. 2019) proves that when sample evidence, helping to stabilize GAN training; when À > 0 tra term maximizes sample entropy, which encourages diversity of generator. Here we consider a Gaussian likelihood function for generated sample x', /=x²)= = exp(-1
0
the extra term equivalent maximizing
the ex-
b)²) which is up to a constant. Its parameter can be estimated by 6 [[x]] Then for generated sample x' = Oz, we have [log ((0z))
===== + [[z]E[=]]
[[]a]² Like the case in WGAN, we consider
E[x] Ejz] = 1. Assume Var[2] = 1 and have B|2?] 1 1 E[2]. Hence, for the analysis on likelihood- (and entropy-) regularized WGAN, we can study the following system:
min max vi - sb
X -)). 8).
(10)
the When = 1, the above objective degrades to D): when X < fiikeihhodd-regulriizie the gradient of regularization term pushes 0 to shrink, which helps for convergence; when X > 0 rearrooyrrgullaiizaton)) the added term forms an amplifiying strength on 0 and leads to divergence. Another issue of reelihooddregularizztiie makes the model converge to a biased distribution, as proved by (Tao et al., 2019). In this case, one verify that the optimum point becomes [ = [-X, 1]. resulting in bias. To avoid this issue, (Tao et al. 2019) proposes to temporally decrease | through training. However, such method would also be stuck in oscillation when |X| gets close zero shown in Fig. F(a)
is that the extra term changes the optimum point and
Finally, consider our proposed model. We also simplify the density estimator as a basic en- ergy model po(x) exp(-1r² Φx) whose score function V, log (x( -I -
- Φ. Then
"Our thcoretical discussions focus on WGAN, and we also compare with original GAN the experiments. Here, adopt the most widely used alternate updating strategy. The simultaneous updating, i.c., 0z-1 0g + 173h and th++1 = the + 7111 0z), would diverge in this case.
16

---

Under review as a conference paper at ICLR 2021
if we specify the two Stein discrepancies in KSD with kernel ((21, = I(m1
12). then ((Praa,, Pg) = Ex (V #1 lo Vx, log μ ((21))
log po (22)
Vag log [((22)]] E [V, Po())) Va (og (())) = (ø + E x. Similarly, one can obtain
S(FG,PE) = (Ø [=] Therefore we arrive at the objective in
\1 (1+ x)²
22 (0 + 6)
min max min e sh
(II)
Interestingly, for 1Y, A the optimum remains the same 10",0 Φ-1 [0, -1]. Then we show that the optimization guarantees convergence to [sb", ,0", .07 Proposition 1. Using alternate SGD (17) geometrically decreases the square norm N. | [ 112 + 10 + 112, for any O < 1 < with X1 = X2 1
N1+1 (1- (1 m)
(12)
Proof. Instead of directly studying the optimization for we first prove the following problem will converge to the unique optimum.
1
min max By: + 0Φ + 62 + 0².
(13)
2
Applying alternate SGD we have the following iterations:
1 = 1/2 7) θ,,
1 (0. ) (1 2n) 1802,
0c+1 = 0, - 7 + 0r+1 + 0,) = - 1y(1 2n) + (1 10]0, - mpr.
Then we obtain the relationship between adjacent iterations:
M++1]
1
0
Ø1+1
0
- 27
7 77
Øt
M
Ø
0
-n
-n(11
2n)
0
0,
We further calculate the eigenvalues for matrix M and have the following equations (assume the eigenvalue as A);
1)-1) + 3n - 1)² + 2n (1 - 7)(a - 1) 217² =0.
One can verify that the solutions the above equation satisfy [^] v(1 - 17 + 172 17 r72).
Then we have the following relationship
T++11
2
1/4 Φ
4/4
Φ++1
lake Φe 0.] M M
X
Φt A.
0.41
6,
2
where Am denotes the eigenvalue with the maximum absolute value of matrix M. Hence, we have
2 + o 0 I (1 7 11 + n - 72) + o3 6.)
We proceed replace v. o and 0 in (13) by v. ø' and 0' respectively and conduct change of variable: let 0' = - 0 and Ø' = =1 6. Then we get the conclusion in the proposition.
As shown in Fig. a). Stein Bridging achieves good convergence to the right optimum. Compared with (9) the objective (TT) adds new bilinear term Φ which acts like connection between the generator and estimator, and two other quadratic terms, which help to penalize the increasing of val- ues through training. The added terms and original terms in [T] cooperate to guarantee convergence
to a unique optimum. In fact, the added terms A1 (1 + 2) + x)² (11) and the original terms db th 0 in WGAN play both necessary roles to guarantee the convergence to the unique optimum J]. If we remove the critic and optimize 0 and e with the remaining loss
points [vb", [0b,00, 0", d*] = [0, 1, 11]. terms, we would find that the training would converge but not necessarily to fsb* 0*] [0. 1] (since the optimum points are not unique in this case). On the other hand, if we remove the estimator, the
system degrades to (9) and would not converge to the unique optimum point [s>*, _0] [0. 1]. If
we consider both of the world and optimize three terms together, the training would converge to
unique global optimum (ah", 0* Φ²) 10, -11]
17

---

Under review as a conference paper at ICLR 2021
D.2 GENERALIZATION TO BILINEAR SYSTEMS
Our analysis in the one-dimension case inspires that we can add affiliated variable to modify the objective and stabilize the training for general bilinear system. The bilinear system of wide
interest for researchers focusing on stability of GAN training ((Goodfellow 2017: Liang & Stokes
2019 Gidel et al.) 2019 Gemp Mahadevan, 2018
Zhang Yu 2020 The general bilinear
function can be written
F( 0) 0- Aap b 0
(14)
where 1b, are both r-dimensional vectors and the objective is min max F( 0) which can be
@ & seen basic form of various GAN objectives. Unfortunately, if we directly use simultancous (resp. alternate) SGD to optimize such objectives, one can obtain divergence (resp. fluctuation). To solve the issue, some recent papers propose several optimization algorithms, like extrapolation from the past ((Gidel et al., 2019)), crossing the curl ((Gemp & Mahadevan (0188) and consensus optimization ((Liagg & Stokes 2019). ). Also, (Liang & Stokes 2019) shows that is the interaction term which generates non-zero values for Vew I and V V that leads to such instability of training. Different from previous works that focused
algorithmic perspective, we propose add new affiliated variables which modify the objective function and allow the SGD algorithm to achieve convergence without changing the optimum points.
Based on the minimax objective of (14 we add affiliated r-dimensional variable & (corresponding the estimator our model) the original system and tackle the following problem:
min max min P(b, 0) + (H)
(15)
0
+
+
where H (() 0) = ( B(O Φ). B (AAT) and a non-negative constant. Theoretically, the new problem keeps the optimum points of (14) unchanged. Let L( Φ F( 0) + aG( 0).
B
Proposition 2. Assume the optimum point of min max F(tb, 0) are ( then the optimum points
0 *
15 would be (ab", f x²) where o= = -
Proof. The condition tells us that Vol 0) and ((b,,))) =0. Then derive the gradients for L(v,0,
Vol(sb Φ. 0) = VoF( ,0) =0
(16)
1
VoL 0.0 = V (vF + Vo H (6,0" 0*)
(B B )0 + +),
(17)
V L Φ 0) 11 (0, 0) / (B B ()+
(18)
Combining 17 and
18 we get 6- - Hence, the optimum point of (55 is [2b* 0" o51 where
0*
The advantage of the new problem is that it can be solved by SGD algorithm and guarantees conver- gence theoretically. We formulate the results in the following theorem.
Theorem 3. For problem min max min L(b,, Φ, 0) using alternate SGD algorithm, i.e.,
0 t
4b++1 = tb. + V v0 ((,,, 1h Φ1),
Φ++1 Φ: v/ v (,,, 1/tt1, 1, (b.), O.11 = O, - 1 VO,,,. tb+11, 1b+1)),
(19)
can achieve convergence to [4b", 0- 0''6"1 where 0² = 0" with at least linear rate of (1 m + (1 + +12 - uit) where 1)1 = mjmmi
110 maz and omin (resp. dmaz) denotes the maximum (resp. minimum) singular value of matrix A.
To prove Theorem 3, can prove more general argument.
18

---

Under review as a conference paper at ICLR 2021
Lemma 1. If we consider any first-onder optimization method on 11 i.e.
1i+1 E vo + span(L((ba Φ. 0), F Φ, 0)), VI E N,
Φ++1 E 20 + span ((t), 0o 0), L(b, Φr, 0)), VI E,,
1++1 E 40 span span Φ, 0o), L(b, Φ. 0:)), Vt E N,
Then we have
hb V (ehr 24)), कe U (Φ: -"), - (0. -²),
where U and V are the singular vectors decomposed by matrix A using SVD decomposition, i.e., A UDV and the triple ([pbe] ( (0. a) sagr follows the update rule with step size oy as the same optimization method on unidimensional problem
1 min max min Ob + 0 0² 202
(20)
V @
with step size where a, denotes the i-th singular value on the diagonal of D.
Proof. The proof is extended from the proof Lemma 3 in (Gidel et al. 2019). The general class of first-order optimization methods derive the following updations:
1+1
1+1
+1+ 40
Pat A 0, c) vo Σ
PaA S0, 0"),
1 1+1
Φ141 = Φu +
Σ
( (B + B ( + +.),
2 ==0
1+1
O
0o
+
Pstt ( - か³) (B + B (0, + +.)).
==0
where Pst, Est. Mst E R depend on specific optimization method (for example, in SGD, Pu 8
(Ase remain as a non-zero constant for Vt and other coefficients are zero).
Using SVD A UDV and the fact 0- = B (UDD UT) D, we have
(ihtu - V (+b V - 4b)
Pa PD
D T 8*)
[+1
U (0111 -U - -0))+
Σ 8=0
Sgt U D(O, -0 +U - x²),
t+1
UT (0.+1- UT (T, -0)
DV T
#=0
and equivalently,
1+1
1h+1 vo
Σ P DTO 0,,
3. = Φ0 S..
t+1
0 0 0. +
Pst D
0, Tz).
8=0
Note that D is rectangular matrix with non-zero elements diagonal block of size Hence, the above y-dimensional problem can be reduced to r unidimensional problems:
1+1
[5441k (ole +
Σ
Paro Oels:
[ = [Zo] +
Σ
Sua ((O,): + (T))))
==0
==0
0 = (0),
Σ Psio (1 + (0.). . (..1.).
The above iterations can be conducted independently in each dimension where the optimization in i-th dimension follows the same updating rule with step size σzz) problem in 20
19

---

Under review as a conference paper at ICLR 2021
Furthermore, since problem (20) can achieve convergence with linear rate of (1 (1- (1+m)
using alternate SGD (the proof is similar to that of (13)), the muuti-diimeniionl problem in (15) can
achieve convergence by SGD with at least a rate of (I in + (1 + 7/2 7/1) where 771 n1 mans
122 = min and maz (resp. min denotes the maximum (resp. minimum) singular value of matrix
A. We conclude the proof for Theorem 4.
Theorem suggests that the added term ( O) with affiliated variables & could help the SGD algorithm achieve convergence the the same optimum points directly optimizing F( 0). Our method is related to consensus optimization algorithm (Liang & Stokes 2019)) which adds regularization term ( 0)I + [ 0)] to (1) resulting extra quadratic terms for and 1 The disadvantage of such method is the requirement of Hessian matrix of F( 0) which
computational expensive for high-dimensional data. By contrast, our solution only requires the first-order derivatives.
E DETAILS FOR IMPLEMENTATIONS
E. SYNTHETIC DATASETS
We provide the details for two synthetic datasets. The Two-Circle dataset consists of 24 Gaussian mixtures where 8 of them are located in an inner circle with radius rı 4 and 16 of them lie in an outer circle with radius r2 8. For each Gaussian component, the covariance matrix is
0.2 0
I, I and the mean value is [r1 cost, r1 sin where 2m-k k 1,... 8, for the 0.2
0
inner circle, and [r2 cos r2 sin d), where 2 k 1,... 16 for the outer circle. We sample 16
N1 2000 points as true observed samples for model training.
The Two-Spiral dataset contains 100 Gaussian mixtures whose centers locate on two spiral-shaped
curves. For each Gaussian component, the covariance matrix is 0.5 0
0.5 Ø2l and the mean
value is [-c cos c1, C1 sin c1). where с₁ [02 cos c22 - 02 sin o2). where 0o 2% + linspace(O,, 0.5, 2 for another spiral. We sample N2 5000 points as true observed samples.
2x linspace (0, 0.5, 50) 2TT, for one spiral, and
E.2 MODEL SPECIFICATIONS AND TRAINING ALGORITHM
In different tasks, we consider different model specifications in order to meet the demand of capacify as well as test the effectiveness under various settings. Our proposed framework (3) adopts Wasser- stein distance for the first term and two Stein discrepancies for the second and the third terms. We can write (3) as a more general form
min D, (Breaat, Pa) + D2PPee PF.) D2 0.6
(21)
where D1, D2, D3 denote three general discrepancy measures for distributions. As stated in our remark, D, can be specified as arbitrary discrepancy measures for implicit generative models. Here also use JS divergence, the objective for valina GAN. To well distinguish them, we call the model using Wasserstein distance (resp. JS divergence) as Joint-W (resp. Joint-JS) in our experiments. On the other hand, the two Stein discrepancies (3) can specified by KSD (as defined by S₁ in ( or general Stein discrepancy with an extra critic (as defined by S in (I)). Hence, the two specifications for D, and the two for D: (Da) compose four different combinations total, and we organize the objectives in each case in Table
In our experiments, we use KSD with RBF kernels for D2 and D3 in Joint-W and Joint-JS on two synthetic datasets. For MNIST with conditional training (given the digit class as model input), also use KSD with RBF kernels. For MNIST and CIFAR with unconditional training (the class is not given as known information), we find that KSD cannot provide desirable results so adopt general Stein discrepancy for higher model capacity.
The objectives in Table appear to be comutationally expensive. In the worst case (using general Stein discrepancy), there are two minimax operations where one is from GAN or WGAN and one is from Stein discrepancy estimation. To guarantee training efficiency, we alternatively update the
20

---

Under review as a conference paper at ICLR 2021
Table 3: Objectives for different specifications of Di (Peral, PG). D2 real, PF) and We specify D1 as Wasserstein distance or JS divergence in our paper and for D2 and D3 we consider the general Stein discrepancy or kernel Stein discrepancy. Here we use W, JS denote Wasser- stein distance and JS divergence respectively, and S, Sz to represent general Stein discrepancy and kernel Stein discrepancy respectively. We omit the gradient penalty term for Wasserstein distance here but use it in experiments.
D1
D2
D3
Objective
ming ming max max, Ex-Feare [dw(x)] Ezapo (du(Go(z))
W
S
s
+ Exe-Paa (Apa [far (x)] + X2 Bizioo (Ap, [G (z))]
ming ming max Ex-r yurr (dh (x)) Buipo (d (Go
W
S
Sa
+ + Exxcs (upo (Ie, x.) )]+ + Xx Bx, upo (up Go(z), Go(z)))
ming mins maxy max,, Ex [log (d (x)) + Ez llog (Ga(())
JS
S
s
+X1Exx Ap [f, (x)] + A2 LAp [f (Gx(z))
ming ming maxs Ex-p [log( d (x])] + Exapo [log(1 de (Gy (z)))
JS
Sk
Sx
+A1 X'xxx'Paans Mp (2, x²) )] + A2 upo Mp.. (Go(z), Go
generator. estimator, Wasserstein critic and Stein critic over the parameters 6, o. v and π respec- tively. Specifically, in one iteration, we optimize the generator over 0 and the estimator over ø with one step respectively, and then optimize the Wasserstein critic over to with na steps and the Stein critic over π with Re steps. Such training approach guarantees the same time complexity order of proposed method as that of GAN or WGAN, and the training time for our model can be bounded within constant times the time for training GAN model. In our experiment, we set Rl = ne = 5 and empirically find that our model Stein Bridging would be two times slower than WGAN on average. We present the training algorithm for Stein Bridging in Algorithm
E.3 IMPLEMENTATION DETAILS
We give the information of network architectures and hyper-parameter settings for our model as well as each competitor in our experiments.
The energy function is often parametrized as a sum of multiple experts expents (Hinton 1999h) and each expert can have various function forms depending on the distributions. If using sigmoid distribution, the energy function becomes (see section 2.1 in (Kim & Bengio, 2017) for details)
Ex(x)
Σ log(1 Elog(1 + - +W)))
(22)
where n(x) maps input to a feature vector and could be specified deep neural network, which corresponds to deep energy model ((Ngiam et al. 2011))
When not using KSD, the implementation for Stein critic f and operation function / in ① has still remained an open problem. Some existing studies like (Hu et al. 2018) set d' = 1 in which situation f reduces a scalar-function from d-dimension input to dimedmmnnsoon scalar value. Such setting can reduce computational cost since large d' could lead to heavy computation for training. Empirically, in our experiments on image dataset, we find that setting d' can provide similar performance to d' = 10 or d' = 100. Hence, we set d' = 1 in our experiment in order for efficiency. Besides. to further reduce computational cost, we let the two Stein critics share the parameters, which empirically provide better performance than two different Stein crities.
Another tricky point is how to design a proper given d' * where the trace operation is not ap- plicable. One simple way is to set I as some matrix norms. However, the issue is that using matrix norm would make hard for SGD learning. The reason is that the I' and the expectation in (I) cannot exchange the order, in which case there is no unbiased estimation by mini-batch samples for the gradient. Here, we specify I' as max poollin over different dimensions of Ap, [f (x)], i.c. the gradient would back- propagate through the dimension with largest absolute value one time. The- oretically, such setting can guarantee the value in each dimension reduces to zero through training and we find it works well in practice.
21

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Training Algorithm for Stein Bridging
REQUIRE; observed training samples (x) ~ Preat-
REQUIRE: Oo. Φo. tho, To, initial parameters for generator, estimator, Wasserstein critic and
Stein critic models respectively. af 0.0 0.0002, 3f 0.9, 3 0.999, Adam syper-parameter for explicit models. a' = 0.0002, Bf = 0.5, 3/2 = 0.999, Adam syper-parameten for implicit models. A 1, A2. weights for D2 and D3 (we suggest increasing X2 from 0 to 1 through training). nh = 5, nh 5 number of iterations for Wasserstein critic and Stein critic, respectively, before one iteration for generator and estimator. B = 100, batch size.
while not converged do
for n 1,... ,ad do
B Sample B true samples fx, [-n from {x];
Sample B random noise { ~ Po and obtain generated samples x; = Go (Z:):: - 1)² // the last term is for gradient penalty in WGAN-GP where &; = GX;; + (1 - c))xi, (0, 1);
Ldis = B -i=1dy (x,(x)) - (Xi)
1/+11 5 -Adammn--aaa Vh, of , 3/ update the Wasserstein critic;
for 1,.... , do
Sample B true samples { from {x):
Sample B random noise { ~ Po and obtain generated samples x; Go(z.)
Laritic = = E. Ap, [x)] + A2 (x;)];
12
13
Th+1 +-
Th, a ,,,,, ,,2%)1 )/ update the Stein critic;
Sample B random noise {2 ~ Po and obtain generated samples x; = (z,);
14
Lest 1/ Ei= AIA [x)] + (f
15
B
Φk+1 + Adamm Lests Ub, a a ,3,,, , y/ update the density estimator;
16
Egen # EEE do (21) + Ag [E, (Ri)];
17
Oc+1 Adam(Lgen. On, a B1, 3/ \// update the sample generator;
18
OUTPUT: trained sample generator (z) and density estimator (x(x).
For synthetic datasets, we set the noise dimension 4. All the generators are specified as three- layer fully-connected (FC) neural network with neuron size 4- 128 102--128-2 22, and all the Wasserstein critics (or the discriminators in adivergence-based GAN) are also three-layer FC network with neuron size 2 - 128 - 128- - 1. For the estimators, we set the expert number as 4 and the feature function nxx) is a FC network with neuron size 2 128 128 4. Then in the last layer we sum the outputs from each expert as the energy value E(x). The activation units are searched within [Leaky tanh, sigmoid, softplus). The learning rate [le 6, le 5, le 1, 3, 1e 2] and the batch size [50, 100, 150, 200). The gradient penalty weight for WGAN is searched in [0, 1,111 1, 10, 100).
For MNIST dataset, we set the noise dimension as 100. All the crities/diseriminators are im- plemented as a four-layer network where the first two layers adopt convolution operations with filter size 5 and stride 2] and the last two layers are FC layers. The size for each layer is 1 64 128-
256 1. All the generators are implemented as four-layer networks where the first two layers are FC and the last two adopt deconvolution operations with filter size 5 and stride 2]. The size for each layer 100- 256 128 64 1. For the estimators, we con- sider the expert number as 128 and the feature function is the same as the Wasserstein critic ex- cept that the size of last layer 128. Then we sum the outputs from each expert as the energy value. The activation units are searched within [ReLU, LeakyReLU, tanh). The learning rate [2e- - 5, 2e - 4, 2e 3, 2e- 2] and the batch size [32, 64, 100, 128). The gradient penalty weight for WGAN is searched in [1, 10, 100, 1000).
For CIFAR dataset, we adopt the same architecture as DCGAN for critics and generators. As for the estimator, the architecture of feature function is the same the critics except the last year where set the expert number as 128 and sum each output as the output energy value. The architectures for Stein critic are the same Wasserstein critic for both MNIST and CIFAR datasets. In other words,
22

---

Under review as a conference paper at ICLR 2021
Table Distances between means generated digits (resp. images) and true digits (resp. images) on MNIST (resp. CIFAR-10).
MNIST
CIFAR
Method
Dis
12 Dis
1 Dis
1g Dis
WGAN-GP
13.80
0.93
80.98
1.72
WGAN+LR
12.91
0.86
82.96
1.81
WGAN+ER
12.26
0.77
72.28
1.59
WGAN+VA
12.38
0.78
69.01
1.53
DGM
12.12
0.79
179.30
3,95
Joint-W
11.82
0.73
64.23
1.41
Table 5: Quantitative results including MMD (lower better), HSR (higher is better) as the metrics for quality of generated samples and KLD (lower is better), JSD (lower is better), AUC (higher is better) as the metrics for accuracy of estimated densities on Two-Circle and Two-Spiral datasets.
Two-Cirlee
Two-Spiral
Method
MMD
HSR
KLD
ISD
AUC
MMD
HSR
KLD
JSD
AUC
GAN
0.0033
0.772
0.0082
0.583
VA
0.0118
0.295
0.0085
0.761
WGAN-GP
0.0010
0,841
0.0090
0,697
WGAN+VA
0.0016
0.835
0.0159
0.618
DEM
2.036
0.431
0.683
1.206
0.315
0.640
EGAN
3,350
0.474
0.616
1.916
0,445
0.499
DGM
0.0040
0.774
2.272
0.445
0.600
0.0019
0.833
1.725
0.414
0.589
Joint-IS
0.0037
0.883
1.104
0.297
0.962
0.0031
0.717
0.655
0.193
0,808
Joint-W
0.0007
0.844
1.030
0.281
0.961
0.0003
0.909
0.364
0.110
0.810
we consider d' = in and further simply & as an average each dimension of Ex-r(AA Empirically found this setting can provide efficient computation and decent performance.
E.4 EVALUATION METRICS
We adopt some quantitative metrics to evaluate the performance of each method on different tasks. In section 4.1, we use two metrics to test the sample quality: Maximum Mean Discrepancy (MMD) and High-quality Sample Rate (HSR), MMD measures the discrepancy between two distributions X and Y, MD = || Σm= ((x,) - 1 Σn = (yy:))l where 2; and yj denote samples from X and Y respectively and + maps each sample to RKHS. Here we use RBF kernel and calculate MMD between generated samples and true samples. HSR statistics the rate of high-quality samples over all generated samples. For Two-Cirlce dataset, we define the generated points whose distance from the nearest Gaussian component is less than σ high-quality samples. We generate 2000 points total and statistic HSR. For Two-Spiral dataset, we set the distance threshold as 50 and generate 5000 points to calculate HSR. For CIFAR, we use the Inception V3 Network in Tensorflow as pre-trained classifier to calculate inception score.
three metrics to characterize the performance for density estimation: KL divergence, JS divergence and AUC. We divide the map into a 300 meshgrid, calculate the unnor- malized density values of each point given by the estimators and compute the KI. and JS divergences between estimated density and ground- truth density. Besides, we select the centers of each Gaussian components as positive examples (expected to have high densities) and randomly sample 10 points within circle around each center as negative examples (expected to have relatively low densities) and rank them according the densities given by the model. Then we obtain the area under the curve (AUC) for false-positive rate v.s. true-positive rate.
In section 4.2,
23

---

Under review as a conference paper at ICLR 2021
8
4 5
6
8
4
0
5
3
3 5
9
5 0
7
6
8
4
2
S
3
1
3
3
0
2
3
6
9
7
6
1 6
/
2
8
6
7
8
7
8
6
6
3
8
0 9
4
7 9
6
0
5
6
5 1
5 3
9
4
8
8
3
3
5
5
8
3
5
$ 3 8 8 3 3 7 9 9
/ 6
6
6
8
4
6
3
7
7
9
0
5
7
5
3
0 0
3
1 0 4 6
/
5
9
b
3
53
20
1
4
9
5
0
8
5
O 4 8 4 8
| 6
5 5
/
0 4
7 4 0 6
9
4
0
7
6
0
1
6
O 0 4 2
6
2
S
3
4
6
9
5
7
23 8 8
0 6
8
9
0
7
(
0
8
6
8
7
2
9
3 / 11228610
4
0
1
3
6
0 4
8
9
0
4
7
9
6 / 5
8 0091080 03
1
3
3
5
4
2
8
6 / 3
5
9
0
8
4
8
4 4
2
6
7
7
6
1
4
4
3 6
4
6
6
2
8
4
3
/ 6
8
a
2
7
4
8
5
7
8
O
5
6
4
(a) Randomly sampled over all digits
(b) Randomly sampled over digits with top 50% densi-
tics
Figure 9: Generated digits given by Joint-W on MNIST.
(a) Randomly sampled over images
(b) Randomly sampled over images with top 50% densi-
ties
Figure 10: Generated images given by Joint-W on CIFAR.
24

---

Under review as a conference paper at ICLR 2021
5
G q 9
(a) Generated digits with highest densities
(b) Generated digits with lowest densities
1
$ $
5
5 5 5
5
5
5 5
9 f 9
9
9
9 S 99999999
(c) Real digits with highest densities
(d) Real digits with lowest densities
Figure 11: The generated digits (and real digits) with the highest densities and the lowest densities given by Joint-W. W.
5 M
5 5
5
5 5
999
9
Generated digits with highest densities
(b) Generated digits with lowest densitics
1
1
G 9 a
999999
9
9
9 9
(c) Real digits with highest densities
(d) Real digits with lowest densities
Figure 12: The generated digits (and real digits) with the highest densities and the lowest densities given by DGM.
f 5
5
5
5
9
9
9 9
9 9
9 9
(a) Generated digits with highest densities
(b) Generated digits with lowest densities
5 5
5
5
6
5
7 Q Q
9
9
(d) Real digits with lowest densities Figure 13: The generated digits (and real digits) with the highest densities and the lowest densities given by EGAN.
(c) Real digits with highest densities
25


Reviews:
[{"review_id": "srAqictryu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n* The idea to bind an explicit and implicit generative model and study the effect on both models is a valid research topic.\n* The method seems novel and inspiring, and the paper also shows a theoretical understanding of the proposed method, which is technically nontrivial.\n* The presentation is clear and pleasing (e.g., content organization, background, Fig. 1). The paper also includes a detailed review on existing works.\n* The paper presents comprehensive experiments, and the results are promising.\n\nCons:\n* On the motivation.\n  - Would it be too costly to train two models for one task just to alleviate the problem of one model? It seems to hide the problems of each model and serves as a black-box solution. The theoretical analysis makes things better, but the explanations may seem to be like \"side effects\" but not a direct solution targeting on the problems. Moreover, both the explicit and implicit models have the same amount of knowledge from data: one model cannot provide more information to the other model beyond the training dataset. How to understand the improvement under this perspective?\n  - For training an explicit model, the mode-collapse behavior may be due to the usage of the Stein discrepancy. Training by maximizing likelihood (i.e., minimizing forward KL divergence) via classical methods e.g. contrastive divergence may already circumvent this problem.\n* On the theory. It may be better to explain why \"By smoothing the Stein critic, the Stein bridge encourages the energy model\nto seek more modes in data instead of focusing on some dominated modes\". How does it make the Stein discrepancy more picky on missing a mode?\n* On the experiments.\n  - I see in the supplement that a hyperparameter searching is conducted, but I did not find the metric to select them. Is it done by AUC / IS / FID / MMD / HSR / manual visualization evaluation? Results of \"WGAN + something\" may be sensitive to hyperparameters and maybe they should not be worse than the vanilla WGAN (taking zero regularization).\n  - In Figs. 4 and 5, how are the density estimation of implicit models GAN/WGAN and samples of explicit models DEM/DGM got visualized? Do they rely on techniques like kernel density estimation or MCMC? If yes, how to make sure these techniques do not affect the outcome?\n  - The definition of High-quality Sample Rate (HSR) may value mode collapse, since it gives a high score if all generated samples are on the center of one mode. For the result in Fig. 8(a), may be the HSR needs to drop to be faithful to data.\n  - The experiment is comprehensive and shows the desired improvements. But maybe a comparison with other explicit model training methods (contrastive divergence, annealed Langevin dynamics, etc.) that avoid mode-collapse is also needed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "orqU_YRUHTm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the task is to train an implicit and an explicit model simultaneously via GAN setting and a new regularizer called \"stein bridge\", which is constructed from the kernel Stein discrepancy between the implicit and explicit models. The idea of adding such regularization, with the notion of mutual regularization of two models, is interesting. The proposed regularization term is clearly presented, the illustration of stablizing the training procedure, and the empirical results are clearly shown and discussed. The sample quality from the generative models are compared.\n\nThere are some parts that remain unclear or can be further emphasized. \n It is said that training both explicit and implicit densities are more helpful to the whole procedure. Despite the cited literature reviews, it is unclear to me, in this paper presentation, why is this so?\nIn the paper, the implicit network is parameterized by \\theta as G_{\\theta} while the explicit EBM is parametrized by \\phi, as p_{\\phi}. \nBefore the Stein bridge is introduced:\nHow do \\theta and \\phi interact? From figure1 it seems they do not interact during training but only coupled via the objective. In addition, which of the model (implicit or explicit) is used as the final outcome?\nAfter the Stein bridge is introduced:\nstein bridge tries to minimize the Stein discrepancy between implicit and explict models. How is the EBM chosen so that the density class is rich enough? How are \\lambda_1 and \\lambda_2 chosen to balance three terms?\n\nHow does the training of generative model compared to the learning procedure in \"Deep energy estimator networks.\" Saremi, Saeed, et al.  2018, which learns a generative model from score-matching based criterion?\n\nThanks for the presentation.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "c_W11bDa70T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adopted Stein's method to connect an explicit density estimator and an implicit sample generator to propose an objective function for deep generative learning.\nThis paper is written well and the organization is very clear.\nHowever, this paper is very similar to a NeurIPS-19 paper (Two Generator Game: Learning to Sample via Linear Goodness-of-Fit Test).\nBesides, the authors didn't cite this paper in the current submission.\n\nFirst, the top-level idea is the same.\n1. They all adopted two generative models. One is explicit and the other is implicit.\n2. They all adopted Stein's method to connect these two generative models.\n\nSecond, important technical details are similar.\n1. They all used energy-based model in the explicit part.\nThe energy-based model is used to mimic the underlying distribution of the real data.\n2. They all used Stein's method to avoid solving the normalization constant.\nStein's method is a likelihood-free method that depends on the distribution only through logarithmic derivatives.\nWhen taking derivatives, the normalization constant will be eliminated.\n\nThird, these two papers have the same target.\nThey hope the explicit generative model characterize the formulation of the distribution and\nthe implicit one produces vivid or genuine-looking images.\n\nThe novel part of this submission is the introduction of kernel Sobolev dual norm and Moreau-Yosida regularization.\n\nThere is a big gap between the optimization formulations in Equation (3), Theorem 1 and Theorem 2 and the experimental results shown in Section 5.\nBesides, there are no open source codes provided.\nIt is very hard for me to figure out the details of the experiments and meantime to check the reproducibility of this paper.\n\nIn summary, I hope that the authors will correctly cite the closely related NeurIPS-19 paper,\nand clearly demonstrate their completely new contributions as compared to the NeurIPS-19 paper.\n\nSince ICLR is a highly selective conference,\nthe originality and significance of one submission will always be in the first priority.\nAlthough the writing quality of this paper is good, I cannot accept this paper in current state.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S9q1AcYRWt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\n\nThe author proposed a novel regularization technique for jointly training of implicit model (IM) and explicit model (EM). This is achieved by connecting the training of IM and EM via Stein discrepancy (SD), which is called Stein bridging. \n\nThe author claimed this regularization can (i) smooth the Wasserstein critic by kernel Sobolev dual norm (ii) smooth the Stein critic by Moreau-Yoshida regularization (iii) stabilize the training of IM. \n\nThe author also theoretically proves the (i), (ii) under Wasserstein distance, and (iii) for a simple toy case.  The author empirically evaluated the EM by inspecting the mode coverage of toy example, ranking digit and OOD detection on more complex data sets. \n\n-----\nReviews:\n\nClarity: The main text of this paper is in general clearly written and easy to follow. However, I have some concerns related to the theoretical analysis in the Appendix. \n\nNovelty: This regularization technique seems to be novel to the best of my knowledge. Although the idea is simple, the author provided some theoretical analysis to back it up. However, it is not enough to fully back the claims made in the main text. Details later.\n\nTechnical soundness and concerns: I have some concerns related to the SD and proof for theorem 1 and 2. \n1. For SD, the author mentioned that for Stein critic $f_s(\\pmb{x})$, it is not necessary $\\mathbb{R}^d\\rightarrow \\mathbb{R}^d$. One can specify a lower dimension $d'<d$ and make $f_s:\\mathbb{R}^d\\rightarrow \\mathbb{R}^{d'}$ as long as $f_s$ belongs to the Stein class. Indeed, this is true for Stein's identity (see Def 2.1 in Liu's paper). However, this does not mean it defines a valid discrepancy measure. The original SD in Gorham's paper assumes $d'=d$ and the trace operator is used to transform $d\\times d$ Stein identity to the scalar value. In that case, Gorham proves its validity by investigating its weak convergence property. My concern is I cannot see the direct generalization from the trace operator to other matrix operators like the ones used in this paper with $d'=1$. In other words, I agree that for two distributions $p$,$q$, when $p=q$, the SD defined in this paper is $0$, but not vice versa. Could the author point out any references or provides any details on the validity of the proposed SD?\n2. I do not fully follow the derivations in Appendix C.1. In page 15, how do you introduce the auxiliary variable $r^2$? Why there are two $\\min$ operations instead of one $\\min$ with jointly optimizing $h$ and $r$? Could you elaborate more on this and also how do you get rid of $r^2$ in the constraints in the second equality?\n3. I am also a bit confused about derivations in Appendix C.2. How do you combine the $\\mathbb{P}$ and $\\gamma$ in one $\\min$ operation instead of $\\min_{\\mathbb{P}}\\min_{\\gamma}$? (In page 15)\n4. In the main text, the author claimed that other objectives can be used for training implicit the model such as JS divergence. However, the theoretical analysis (theorem 1 and 2) only shows the regularization effect of Stein bridging is only for Wasserstein-1. So the analysis won't hold for other objectives like JS divergence. \n5. It is known that Stein based divergence is a weak objective for training EBM (see Liu 2016). Therefore the regularization technique may help a lot, like the mode coverage demonstrated in the experiment. I wonder if SOTA training method for EBM is used (like SSM in Song 2020, Song 2019), does this regularization help the training, because this regularization is not cheap to compute (higher than the SOTA method for EBM).\n6. In figure 4, I cannot find DEM and EGAN in the density plot.\n7. In table 2, it seems that the training of DEM is failing as the AUC is closed to 0.5. Any guess on why it fails? How do you pre-process the data set? Do you add different scales of noise in the images to smooth it for the EBM to learn the distribution (like the trick used in Song 2019)?  \n\nSummary:\nI am quite interested in this approach. But I am a bit concerned about the theoretical analysis and the true advantage of training EBM with Stein bridging compared to the cheaper SOTA EBM method.\n--------\nLiu, Qiang, and Yihao Feng. \"Two methods for wild variational inference.\" arXiv preprint arXiv:1612.00081 (2016).\n\nSong, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" Uncertainty in Artificial Intelligence. PMLR, 2020.\n\nSong, Yang, and Stefano Ermon. \"Generative modeling by estimating gradients of the data distribution.\" Advances in Neural Information Processing Systems. 2019.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents "stein bridge", a joint training framework that connects an explicit (unnormalized) density estimator and an implicit sample generator via Stein discrepancy. The idea and methodology are valid and of interest. But the raised concerns were not properly addressed. 
Decision: Reject