Efficient Sampling for Generative Adversarial Networks with Reparameterized Markov Chains
Authors: Yifei Wang
Yisen Wang
Jiansheng Yang
Zhouchen Lin
Venue:  (2021)
Keywords: Generative Adverarial Networks
Sampling
Markov chain Monte Carlo
Reparameterization

Abstract: Recently, sampling methods have been successfully applied to enhance the sample quality of Generative Adversarial Networks (GANs). However, in practice, they typically have poor sample efficiency because of the independent proposal sampling from the generator. In this work, we propose REP-GAN, a novel sampling method that allows general dependent proposals by REParameterizing the Markov chains into the latent space of the generator. Theoretically, we show that our reparameterized proposal admits a closed-form Metropolis-Hastings acceptance ratio. Empirically, extensive experiments on synthetic and real datasets demonstrate that our REP-GAN largely improves the sample efficiency and obtains better sample quality simultaneously.

Under review as a conference paper at ICLR 2021
EFFICIENT SAMPLING FOR GENERATIVE ADVERSAR- IAL NETWORKS WITH REPARAMETERIZED MARKOV CHAINS
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, sampling methods have been successfully applied to enhance the sample quality of Generative Adversarial Networks (GANs). However, in practice, they typically have poor sample efficiency because of the independent proposal sam- pling from the generator. In this work, we propose REP-GAN, a novel sampling method that allows general dependent proposals by REParameterizing the Markov chains into the latent space of the generator. Theoretically, we show that our repa- rameterized proposal admits closed- Metropolis- Hasttnngs acceptance ratio. Empirically, extensive experiments on synthetic and real datasets demonstrate that REP-GAN largely improves the sample efficiency and obtains better sample quality simultaneously.
INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al. (014) have achieved a great success on generating realistic images in recent years (Karras et al.] 2019 Brock et al.) 2019). Unlike previous models that explicitly parameterize the data distribution, GANs rely on an alternative optimization between generator and a discriminator to learn the data distribution implicitly. However, in practice, samples generated by GANs still suffer from problems such as mode collapse and bad artifacts.
Recently, sampling methods have shown promising results on enhancing the sample quality of GANs by making use of the information in the discriminator. In the alternative training scheme of GANs, the generator only performs few updates for the inner loop and has not fully utilized the density ratio information estimated by the discriminator: Thus, after GAN training. the sampling methods propose to further utilize this information to bridge the gap between the generative distribution and the data distribution in fine-grained manner. For example, DRS Azadi et al. 2019) applies rejection sampling. and MH-GAN (Turner et al. 2019) adopts Markov chain Monte Carlo (MCMC) sampling for the improved sample quality of GANs. Nevertheless, these methods still suffer lot from the sample efficiency problem. For example, as will be shown in Section MH-GAN's average acceptance ratio on CIFARIO 10 be lower than 5%, which makes the Markov chains slow mix. As MH-GAN adopts an independent proposal i.e.. qxx' x) = q(x'), ), the difference between samples can be so large that the proposal gets rejected easily.
To address this limitation, we propose to generalize the independent proposal to general dependent proposal (xx)x). To the end, the proposed sample can be refinement of the previous one, which leads to a higher acceptance ratio and better sample quality. We can also balance between the exploration and exploitation of the Markov chains by tuning the step size. However, it is hard to design proper dependent proposal in the high dimensional sample space X' because the energy landscape could be very complex (Neal et al. 2010).
Nevertheless, we notice that the generative distribution (x) of GANs is implicitly defined as the push-forward of the latent prior distribution po (z). and designing proposals in the low dimensional latent space is generally much easier. Hence, GAN's latent variable structure motivates to design a structured dependent proposal with two pairing Markov chains, one in the sample space and the other in the latent space Z. As shown in Figure [ given the current pairing samples (Zz, xt). we draw the next proposal x' in a pottom-to-up way: 1) drawing a latent proposal z' following q(22 )x)); 2) pushing it forward through the generator and getting the sample proposal x' G(z)):

---

Under review as a conference paper at ICLR 2021
accept
reject
accept
Selector
Ik
Ck+3
Generator
G
G
G
SSN
2
2-++1
2
2
Latent transition
Figure 1: Illustration of REP-GAN's reparameterized dependent proposal with two pairing Markov chains, one in the latent space 2, and the other in the sample space X.
assigning Xx+1 x' if the proposal x' is accepted, otherwise rejected X = X By utilizing the underlying structure of GANs, the proposed reparameterized sampler becomes more efficient the low-dimensional latent space. We summarize our main contributions as follows:
We propose a structured dependent proposal of GANs, which reparameterizes the sample- level transition x + x' into the latent-level + z' with two pairing Markov chains. We prove that our reparameterized proposal admits tractable acceptance criterion.
Our proposed method, called REP-GAN, serves as unified framework for the existing sam- pling methods of GANs. It provides better balance between exploration and exploitation by the structured dependent proposal, and also corrects the bias of Markov chains by the acceptance-rejection step.
Empirical results demonstrate that REP-GAN achieves better image quality and much higher sample efficiency than the state-of-the-ee methods both synthetic and real datasets.
2
RELATED WORK
Although GANs able synthesize high-quality images, the minimax nature of GANs makes quite unstable, which usually results in degraded sample quality. A vast literature has been developed fix the problems of GANs ever since, including novel network modules (Miyato et al. 2018). training mechanism Metz et al. 2017), and alternative objectives (Arjovsky et al. 2017).
Moreover, there is another line of work using sampling methods to improve the sample quality of
GANs. DRS (Azadi et al.) (019)) firstly proposes to use rejection sampling. MH-GAN (Turner et al. 2019) instead uses the Metropolis-Hann Hasting (MH) algorithm with an independent proposal. DDLS (Che et al. 2020) and DCD (Song et al.) 2020) apply gradient-based proposals by viewing GAN as an energy-based model. [Tanaka (anaka)(2019) (2019) proposes similar gradient-based method named DOT from the perspective of optimal transport.
Table I: Comparison of sampling methods for GANs in terms of three effective sampling mechanisms.
Method
Rejection step
Markoy chain
Latent gradient proposal
GAN
DRS Azadi et al.
2019
XX X
MH-GAN (Turmer et al.
2019
DDLS (Che et al. 2020
X
REP-GAN (ours)
2

---

Under review as a conference paper at ICLR 2021
Different from them, our REP-GAN introduces structured dependent proposal through latent reparameterization, and includes all three effective sampling mechanisms, the Markov Chain Monte Carlo method, the scceptance-rejeettinn step, and the latent gradient- based proposal, to further improve the sample efficiency. As shown in Table many existing works are special cases of our REP-GAN.
Our method also belongs the thread of works that combine MCMC and neural networks for better sample quality. Previously, some works combine variational autoencoders (Kingma & Welling 2014)
and MCMC bridge the amorization gap (Salimans al. 2015 Hoffmann [201]] 2017|Li al. 2017)
while others directly learn a neural proposal function for MCMC (Song et al. 2017 Levy et al. 2018
Wang et al., 2018 Our work instead reparameterizes the ligh-dimensional sample- transition into a simpler oow-dimensional latent space via the learned generator network.
3 BACKGROUND
3.1 GAN
GAN models the data distribution px(x) implicitly with generator G 2 -> X' mapping from low-dimensional latent space 2 to a high-dimensional sample space X,
x = G(z), Z z~
Po(2),
(I)
where the sample follows the generative distribution Py (x) and the latent variable z follows the prior distribution pz), e.g., a standard normal distribution (0,, I). In GAN, a discriminator D X' [0, 1] is learned distinguish samples from pa(x) and (x in an adversarial way
min max Ex~pa(x) log( (xx)) + (o)) log(1 D(G(z))).
G
Goodfellow et al. (2014) point out that an optimal discriminator D implies the density ratio between the data and generative distributions
Pa(x)
Pa(x) 1 Px(x)
D(x) pa(x) + (x))
(3)
D(x)
3.2 MCMC
Markov Chain Monte Carlo (MCMC) refers to kind of sampling methods that draw a chain of samples X1:K E 2 from target distribution Pe (x).. We denote the initial distribution as Pu(x) and the proposal distribution as q(x' (x')x)). (xx). With the Metropolis-Hastinn (MH) algorithm, we accept the proposal x' ((x((xa) with probability
a (x', xx) min (1, Pt (x') (xx|x')
[0. 1].
Pe (xx) 9 (x' xx.
If x''' is accepted, X x', otherwise x Xx. Under mild assumptions, the Markov chain is guaranteed to converge to Pt (x) as K + co. In practice, the sample efficiency of MCMC crucially depends on the proposal distribution trade off between exploration and exploitation.
THE PROPOSED REP-GAN
In this section, first review MH-GAN and point out the limitations. We then propose our structured dependent proposal to overcome these obstacles, and finally discuss its theoretical properties as well as practical implementations.
4.1 FROM INDEPENDENT PROPOSAL TO DEPENDENT PROPOSAL
MH-GAN Turner et al. 2019) first proposes to improve GAN sampling with MCMC. Specifically, given a perfect discriminator and a descent (but imperfect) generator G after training, they take the data distribution (x(x) the target distribution and use the generator distribution Pg (x() as independent proposal
x' q (x' xxx) (x') (x)).
(5)
3

---

Under review as a conference paper at ICLR 2021
With the MH criterion (Eqn.
and the density ratio (Eqn.
(3, ) should accept x' with probability
Pa (xx) q (Xcx) Pa (xx) (xx)
D (xx')- (
OMH (x', xx) min 1,
min
(6)
However, to achieve tractability, MH-GAN adopts an independent proposal g(x') with poor sample efficiency. As the proposed sample x' is independent of the current sample X3. the difference between the two samples can be so large that it results in a very low acceptance probability. Consequently, samples can be trapped in the same place for a long time, leading to very slow mixing of the chain.
A natural solution is to take a dependent proposal q(x ((x|(x+) that will propose a sample x' close to the current one X3. which is more likely to be accepted. Nevertheless, the problem of such a dependent proposal is that its MH acceptance criterion
Pa (x') (Xxlxx) 1, Pa (xxx q (x') xxxx
ODEP (x', min
is generally intractable because the data density pa(x) is unknown. Besides, it is hard to design a proper dependent proposal q(x' xx)) in the high dimensional sample space X' with complex landscape. These obstacles prevent us from adopting dependent proposal that is more suitable for MCMC.
4.2 A TRACTABLE STRUCTURED DEPENDENT PROPOSAL WITH REPARAMETERIZED MARKOV CHAINS
As discussed above, the major difficulty of a general dependent proposal q(x' |xx) is to compute the MH criterion. We show that it can be made tractable by considering additional pairing Markov chain in the latent space.
As know, samples of GANs lie in low-dimensional manifold induced by the push-forward of the latent variable. Suppose that at the k-th step of the Markov chain, we have a GAN sample x5; with latent Zj. Instead of drawing sample x' directly from sample-level proposal distribution q(x' xx:). we first draw latent proposal z' from a dependent latent proposal distribution q(z' (zz). Afterward, we push the latent z' forward through the generator and get the output x' as our sample proposal.
As illustrated in Figure T. our bottom-to-up proposal relies on the transition reparameterization with two pairing Markov chains in the sample space X' and the latent space Z. Hence we call it a REP (reparameterized) proposal. Through a learned generator, we transport the transition X - x' in the high dimensional space X' into the low dimensional space Z, 24 - z', which enjoys much better landscape and makes it easier to design proposals in MCMC algorithms. For example, the latent target distribution is nearly standard normal when the generator is nearly perfect. In fact, under mild conditions, the REP proposal distribution QREP (x|xx)) and the latent proposal distribution g(z' [zx) are tied with the following change of variables (Gemici et al. 2016 Ben-Israel 1999)
log QREP (x' (xx) = log g(x = log [zx) - log det J J2's
(8)
2
where Jg denotes the Jacobian matrix of the fosh-forward G at z, i.e., [Jeli 0x 0x, G(z).
Nevertheless, it remains unclear whether we can perform the MH test to decide the acceptance of the proposal x'. Note that general dependent proposal distribution does not meet tractable MH acceptance criterion (Eqn. 7). Perhaps surprisingly, it can be shown that with our structured REP proposal, the MH acceptance criterion is tractable for general latent proposals ((z'(zx).
Theorem 1. Consider a Markov chain of GAN samples X1:K with initial distribution (x)).. For step k ++1 1, we accept our REP proposal x' "QREP(X* (x') (xx) with probability
OREP (x'xx)) = min (1, Po (o(zz)q(zz )q(zelz')
D(xx)-1-1 D(x')
(9)
Po
f.e. Xe++1 = x' if is accepted and x xx otherwise. Further assume the chain is irreducible, aperiodic and not transient. Then, according to the Metropolis- HatropolisHHastinss algorithm, the stationary distribution of this Markov chain is the data distribution pa(x) Gelman et al. 2013).
4

---

Under review as a conference paper at ICLR 2021
Proof. Note that similar Eqn (8, also have the change of variables between Pa (x) and pa (z),
log (x))x=(( = log log det Jz Jz.
(10)
According to Gelman et al. (2013), the assumptions that the chain is irreducible, aperiodic, and not transient make sure that the chain has unique stationary distribution, and the MH algorithm ensures that this stationary distribution equals to the target distribution pa(x). Thus we only need to show
that the MH criterion in Eqn. ( holds. Together with Eqn. (3) and (8). we have
Pa (x') ( (x')
Pa
(z') (det JT J2.) - Pa (xx
OREP (x', xx.
Pa (xx) (x' X1)
Pa (xx)g (det Ju Jz') Py (x (2(xx))0)) (x))))(()))))))
g(zz|z') ( (det Jz Jo.)
po po ( (det J.. (D(xx) -1
(11)
((z1|zz) ) J Jz Po(zz)) (det Jz J2.) (D(x')
po (Zz (D
po )a Zn (D(x))11-)
Hence the proof is completed.
The theorem above demonstrates the following favorable properties our method:
The diseriminator score ratio is the same ( xx), but MH-GAN is restricted to specific independent proposal. Our method instead works for any latent proposal ((z'|zz). When take ((z' (zz) = (o our method reduces MH-GAN.
Compared to aDEP (x', Xx) of general dependent proposal (Eqn. 77). the unknown data distributions terms are successfully cancelled in the reparameterized acceptance criterion.
The reparameterized MH acceptance criterion becomes tractable only involves the latent priors, the latent proposal distributions, and the discriminator scores.
Combining the REP proposal qReP (x') xx)) and its tractable MH criterion GREP xx), we have developed a novel sampling method for GANs, coined REP-GAN, See [p detailed description. Moreover, our method can serve as general approximate inference technique for Bayesian models by bridging MCMC and GANs. Previous works (Marzouk et al. 2016 Titsias 2017, Hoffman et al. 2019) also propose to avoid the bad geometry complex probability measure by reparameterizing the Markov transitions into simpler measure. However, these methods are limited to explicit invertible mappings without dimensionality reduction. In our work, we first show that it is also tractable to conduct such model-based reparameterization with implicit models like GANs.
4.3 A PRACTICAL IMPLEMENTATION
REP-GAN enables to utilize the vast literature of existing MCMC algorithms (Neal et al. 2010) to design dependent proposals for GANs. We take Langevin Monte Carlo (LMC) as an example. As an Maler-Maruyama discretization of the Langevin dynamics, LMC updates the Markov chain with
X
X + 2
V (xx))(xx) + VT e, e~ e~ I(0,I,
(12)
for a target distribution p. (x). Compared to MH-GAN, LMC utilizes the gradient information to efficiently. However, if we directly take the (unknown) data
explore the energy landscape distribution pa(x) as the target distribution pe (x), LMC does not meet a tractable update rule.
As discussed above, the reparameterization of REP-GAN makes it easier design transitions in the low-dimensional latent space. Hence, we instead propose to use LMC for the latent Markov chain. We assume that the data distribution also lies the fow-dimensional manifold induced by the generator, i.c., Supp (pa) C Im(G). This implies that the data distribution (x(x) also has pairing distribution in the latent space, denoted as pe (z). They are tied with the change of variables
log (xx) O(x) = (z())
log det JT Ja.
(13)
2
5

---

Under review as a conference paper at ICLR 2021
Taking (z) as the (unknown) target distribution of the latent Markov chain, we have the following
Latent LMC (L2MC) proposal
T V (rg))(zz) + V
z' 21
+
2
(r(za) (det Jz. Jai)
-
2/. +
2
V log
(
(det JI.
)-
+ Vz logpo(zz) + VF-E
(14)
+
7
v, log
Pd (x(x))
+
T
Vx log po (Zu) + v
2
Pgl (xx)
2
2 log D (xx) 1)
2 V. (ox)o(zz) + VT.e, c~
2
where X -(zz)) As sec, L2MC is made tractable by our structured dependent proposal with pairing Markov chains. DDLS (Che et al.) 2020) proposes a similar Langevin proposal by formalizing GANs derivation through reparameterivation. Our major difference to DDLS is that REP-GAN also includes tractable MH correction step (Eqn. ()) which accounts for the numerical errors introduced by the discretization and ensures that detailed balance holds.
implicit energy-based model, while here provide straightforward
4.4 EXTENSION TO WGAN
Our method can also be extended to other kinds of GAN, like Wasserstein GAN (WGAN) (Arjossky et al. 2017). The WGAN objective is
min max Ex-pa(x) [D(x)] Ex-py (x) [D(x)}
(15)
G
where D X + I is restricted to be Lipschitz function. Under certain conditions, WGAN also implies an approximate estimation of the density ratio (Che et al. 2020).
(x) Py (x)
Pa(x)
D(x) ~ log
+ const
e(p(D(x)) const.
(16)
Pg (x(x)
Following the same derivations as in Eqn. (T]) and 14, we will have the WGAN version of REP-GAN. Specifically, with X G(zz), the L2MC proposal follows
z' V ((xx) 2
Vx log (o(x)) VT e, E~ c(0,0,
(17)
and the MH acceptance criterion is
QREP-W (x' xx) min (1, g(zz|zz))p (o(z')
exp (D(x')
(18)
q(<' 2u (ol))))
exp (D(xx))
5 EXPERIMENTS
We show our empirical results both on synthetic and real image datasets.
5.1 SYNTHETIC DATA
Following DOT (Tanaka 2019) and DDLS (Che al. 2020 we apply REP-GAN to the synthetic
Swiss Roll dataset, where data samples lie on a Swiss roll manifold in the two-dimensional space. We construct the dataset by scikit-learn with 100,000 samples. and train a WGAN as in Tanaka (2019). where both the generator and discriminator are fully connected neural networks with leaky ReLU nonlinearities. We optimize the model using the Adam optimizer, with a = 0.0001, 31 = 0.5, 82 = 0.9. After training, draw 1,000 samples with different sampling methods.
As shown in Figure with appropriate step size (T = 0.01), the gradient-based methods (DDLS and REP-GAN) outperform independent proposals (DRS and MH-GAN) by large margin, while DDLS is more discontinuous on shape compared to REP-GAN In DDLS, when the step size becomes too large (r 0.1, 1), the numerical error of the Langevin dynamics becomes so large that the chain either collapses or diverges. In contrast, those bad proposals are rejected by the MH correction steps of REP-GAN, which prevents the mishehavior of the Markov chain.

---

Under review as a conference paper at ICLR 2021
MH.GAN
DDLS (aaL00.01
REP-GAN (lau=0.01)
4
44
-1
-40
Vanilla Sampling
DOLS (AUR0.1)
REP-GAN (bunn..))
DDLS (BUm1)
GEP (1Nu)1)
3
43
40
Figure 2: Visualization of samples with different sampling methods on the Swiss Roll dataset. Here tau denotes the Langevin step size in Eqn. (17).
Table Inception Scores (IS) of different sampling methods on CIFAR-10 10 and CelebA. For GAN,
DRS and MH-GAN, we report the results in Turner et al. 2019 The results are given by our implementation.
CIFAR-10
CelebA
Method
DCGAN
WGAN
DCGAN
WGAN
GAN
2.879
3.073
2.332
2.788
DRS Azadi et al. 2019
3.073
3.137
2.869
2,861
MH-GAN Turner
2019
3.379
3.305
3.106
2.889
DDLS (Che et al. 2020
3,518
3.547
2.534
2.862
REP-GA REP-GAN (ours)
3.851
4.035
2.686
2.943
5.2 REAL IMAGE DATA
Following MH-GAN (Turner et al. 2019 we conduct experiments on two real-world image datasets, CIFAR-10 and CelebA, for DCGAN (Radford al., (215)) and WGAN (Arjovsky et al., (017).
Following the conventional evaluation protocol, we initialize each Markov chain with a GAN sample,
run for 640 steps, and take the last sample for evaluation. We collect 50,000 samples to evaluate the Inception Scord") Salimans et al. 2016). The step size of our L2MC proposal is 0.01 CIFAR-10 10 and 0.1 on CelebA. We calibrate the discriminator with Logistic Regression as in Turner et al. (2019).
From Table we plot the Inception Score and acceptance ratio per epoch in Figure based on our re-implementation. Although the training process of GANs is known to be very unstable, our REP-GAN can still outperform previous sampling methods both consistently (superior most epochs) and significantly (the improvement is larger than the error bar) shown in the left panel of Figure In the right panel, we find that the average acceptance ratio of MH-GAN is lower than 0.2 in most cases, while REP-GAN has acceptance ratio of 0.4-0.8, which known algorithms. We also notice that the acceptance ratio goes down as the training continues. We suspect this is because the distribution landscape becomes complex and a constant sampling step size will produce more distinct samples that are more likely to get rejected.
method outperforms state-of-the-art methods on both datasets. We also
a good tradcoff for MCMC
For fair comparison, training and evaluation follows the the official code of MH-GAN Turner al. https: //gi thub. com/ubees research/netreeeee
2019
hast waastinggs gans

---

Under review as a conference paper at ICLR 2021
GAN DOLS REI--GA
MH-GAN REP-CAN
20
Inceplian Score
Acceptance ratlo
Epoch
Epoch
Figure 3:
Average Inception Score (left) and acceptance ratio (right) training epoch CIFAR-10
based on our re-implementation. The standard deviation shown with shaded error bar (left).
Table Ablation study of our REP-GAN with Inception Scores (IS) and acceptance ratios (Accept) averaged over five adjacent checkpoints. IND refers to the independent proposal of MH-GAN, REP refers to our REP proposal. MH denotes the MH rejection step the corresponding sampler.
CIFAR10
CelebA
Proposal
MH
DCGAN
WGAN
DCGAN
WGAN
Accept
IS
Accept
IS
Accept
IS
Accept
IS
IND
X
3,769
3,557
2,431
2.799
REP
X
3.312
3.500
2.465
2.868
IND
0.033
3.735
0.084
3.509
0.678
2.500
0.747
2.879
REP
0.363
3.820
0.447
3.598
0.628
2.476
0.955
2.894
that without the MH correction step, the Langevin steps often result in worse sample quality. Mcanwhile, the acceptance is very small on CIFAR-10 10 without the dependent REP proposal. improves over the baseline and obtains the best Inception Score on each dataset. The only exception DCGAN on CelebA, where the independent proposal outperforms our REP proposal with a higher acceptance ratio. We believe that this is because the human face samples of CelebA are very alike among each other, such that independent samples from the generator can also easily accepted. Nevertheless, the acceptance ratio of the independent proposal can be much lower on datasets with diverse sources of images, like CIFAR-10 10.
Ablation study. From Table 3
result, our REP-GAN (REP+MH) is the only setup that consistently
Figure 4: The first 20 steps of two Markov chains with the same initial samples. The chains are generated by CH-GAN (top), DDLS (middle), and REP-GAN (bottom).
Markov chain visualization. In Figure demonstrate two Markov chains sampled with different methods. We can see that MH-GAN is often trapped in the same place because the independent proposals. DDLS and REP-GAN instead gradually refine the samples with gradient steps. In addition,
8

---

Under review as a conference paper at ICLR 2021
Figure 5: Visualization of Markov chains of our REP proposals (i.e., REP-GAN without the MH rejection steps) with a large step size (T = 1).
compared the gradient-based methods, we can that the MH rejection steps of REP-GAN help avoid some bad artifacts in the images. For example, in the camel-like images marked in red, the body of the camel is separated in the sample of DDLS (middle) while it is not in the sample of REP-GAN (bottom). Note that, the evaluation protocol only needs the last step of the chain, thus we prefer small step size that finetunes the initial samples for better sample quality. As shown in Figure our REP proposal can also produce very diverse images with a large step size.
CONCLUSION
In this paper, we have proposed a novel method, REP-GAN, to improve the sampling GAN. We devise structured dependent proposal that reparameterizes the sample-level transition of GAN into the latent-level transition. More importantly, we first prove that this general proposal admits tractable MH criterion. Experiments show method does not only improve sample efficiency but also demonstrate state-of-the-ar sample quality on benchmark datasets over existing sampling methods.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. ICML, 2017.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrimi- nator rejection sampling. ICLR, 2019.
Adi Ben-Israel. The change-of-variables formula using matrix volume. SIAM Journal on Matrix Analysis and Applications, 21(1):300-31,, 1999.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. ICLR, 2019.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein,, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly energy-based model and you should use discriminator driven latent sampling. ICML., 2020.
Andrew Gelman, John B Carlin, Hal S Stern, David Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows Riemannian manifolds, arXiv preprint arXiv:1611.02300. 2016.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NewrIPS, 2014.

---

Under review as a conference paper at ICLR 2021
Matthew Hoffman, Pavel Sountsov, Joshua V Dillon, lan Langmore, Dustin Tran, and Srinivas Vasudevan. Neutra-lizing bad geometry in Hamiltonian Monte Carlo using neural transport. arXiv preprint 1xXi::1903.03044, 2019.
Matthew D Hoffman. Learning deep latent gaussian models with markov chain monte carlo. In ICML, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. NeurIPS, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Daniel Levy, Matthew D Hoffman, and Jascha Soht-Dickstein. Generalizing Hamiltonian Monte Carlo with neural networks. ICLR, 2018.
Yingzhen Li, Richard Turner, and Qiang Liu. Approximate inference with amortised meme. arXiv preprint arXiv: 7ee.:00772..333, 2017.
Youssef Marzouk, Tarek Moselhy, Matthew Pamo, and Alessio Spantini. An introduction to sampling via measure transport. arXiv preprint arXiv: 05002, 2016.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. ICLR, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Radford M Neal et al. MCMC using Hamiltonian dynamics. Handbook of markov chuin monte carlo, 54:1 13-162, 2010.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Tim Salimans, Diederik Kingma, and Max Welling, Markov chain monte carlo and variational inference: Bridging the gap. In ICML, 2015.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. NeurlPS, 2016.
Jiaming Song. Shengjia Zhao, and Stefano Ermon. A-NICE-MC: Adversarial training for MCMC. NeurlPS, 2017.
Yuxuan Song, Qiwei Ye, Minkai Xu, and Tie-Yan Liu. Discriminator contrastive divergence: semi-amortized generative modeling by exploring energy of the discriminator. arXiv preprint arXiv:2004.0170. 2020.
Akinori Tanaka. Discriminator optimal transport. NeurIPS, 2019.
Shichang Tang. Lessons learned from the training gans on artificial datascts. IEEE Access, 165044-165055, 2020.
Michalis K Titsias. Learning model reparametrizations:: implicit variational inference by fitting MCMC distributions. arXiv preprint arXiv: 1708.0177 2017.
Ryan Turner, Jane Hung, Yunus Saatci, and Jason Yosinski. Cetropolis-Hastinge generative adversar- ial networks. ICML, 2019.
Tongzhou Wang, Wu, Dave Moore, and Stuart J Russell. Meta-learning meme proposals. In NeurIPS, 2018.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
ASSUMPTIONS AND IMPLICATIONS
Note that our method needs few assumptions on the models for our analysis hold. Here we state them explicitly and discuss their applicability and potential impacts.
Assumption 1. The generator mapping G IR - Rn m) is injective, and its Jacobian matrix of size m x n, has full column rank for all 2 E IR.
G(x)
Oz
For the change of variables in Equ. 8 and (10) to hold, according to Ben-Israel (1999). we need the mapping to be injective and its Jaobian should have full column rank. A mild sufficient condition for injectivity is that the generator only contains (non-degenerate) affine layers and injective non- linearities, like LeakyReL.U. It is not hard to show that such condition also implies the full rankness of the Jacobian. In fact, this architecture has already been found to benefit GANs and achieved state-of-the-art results (Tang 2020). The affine layers here also likely be non-degenerate because their weights are randomly initialized and typically will not degenerate in practice during the training of GANs.
Assumption 2. The discriminator D offers perfect estimate the density ratio between the generative distribution Pg (x) and the data distribution Pa(x) as in Eqn.
This is a common, critical, but less practical assumption among the existing sampling methods of GANs. It is unlikely hold exactly in practice, because during the alternative training of GANs, the generator is also changing all the time, and the a few updates of the discriminator cannot fully learn the corresponding density ratio. Nevertheless, density ratio which explains why the sampling methods can consistently improve over the baseline at each epoch.
think it capture certain extent information of
From our understanding, the estimated density ratio is enough to push the generator better but not able bring it up to the data distribution. This could be the reason why the Inception scores obtained by the sampling methods, can improve over the baselines but cannot reach to that of real data and fully close the gap, even with very long run of the Markov chains.
Hence, there is still much room for improvement. To list few, one can develop mechanisms that bring more accurate density ratio estimate, or relax the assumptions for the method hold, or establishing estimation error bounds. Overall, we believe GANs offer an interesting alternative scenario for the development of sampling methods.
A.2 ALGORITIM PROCEDURE
We give detailed description of the algorithm procedure of our REP-GAN Algorithm !
Algorithm 1 GAN sampling with Reparameterized Markov chains (REP-GAN)
Input:
trained GAN with (calibrated) discriminator / and generator G, Markov chain length K, latent prior distribution po (z), latent proposal distribution g(z'lzz, (zzl Output: an improved GAN sample xx;
Draw an initial sample x 1) draw initial latent ž ~ Po(z) and 2) push forward x G(zi); for each step A: E [1, K 1] do
Draw REP proposal x' ~ (REP(x' x+): 1) draw a latent proposal z' ~ q(z' [zz), and 2) push forward x' G(z');
Calculate the MH acceptance criterion OREr(XE, x') following Eqn. (9:: Decide the acceptance of x' with probability onEr (nEr(X;;,")) xx);
if x' is accepted then
Let X = x', Zi+1 =
else
Let X6+1 = Xk. 2b+1 = 2
end if
end for
11

---

Under review as a conference paper at ICLR 2021
Table 4: Fréchet Inception Distance (FID) of different MCMC sampling methods on CIFAR-10 and
CelebA based on our re-implementation.
CIFAR-10
CelebA
Method
DCGAN
WGAN
DCGAN
WGAN
GAN
100.363
153.683
227.892
207.545
MH-GAN (Trrner al.
2019
100.167
143,426
227.233
207.143
DDLS (Che et al. 2020
145.981
193.558
269.840
232.522
REP-GAN (ours)
99.798
143.322
230.748
207.053
Table 5: Comparison of computation cost (measured in seconds) of gradient-baset MCMC sampling methods of GANs. We report the total time to sample a batch of 500 samples with DCGAN on a NVIDIA 1080 Ti GPU. We initialize the chain with GAN samples and run each chain for 640 steps.
DDLS REP-GAN (ours)
REP-GAN w/o MH correction
88.94s
88.85s
87.62s
A.3 ADDITIONAL EMPIRICAL RESULTS
Here we list some additional empirical results of our methods.
Fréchet Inception Distance (FID). We additionally report the comparison of Fréchet Inception Distance (FID) in T Because previous works do not report FID on these benchmarks, we report re-implementation results instead. We in Table and our method is superior in most cases.
the ranks consistent with the Inception scores
Computation overhead. In Table [] compare different gradient-based sampling methods of GANs, Comparing DDLS and our REP-GAN, they take 88.94s and 88.85s, respectively, hence the difference is negligible. Without the MH-step, our method takes 87.62s, meaning the additional MH-step only costs 1.4% computation overhead, which is also negligible, but it brings significant improvement of sample quality as shown Table 3
Markov chain visualization on CelebA. We demonstrate two Markov chains CelebA with different MCMC sampling methods of WGAN in Figure We can see that on CelebA, the acceptance ratio of MH-GAN becomes much higher than that on CIFAR-10. Nevertheless, the sample quality is still relatively low. comparison, the gradient-based method can gradually refine the samples with Langevin steps, and our REP-GAN can alleviate image artifacts with MH correction steps.
A.4 MULTI-MODAL EXPERIMENTS
Aside from the manifold learning example shown in Figure 2, we additionally conduct experiments illustrate the performance of our sampling methods for multi-modal distributions.
25- AS-Gaussinns. To begin with, we consider the 25-Gaussians dataset widely discussed in previous
work (Azadi et al. 2019 Turner et al. 2019 Che al. 2020). The 25-Gaussians dataset is generated by mixture of twenty-five five -io-dimensional isotropic Gaussian distributions with variance 0.01, and means separated by 1, arranged in a grid. We train small Wasserstein GAN model with the standard WGAN-GP objective following the setup in Tanaka (2019). After training, we draw 1,000 samples with different sampling methods. Similarly, we starts a Markov chain with a GAN sample, run it for 100 steps, and collect the last example for evaluation.
As shown in F compared to MH-GAN, the gradient-based methods (DDLS and ours) produce much better samples close to the data distribution with proper step size. Comparing DDLS and ours, DDLS tends to concentrate so much on the mode centers that its standard deviation can be even smaller than that of data distribution. Instead, our method preserves more sample diversity while concentrating on the mode centers. When the step size becomes larger, the difference becomes more obvious. When 0. marked with blue circles, the samples of DDLS become concentrated
12

---

Under review as a conference paper at ICLR 2021
Figure 6: Visualization of the Markov chains of MH-GAN (top), DDLS (middle), and REP-GAN (bottom) on CelebA with WGAN backbone.
Training samples
MH-GAN
(1NN=0.01)
REP-CAN (mo.-0.01)
Vanill Sampling
DOLS (1))0.1)
REP-GNN (W))0.3)
GCLS ((UN1)
REP-GAN (1APR1)
41
missing modes
Figure 7: Visualization samples with different sampling methods on the 25-Gaussians dataset. Here denotes the Langevin step size in Eqn.
that some modes missed. When comparison, our REP-GAN does not suffer from these issues with the MH correction steps accounting for the bias introduced by numerical errors.
samples of DDLS diverge far beyond the 5x5 grid. In
Scale to more modes. In the above, we have experimented w.r.t. a relatively easy scenario where the multi-modal distribution only has 5x5 modes (n 5 modes along each axis). In fact, the distinctions between the sampling methods become even more obvious when we scale to more modes. Specifically, shown in Figure we also compare them w.r.t. mixture of Gaussians with 9x9 and 13x13 modes, respectively. The rest of the setup is similar to 25-Gaussians. Note that throughout the experiments in this part, we adopt proper step size, 0.01, for the gradient-base methods (DDL. and REP-GANT by default.
Under the more challenging scenarios, we can see that the gradient- methods still consistently outperforms MH-GAN, Moreover, our REP-GAN has more clear advantage over DDLS. Specifi- cally, for 9x9 modes, our REP-GAN produces samples that are less noisy (i.e., less examples distinct from the modes), while preserving all the modes. For 9x9 modes. DDLS makes critical mistake that it drops one of the modes (left down corner, marked with red circle) during the Markov chain update. As discussed above, believe this is because DDLS has bias towards regions with high
13

---

Under review as a conference paper at ICLR 2021
Vanilla sampling
MH-GAN
DDLS
REP-GAN (ours)
n=
n=13
Figure 8: Visualization of the multi-modal experiments with more modes. Specifically two cases are considered, 9x9, and 13x13 mixture of Gaussian. The true data points are shown grey in background, and the generated points with different sampling methods are shown in blue.
probability, while ignoring the diversity of the distribution. In comparison, REP-GAN effectively prevents such bias by the MH correction steps.
14


Reviews:
[{"review_id": "UXfVdn2zWvV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is a good idea. It is based on the notion of following the same line of reasoning as the prior work of MH-GAN. However, it makes the crucial advance of allowing the use of gradient information for the far more efficient HMC-type samplers.\n\nThe paper is able to do much of the sampling in the latent space, which enables the use of gradient-based sampling. One nitpick would be that I suspect there is a latent assumption that the mapping of z -> x is injective (but need not be bijective like in normalizing flows). However, this is not explicitly stated at all in the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q32udMOfAE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe paper proposes an MCMC based sampling mechanism for GANs. In contrast to earlier work, the proposal distribution is conditioning conditioned on the previous state (here in latent space), which is supposed to help sampling efficiency. This is achieved by a clever re-parametrization of intermediate steps of the MCMC chain. As an example, the authors provide a Langevin version (which uses gradient information) of their method.\n\nI enjoyed reading this well-written paper and think the re-parameterized MCMC chain is a very neat idea that fits very naturally in the GAN framework. The paper is of quite incremental nature though: little to no theory contribution, only some incremental empirical benefits. I also have some doubts whether the comparisons are fair in terms of compute and effective sample size (more below).\n\nBelow are a few points to consider:\n\n**Effective sample size**\nWhat puzzles me here is the fact that while the independent proposal of Turner et al gets stuck in certain parts of the space, the Langevin updates of the proposed method (Figure 4) do have less variability. I assume the incremental improvement happen due to relatively small step sizes in the Langevin sampler? This would have a bad effect on the effective sample size. Does the MCMC chain eventually visits other parts of the space, or does it need to be restarted for every sample created? This does not seem to be true for the MH sampler, which despite its low acceptance rate produces samples from the entire space. Could the authors clarify this, and if possible provide measure of effective sample size for a fixed number of iterations?\n\n**Compute**\nThe authors do neither discuss nor evaluate the computational load compared to other methods, especially performance metrics as a function of compute would be interesting. Or maybe the computational load is comparable for all methods?\n\n**Title**\nThe term \"coupled\" is currently heavily used within the MCMC community, e.g. [2], and it denotes a different technique. I would suggest replacing it by \"re-parameterized\" which would be more accurate in that respect.\n\n**\"Proof\" for convergence**\nThe abstract claims that convergence to the true data distribution is proved, but that is really stretching the term. All the paper contains is a valid MH ratio. But that alone does not prove convergence, for which the authors would need to go into the weeds of MCMC convergence, with conditions on the target distribution. This is especially true for the Langevin proposal, e.g. [1].\n\n**Experiments**\nI would suggest to move the Swiss roll dataset to the Appendix and replace it with something more interesting, preferably higher dimensional. This would make the paper stronger. In fact some analysis how all the different sampling methods compare as a function of dimensional (or \"difficulty\") would make it even more so.\n\n\n[1] Roberts, G. O., & Tweedie, R. L. (1996). Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli.\n\n[2] Jacob, P. E., O'Leary, J., & Atchadé, Y. F. (2017). Unbiased markov chain monte carlo with couplings. JRSS-B.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9-QGS49AFcm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nIn this work, the authors propose coupling markov chain GAN (CMC-GAN), a technique that allows to draw GAN samples by running an MCMC chain that uses the generator to produce proposals, and the discriminator to compute acceptance probabilities.\n\nWhereas prior work by Turner et al. (2019) uses an independent proposal, for each sample, the present work uses a small random perturbation of the last sample *in the latent space* to create the proposal. \nGoodfellow et al. 2014 showed that in the large data limit and large model capacity limit for absolutely continuous distributions, the output of the optimal discriminator represents encodes the density ratio of target and generated data.\nUsing this result, the authors of the present work derive a tractable expression for the acceptance probabilities that leads to an unbiased estimate of the target distribution in settings where the results of Goodfellow et al. 2014 hold.\nThey also show that an analogue acceptance criterion can be derived for the popular WGAN. \nThe authors show experimental results on the synthetic swiss roll dataset as well as CIFAR10 and CELEBA with DCGAN and WGAN architecture, showing improved sample quality (measured by IS on the image datasets) compared to standard GAN and competing MCMC approaches, and improved acceptance rate compared to GAN. \n\n### Decision\nI believe that proposes the most natural way for performing MCMC sampling on GANs. Both the dependent sampling based on the latent space and the acceptance mechanism based on the discriminator are natural and clean and, I'm sure will end up useful one way or another.\nWhere the paper is lacking, in my opinion, is in making a case *why* and *when* one should be using MCMC based methods in the first place when using GANs. \nWhile I do find the results on the swiss roll experiments are convincing, I am not convinced by the results on image datasets. \nThe improvements seem to be minor, inconsistent, and on baselines performing worse than commonly reported in the literature.\nGiven how clean the proposed method itself is authors should improve the motivation by trying to find a more convincing application. I think it would be a wasted opportunity for what could be a very nice paper to accept it without such improvements.\n\n### Questions/Suggestions\n\n- The only motivation for the use of MCMC techniques is to not \"waste the discriminator\". Why should I expect the discriminator to be any better than the generator that it was trained with? Are there situations where one would naturally end up with a \"better\" discriminator than generator? Given that ordinary GAN sampling has 100 percent sample efficiency, I feel that a stronger argument in favor of using any MCMC based-technique in the first place is necessary.\n\n- While I do find the experiments on swiss roll convincing, I am confused by the experiments on image GANs. **First**, the inception scores given in Table 2 seem significantly below those usually reported in the literature. **Second**, the experiments in Figure 3 show a minimal improvement of CMC-GAN compared to ordinary GAN sampling, in particular compared to the wide oscillation of IS over the different epochs. In particular, Figure 3 and Table 2 do not seem to be consistent. **Third**, it is my understanding that it does not make much sense to evaluate the IS on CELEBA. Usually, FID is used instead. \n\n- Recent work such as [Berard et al.](https://arxiv.org/abs/1906.04848), [Schafer et al.](https://arxiv.org/abs/1910.05852),  [Arjovsky and Bottou 2017](https://arxiv.org/abs/1701.04862) suggests that the behavior of discriminator training on image GANs might be radically different than in the idealized case analyzed by Goodfellow et al. in 2017. \nCould this be the reason why the results on image data sets fall short of those obtained in the absolutely continuous swiss roll dataset. Maybe techniques such as in [Dieng et al. 2019](https://arxiv.org/abs/1910.04302) could be useful?\n\n- Although it has a somewhat different goal, it might still be useful to cite [Song et al.](https://arxiv.org/abs/1706.07561).\n\n=============================================================================================\n\nAfter discussion with the authors and reading the other reviews I still believe that the paper should not be accepted and therefore stay with my original review. While the authors have shown improvement over previous work (MH-GAN) using a very natural idea, this previous work in turn has not provided sufficient evidence that MCMC-GANs are useful.\nI believe that we should not accept further MCMC-GAN papers, before this methodology has shown any improvement on a plausible use-case.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2gmInCq9w5q", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes an MCMC sampling strategy for GANs. The idea is clear: for high-dimensional x, making a good proposal is difficult, so they propose to do that in the latent space.  Then they use a similar strategy as MH-GAN to compute a rejection strategy. The difference between the two methods is that proposal in MH-GAN does not depend on x while the proposed method does, and the argument is that it results in a higher acceptance rate. \n\nThe idea is interesting, and the paper is relatively well-written, but the execution and experiment left a lot to be desired.\n\n* One of the arguments of the paper (in the related work) the method overcomes sample low sample diversity. No metric is reported for that. The authors need to report LPIPS [1] and compare different methods. \n\n*  Fig 4 shows very poor diversity; all samples seem identical to each other, and honestly, it seems that the GAN collapsed. \n\n* The results in Fig 3 is also very strange. First of all, what is the point of Fig 3 (left)? All IS scores are roughly going up, but this figure is no comparison and does not show any if one method is doing better.  Fig 3 (right) is more concerning. One expects that as the epoch increases, the generator becomes better at generating samples. The acceptance rate improves (b/c G generates more realistic results and discriminator cannot tell them apart); why the acceptance rates are going down?\n\n* Fig 2 is also presented in a confusing way. All methods except the second column from left almost perform the same and propose methods is in both columns (!!) What does each column, row mean? Given what I read in the related work, I expected to see the model's performance in the sample a multi-modal distribution. In fact, I want to see that: 100 Gaussian distributions centered at the 10x10 grids and small enough variance. I want to see how this method performs in recovering centers of the clusters (you can see examples in the literature). \n\n* Given that finally, the sampling strategy becomes very similar to the recent energy-based method. It is fair to say comparison is required. Yes, GAN is and the energy-based methods are different but the sampling strategy (the Langevian dynamic) are very similar and I would like to see how those two methods perform in term of quality of the samples. \n\n\n* Finally, if one views this method as a general strategy for generating samples using GAN, perhaps you can think of this approach as an approximate inference approach for inference in a complicated Baysian model. This paper does not focus on that and I don't expect authors to do any experiment for that but it can useful there.\n \n\n\n[1] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The Unreasonable\nEffectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The consensus among the reviewers is that this is a borderline paper: its main idea is sensible and natural. Unfortunately, while the reviewers appreciated the authors' responses to their comments, they felt that the paper failed to demonstrate the usefulness of the idea beyond toy-datasets. The latter would considerably strengthen this paper.
Decision: Reject