Intervention Generative Adversarial Nets
Authors: Jiadong Liang
Liangyu Zhang
Cheng Zhang
Zhihua Zhang
Venue:  (2021)

Abstract: In this paper we propose a novel approach for stabilizing the training process of Generative Adversarial Networks as well as alleviating the mode collapse problem. The main idea is to incorporate a regularization term that we call intervention into the objective. We refer to the resulting generative model as Intervention Generative Adversarial Networks (IVGAN). By perturbing the latent representations of real images obtained from an auxiliary encoder network with Gaussian invariant interventions and penalizing the dissimilarity of the distributions of the resulting generated images, the intervention term provides more informative gradient for the generator, significantly improving training stability and encouraging modecovering behaviour. We demonstrate the performance of our approach via solid theoretical analysis and thorough evaluation on standard real-world datasets as well as the stacked MNIST dataset.


Under review as a conference paper at ICLR 2021
INTERVENTION GENERATIVE ADVERSARIAL NETS
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper we propose a novel approach for stabilizing the training process of Generative Adversarial Networks as well as alleviating the mode collapse problem. The main idea is to incorporate regularization term that we call intervention into the objective. We refer to the resulting generative model as Intervention Genera- tive Adversarial Nerworks (IVGAN). By perturbing the latent representations of real images obtained from an auxiliary encoder network with Gaussian invariant interventions and penalizing the dissimilarity of the distributions of the resulting generated images, the intervention term provides more informative gradient for the generator, significantly improving training stability and encouraging mode- covering behaviour. We demonstrate the performance of our approach via solid theoretical analysis and thorough evaluation on standard real-world datasets as well the stacked MNIST dataset.
INTRODUCTION
As one of the most important advances in generative models in recent years, Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) have been attracting great attention in the machine learning community. GANs aim to train a generator network that transforms simple vectors of noise to produce "realistic" samples from the data distribution. In the basic training process of GANs, discriminator and a target generator are trained in an adversarial manner. The discriminator tries to distinguish the generated fake samples from the real ones, and the generator tries fool the discriminator into believing the generated samples to be real.
Though successful, there are two major challenges in training GANs: the instability of the training process and the mode collapse problem. To deal with these problems, one class of approaches focus on designing more informative objective functions (Salimans et al. 2016 Mao al.) 2016 Kodali et al. 2018 Arjovsky& & Bottou, Arjovsky et al.) 2017] Gulrajjni et al. 2ho7 et al.) 2019). For example, Mao al. (2016) proposed Least Squares GAN (LSGAN) that uses the least squares Ioss to penalize the outlier point more harshly. [Arjovsky & Bottou discussed the role of the Jensen-Shannon divergence in training GANs and proposed WGAN (Arjovsky et al.) 2017) and WGAN-GP (Gulrajani et al. 2017) that use the more informative Wasserstein distance instead. Other approaches enforce proper constraints on latent space representations to better capture the data distribution (Makhzani al. 2015 Larsen et al., 2015 Che et al.) 2016 Tran al. 2018). A representative work is the Adversarial Autoencoders (AAE) (Makhzani et al. 2013) which uses the discriminator to distinguish the latent representations generated by encoder from Gaussian noise. Larsen al. (2015) employed image representation in the discriminator as the reconstruction basis of a VAE. Their method turns pixel-wise loss feature-wise, which can capture the real distribution more simply when some form of invariance is induced. Different from VAE-GAN, Che et al. (2016) regarded the encoder as an auxiliary network, which can promote GANs to pay much attention on missing mode and derive an objective function similar to VAE-GAN, A more detailed discussion of related works can be found in Appendix C
In this paper we propose novel technique for GANs that improve both the training stability and the quality of generated images. The core of our approach is a regularization term based on the latent representations of real images provided by encoder network. More specifically, auxiliary intervention operations that preserve the standard Gaussian (e.g., the noise distribution) to these latent representations. The perturbed latent representations are then fed into the generator to produce intervened samples. We then introduce a classifier network to identify the right intervention operations that would have led to these intervened samples. The resulting negative cross-entropy loss
apply

---

Under review as a conference paper at ICLR 2021
is added as regularizer to the objective when training the generator. We call this regularization term the intervention loss and our approach Inter Vention Generative Adversarial Nets (IVGAN).
We show that the intervention loss is equivalent with the JS-divergence among multiple intervened distributions. Furthermore, these intervened distributions interpolate between the original generative distribution of GAN and the data distribution, providing useful information for the generator that previously unavailable in common GAN models (see a thorough analysis on a toy example in Example We show empirically that our model can be trained efficiently by utilizing the parameter sharing strategy between the discriminator and the classifier. The models trained on the MNIST, CIFAR- 10, LSUN and STL-10 datasets successfully generate diverse, visually appealing objects, outperforming state-of-ther thr- baseline methods such as WGAN- and MRGAN in terms of the Fréchet Inception Distance (FID) (proposed (Heusel et al., 2017 ). We also perform series of experiments on the stacked MNIST dataset and the results show that our proposed method can also effectively alleviate the mode collapse problem. Moreover, an ablation study is conducted, which validates the effectiveness of the proposed intervention loss.
In summary, our work offers three major contributions as follows. (i) We propose a novel method that can improve GAN's training as well as generating performance. (ii) We theoretically analyze our proposed model and give insights on how it makes the gradient of generator more informative and thus stabilizes GAN's 's training. (iii) We evaluate the performance of our method on both standard real-world datasets and the stacked MNIST dataset by carefully designed expriments, showing that approach is able to stabilize GAN's training and improve the quality and diversity of generated samples as well.
PRELIMINARIES
Generative adversarial nets The basic idea of GANs is to utilize a discriminator to continuously push a generator to map Gaussian noise to samples drawn according an implicit data distribution. The objective function of the vanilla GAN takes the following form:
min max { (VD,G) Eprpeara log(D(x)) + Ez log(1 - D(G())))
(1)
where Ps prior distribution (e.g., the standard Gaussian). can be easily seen that when the discriminator reaches its optimum, that is, (x)) = Pduta (#) (Z)) the objective is equivalent to the
Pdtta(w)
Jensen-Shannon (JS) divergence between the generated distribution pa and data distribution Pdaata:
JS(pG (Pdota) e 2 KL(PGl PG Pdata
PG Pdata
KL
2
Minimizing this JS divergence guarantees that the generated distribution converges to the data distribution given adequate model capacity.
Multi-distribution JS divergence
The JS divergence between two distributions pi and p2 can be
rewritten as
JS(pillp2) = H1 P1 + 2 - H1
1 (1(p))
where (p) denotes the entropy of distribution p. We observe that the JS- -S-divergence can be inter- preted as the entropy of the mean the two distribution minus the mean of two distributions' entropy. So it is immediate to generalize the JS- JS-divergence to the setting of multiple distributions. In particular, we define the JS-divergence of p1. P1.P2..... with respect to weights Ti, T2..... TT ②Σ T = and
T, 0)
JS SS (pi, P2..... Pm) A H
T.P:)
क.
T. (p:)..
The two-distribution case described above is actually special case of the 'multi-JS divergence', where Ji = T2 = }. When n. > 0 Vi, can be found immediately by Jensen's inequality that JS= T (p1. Po..P.....PP. 0 if and only if pı P2
Pu-
2

---

Under review as a conference paper at ICLR 2021
Autoencoder
X- data
E
E(x)
G(E(x)) x--Paato
0,(E(x))
G(O,(E(x)))-o
x- duta
D
JSD
Intervention
O2(EGx))
G(0,(E())))PG
Multi
G
z-,1(011)
G(x)~Po
(x()())
G(O,(E(x)))
2-1(
G(x)-Po
Vanilla GAN
Intervention GAN
Figure 1: Comparison between the vanilla GAN loss and the Intervention loss. Here the intervened
samples generated through different intervention operations, namely O1,-.. OK-
3 METHODOLOGY
Training GAN has been challenging, especially when the generated distribution and the data distribu- tion far away from each other. In such cases, the discriminator often struggles provide useful information for the generator, leading to instability and mode collapse problems. The key idea behind our approach that we construct auxiliary intermediate distributions that interpolate between the generated distribution and the data distribution. To do that, we first introduce an encoder network and combine with the generator learn the latent representation of real images within the framework of a standard autoencoder. We then perturb these latent representations with carefully designed intervention operations before feeding them into the generator to create these auxiliary interpolating distributions. A classifier is used distinguish the intervened samples, which leads an intervention loss that penalizes dissimilarity of these intervened distributions. The reconstruction loss and the intervention loss are added regularization terms the standard GAN loss for training. We start with an introduction of some notation and definitions.
Definition (Intervention), Let O be transformation on the space of d-dimension random vectors and IP be probability distribution whose support is in IRd, We call O a P-intervention for any d-dimensional random vector X. X P -(X)~~ IP
Since the noise distribution in GAN models is usually taken to be standard Gaussian, we use the standard Gaussian distribution the default choice of P and abbreviate the IP - as intervention, unless otherwise claimed. One of the simplest groups of interventions is block substitution. Let E I be a random vector, N and k:dd We slice Z into i: blocks so that every block is in R A block substitution intervention O is to replace the ith block of Z with Gaussian noise, unless otherwise specified. Note that our theoretical analysis as well as the algorithmic framework do depend on the specific choice the intervention group.
1, ....., We will use block substitution interventions in the of the paper
Notation We use E, G. f f to represent encoder, generator, discriminator and classifier, respec- tively. Here and later, Preat means the distribution of the real data X, and p. is the prior distribution of noise defined on the latent space (usually is taken to be Gaussian). Let O., 1, 1.... k denote k different interventions and p; be the distribution of intervened sample X, created from O (namely X, ((()))()))).
Intervention loss latent representation that is generated by encoder network E, we sample intervention O from a complete group S {O,.... O } and obtain the corresponding intervened latent variable (z(z) with label e,. These perturbed latent representations then fed into the generator produce
The intervention loss is the core of our approach. More specifically, given a
3

---

Under review as a conference paper at ICLR 2021
intervened samples. We then introduce auxiliary classifier network to identify which intervention operations may lead to these intervened samples. The intervention loss Liv (G, E) is simply the resulting negative cross-entropy loss and add that regularizer to the objective function when training the generator. As we can see, the intervention loss is used to penalize the dissimilarity of the distributions of the images generated by different intervention operations. Moreover, it can be noticed that the classifier and the combination of the generator and the encoder are playing two-player adversarial game and will train them in an adversarial manner. In particular, we define
LIv B) = min Velases
where
Valass EA-)(())) Ex'op c log f(x').
(3)
Theorem (Optimal Classifier). The optimal solution of the classifier the conditional probability of label given X', where X' is the intervened sample generated by the intervention operation sampled from S. And the minimum of the cross entropy loss is equivalent with the negative of the Jensen Shannon divergence among (PI.P2...., That is,
Pi(x) and Liv B) (1, + Const. (x) 2)-1P((4)
(4)
The proof can be found in Appendix A.1 Clearly, the intervention loss is an approximation of the multi-JS divergence among the intervened distributions {ps : [k]]. the intervention reaches its global minimum, we have P1 = 12 and only if the supports of these & distributions do not intersect with each other. This way, the probability that the 'multi" JS-divergence has constant value is much smaller, which means the phenomenon of gradient vanishing should be rare in IVGAN. Moreover, as shown in the following example, due to these auxiliary intervened distributions, the intervention is able pro- vide more informative gradient for the generator that is not previously available in other GAN variants.
Ps. And it reaches the maximum log k
Example 1 (Square fitting). Let Xy be a random vector with distribution U(x), where a x And X1 U(3,, where 3 = [a - 2 O S a S 1. Assuming we have a perfect discriminator (or classifier), we compute the vanilla GAN loss (i.e. the JS JS-divergence) and the intervention loss between these two distribations, respectively,
and
B
)5(X)l(X1) = log
1/2
In order to compute the intervention loss we need figure out two intervened sumples' distri- butions evolved from U(a)) and U(3). Y1
Figure 2: The supports of the two orig- inal distribution are the squares with black border, and the supports of the syn- thetic distributions are the area enclosed by red and blue dotted line, respectively.
U(m1): 71 = - 1. and Y2 2: Then the intervention loss the multi JS-divergence among these four distributions:
u(r2): 7/2 = 2. 2
1
2:
Liv JS(Xo; X1; Y2)
/
1
1
/ log 2
1 d(x)H(()) = log
log 4
-du-
2 [p(A9)-H(A))
A=
4
log 2
2 2(2 a) H(u)) -llo 2)a Const.
Here A is the shaded part in Figure 2 and = observation is that the intervention loss is a function of parameter a and the traditional GAN loss is always constant. When we replace the JS with other f-divergence, the metric between U(a)) and U(3 would still remain constant. Hence in this situation, we can not get any information from the standard JS for training of the generator but the intervention loss works well.
= (QUBUM
The most important
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Intervention GAN
Input learning rate a, regularization parameters À and μ. dimension of latent space, number is of blocks in which the hidden space is divided, minibatch size n, Hadamard multiplier
1: for number of training iterations do
2: Sample minibatch 2 = 1, ..., n, 2j ~ Ps
3: Sample minibatch 2z ..... n. 2 Preal
4: for number inner itcration do
5: W < -x(x))) 1,....?
6:
Sample Gaussian noise e
7:
Sample ij [k]. 1,.,7
8:
x', + G (w,))
9:
Update the parameters of D by: a Ved Lads (0p)
10:
fD t 0D
2n
Update the parameters of by:
12:
0 + ,=1
E log fú, (a'))
13:
Calculate LAdo and LIv
14:
Update the parameter of G by:
0g + 0g + Voc [Add + &&recon + uLiv) (v)
15:
16:
Update the parameter of E by:
OF OF + Vo0 [rrecon + =urr)
17:
Reconstruction loss In some sense we expect our encoder to be a reverse function of the generator. So is necessary for the objective function have to push the map composed of the Encoder and the Generator to have the ability to reconstruct the real samples. Not only that, we also hope that the representation can be reconstructed from samples in the pixel space.
Formally, the reconstruction loss can be defined by the fy-norm (p 2 1) between the two samples, or in the from of the Wasserstein distance between samples if images regarded histogram. Here we choose to use the é -nomr the reconstruction loss:
Lrecon Ex-preal [|(((()))XXX)) -C -|II E Baa [BG(() OF (5)
Theorem (Inverse Distribution). Suppose the cumulative distribution function of O2(2) is q. For any given positive real number e, there exist 5 >> O such that if Crecon LIV S 5, then Vi. E [k]. sup (r)
(r) Sc.
The proof is in A.2
Adversarial loss The intervention loss and reconstruction loss be added regularization terms to the adversarial loss in many GAN models, e.g., the binary cross entropy loss in vanilla GAN and the least square loss in LSGAN. In the experiments, we use LSGAN (Mao et 2016) and DCGAN (Radford et al. 2015) as our base models, and name the resulting IVGAN models IVLSGAN and IVDCGAN respectively.
Now that we have introduced the essential components in the objective of IVGAN, we can write the loss function of the entire model:
Lmodel = LAds + XCreoon + uLvv,
(6)
where À and μ the regularization coefficients for the reconstruction loss and the intervention loss respectively. We summarize the training procedure in Algorithm [ A diagram of the full workflow of our framework can be found in Figire
5

---

Under review as a conference paper at ICLR 2021
Reconstractio Loss
Intervention
X data
Intervention
Classifier
Loss
2 P=
Parameter Skaring
Adversarial
Loss
Figure Full workflow of our approach.
EXPERIMENTS
In this section we conduct a series of experiments to study IVGAN from multiple aspects. First we evaluate IVGAN's performance on standard real-world datasets. Then we show IVGAN's ability to tackle the mode collapse problem on the stacked MNIST dataset. Finally, through an ablation study we investigate the performance of our proposed method under different settings of hyperparameters and demonstrate the effectiveness of the intervention loss.
We implement our models using Py (Paszke et al. 2019) with the Adam optimizer (Kingma & Ba 2015). Network architectures are fairly chosen among the baseline methods and IVGAN (see Table B.I in the appendix for more details). The classifier
to compute the intervention loss shares the parameters with the discriminator except for the output layer. All input images are resized to have 64 x 64 pixels. We 100-dimensional standard Gaussian distribution the prior p.. We deploy the instance noise technique as (Jenni Favaro 2019). One may check Appendix [8.2 for detailed hyperparameter settings. All experiments are run one single NVIDIA RTX 2080Ti GPU. Although IVGAN introduces extra computational complexities to the original framework of GANs, the training cost of IVGAN within an acceptable range due the application of strategies like parameter sharing.
6
9
9
8
8
2
3
1
Figure 4: Random samples of generated images on MNIST, CIFAR-10. 10, LSUN and STL-10.
Real-world datasets experiments We first test IVGAN four standard real-world datasets, including CIFAR-10 (Krizhevsky 2009), MNIST (Lecun et al. 1998). STL-10 (Coates et al. 2011). and subclass named "church outdoor" of the LSUN dataset (Yu et al. 2015), to investigate its training stability and quality of the generated images. We use the Frécher Inception Distance (FID)
(Heusel al. 2017) measure the performance of all methods.
The FID results are listed in Table [ and the training curves of the baseline methods and IVGAN on four different datasets are shown in Figure 5 We see that on each datasets, the IVGAN counter- parts obtain better FID scores than their corresponding baselines. Moreover, the figure of training curves also suggests the learning processes of IVDCGAN and IVLSGAN smoother and steadier compared to DCGAN, LSGAN or MRGAN (Che et al. 2016). and converge faster than WGAN-GP. Samples of generated images on all datasets are presented in Figure
Stacked MNIST experiments The original MNIST dataset contains 70K images of 28 x 28
handwritten digits. Following the same approaches in Metz et al. (2017); Srivastaaa et al. (0117) : Lin
et al. 2018), we increase the number of modes of the dataset from 10 to 1000 10 x 10 x 10 by
Empirically IVGANs are approximately times slower than their corresponding haseline methods.

---

Under review as a conference paper at ICLR 2021
Table I: Minimum of FIDs on different Datasets. The FID results are calculated every 10 epochs, and are averaged over five independent runs. Lower is better.
Methods
MNIST
CIFARIO
LSUN bechrcch_outtore
STL-10
DCGAN
9.87 + 1.18
37.30 + 2.93
23.18 + 0.98
45.13 + 0.95
LSGAN
12.06 + 0.88
34.43 2.65
29.53 + 1.10
57.39 + 1.61
WGAN-GP
10.83 0.66
40.03 1.62
26.08 + 0.60
48.52 = 0.61
MRGAN
8.52 + 1.19
32.52 + 2.89
21.03 + 0.90
44.09 + 1.56
IVDCGAN
8.27 + 1.25
30.19 + 1.40
20.30 + 0.80
42.86 + 1.37
IVLSGAN
8.33 + 0.25
27.58 0.79
18.99 + 0.88
41.85 + 1.17
CIFAR1O
LSUN(Church, ISUN(Churchouutdoor
200
MRGAN
MRGAN
180
WGAN-GP
175
WGAN-GP
160
150
FID Score
120
LSGAN
MDCGAN
FID Score
125
MVCGGNN
LSGAN
NLSGAN
MLSGAN
75
50
20
100
150
200
250
350
400
100
150
200
250
Epochs
Epochs
Figure 5: Training curves of different methods in terms of FID on different datasets, averaged over five runs. Left: CIFARIO. Right: Church Outdoors. Note that the raise of curves in the later stage may indicate mode collapse.
stacking three random MNIST images into 28 x 28 x RGB image. The metric we use to evaluate model's robustness to mode collapse problem is the number modes captured by the model, as well the KL divergence between the generated distribution over modes and the true uniform distribution. The mode generated imaged is found from pre-trained MNIST digit classifier.
Our results shown in Table It can be that model works very well to prevent the mode collapse problem. Both IVLSGAN and IVDCGAN are able to reach 1,000 modes and greatly outperforms early approaches mitigate mode collapse, such VEEGAN (Srivastava et al. 2017 and Unrolled GAN (Metz et al.) 2017). Moreover, the performance of our model is also comparable method that is proposed images generated randomly by our model as well as the baseline methods.
recently, such as the PacDCGAN (Lin al. 2018). Figure shows
Table 2: Results of our stacked MNIST experiments. The first four rows are directly copied from
(Lin et al. 2018) and (Srivastava et al. 2017). And the last three rows are obtained after training each model for 100K iterations, respectively.
Modes
KL Divergence
DCGAN
78.9
4.50
VEEGAN
150.0
2.95
Unrolled GAN
48.7
4.32
PacDCGAN
1000
0.06
LSGAN
53
3.88
IVLSGAN
1000
0.07
IVDCGAN
1000
0.08
Ablation study Our ablation study is conducted on the CIFAR- 10 dataset. First, we show the effectiveness of the intervention loss. We consider two cases, IVLSGAN without the intervention loss (μ- = 0), and standard IVLSGAN (μ = 0.5). From Figure we can find that the intervention loss makes the training process much smoother and leads to lower FID score in the end.
7

---

Under review as a conference paper at ICLR 2021
6
7
6
Figure 6: Sampled images on the stacked MNIST dataset. Left: Ground-truth. Middle: LSGAN. Right: IVLSGAN. Images generated by our method are more diverse.
We also investigate the performance our model using different number of blocks for the block substitution interventions and different regularization coefficients for the intervention loss. The results are presented in T It can be noticed that to some extent our models' performance is not sensitive to the choice of hyperparameters and performs well under several different hyperparameter settings. However, when the number of blocks or the scale of IV loss becomes too large the performance of model gets worse.
CIFAR10
Table Minimum FID scores of IVLSGAN under different hyperparameter settings on the CIFARIO dataset, calculated every 10 epochs.
without intervention loss with intervention loss
180
FID Score
140
FID score
80
0.5
k 2
29.2
60
= 0.5
28.2
40
0.5
= 10
41.2
= 0.5
= 20
36.1
35
25
Epochs
O
= 34.5
0.25
29.6
0.5 J
k =
28.2
Figure 7: Training curve IVLSGAN, with and without the intervention loss.
39.7
CONCLUSION
We have presented a novel model, intervention GAN (IVGAN), to stabilize the training process of GAN and alleviate the mode collapse problem. By introducing auxiliary Gaussian invariant interventions to the latent space of real images and feeding these perturbed latent representations into the generator, have created intermediate distributions that interpolate between the generated distribution of GAN and the data distribution. The intervention loss based on these auxiliary intervened distributions, together with the reconstruction loss, are added as regularizers to the objective to provide more informative gradients for the generator, significantly improving GAN's training stability and alleviating the mode collapse problem as well.
We have conducted a detailed theoretical analysis of our proposed approach, and illustrated the advantage of the proposed intervention loss on a toy example. Experiments on both real-world datasets and the stacked MNIST dataset demonstrate that, compared to the baseline methods, IVGAN variants are stabler and smoother during training, and are able to generate images of higher quality (achieving ofate-of-the-aa art FID scores) and diversity. We believe that proposed approach can also be applied to other generative models such as Adversarial Autoencoders (Makhzani et al. 2013). which we leave future work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
M Arjovsky and L Bottou. Towards principled methods for training generative adversarial networks. arxiv 2017. arXiv preprint S.Kiv:1701.44550
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214 224,222,422 2017.
Tong Che, Yanran Li, Athul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. 2016.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223, 2011.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2672 2680, 2014. URL http: //papers n wwt.:ippppcpppppccooooo.co http:////..33eeeeraaeeeeeeeeeeee ll
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626 6637, 2017.
Simon Jenni and Paolo Favaro. On stabilizing generative adversarial training with noise, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: 0000000982... 2018.
D. Kingma and J. Ba. Adam: method for stochastic optimization. ICLR, 2015.
Naveen Kodali, James Hays, Jacob Abernethy, and Zsolt Kira. On convergence and stability of gans. 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. CoRR, 1 2015. URt.http://axxiv.ooo
abs/1512, 09300
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6:111)22782233 -2244, 1998.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems 31, pp. 1598-1507. Curran Associates, Inc., 2018. URL http: httt::://ppprssssssts//sa nips wwtpppppp..cpprrr///.ppppppppp.e 7423 pacgan the power- ooftwor samples in www.ggeeeiiieeee adversarial Seciiial--ettttee networks.
pdf
Alireza Makhzani, Jonathon Shlens, NavdeepJaing and Ian J. Goodfellow, Adversarial autoencoders. CoRR, abs/1511.05644 2015.
05644
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, and Zhen Wang. Multi-class generative adversarial networks with the L2 loss function. CoRR, 7/ //aaxiv.. wwwpwxwwwwwwww.....p.oo 04076
2016. URL http:

---

Under review as a conference paper at ICLR 2021
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL [RR[:tttpa://ooeeeeeeee net/forum? i dddmmmdddddddddll......
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8026-8037. Curran Associates, Inc., 2019. http:tttpppppppr...tttt nips wwt.appperr.iisscccceppppeeeeeee
9015 pdf
library.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint urXiv:1511.06434 2015.
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3308 -3088-331,101 2017.
Mgoc-Trung Tran, Tuan-Anh Bui, and Ngai-Man Cheung. Dist-gan: An improved gan using distance constraints. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 370-385, 2018.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint (00.::::506033555 2015.
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. arXiv preprint 17X1v:1902.055687,1 2019.
A PROOFS
A.I PROOF OF THEOREM 1
Proof. The conditional probability of X' given label can be written as P(X)(e.) >>(XX"), further
P(X', e4) tp.. And we denote the marginal distribution of as p(x) = - E=1
(z)). Cause the
activation function at the output layer of the classifier is softmax, we can rewrite the loss function into a more explicit form:
Valass (f) ( c. log f(x') EisecyyEss log f((x)
1 b b:
p(x) { {- ---ple p(e)(x) log h())))))))))))()))) dr.
Σ fo
{-Σ
10

---

Under review as a conference paper at ICLR 2021
& ((e)l))))))) = 1. And notice that Σ p(e,kz) = 1. By Jensen's
Let (x)
fa(z) plaaa then Σ
inequality, we have:
(og)z)))ogg Σ
logg: + H(v(-1=)) 2 log Σ p(eclx))::1) + H(x(+)))
==1
log + H(p(-Iz)) ( (p((==))
And Vdlss (f)) p(x) H (+))())))) if and only if g. (a) 97(x) (a)) for any + j. which means
k
that (x)
[()] =r Vi E [k.] where r E R. Notice that Σ li (x) = 1, it is not difficult to get that
(x) = p(). The loss function becomes
Σ (+(x)) logp(e///)d=== = - +
H(p)) + log
(S(p,, log
A.2 PROOF OF THEOREM 2
Proof. According Theorem for given real number é1, we can find another 81, when intervention loss is less than 51, the distance between p: and Pj under the measurement of JS di-ergeeeenc is less than é1. And because JS -S-diverge and Total Variance distance (TV) are equivalent in the sense of convergence. So we can bound the TV-distance between y); and pj by their JS JS-divergence. Which means that f lp: py I 6 when the intervention loss is less than 61 (we can according to the cp to finding the appropriate 61). Using this conclusion we can deduce P(E(((((())) < r) - (())((()())) S r)] S co. where r is an arbitrary vector in Rd. Further, we have:
< <)) <((((() S (r:])O(>)) > 6)1 + (P(O.(z) (|; E(G(O, < 8) - -(m:)00(())-(((()(()))) s <6)
E(G(O,(=))) > 6)
+ (000)))({r(((())))BEG000))) -
(8)
We control the three terms the right side of the inequality sign respectively.
P(O.(z) I (1),)))>) E(G(O,(=))) > 6)
8) [EJO,(2) E(G(O,(=)))
(9)
S -P(IO,()-
And the last term can be bounded by the reconstruction loss. The same trick can be used on ((G(O)
P(O,(2) s r: = 8). Moreover, we have
P(B(G(O.(E)) < (r)())))))((( ((G(O,())))] 8) <P(O,()) < (r: E(G S 6) s P(E(G(O,(2)) 5
(10)
Notice that lim P(E(G(O((=)) < r 6) P(E(G(O,(=)) - r). Let (r,
=
6-0
(P(F(G(O((E)) $ r- 5)) P(F(G(( < r) then the last term of inequality A.2 can
be bounded as:
IP(O,(s) < r: ((G())))GG((()))
S ((EEG(()(()) - ())) (r, 0) + (r,
- 8)
(11) Every term on the right hand of the inequality can be controlled close by the inequalities mentioned above.
11

---

Under review as a conference paper at ICLR 2021
B EXPERIMENTAL DETAILS
B.1 NETWORK ARCHITECTURES
Table 4:
The NN architecture used by us, where CONV denotes the convolutional la yer, TCONV denotes the transposed convolutional layer, FC denotes the fully-connected layer, BN denotes the batch normalization layer, and (K4, S1, 0512) denotes a layer with kernel of size 4, stride 1, and 512 output channels.
D
G
E
INPUT 64x64x3
INPUT TCONV(K4, SI, 0512)
INPUT 64x64x3
CONV(K4, S2, 064)
CONV(K4, S2, 064)
BN, LeakyReLU
BN, ReLU TCONV(K4, S2, O256)
LeakyReLU
CONV(K4, S2, O128)
CONV(K4, S2, O128)
BN, LeakyReLU CONV(K4, S2, O256) BN, LeakyReLJU
BN, ReLU TCONV(K4, S2, O128)
BN, LeakyReLU
CONV(K4, S2, O256) BN, LeakyRel.U
BN. ReLU TCONV(K4, S2, 064)
CONV(K4, S2, O512)
CONV(K4, S2, O512)
BN, LeakyReLU FC(O))
BN, ReLU TCONV(K4, S2, 03)
BN, Leaky
CONV(K4, S2, 0100)
LOSS
Tanh
BN
B.2 HYPERPARAMETER SETTINGS
Table 5: Detailed hyperparameter settings our experiments.
Settings of the Adam optimizer
Other Hyperparameters
Irp
Irg
brg
BatchSize
81
82
DCGAN
0.0001
0.0001
64
0.5
0.999
LSGAN
0.0002
0.0002
64
0.5
0.999
WGANGP
0.0001
0.0001
64
0.5
0.9
Nidi = 5
AGP = 10
MRGAN
0.0002
0.0002
0.01
64
0.5
0.999
A 0.25
A 0.25
IVDCGAN
0.0002
0.0002
0.005
64
0.5
0.999
1 0.5
== =0.25
h: = 4
IVLSGAN
0.0002
0.0002
0.005
64
0.5
0.999
1 0.5
= -0.5
k 4
C RELATED WORK
In order to address GAN's unstable training and mode missing problems, many researchers have turned their attention the latent representations of samples. Makhzani et al. (2015) proposed the Adversarial Autoencoder (AAE). As its name suggests, AAE is essentially a probabilistic autoencoder based the framework of GANs. Unlike classical GAN models, in the setting of AAE the discriminator's 's task is to distinguish the latent representations of real images that are generated by an encoder network from Gaussian noise. And the generator and the encoder are trained to fool the discriminator well as reconstruct the input image from the encoded representations. However, the generator can only be trained by fitting the reverse of the encoder and cannot get any information from the latent representation.
The VAE-GAN (Larsen et al. 2015) combines the objective function from a VAE model with a GAN and utilizes the learned features in the discriminator for better image similarity metrics, which is of great help for the sample visual fidelity. Considering the opposite perspective, Che et al. (016 claim that the whole learning process of generative model can be divided into the manifold learning phase and the diffusion learning phase. And the former one considered to be the source of the mode missing problem. (Che et al. 2016) then proposed Mode Regularized Generative Adversarial
12

---

Under review as a conference paper at ICLR 2021
Nets which introduce reconstruction loss term to the training target of GAN penalize the missing modes. is shown that it actually ameliorates GAN's 'mode missing' -prone weakness to some extent. However, both of them fail to fully excavate the impact of the interaction between VAEs and GANs.
Kim & Mnih 2018) proposed Factor VAE where a regularization term called total correlation penalty is added to the traditional VAE loss. The total correlation is essentially the Kullback-Leibler
divergence between the joint distribution p(²), 22.... 2d) and the product of marginal distribution p(z;). Because the closed forms of these distribution unavailable, Factor VAE uses adversarial training to approximate the likelihood ratio.
13


Reviews:
[{"review_id": "m3Z4GQmwEDz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces \"Intervention GAN (IVGAN)\", an algorithm proposed to stabilize the training process os GANs as well as alleviating the mode collapse problem. In IVGAN the authors feed O(x) to the generator with O different possible transformations and train the discriminator to also differentiate which transformation originated the sample.\n\nThe approach is extremely poorly motivated. Why transformations like this have anything to do with mode colapse or training stability is not at all at the level of this conference. This paper is literally written as \"We came up with this method and we tried out whether it works or not, the conclusion is it doesn't really hurt in our limited experiments, sometimes it helps by an epsilon\". This is really not up to the standards of ICLR, I would expect at least a tiny step towards solving a fundamental problem with a clear motivation and research path, or at the very least better experimental results.\n\nThere is no quantification of training stability, and the only qualitative experiment in that regard is figure 5, which doesn't really provide a strong argument for the model in my opinion. If they showed they could train faster difficult models (such as bigger ones like SGAN) or they could train models for new tasks where previous methods fail that would be better, but this is really quite underwhelming.\n\nThe only quantification for mode dropping is on mnist, a dataset that's widely accepted as useless in the current state of generative modelling. Furthermore, important baselines like wgangp are missing.\n\nFurthermore, the language and the overclaiming of the paper is very strong: \"our approach is able to stabilize GAN's training and improve the quality and diversity of generated samples as well\". Do the authors really think that their approach was finaly able to stabilize GAN training? The way these sentences are written make it seem like IVGAN is as easy to train in imagenet as training a resnet with cross-entropy for supervised learning. It is simply unacceptable given the limited experiments (tiny datasets, small improvements, far from the current state of generative modelling).\n\nHonestly, I feel very dissapointed about how this paper presents itself. Poorly motivated ideas with an epsilon improvement are not meant for top tier conferences, and the degree to what this paper overclaims is not acceptable either.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_OwDUeqH4e", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a regularization term in GANs to both improve training stability and remedy mode collapse.  The approach, as I understand it, is the follows:\n* Real images are encoded into a latent representation (encoder learned jointly with separate loss)\n* From an available set of perturbations or interventions, this latent representation is transformed. This work does a block substitution of a particular slice of the latent representation vector.\n* The intervened latent representation is given to the generator which produces an intervened sample.\n* A classifier determines which intervention the intervened sample belongs to.\n\nThe authors describe how this intervention loss is equivalent to a multi-JSD. This was a rather intricate algorithm and I think it’s generally well-explained. This paper demonstrates clear empirical improvements of the method on many benchmarks of interest. I therefore recommend acceptance.\n\nQuestions to authors:\n* This work focuses on the block-intervention, however, does the nature of the intervention matter?\n* Does this intervention loss have applicability outside of GANs?\n* Are the p_real latent distributions approximately Gaussian? Does a mismatch with the latent distribution used to produce fake images impact performance? \n\nNotes:\n* Regarding the comment “Training GAN has been challenging, especially when the generated distribution and the data distribution are far away from each other. In such cases, the discriminator often struggles to provide useful information for the generator, leading to instability and mode collapse problems”, Fedus et al. (2017) showed non-overlapping data distributions are learnable by a non-saturating GAN. You may want to qualify this statement. In practice, we never use a *perfect* discriminator.\n* Example 1 gave nice intuition on this complicated method.\n* Nit: Remove subjective claims in your writing -- this will strengthen the presentation, e.g. “...via solid theoretical analysis...”.\n* Nit: Using the word “novel” is redundant since you are proposing it, e.g. “In this paper we propose a novel technique for GANs”\n* Nit: Recommend converting your Figures from PNG to PDF to improve the image quality, especially for online viewers.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P7HFrvRZOes", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Intervention Generative Adversarial Nets\n\nSummary\n\nThis paper proposes a GAN training procedure where images are encoded into latents, which are perturbed and passed through a generator/decoder, and the discriminator objective is augmented to encourage it to classify the perturbation. Results are presented for generative image modeling on several datasets and compared against DCGAN, LSGAN, and WGAN-GP.\n\nMy take\n\nThis paper is another one of a common variant of GAN papers which makes small changes to the objective and training setup and shows small gains relative to old baselines on small-scale tasks. This paper’s empirical results are unfortunately lacking and there are no substantial or new theoretical insights, making this paper a clear reject.\n\nDetailed review\n\nSignificance\n\nThis paper proposes yet another GAN training procedure, based on an encoder-decoder-discriminator setup (as has been done many times before, especially in VAE-GAN setups). The change to the vanilla training setup is effectively quite small, but introduces a 100% compute overhead relative to its baselines. My primary complaint with this paper is that for such a small change to be of interest to the community (especially given its tremendous compute cost and any added implementation complexity) there must be substantial evidence that doing so is beneficial, and the empirical results in this paper are quite weak.\n\nThis paper compares against old baselines and obtains numbers which do not justify the computational overhead even relative to these old baselines. This paper should include comparison against more recent work if it wishes to prove its relevance--a quick look at an online leaderboard shows CIFAR-10 (https://paperswithcode.com/sota/image-generation-on-cifar-10) has tons of recent work which the authors could compare to or build off of. WGAN-GP is far from state-of-the-art on any of the tested benchmarks, and that the authors have made any sort of SOTA claims is misleading, and not acceptable.\n\nClarity\n\nThe paper is reasonably well written and easy to follow.\n\nMethodological soundness\n\nThe paper’s methodology is reasonably sound, and not explicitly cause for concern.\n\nOriginality\n\nThis paper is not especially original. There are dozens, if not hundreds, of GAN training papers which introduce small changes to the (now 6 year old) vanilla GAN or VAE-GAN objective, and there are no ideas in this paper which I would consider espeically new or novel.\n\nMisc\n\n-The authors seem to conflate mode collapse (when large regions of Z map to small regions of G(Z)) with mode dropping (when modes in X are not represented in G(Z)). These are two separate phenomena and while they co-occur, they should not be considered the same.\n\n-There is no discussion of previous work on “intervention,” or any of the related concepts. Being a critical element of the proposed method, this should be discussed in more detail (rather than just focusing on related GAN work).\n\n-There is an obvious connection to InfoGAN, which should be cited.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5j3kymreqe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for stabilizing the training of GAN as well as overcoming the problem of mode collapse by optimizing several auxiliary models. The first step is to learn a latent space using an autoencoder. Then, this latent space is \"intervened\" by a predefined set of $K$ transformations to generate a set of distributions $p_k$. A classifier is then taught to distinguish between $p_k$. Eventually, the weights of the classifier are shared with those of the discriminator network to produce the desired stabilization/diversification effect. In other words, the authors propose to stabilize GANs by intervening with the discriminator. This is done by sharing its weights with a classifier that trains on a perturbed latent distribution that is somehow related to the original problem via the prior assumption imposed.\n\nThroughout the paper, the authors use a \\textbf{block substitution} transformation as the intervening transformation. In this case, the kth transformation replaces the kth block in the embedding $e(X)$ of a sample $X$ with Gaussian noise. The authors also present a theorem relating the value of the classification loss function at the optimal solution to the Jensen Shannon divergence between the intervened distributions.\nFinally, the authors apply their approach to generate images using several datasets (MNIST-10, CIFAR, LSUN, STL-10). Figure 5 demonstrates that the approach outperforms the previous methods.\n\nIssues:\nA. The use of block transformations is not well motivated. Since it seems like an arbitrary design choice, I would appreciate it if the authors could justify its use.\nB. The approach seems quite cumbersome and adds non-negligible overhead compared to the baseline. Furthermore, looking at Figure 7, the gain appears marginal.\nC. The authors introduce a high order extension of JS divergence, denoted Multi-distribution JS divergence, and show that standard JS divergence is a special case for $k=2$ and $\\pi_1=\\pi_2=0.5$. The authors then use this quantity in Theorem 1 to relate the intervention loss to JS divergence. However, it is hard for me to see this logical move as valid proof of anything. Although the high-order JS divergence is intuitive, it is still a subjective quantity that the authors define and then use to make a formal statement. I think it's acceptable to use the high-order JS divergence as intuition about the intervention loss, but problematic to use it as proof of anything.\nD. It is difficult for me to evaluate \"diversity\" based on a few samples, particularly in the \"stacked\" MNIST images my eyes cannot read.\n\nPersonal note:\nThe reviewer admits his unfamiliarity with recent research in this field. Therefore, he cannot assess the novel nature of the method nor its significance. I see this as borderline work. However, with no such score available, I gave it a slightly positive rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Nominally, the scores on this paper were pretty split. 
In reality, I concur with the 2 and the 3. 
The 6 acknowledges being unfamiliar w/ the GAN literature, and I think the 7 is being too permissive about the baselines. 

The empirical evaluation here is simply not up to par for a major machine learning conference. 
As reviewers have mentioned, the baselines are out of date, and even then the improvements are marginal. 
It's totally fine to have a marginal improvement if the proposed technique is very new and interesting and the baselines 
are taken seriously, but unfortunately I don't believe that's the case here.
Thus, I recommend rejection.
Decision: Reject