Year: 2020

Abstract:
In this paper we propose a novel approach for stabilizing the training process of Generative Adversarial Networks as well as alleviating the mode collapse problem. The main idea is to incorporate a regularization term that we call intervention into the objective. We refer to the resulting generative model as Intervention Generative Adversarial Networks (IVGAN). By perturbing the latent representations of real images obtained from an auxiliary encoder network with Gaussian invariant interventions and penalizing the dissimilarity of the distributions of the resulting generated images, the intervention term provides more informative gradient for the generator, significantly improving training stability and encouraging modecovering behaviour. We demonstrate the performance of our approach via solid theoretical analysis and thorough evaluation on standard real-world datasets as well as the stacked MNIST dataset.

heading: 1 INTRODUCTION
text: As one of the most important advances in generative models in recent years, Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have been attracting great attention in the machine learning community. GANs aim to train a generator network that transforms simple vectors of noise to produce “realistic” samples from the data distribution. In the basic training process of GANs, a discriminator and a target generator are trained in an adversarial manner. The discriminator tries to distinguish the generated fake samples from the real ones, and the generator tries to fool the discriminator into believing the generated samples to be real.
Though successful, there are two major challenges in training GANs: the instability of the training process and the mode collapse problem. To deal with these problems, one class of approaches focus on designing more informative objective functions (Salimans et al., 2016; Mao et al., 2016; Kodali et al., 2018; Arjovsky & Bottou; Arjovsky et al., 2017; Gulrajani et al., 2017; Zhou et al., 2019). For example, Mao et al. (2016) proposed Least Squares GAN (LSGAN) that uses the least squares loss to penalize the outlier point more harshly. Arjovsky & Bottou discussed the role of the Jensen-Shannon divergence in training GANs and proposed WGAN (Arjovsky et al., 2017) and WGAN-GP (Gulrajani et al., 2017) that use the more informative Wasserstein distance instead. Other approaches enforce proper constraints on latent space representations to better capture the data distribution (Makhzani et al., 2015; Larsen et al., 2015; Che et al., 2016; Tran et al., 2018). A representative work is the Adversarial Autoencoders (AAE) (Makhzani et al., 2015) which uses the discriminator to distinguish the latent representations generated by encoder from Gaussian noise. Larsen et al. (2015) employed image representation in the discriminator as the reconstruction basis of a VAE. Their method turns pixel-wise loss to feature-wise, which can capture the real distribution more simply when some form of invariance is induced. Different from VAE-GAN, Che et al. (2016) regarded the encoder as an auxiliary network, which can promote GANs to pay much attention on missing mode and derive an objective function similar to VAE-GAN. A more detailed discussion of related works can be found in Appendix C.
In this paper we propose a novel technique for GANs that improve both the training stability and the quality of generated images. The core of our approach is a regularization term based on the latent representations of real images provided by an encoder network. More specifically, we apply auxiliary intervention operations that preserve the standard Gaussian (e.g., the noise distribution) to these latent representations. The perturbed latent representations are then fed into the generator to produce intervened samples. We then introduce a classifier network to identify the right intervention operations that would have led to these intervened samples. The resulting negative cross-entropy loss
is added as a regularizer to the objective when training the generator. We call this regularization term the intervention loss and our approach InterVention Generative Adversarial Nets (IVGAN).
We show that the intervention loss is equivalent with the JS-divergence among multiple intervened distributions. Furthermore, these intervened distributions interpolate between the original generative distribution of GAN and the data distribution, providing useful information for the generator that is previously unavailable in common GAN models (see a thorough analysis on a toy example in Example 1). We show empirically that our model can be trained efficiently by utilizing the parameter sharing strategy between the discriminator and the classifier. The models trained on the MNIST, CIFAR-10, LSUN and STL-10 datasets successfully generate diverse, visually appealing objects, outperforming state-of-the-art baseline methods such as WGAN-GP and MRGAN in terms of the Frèchet Inception Distance (FID) (proposed in (Heusel et al., 2017)). We also perform a series of experiments on the stacked MNIST dataset and the results show that our proposed method can also effectively alleviate the mode collapse problem. Moreover, an ablation study is conducted, which validates the effectiveness of the proposed intervention loss.
In summary, our work offers three major contributions as follows. (i) We propose a novel method that can improve GAN’s training as well as generating performance. (ii) We theoretically analyze our proposed model and give insights on how it makes the gradient of generator more informative and thus stabilizes GAN’s training. (iii) We evaluate the performance of our method on both standard real-world datasets and the stacked MNIST dataset by carefully designed expriments, showing that our approach is able to stabilize GAN’s training and improve the quality and diversity of generated samples as well.

heading: 2 PRELIMINARIES
text: Generative adversarial nets The basic idea of GANs is to utilize a discriminator to continuously push a generator to map Gaussian noise to samples drawn according to an implicit data distribution. The objective function of the vanilla GAN takes the following form:
min G max D
{ V (D,G) , Ex∼pdata log(D(x)) + Ez∼pz log(1−D(G(z))) } , (1)
where pz is a prior distribution (e.g., the standard Gaussian). It can be easily seen that when the discriminator reaches its optimum, that is, D∗(x) = pdata(x)pdata(x)+pG(x) , the objective is equivalent to the Jensen-Shannon (JS) divergence between the generated distribution pG and data distribution pdata:
JS(pG‖pdata) , 1
2
{ KL(pG‖
pG + pdata 2 ) +KL(pdata‖ pG + pdata 2 )
} .
Minimizing this JS divergence guarantees that the generated distribution converges to the data distribution given adequate model capacity.
Multi-distribution JS divergence The JS divergence between two distributions p1 and p2 can be rewritten as
JS(p1‖p2) = H( p1 + p2 2 )− 1 2 H(p1)− 1 2 H(p2),
where H(p) denotes the entropy of distribution p. We observe that the JS-divergence can be interpreted as the entropy of the mean of the two distribution minus the mean of two distributions’ entropy. So it is immediate to generalize the JS-divergence to the setting of multiple distributions. In particular, we define the JS-divergence of p1, p2, . . . , pn with respect to weights π1, π2, . . . , πn ( ∑ πi = 1 and πi ≥ 0) as
JSπ1,...,πn(p1, p2, . . . , pn) , H( n∑ i=1 πipi)− n∑ i=1 πiH(pi). (2)
The two-distribution case described above is actually a special case of the ‘multi-JS divergence’, where π1 = π2 = 12 . When πi > 0 ∀i, it can be found immediately by Jensen’s inequality that JSπ1,...,πn(p1, p2, . . . , pn) = 0 if and only if p1 = p2 = · · · = pn.

heading: 3 METHODOLOGY
text: Training GAN has been challenging, especially when the generated distribution and the data distribution are far away from each other. In such cases, the discriminator often struggles to provide useful information for the generator, leading to instability and mode collapse problems. The key idea behind our approach is that we construct auxiliary intermediate distributions that interpolate between the generated distribution and the data distribution. To do that, we first introduce an encoder network and combine it with the generator to learn the latent representation of real images within the framework of a standard autoencoder. We then perturb these latent representations with carefully designed intervention operations before feeding them into the generator to create these auxiliary interpolating distributions. A classifier is used to distinguish the intervened samples, which leads to an intervention loss that penalizes the dissimilarity of these intervened distributions. The reconstruction loss and the intervention loss are added as regularization terms to the standard GAN loss for training. We start with an introduction of some notation and definitions. Definition 1 (Intervention). Let O be a transformation on the space of d-dimension random vectors and P be a probability distribution whose support is in Rd. We call O a P-intervention if for any d-dimensional random vector X , X ∼ P⇒ O(X) ∼ P.
Since the noise distribution in GAN models is usually taken to be standard Gaussian, we use the standard Gaussian distribution as the default choice of P and abbreviate the P-intervention as intervention, unless otherwise claimed. One of the simplest groups of interventions is block substitution. Let Z ∈ Rd be a random vector, k ∈ N and k|d. We slice Z into k blocks so that every block is in R d k . A block substitution intervention Oi is to replace the ith block of Z with Gaussian noise, i = 1, . . . , dk . We will use block substitution interventions in the rest of the paper unless otherwise specified. Note that our theoretical analysis as well as the algorithmic framework do not depend on the specific choice of the intervention group.
Notation We use E,G,D, f to represent encoder, generator, discriminator and classifier, respectively. Here and later, preal means the distribution of the real data X , and pz is the prior distribution of noise z defined on the latent space (usually is taken to be Gaussian). Let Oi, i = 1, . . . , k denote k different interventions and pi be the distribution of intervened sample Xi created from Oi (namely Xi = G(Oi(E(X)))).
Intervention loss The intervention loss is the core of our approach. More specifically, given a latent representation z that is generated by an encoder network E, we sample an intervention Oi from a complete group S = {O1, . . . , Ok} and obtain the corresponding intervened latent variable Oi(z) with label ei. These perturbed latent representations are then fed into the generator to produce
intervened samples. We then introduce an auxiliary classifier network to identify which intervention operations may lead to these intervened samples. The intervention loss LIV (G,E) is simply the resulting negative cross-entropy loss and we add that as a regularizer to the objective function when training the generator. As we can see, the intervention loss is used to penalize the dissimilarity of the distributions of the images generated by different intervention operations. Moreover, it can be noticed that the classifier and the combination of the generator and the encoder are playing a two-player adversarial game and we will train them in an adversarial manner. In particular, we define
LIV (G,E) = −min f Vclass, where Vclass = Ei∼U([k])Ex′∼pi − eTi log f(x′). (3)
Theorem 1 (Optimal Classifier). The optimal solution of the classifier is the conditional probability of label y given X ′, where X ′ is the intervened sample generated by the intervention operation sampled from S. And the minimum of the cross entropy loss is equivalent with the negative of the Jensen Shannon divergence among {p1, p2, ..., pk}. That is,
f∗i (x) = pi(x)∑k j=1 pj(x) and LIV (G,E) = JS(p1, p2, ..., pk) + Const. (4)
The proof can be found in Appendix A.1. Clearly, the intervention loss is an approximation of the multi-JS divergence among the intervened distributions {pi : i ∈ [k]}. If the intervention reaches its global minimum, we have p1 = p2 = · · · = pk. And it reaches the maximum log k if and only if the supports of these k distributions do not intersect with each other. This way, the probability that the ‘multi’ JS-divergence has constant value is much smaller, which means the phenomenon of gradient vanishing should be rare in IVGAN. Moreover, as shown in the following example, due to these auxiliary intervened distributions, the intervention is able to provide more informative gradient for the generator that is not previously available in other GAN variants.
Example 1 (Square fitting). Let X0 be a random vector with distribution U(α), where α = [− 12 , 1 2 ] × [− 1 2 , 1 2 ]. And X1 ∼ U(β), where β = [a− 12 , a+ 1 2 ]× [ 1 2 , 3 2 ] and 0 ≤ a ≤ 1. Assuming we have a perfect discriminator (or classifier), we compute the vanilla GAN loss (i.e. the JS-divergence) and the intervention loss between these two distributions, respectively,
• JS(X0‖X1) = log 2.
• In order to compute the intervention loss we need figure out two intervened samples’ distributions evolved from U(α) and U(β). Y1 ∼ U(γ1); γ1 = [− 12 , 1 2 ] × [ 1 2 , 3 2 ] and Y2 ∼
U(γ2); γ2 = [a − 12 , a + 1 2 ] × [− 1 2 , 1 2 ]. Then the intervention loss is the multi JS-divergence among these four distributions:
LIV = JS(X0;X1;Y1;Y2) = − ∫ Ac 1 4 log 1 4 dµ− ∫ A 1 2 log 1 2 dµ−H(X0) = log 2 2 [µ(Ac)+µ(A)]
= log 2
2 × 2(2− a)−H(X0) = −(log 2)a− Const.
Here A is the shaded part in Figure 2 and Ac = {α ∪ β ∪ γ1 ∪ γ2} \A. The most important observation is that the intervention loss is a function of parameter a and the traditional GAN loss is always constant. When we replace the JS with other f -divergence, the metric between U(α) and U(β) would still remain constant. Hence in this situation, we can not get any information from the standard JS for training of the generator but the intervention loss works well.
Algorithm 1 Intervention GAN Input learning rate α, regularization parameters λ and µ, dimension d of latent space, number k of blocks in which the hidden space is divided, minibatch size n, Hadamard multiplier ∗
1: for number of training iterations do 2: Sample minibatch zj , j = 1, ..., n, zj ∼ pz 3: Sample minibatch xj , j = 1, ..., n, xj ∼ preal 4: for number of inner iteration do 5: wj ← E(xj), j = 1, ..., n 6: Sample Gaussian noise 7: Sample ij ∈ [k], j = 1, ..., n 8: x′j ← G(Oij (wj)) 9: Update the parameters of D by:
10: θD ← θD − α2n∇θDLadv(θD) 11: Update the parameters of f by:
12: θf ← θf + αn∇θf n∑ j=1 log fij (x ′ j) 13: Calculate LAdv and LIV 14: Update the parameter of G by: 15: θG ← θG + αn∇θG { L̂Adv + λL̂recon + µL̂IV
} 16: Update the parameter of E by: 17: θE ← θE + αn∇θE { λL̂recon + µL̂IV }
Reconstruction loss In some sense we expect our encoder to be a reverse function of the generator. So it is necessary for the objective function to have a term to push the map composed of the Encoder and the Generator to have the ability to reconstruct the real samples. Not only that, we also hope that the representation can be reconstructed from samples in the pixel space.
Formally, the reconstruction loss can be defined by the `p-norm (p ≥ 1) between the two samples, or in the from of the Wasserstein distance between samples if images are regarded as a histogram. Here we choose to use the `1-norm as the reconstruction loss:
Lrecon = EX∼preal‖G(E(X))−X‖1 +Ei∼U([k])Ex,z∼preal,pz‖E(G(Oi(z)))−Oi(z)‖1. (5)
Theorem 2 (Inverse Distribution). Suppose the cumulative distribution function of Oi(z) is qi. For any given positive real number , there exist a δ > 0 such that if Lrecon +LIV ≤ δ , then ∀i, j ∈ [k], sup r ‖qi(r)− qj(r)‖ ≤ .
The proof is in A.2.
Adversarial loss The intervention loss and reconstruction loss can be added as regularization terms to the adversarial loss in many GAN models, e.g., the binary cross entropy loss in vanilla GAN and the least square loss in LSGAN. In the experiments, we use LSGAN (Mao et al., 2016) and DCGAN (Radford et al., 2015) as our base models, and name the resulting IVGAN models IVLSGAN and IVDCGAN respectively.
Now that we have introduced the essential components in the objective of IVGAN, we can write the loss function of the entire model:
Lmodel = LAdv + λLrecon + µLIV , (6)
where λ and µ are the regularization coefficients for the reconstruction loss and the intervention loss respectively. We summarize the training procedure in Algorithm 1. A diagram of the full workflow of our framework can be found in Figire 3.

heading: 4 EXPERIMENTS
text: In this section we conduct a series of experiments to study IVGAN from multiple aspects. First we evaluate IVGAN’s performance on standard real-world datasets. Then we show IVGAN’s ability to tackle the mode collapse problem on the stacked MNIST dataset. Finally, through an ablation study we investigate the performance of our proposed method under different settings of hyperparameters and demonstrate the effectiveness of the intervention loss.
We implement our models using PyTorch (Paszke et al., 2019) with the Adam optimizer (Kingma & Ba, 2015). Network architectures are fairly chosen among the baseline methods and IVGAN (see
Table B.1 in the appendix for more details). The classifier we use to compute the intervention loss shares the parameters with the discriminator except for the output layer. All input images are resized to have 64× 64 pixels. We use 100-dimensional standard Gaussian distribution as the prior pz . We deploy the instance noise technique as in (Jenni & Favaro, 2019). One may check Appendix B.2 for detailed hyperparameter settings. All experiments are run on one single NVIDIA RTX 2080Ti GPU. Although IVGAN introduces extra computational complexities to the original framework of GANs, the training cost of IVGAN is within an acceptable range1 due to the application of strategies like parameter sharing.
Figure 4: Random samples of generated images on MNIST, CIFAR-10, LSUN and STL-10.
Real-world datasets experiments We first test IVGAN on four standard real-world datasets, including CIFAR-10 (Krizhevsky, 2009), MNIST (Lecun et al., 1998), STL-10 (Coates et al., 2011), and a subclass named “church_outdoor" of the LSUN dataset (Yu et al., 2015), to investigate its training stability and quality of the generated images. We use the Frèchet Inception Distance (FID) (Heusel et al., 2017) to measure the performance of all methods.
The FID results are listed in Table 1, and the training curves of the baseline methods and IVGAN on four different datasets are shown in Figure 5. We see that on each datasets, the IVGAN counterparts obtain better FID scores than their corresponding baselines. Moreover, the figure of training curves also suggests the learning processes of IVDCGAN and IVLSGAN are smoother and steadier compared to DCGAN, LSGAN or MRGAN (Che et al., 2016), and converge faster than WGAN-GP. Samples of generated images on all datasets are presented in Figure 4.
Stacked MNIST experiments The original MNIST dataset contains 70K images of 28 × 28 handwritten digits. Following the same approaches in Metz et al. (2017); Srivastava et al. (2017); Lin et al. (2018), we increase the number of modes of the dataset from 10 to 1000 = 10× 10× 10 by
1Empirically IVGANs are approximately 2 times slower than their corresponding baseline methods.
stacking three random MNIST images into a 28×28×3 RGB image. The metric we use to evaluate a model’s robustness to mode collapse problem is the number of modes captured by the model, as well as the KL divergence between the generated distribution over modes and the true uniform distribution. The mode of a generated imaged is found from a pre-trained MNIST digit classifier.
Our results are shown in Table 2. It can be seen that our model works very well to prevent the mode collapse problem. Both IVLSGAN and IVDCGAN are able to reach all 1,000 modes and greatly outperforms early approaches to mitigate mode collapse, such as VEEGAN (Srivastava et al., 2017), and Unrolled GAN (Metz et al., 2017). Moreover, the performance of our model is also comparable to method that is proposed more recently, such as the PacDCGAN (Lin et al., 2018). Figure 6 shows images generated randomly by our model as well as the baseline methods.
Ablation study Our ablation study is conducted on the CIFAR-10 dataset. First, we show the effectiveness of the intervention loss. We consider two cases, IVLSGAN without the intervention loss (µ = 0), and standard IVLSGAN (µ = 0.5). From Figure 7 we can find that the intervention loss makes the training process much smoother and leads to a lower FID score in the end.
We also investigate the performance of our model using different number of blocks for the block substitution interventions and different regularization coefficients for the intervention loss. The results are presented in Table 3. It can be noticed that to some extent our models’ performance is not sensitive to the choice of hyperparameters and performs well under several different hyperparameter settings. However, when the number of blocks or the scale of IV loss becomes too large the performance of our model gets worse.

heading: 5 CONCLUSION
text: We have presented a novel model, intervention GAN (IVGAN), to stabilize the training process of GAN and alleviate the mode collapse problem. By introducing auxiliary Gaussian invariant interventions to the latent space of real images and feeding these perturbed latent representations into the generator, we have created intermediate distributions that interpolate between the generated distribution of GAN and the data distribution. The intervention loss based on these auxiliary intervened distributions, together with the reconstruction loss, are added as regularizers to the objective to provide more informative gradients for the generator, significantly improving GAN’s training stability and alleviating the mode collapse problem as well.
We have conducted a detailed theoretical analysis of our proposed approach, and illustrated the advantage of the proposed intervention loss on a toy example. Experiments on both real-world datasets and the stacked MNIST dataset demonstrate that, compared to the baseline methods, IVGAN variants are stabler and smoother during training, and are able to generate images of higher quality (achieving state-of-the-art FID scores) and diversity. We believe that our proposed approach can also be applied to other generative models such as Adversarial Autoencoders (Makhzani et al., 2015), which we leave to future work.

heading: A PROOFS

heading: A.1 PROOF OF THEOREM 1
text: Proof. The conditional probability ofX ′ given label can be written asP(X ′|ei) = pi(X ′), so further
P(X ′, ei) = 1 kpi. And we denote the marginal distribution of x as p(x) = 1 k k∑ i=1 pi(x). Cause the activation function at the output layer of the classifier is softmax, we can rewrite the loss function into a more explicit form:
Vclass(f) = Ei∼U [k]Ex′∼pi − eTi log f(x′) = Ei∼U [k]Ex′∼pi − log fi(x)
= 1
k ∫ k∑ i=1 −pi(x) log fi(x)dx = ∫ p(x) { − k∑ i=1 p(ei|x) log fi(x) } dx.
Let gi(x) = fi(x) p(ei|x) , then k∑ i=1 p(ei|x)gi(x) = 1. And notice that k∑ i=1 p(ei|x) = 1. By Jensen’s inequality, we have: k∑ i=1 −p(ei|x) log fi(x) = k∑ i=1 −p(ei|x) log[gi(x)p(ei|x)]
= k∑ i=1 −p(ei|x) log gi(x) +H(p(·|x)) ≥ log k∑ i=1 p(ei|x)gi(x) +H(pi(·|x))
= log 1 +H(p(·|x)) = H(p(·|x)). And Vclass(f∗) = ∫ p(x)H(pi(·|x))dx if and only if g∗i (x) = g∗j (x) for any i 6= j, which means
that f ∗ i (x) p(ei|x) = r ∀i ∈ [k], where r ∈ R. Notice that k∑ i=1 f∗i (x) = 1, it is not difficult to get that f∗i (x) = p(ei|x). The loss function becomes
1
k ∫ k∑ i=1 −pi(x) log p(ei|x)dx = −H(x) + k∑ i=1 1 k H(pi) + log k
= −JS(p1, p2, ..., pk) + log k
(7)

heading: A.2 PROOF OF THEOREM 2
text: Proof. According to Theorem 1, for a given real number 1, we can find another δ1, when intervention loss is less than δ1, the distance between pi and pj under the measurement of JS-divergence is less than 1. And because JS-divergence and Total Variance distance (TV) are equivalent in the sense of convergence. So we can bound the TV-distance between pi and pj by their JS-divergence. Which means that ∫ |pi − pj |dx ≤ 0 when the intervention loss is less than 1 (we can according to the 0 to finding the appropriate 1). Using this conclusion we can deduce |P (E(G(Oi(z))) ≤ r)− P (E(G(Oj(z))) ≤ r)| ≤ 0, where r is an arbitrary vector inRd. Further, we have:
|P (Oi(z) ≤ r)− P (Oj(z) ≤ r)| ≤ |P (Oi(z) ≤ r; ‖Oi(z)− E(G(Oi(z)))‖ > δ)| + |P (Oj(z) ≤ r; ‖Oj(z)− E(G(Oj(z)))‖ > δ)|+ |P (Oi(z) ≤ r; ‖Oi(z)− E(G(Oi(z)))‖ ≤ δ) − P (Oj(z) ≤ r; ‖Oj(z)− E(G(Oj(z)))‖ ≤ δ)|
(8) We control the three terms on the right side of the inequality sign respectively.
P (Oi(z) ≤ r; ‖Oi(z)− E(G(Oi(z)))‖ > δ)
≤ P (‖Oi(z)− E(G(Oi(z)))‖ > δ) ≤ E‖Oi(z)− E(G(Oi(z)))‖
δ
(9)
And the last term can be bounded by the reconstruction loss. The same trick can be used on P (Oj(z) ≤ r; ‖Oj(z)− E(G(Oj(z)))‖ > δ). Moreover, we have
P (E(G(Oi(z))) ≤ r − δ)− P (‖Oi(z)− E(G(Oi(z)))‖ > δ) ≤P (Oi(z) ≤ r; ‖Oi(z)− E(G(Oi(z)))‖ ≤ δ) ≤ P (E(G(Oi(z))) ≤ r + δ)
(10)
Notice that lim δ→0 P (E(G(Oi(z))) ≤ r ± δ) = P (E(G(Oi(z))) ≤ r). Let si(r, δ) = |P (E(G(Oi(z))) ≤ r ± δ)) − P (E(G(Oi(z))) ≤ r)| then the last term of inequalityA.2 can be bounded as: |P (Oi(z) ≤ r;‖Oi(z)− E(G(Oi(z)))‖ ≤ δ)− P (Oj(z) ≤ r; ‖Oj(z)− E(G(Oj(z)))‖ ≤ δ)|
≤|P (E(G(Oi(z))) ≤ r)− P (E(G(Oj(z))) ≤ r)|+ P (‖Oi(z)− E(G(Oi(z)))‖ > δ) + si(r, δ) + sj(r, δ)
(11) Every term on the right hand of the inequality can be controlled close to 0 by the inequalities mentioned above.

heading: B EXPERIMENTAL DETAILS

heading: B.1 NETWORK ARCHITECTURES

heading: B.2 HYPERPARAMETER SETTINGS

heading: C RELATED WORK
text: In order to address GAN’s unstable training and mode missing problems, many researchers have turned their attention to the latent representations of samples. Makhzani et al. (2015) proposed the Adversarial Autoencoder (AAE). As its name suggests, AAE is essentially a probabilistic autoencoder based on the framework of GANs. Unlike classical GAN models, in the setting of AAE the discriminator’s task is to distinguish the latent representations of real images that are generated by an encoder network from Gaussian noise. And the generator and the encoder are trained to fool the discriminator as well as reconstruct the input image from the encoded representations. However, the generator can only be trained by fitting the reverse of the encoder and cannot get any information from the latent representation.
The VAE-GAN (Larsen et al., 2015) combines the objective function from a VAE model with a GAN and utilizes the learned features in the discriminator for better image similarity metrics, which is of great help for the sample visual fidelity. Considering the opposite perspective, Che et al. (2016) claim that the whole learning process of a generative model can be divided into the manifold learning phase and the diffusion learning phase. And the former one is considered to be the source of the mode missing problem. (Che et al., 2016) then proposed Mode Regularized Generative Adversarial
Nets which introduce a reconstruction loss term to the training target of GAN to penalize the missing modes. It is shown that it actually ameliorates GAN’s ’mode missing’-prone weakness to some extent. However, both of them fail to fully excavate the impact of the interaction between VAEs and GANs.
Kim & Mnih (2018) proposed Factor VAE where a regularization term called total correlation penalty is added to the traditional VAE loss. The total correlation is essentially the Kullback-Leibler divergence between the joint distribution p(z1, z2, . . . , zd) and the product of marginal distribution p(zi). Because the closed forms of these two distribution are unavailable, Factor VAE uses adversarial training to approximate the likelihood ratio.

Review summaries:
The paper proposes a method for stabilizing the training of GAN as well as overcoming the problem of mode collapse by optimizing several auxiliary models. The first step is to learn a latent space using an autoencoder. Then, this latent space is "intervened" by a predefined set of $K$ transformations to generate a set of distributions $p_k$. A classifier is then taught to distinguish between $p_k$. Eventually, the weights of the classifier are shared with those of the discriminator network to produce the desired stabilization/diversification effect. In other words, the authors propose to stabilize GANs by intervening with the discriminator. This is done by sharing its weights with a classifier that trains on a perturbed latent distribution that is somehow related to the original problem via the prior assumption imposed.