Intervention Adversarial Auto-Encoder
Authors: Yang Hu
Cheng Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Generative Models
Adversarial Training

Abstract: In this paper we propose a new method to stabilize the training process of the latent variables of adversarial auto-encoders, which we name Intervention Adversarial auto-encoder (IVAAE). The main idea is to introduce a sequence of distributions that bridge the distribution of the learned latent variable and its prior distribution. We theoretically and heuristically demonstrate that such bridge-like distributions, realized by a multi-output discriminator, have an effect on guiding the initial latent distribution towards the target one and hence stabilizing the training process. Several different types of the bridge distributions are proposed. We also apply a novel use of Stein variational gradient descent (SVGD), by which point assemble develops in a smooth and gradual fashion. We conduct experiments on multiple real-world datasets. It shows that IVAAE enjoys a more stable training process and achieves a better generating performance compared to the vanilla Adversarial auto-encoder (AAE)

Under review as a conference paper at ICLR 2022
INTERVENTION ADVERSARIAL AUTO-ENCODER
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we propose new method to stabilize the training process of the latent variables of adversarial auto-encoders, which we name Intervention Adversarial auto-encoder (IVAAE). The main idea is to introduce sequence of distributions that bridge the distribution of the learned latent variable and its prior distribution. We theoretically and heuristically demonstrate that such bridge-like distributions, realized by a multi-output discriminator, have an effect on guiding the initial la- tent distribution towards the target one and hence stabilizing the training process. Several different types of the bridge distributions are proposed. We also apply a novel use of Stein variational gradient descent (SVGD) (Liu & Wang 2016), by which point assemble develops in smooth and gradual fashion. We conduct experiments on multiple real-world datasets. It shows that IVAAE enjoys more stable training process and achieves better generating performance compared to the vanilla Adversarial auto-encoder (AAE) (Makhzani & Shlens 2013).
INTRODUCTION
Generative models are widely used for image and texture production. Among them there are two base models which are most appealing scholars for their elegant theoretical foundation and close combination with neural network, namely Variational Auto-encoders (VAD)(Kingma & Welling 2013) and Generative adversarial network (GAN)(Goodfellow lower bound of the logllikelihood called ELBO which decomposes to a reconstruction loss term and a regularization term within an auto-encoder structure. GAN, however, goes beyond the like- lihood concern, trying to generate data directly using learned map from the latent space that is trained an adversarial manner. Both models have some weakness. For example, GAN's training is quite unstable and sometimes may have mode collapse problem. VAE usually produces less sharp pictures. The regularity of the latent space is also an important concern for VAE that affects the quality of the generated images.
al., 2014). VAE maximizes
There're re many attempts to alleviate the above drawbacks. Adversarial Auto-encoders (AAE) com- bines the techniques of both VAE and GAN network. It imposes discriminator on the latent space of an auto-encoder to classify the latent distribution p(z) and q(z). where the adversar- ial term serves as the regularizer. Such choice improves the regularization effect of the latent variable and alleviates the mode collapse problem due to an auto-encoder structure. Wasserstein Auto-Encodersi (Tosstoonn et al. (017)) generalize different regularization approaches on la- tent space, proposing adversarial method (WAE-GAN) and kernel based method (WAE-MMD). It provides AAE with theoretical support as well. Intervention Generative Adversarial Network (IV- ( et al.) 2020), another attempt to combine GAN with the encoder structure, stabilizes GAN's ' training process by intervening on the latent distribution so that the reconstructed data dis- tribution could approach the target more robustly.
In this work, we will propose this novel regularization method on AAE models. As basically an auto- encoder structure, the key question is how to let the original distribution in latent space approach the prior distribution. We intervened on the encoded data, which inherently brings intervened dis- tribution. We will show that loss term in our model can be transformed to certain measure of the overlap among multiple distributions. When minimizing the loss in terms of these constructed variables. expect not only the original distribution to approach the target one, but the distribution of the intervened data (we call bridge distribution) to serve as a guidance to have such approaching process more fast and robust well.

---

Under review as a conference paper at ICLR 2022
2 METHODOLOGY
Training a GAN model has always been a challenge (e.g., the process is known to be unstable and prone to mode collapse). AAE's auto-encoder structure alleviates the problem with reconstruc- tion term. first updates the auto-encoder upon the reconstruction loss, then use the discriminator distinguish the encoded latent variables and generated ones following Gaussian distribution, up- dating the discriminator and encoder in an adversarial manner. However, such architecture doesn't I completely solve GAN's 's problem. Decreasing the JS divergence has a relatively low approaching efficiency between two distributions. Adversarial training on latent space doesn's 't avoid this problem at all. Therefore there's s a desire to apply a more powerful regularization term to have the approach- ing process more efficiently and stable. We consider to construct a series of "bridge distributions" by certain transformations so that they in a sense lie between the encoded distribution and target onc. We let the discriminator to classify these multiple distributions, and update the encoder to let them approach each other. Intuitively such bridge distributions would serve as a guidance to help the la- tent variables follow the target distribution more quickly and robustly, but we need more theoretical arguments first.
Before expound on further details, want to turn our attention how construct the transformations mentioned above. We first explain the meaning of intervention. is similar to what is exhibited in IVGAN([Lang al. 2020 For completeness show it here in our context.
Definition 1 Let L be the set of all random variables in Re on the probability space (St, L,, P). T is the set of all mappings from Ra to Rd. For a distribution function with support in Rd, we call T E I "P-intervention" " if X P = T(X) P. for all X E Ex Sf T(X) ~ P, V S= = N~11 ~,,,
satisfies that
we denote S "Complete Intervention Group".
A rough understanding of the above definition is that complete intervention group let the meet of subsequent multiple distributions embed their final distributing pattern, i.e. P(x). Combining the idea in our model, the intervention can be performed on our encoded data. We hope when these originally different distributions approach each other, they all approach the prior distribution, which is exactly the feature of the transformation proposed above. Therefore the complete intervention group can serve to ensure the identifiability of our method.
To be more concrete, we set the IP to be standard Gaussian and propose several intervention pattems that possess the above feature.
A simplest example of F-intervention is to replace some of the dimension with normal distributed one. Therefore we have our Blockwise substitution: Rd space is divided equally by t parts, where t/d. For 0 & t, let Ti be the transformation that substitute the first blocks with random variables following standard normal distribution independent with the original variable. Then we get a complete intervention group (Tk)-
There's 's other ways to create intervention groups For example, we let be replaced by æcos( 27 + zsin( ko 2r ). for &s 1, I..,,, where z follows standard normal distribution independent with x. We call such construction pattern Radial substitution.
Except for the two substitution method, we also apply Stein variational gradient descent (SVGD) (Liu & Wang (016) to construct the bridge distribution by iteratively updating points. By setting the target distribution the updated particles will approximately approach P. More details will be specified in the next section.
Now we propose our model called Intervention Adversarial Auto-Encoder (IVAAF). [Figure shows its specific structure, which contains encoder E, decoder G and discriminator D. Images r is fed into the É to get the encoded latent code z, which is decoded by to obtain the reconstructed images The encoded is transformed by complete intervention group Tę (To represent the identical mapping) to get 2 which is discriminated by with output of size t + 1. Through maximizing and minimizing alternatively a cross entropy loss, D and E is both updated.
Intervention Loss
2

---

Under review as a conference paper at ICLR 2022
To(z)
T2(2)
X Xdaa
E
Z
Intervention
Intervention loss
T((z)
G
Reconstruction Loss
Figure 1: The structure of our model
The key distinction of our model from AAE the intervention loss. More specifically, when encoded
latent Exx) is obtained, for every k; 0 < k < t, we intervene by Tи to get 2j.- Then we feed size output d, and compute the cross entropy between d
zg into the discriminator obtain and labels ex, averaging on k; finally to update discriminator by gradient flow. The encoder update in the same manner.
Therefore the intervention loss is an unbiased estimator of its theoretical form as follow:
LIv [E, D) -Fa-erts Par-pasta [log( D& (T& (F((a)))
D& = -Ee-arte Ee [Do (z))]
(1)
where DR. Ph represent the (I: 1)th digit of D output, the distribution of 24 Tx(((x))). respec- tively.
Then we give theoretical results that evidence the rationality of proposed models.
Theorem 2 Given I + latent variables {=x) with density function support in Rd. and their probabilistic density function {ps Mi the classifier D is trained to minimize the cross entropy loss as equation 1 under the constraint that Ei=0 D5(2) 1. Then the optimal classifier satisfies:
pc(z)
(2(2) Σj
And the corresponding optimal intervention loss can be expressed as follows:
LIv [F, D) = .P...t) + log(1 + 1)
where JS(q), (2, (b) - == ード
Proof: See Appendix.
Theorem 2 tells us that when the discriminator reaches optimal, the intervention loss can be re- garded certain distance among multiple distributions. We know when & the loss becomes the original JS divergence which conforms with GAN. However, it is known that JS divergence isn't perfect measure because the gradient would be zero if two variables have distributions with disjoint support. According to the theorem, situation for multiple distributions will suffer less from such problem, for bridge distributions provide greater chances for intersection which consequently produces gradient. Therefore it would be more stable to update the encoder.
We now further illustrate the advantage the multi-object adversarial training. Although GAN's convergence has been theoretically GrodeeGGo al. 2014), the training process, is well- known delicate and unstable. One important reason is that the optimal discriminator can always achieve too perfect matter how close it is between the generated data manifold and the real data manifold (Arjovsky & Bottou, (017). One cause for this is that two manifolds is disjoint with probability 1 due their low dimension & Bottou 2017) the data space.

---

Under review as a conference paper at ICLR 2022
Algorithm Intervention Adversarial Auto-Encoder (IVAAE)
Input: learning rate:
dimension of the space: d, number of bridge distributions: t Interventions:
TR. 0 < k I I, hyperparameters: X, fr, μ2, minibatch size: m..
Output: 0p. Oz. 0g
1: for number of training iterations do
2: Sample ) from the training set
3: Sample {2, jj=1 from the prior P(z)
4: Compute 2 E(zz), 1.....
5: Compute R = G(2)), = =
6: for k O.... do
7:
Compute the intervened latent variables 8: 2j = ( ,,.,
9:
end for
Update the parameters of D by:
@D t 0D mi Mi Ven Σド= Ek (2)
12:
Calculate Lrecon, LIv Update the parameters of G by:
13:
14:
OG + 0c + at Vegdereoo
Update the parameters of E by:
Bg + Ag
a Vo (XCrecon + n2Liv}
17: end for m
return pp. 0gg 0G
Different from GAN, AAE applies discriminator the latent distribution, therefore what it learns isn't low-rank manifold, but continuous distribution with full dimension. But the problem doesn't 't get solved in AAE setting. We address it in heuristic way. We run AAE real-world dataset for 50 epochs, then fix the encoder and decoder and update the discriminator alone. We observe how perfect the discriminator will develop. Figure F records the changing curve of adversarial loss every iteration. According to Figrue 2 the adversarial loss still decreases remarkably toward 0, especially when latent dimension goes high. That's because of the limit batch size. IVAAE could alleviate this problem by boosting the difficulty for discriminator to classify more than two distributions and apply our models IVAAE. We choose = 4, block- wise substitution, with the rest of structure all the same as the AAE model. The loss is recorded with fixed encoder and decoder after 50 epochs, too. We note that although the adversarial loss is decreasing well, the drop is much more moderate than AAE. We also examine the consequence of such loss decline. We choose the adversarial loss for encoder: -Fpasta [log( D F(x)))] and record the gradient norm of the encoder after the encoder and decoder fixed. According to Figure AAE is less stable compared to IVAAE
equivalently introducing more particles.
AAE
IVAAE
109
10
10-1
aeverarbal loxs
10-3
adversrtal
10-4
1.:4
dim
4mm
dim
dmm -32
dim 64
dim -44
1000
2000
4000
1000
2000
4000
training Iterations
training Rerations
Figure 2: We train the model for 50 epochs. Left: AAE; Right: IVAEE (t = 4, biock. Then we fix the encoder and decoder to run the discriminator alone. record the loss change every iteration. The dimension of latent space is chosen at 16, 32, 64.
4

---

Under review as a conference paper at ICLR 2022
Gradient of the Encoder
10a
10a
[|VoL D)1
10a
100
10"1
AAE dim 16
10-2
MAAE
16
1000
2000
3000
4000
5000
6000
training iterations
Figure 3: We choose the latent dimension at and run models for 50 epochs. Then we record the change of encoder's gradient with the encoder and decoder fixed. Note that Lado (E, D) ([[log(((((x))) is used as the encoder loss.
RELATED WORK
3.1 GENERATIVE MODELS
Since the appearance of VAE and GAN model, large amount of attempts have been conducted to improve their performance on generating data. Many of them put their attention on the latent space. 2-stage VAE & Wipf 2019) diagnoses the poor generate quality of VAE model and attributes it to the mismatch of the distribution between the encoded g(z) and the prior p(z). They handles the problem improves the model's sample quality. RAE(Ghosh et al., 2019) view VAE from a deterministic angle. It abandons the stochastic component and replace it with proposed regularization term, which provide an alternative to encode smooth and meaningful latent variables. Info VAE(Zhao al. 2019) enhance VAE model by introducing to ELBO mutual information term between and 2, which is generalization of both VAE and AAE. It improves the quality of the variational posterior and strengthen the bond between data space and latent features. Our method, different from the above, dedicates to improve regularity of latent space by enhancing and stabilizing the training process. applies solid schemes and techniques while not deviating from the original target.
imposing another VAE structure
learn the latent distribution, which
3.2 SVGD
Stein variational gradient descent (SVGD) is proposed as variational inference algorithm (Liu & Wang 2016). The method iteratively transports set of particles to match the target distribution by applying form of functional gradient descent that minimized KL divergence among reproducing kernel Hilbert space (RKHS). By using the assignment as follow:
J+1
k222. V log p(2)) + Vzh(e),2))
where the represents the kernel of RKHS, m batch size, the particles 2; gradually approximate
the target distribution p(z). We apply the method in the construction of bridge distribution of the
IVAAE model. We choose step size = 1, kernel function RBF k(z, 2') = exp(- z' l
with bandwidth h and set the target distribution the prior distribution p(z), which standard
5

---

Under review as a conference paper at ICLR 2022
normal in our context. To calculate the intervention loss, we obtain the batched encoded data E( = 1...m. m. We plug it in the iterative function above to update it t times to get { M= Then we use {z'} as the input of discriminator to obtain the loss term and do the back propagation all the way back. This is computationally practical because of the trivial form of Vlog p(=).
4 EXPERIMENTS
In this section we conduct experiments on multiple datasets including MNIST (LeCun) 1998), CI- FARIO (Krizhevsky & Hinton| 2010), CelebA (Liu et al. 2018). We use different patterns of bridge distribution in IVAAE and compare the performance to those of the related baseline method with similar architecture. By exhibiting multiple measures during the train and showing the generated pictures, we empirically demonstrate that the performance of our methods is superior on training stability and generating images quality. The measures we use include FID (Heusel et al. 2017), popular index to measure the generating variety and quality, and Frechet Latent Distance (FLD), a similar form of distance we propose to judge the latent distribution. The bridge distributions we employ include blockwise and radial patterns, corresponding method denoted by IVAAE and IVAAErads respectively. We choose t = 4 in all of our experiments. The competing approaches we apply include WAE-GAN, for it generalizes the AAE algorithm, and WAE-MMD well.
We use Pytorch to implement our models. In IVAAE and other baseline models adopt similar architecture in order to give persuasive conclusion. We use deterministic models only. For all datasets and models, we set hyperparameters as follows: learning rate and decoder, a = 0.0005 for discriminator, t = 4, À = 1, μ1 FT A2 = 2 learning rate decays by multiplying 0.5 after 30 epochs, further multiplying 0.2 after 50 epochs, and 0. after 100 epochs. We run all the codes for 300 epochs. Note that the intervention term in IVAAE is down weighted by E+I because the derivatives of logrithmatic function becomes 2 7+I greater when striking balance at - instead of in binary conditions. There's noise added in any layers.
0.001 for encoder
We calculate FID by generating 20k latent variables from the prior and feeding them the trained decoder. We randomly choose 60k images, get the encoded z, and use the Frechet distance to measure its difference with the standard Gaussian:
FLD(22) = ||=1 Tr(E. - 22 EV
where (4 and Σg is the means and covariance matrix of 2, respectively. We choose the model with the lowest FID score and calculate its FLD score to measure the regularization performance of latent variables.
MNIST
The MNIST dataset contains 60K images of 28 x 28 handwritten digits. Before we use them, we resize the images to 64 x 64 pixels. Instead of going for the best model performance, we adopt the same architecture as WAE (Tolstikhin et al. 2017) do to compare the results among those models with same structure. We choose the latent size 16, batch size 64. The network structure is two fully connected linear layers for encoder, decoder and discriminator. The only difference in WAE and IVAAE's model structure is the output size of discriminator. See Table B] in appendix. We use learning rate distinct with the ground settings: 0.0001 for decoder and discriminator, 0.00005 for the encoder. We run each model for 300 epochs and learning rate is divided by 2 every 100 epochs.
CelebA
CelebA dataset contains over 200k images of humans faces. We apply transform to crop 140 x 140 pixels from the central of images and then resize to 64 x 64 digits before we use them. The architecture adopt is similar to what is recommended in WAE. The dimension of latent space is 64. We use convolutional network for encoder and decoder with x 5 or 6 x 6 size filters, and a network composed of four fully connected linear layers of 512 elements for the discriminator. See Table 4 in appendix. Batch size is set at 100.

---

Under review as a conference paper at ICLR 2022
CIFARIO
To implement methods on CIFARIO we set dimension of latent space 128. We reshape the CIFARIO images to 64 x 64 size. The network structure is completely the same the network for CelebA, except for the output size encoder and input size for decoder and discriminator. The latent dimension is relatively large considering batch size we choose 128. We will note that WAE-GAN fails to converge after 30-40 epochs while IVAAE is stable during the whole training
Except for block-wise replacement and weighted Gaussian, we also use the SVGD & Wang 2016) technique as third method to construct bridge distribution, We obtain particle assembles from encoded batched data by iterative assignment (equation 2). A notable question is, once the batch size is limited, the particle assemble zf is not approaching standard normal but with shrunk variance lower than 1. Therefore we need a high width h which could enlarge the repulsive force between points. The discrepancy is also amplified when the data's dimension goes higher and batch size goes smaller. Therefore we implement the method only on MNIST dataset which needs relatively low latent dimension.
We let and choose RBF kernel. The step size is set to 1. We obtain {=')==] for every two iterations, while 2' is sampled from standard Gaussian. SVGD degrades gradient descent of MAP when width h = O, while high h would cause the convergence to slowing down, so finally we choose h = 100hg. ho = wed' of the pairwise distance between the current points Besides, we add 0.1 Gaussian noise on the particles after every iteration.
logn as recommended (Liu & Wang. 2016), where med is the median
Note that SVGD process points update in a relative smooth way by adding deviation on former points. It is similar to the radial substitution. Moreover, when using equation 2 to update latent point, all points in current batch are involved, therefore the tendency of one point towards Gaussian distribution is directly affected by force from other points. Such interaction become stronger under great bandwidth h. Intuitively we hope this feature would improve the hidden layer's 's regularity. To measure such regularity We fix the parameters of encoder after training WAE-GAN, IVAAEpad. IVAAEzugd models, respec- tively. Then two-layer MLP with 1000 nodes each is connected to the encoder's last hidden layer to do supervised learning. We set batch size 60, learning rate 0.0001, and evaluate the classification error rate every 100 training iterations. The result is favorable our argument as shown in Figure
certain extent,
utilize the trained encoder to do classification work.
5
MNIST
CIFAR10
WARE GAN
IARE GAN IVAA NAAE....
IVARE
IAAT...
80-
200
150-
300
120
340
training aportts
training spochs
Figure 4: FID curves for different methods calculated during the training. Note that WAE-GAN's training process is not stable CIFAR10.
7

---

Under review as a conference paper at ICLR 2022
Classification Error Rate
0.08
WAE-GAN
MAAE
0.07
IVAAEs
0.06
0.05
0.04
0.03
0.02
5000
10000
15000
20000
25000
30000
training iterations
Figure 5: MNIST classification error rate for methods WAE-GAN, IVAAEred, IVAAE Param- eters of the encoder is fixed after pretrained.
Table 1: FID score for different methods on multiple datasets. FID results are calculated every 10 epochs and we choose the minimum. Lower score means generated images are better. * means we don't for the best performance across different structure, but for comparison purpose only.
Methods
MNIST
CelebA
CIFARIO
WAE-MMD
67"
36
WAB-GAN
53"
33
109
IVAAE
49"
34
117
IVAAErad
50
33
80
IVAAE
49"
Table 2: FLD score for different methods on multiple datasets. We choose FLD when FID reaches the best. Lower score means the encoded latent distribution is closer the Guassian prior.
Methods
MNIST
CelebA
CIFAR10
WAE-MMD
0.05
0.34
10.9
WAE-GAN
5.5
1.5
15.1
IVAAE,..
0.2
0.22
4.0
IVAAE,
0.07
0.18
11.8
IVAAEsugd
5.7
From the result we could notice that WAE-GAN's training stability and speed lower than IVAAE on dataset we study. It conforms to what have been discussed previous section. When the latent size reach higher as 128 (with batch size 128), WAE-GAN suffers from intense oscillation. Every time the model converges increases sharply. IVAAE performs well under the situation we inspect, and the FID score and the FLD [Figure 6 when FID reaches lowest is overall better [Table I 2 Curiously, in terms of the performance, among patterns choose there seems be pattern that are superior to the other on every datasets. On CIFAR10 we notice that IVAAE,ad attain far lower FID score than IVAAE while the FLD score goes to the opposite. There seems to be trade-off relation between the generate quality and the regularity (how close to the standard normal) of latent variables. an interesting phenomenon explore in the future work.
certain degree, the adversarial loss decreases and reconstruction loss
8

---

Under review as a conference paper at ICLR 2022
WAE-GAN
IVAAE, rdd
14
1.4
34
94
8-
8
10
20
30
40
50 60
10 20
30 40
50
Figure 6: Heat maps for covariance matrix of the encoded for different models when converging. We choose CelebA with the latent dimension as 64. The brightness represent absolute value of the corresponding entry. The brighter the closer to The darker the closer to O.
(a) WAF-GAN
(b) IVA AV
(c) IVAA AVAAFrad
Figure 7: CelebA images randomly generated. IVAAEpad-
Left:
WAE-GAN; Mid: IVAAE
Right:
5 CONCLUSION
In this paper propose generative model which possess robust and stable training property by leveraging interventions on latent variables and classifying them. creatively applies the multiple- output structure to discriminate latent distributions can be regarded as certain bridge distributions which work as a guidance to pull the encoded variables the target distribution. By conducting heuristic loss trace for discrimina- tor, we demonstrate that multiple classifying have a significant stabilizing effect on the adversarial training practice.
two distributions. It is theoretically proved that the intervened
To structure different bridge distribution, we also apply the SVGD method to smoothly update the latent variables. We successfully combine the method in our adversarial training, and to some extent demonstrate the hidden space enjoys better regularity as well.
Besides. multiple experiments are conducted on real-world datasets. By using measures on both images generating quality and latent regularity, we conclude a better performance of our models compared the baseline models. The data results coincide with our inference on the stability of model training.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiy preprint arXiv: uxXiv:1701.04866 2017.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv: xrXiv11903.05785 2019.

---

Under review as a conference paper at ICLR 2022
Partha Ghosh, Mehdi SMM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12436. 2019.
Ian J. GGodfelllw, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 14Xi::1406.2661 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. arXiv preprint
(xXiv:7706008000 2017.
Diederik Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar- 10. Unpublished manuscript, 40(7): 2010.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
Jiadong Liang, Liangyu Zhang, Cheng Zhang, and Zhihua Zhang. Intervention generative adversar- ial networks. arXiv preprint 00Xiv:2008.03372,, 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. arXiv preprint vrXiv:1608, 1408,0444, 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(201 112101):11,200 2018.
Alireza Makhzani and Jonathon Shlens.
Adversarial autoencoders.
arXiv preprint
arXiv:1511.05644, 2015.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. encoders. arXiv preprint arXiv:1711.015555 2017.
Wasserstein auto-
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the aaai conference on artificial intelligence, vol- ume 33, pp. 0885-5892. 2019.
10

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOF OF THEOREM 2
Liv(E,D) = - (d Ezmp, (xg())())))
pa (2))
J. p(z)
p(ex [exl>>log Dz(2)de
k=
Last equation holds because p(exlz) play)pu(z)
pa(a) ((+1)p(=]
p(2)
Notice that
Dx(2)
Σ ((e)
Σ k=U
plea (lex|=))o0 |=)gog
H[p(+|=))
p(ex|z)
Dx(2)
2 - log
p(ex|z)
+ -H(p(+z))
p(ex|z)
1=
=H(p(11>)
The equality holds when D2(2) & p(ex/z) a pu (x). i.. Di(z) = Σ Ej-oP Plugging (2) into the loss term, we obtain:
pE(2)
Liv(E, D)
Σ k=0
pn(2)logDa(z)dz
b+7
pk(2)]og pk(2) dz
t k=u
Σ (z))
(R(z)
plo(z) log
log(t 1) dz p(2)
8
k=0
t KL(pxllp) log(t + 1)
So we have our theorem proved.
A.2 NETWORK STRUCTURE
Table 3: The network architecture we use for MNIST, FC represents fully connected layer.
E INPUT 64 64 FC(1024) ReLU FC(1024, 1024) ReLU FC(1244, 16)
G
D INPUT = FC(16, 1024) ReLU FC(1024, 1024)
INPUT = FC(16, 1024) ReLU FC(1024, 1024)
ReLU
ReLU
FC(64 x 64), Tanh
FC( + 1)
11

---

Under review as a conference paper at ICLR 2022
Table 4: The network architecture we use for CelebA and CIFARIO. CONV(C, K, S) represents convolutional layer with C channels, KxK-size kernels, and S strides. DCONV represents decon- volutional layer, parameter meaning similar with CONV. BN represents batch normalization layer.
G
INPUT 64 64 3 CONV(C128, K5, S2) BN, ReLU CONV(C256, K5, S2)
INPUT
INPUT
FC(1024 x 8 X 8)
FC(512)
DCONV(C512, S1)
ReLU
BN. ReLU DCONV(C256, SI)
FC(512, 512)
BN, ReLU CONV(C512,, K5, S2)
ReLU
BN, ReLU DCONV(C128, K5, S2)
FC(512, 512)
BN, ReLU CONV(C1024, K5, S2)
ReLU
BN, ReLU DCONV(CI, K4, SO)
FC(512, 512)
BN, ReLU FC(C1)
ReLU
Tanh
FC(t 1)
12


Reviews:
[{"review_id": "ljL_Ex7GRy", "reviewer": "Reviewer_Nq3D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper targets on several serious concerns about generative modeling, such as unstable training process and mode collapse.\n\nWeaknesses:\n1. The novelty of this paper is not significant enough as similar ideas have been developed for GAN in \"Jiadong Liang, Liangyu Zhang, Cheng Zhang, and Zhihua Zhang. Intervention generative adversarial networks.\"\n2. The experiment evaluation is not sufficient as the only comparing method is WAE. There are many methods targeting on unstable training process and mode collapse in GAN, and more comparisons are needed, which should also be included in related work and reference.\n3. The improvement shown in experiments is marginal and not significant.\n4. As the key component of IVAAE, page 2 shows two substitutions, more systematic discussion is needed on how to develop and assess substitutions.\n5. On page 4, \"We address it in a heuristic way.\" Detailed settings for this heuristic study should be provided, why can this particular setting be representative and generalize to other settings?", "summary_of_the_paper": "This paper presents Intervention Adversarial auto-encoder to stabilize the training process of the latent variables of adversarial auto-encoders. Intervention Adversarial auto-encoder adopts a sequence of distributions to bridge the distribution of the learned latent variable and its prior distribution. The bridge distributions are implemented by a multi-output discriminator, which guides the initial latent distribution to the target distribution with a stable training process. The paper discusses several different types of the bridge distributions and applies\nStein variational gradient descent. Experiments on multiple real-world datasets are conducted.", "summary_of_the_review": "This paper targets on some important issues in generative modeling such as unstable training process and mode collapse. Novelty of this paper is low and evaluation is not sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VunKqP972qE", "reviewer": "Reviewer_tnv3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength:\nThe intervention loss for regularization seems novel in VAE training. But it's far below the bar of ICLR. I will describe the weakness as:\n\n## Weakness:\n1. Lots of related works are missing in this paper, e.g. ARAE [1] and VAE-SVGD [2].\n\n2. Though the method is motivated well, the proposed method doesn't seem to solve the problem of VAE, especially considering only WAE as baselines.\n\n3. The experiments are weak. Recall the baseline works that have been published several years ago, this paper is not even close to them.\n\n[1] Adversarially Regularized Autoencoders\n\n[2] VAE Learning via Stein Variational Gradient Descent", "summary_of_the_paper": "This paper introduces an intervention adversarial autoencoder and claims this could improve VAE learning. ", "summary_of_the_review": "I would suggest the authors choose another conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4InGgIcyZXP", "reviewer": "Reviewer_LAu5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n+ The topic is interesting.\n\nCons\n- The technical correctness is very concerning. There are many claims left unsupported (via references or experiments) or poorly explained at best. The most serious ones are about stability claims, including \"according to Figure 3, AAE is less stable compared to IVAAE\" on page 4 and \"we propose a generative model which possess robust and stable training\" on page 9. The only result which very weakly (given the limited scale of experiments) suggests better stability in the paper is the CIFAR10 panel in Figure 4, which is clearly insufficient to support the aforementioned claims. Also, Algorithm 1 has many forward and backward (backprop) passes (some likely redundant), but the speed (based on wall-clock time) is never discussed in the paper.\n- The novelty is quite poor. The intervention loss (Liang et al., 2020) is only slightly improved by introducing the radial substitution, which is very close to mixup in the latent space [1,2]. The addition of SVGD is somewhat random and poorly justified. Given that Gaussianization has been studied for a long time [3,4,5], it's unclear why the much more generic SVGD should be used instead. The \"proposed\" FLD is simply just [6] and not helpful in explaining the results (since it's uncommon). Normality tests [7] which can directly give probabilities would be much better in my opinion.\n- The significance is quite low. The scale of the experiments and the number of baselines are simply too small to convince the research community that the proposed algorithms are indeed better and the results will generalize. The FID scores and the quality of the generated images are also clearly too low for today's standards. Theorem 2 is basically reiterating (Liang et al., 2020).\n- The writing is poor and hard to follow given the amount of errors. Also, Eq 2 is clearly missing some subscripts. Fig 4 and 7 are not referenced in the maintext. AAE was authored by Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly and Ian Goodfellow, not just Makhzani and Shlens.\n\n[1] On Adversarial Mixup Resynthesis, NeurIPS, 2019.\\\n[2] Manifold mixup: Better representations by interpolating hidden states, ICML, 2019.\\\n[3] Gaussianization, NeurIPS, 2000.\\\n[4] Iterative gaussianization: from ica to random rotations, IEEE TNN, 2011.\\\n[5] Gaussianization flows, AISTATS, 2020.\\\n[6] The Fréchet distance between multivariate normal distributions, Journal of Multivariate Analysis, 1982.\\\n[7] https://en.wikipedia.org/wiki/Normality_test", "summary_of_the_paper": "The paper aims to improve AAEs with the intervention loss (Liang et al., 2020) and SVGD (by constructing \"bridge distributions\") and demonstrates some empirical results.", "summary_of_the_review": "Given the concerning technical correctness, poor novelty, significance and writing, I recommend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims at improving AAEs with an intervention loss. While the topic is important, the reviewers agree that

- The paper has poor clarity,
- The related work is not adequately put into perspective,
- There are concerns with technical correctness,
- Experimental evidence is lacking,

As the authors have not addressed any of these concerns, the paper can not be accepted in its current form.
Decision: Reject