LIA: Latently Invertible Autoencoder with Adversarial Learning
Authors: Jiapeng Zhu
Deli Zhao
Bolei Zhou
Bo Zhang
Venue:  (2020)
Keywords: variational autoencoder
generative adversarial network

Abstract: Deep generative models such as Variational AutoEncoder (VAE) and Generative Adversarial Network (GAN) play an increasingly important role in machine learning and computer vision. However, there are two fundamental issues hindering their real-world applications: the difficulty of conducting variational inference in VAE and the functional absence of encoding real-world samples in GAN. In this paper, we propose a novel algorithm named Latently Invertible Autoencoder (LIA) to address the above two issues in one framework. An invertible network and its inverse mapping are symmetrically embedded in the latent space of VAE. Thus the partial encoder first transforms the input into feature vectors and then the distribution of these feature vectors is reshaped to fit a prior by the invertible network. The decoder proceeds in the reverse order of the encoder's composite mappings. A two-stage stochasticity-free training scheme is designed to train LIA via adversarial learning, in the sense that the decoder of LIA is first trained as a standard GAN with the invertible network and then the partial encoder is learned from an autoencoder by detaching the invertible network from LIA.  Experiments conducted on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA for inference and generation.

Under review as a conference paper at ICLR 2020
LIA: LATENTLY INVERTIBLE AUTOENCODER WITH ADVERSARIAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep generative models such as Variational AutoEncoder (VAE) and Generative Adversarial Network (GAN) play an increasingly important role in machine learn- ing and computer vision. However, there are two fundamental issues hindering their real-world applications: the difficulty conducting variational inference VAE and the functional absence of encoding worldworld samples in GAN. In this paper, we propose novel algorithm named Latently Invertible Autoencoder (LIA) address the above two issues in one framework. An invertible network and its inverse mapping are symmetrically embedded in the latent space of VAE. Thus the partial encoder first transforms the input into feature vectors and then the distri- bution of these feature vectors is reshaped to fit a prior by the invertible network. The decoder proceeds in the reverse order of the encoder's composite mappings. A two-stage stochasticity- -free training scheme is designed to train LIA via adver- sarial learning, in the sense that the decoder of LIA is first trained as a standard GAN with the invertible network and then the partial encoder is learned from an autoencoder by detaching the invertible network from LIA. Experiments conducted on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA for inference and generation
INTRODUCTION
Deep generative models play a more and more important role cracking challenges in computer vision as well as in other disciplines, such as high-qualit, image generation (Tsola et al. 2017]
Zhu et al. 2017 7 Karras et al. 2018a b. Brock et al. 2018). 8), text-to-speech transformation (van den
Oord et al.] 201 2017), information retrieval (Wang et al. 2017) 3D rendering (Wu al. 2016
Eslami et al. 2018), and iignal-to-imag acquisition (Zhu et al. 2018). Overall, the generative
models fall into four categories: autoencoder and its most important variant of Variational Auto- Encoder (VAE) (Kingma & Welling, 2013), auto-regressive models (van den Oord et al. 2016b a)
Generative Adversarial Network (GAN) (Goodfellow al. 2014), and normalizing flows (NF) (Tabak Tabak & Turner 2013) Rezende Mohamed] (015)).
& Vanden-Fijndenn 2010
Here we compare these models through the perspective of data dimensionality reduction and recon- struction. To be formal, let æ be data point in the de-dimensional observable space Rds and be its corresponding low-dimensional representation in the feature space Rd,, The general formulation of dimensionality reduction is
J:Ra= + Rdd 2 -- - [(x),
where f()) is the mapping function and d, < d. The manifold learning aims requiring under various constraints on [Tenebbaum] I et al. 22000] Roweis & Saul 2000). However, the sparsity of data points in high-dimensional space often leads to model overlitling, thus necessitating research on opposite mapping from to x, i.e.
g : Rd Rd 4 -gy))
where g(-) is the opposite mapping function with respect to f(-), to reconstruct the data. In general, the role of g(-) regularizer to f(.)) or a generator to produce more data. The autoencoder is
We will make source code of L.A publicly available

---

Under review as a conference paper at ICLR 2020
of mapping x: 2 : ă. A common assumption in autoencoder is that the variables in low- dimensional space are usually sampled from a prior distribution P(z; 0) such as uniform Gaussian. To differentiate from y, we let z represent the low-dimensional vector following the prior distribution. Thus we can write
g R + Rde, x x g(z), P)z:0).
It is crucial to establish such dual maps = - f(x)) and æ - =(z). In the parlance of probability, the process of x + 2 f(z) is called inference, and the other procedure of z + a g(2) is called sampling or generation. VAE is capable of carrying out inference and generation in one framework by two collaborative functional modules. However, it known that many cases VAEs are only
able to generate blurry images due to the imprecise variational inference. To see this, we write the approximation of the marginal oog-likelihod
log p(z) log / p(z) 2) 2 -LLg((/ ((0
(1)
where KL[g(z|as x))
[p(2]] is the Kullback-Leible divergence with respect to posterior probability (<z|z²) and prior p(z). This lower-bound log likelihood usually produces imprecise inference. Furthermore, the posterior collapse frequently occurs when using more sophisticated decoder mod- els (Bowman et al.
2015; Kingma al. 2016). These two issues greatly limit the generation capability of the VAE. On the other hand, GAN is able to achieve photo-realistic generation re- sults (Krrrs et al. 2018a b). However, its critical limitation is the absence of the encoder f(x) for carrying inference on real images. Effort has been made on learning an encoder for GAN under the framework of VAE, however the previous two issues of learning VAE still exist. Normalizing flows perform the exact inference and generation with one architecture by virtue of invertible networks (Kingma & Dhariwal 2018). But it requires the dimension d, of the data space to be identical to the dimension de of the latent space, thus posing computational issues due to high
complexity of learning deep flows and computing the Jacobian matrices.
Inspired by recent success of GANs (Karras et al., 2018al b) and normalizing flows (Kingma et al.. 2016 Kingma& & Dhariwal 2018), we develop a new model called Latently Invertible Autoencoder (LIA). LIA utilizes an invertible network bridge the encoder and the decoder of VAE in a symmetric
manner. We summarize its key advantages as follows:
The symmetric design of the invertible network brings two benefits. The prior distribution can be exactly fitted from an unfolded feature space, thus significantly easing the inference problem. Besides, since the latent space detached, the autoencoder can be trained without variational optimization thus there is no approximation here.
The two-stage adversarial learning decomposes the LIA framework into a Wasserstein GAN (only prior needed) and standard autoencoder without stochastic variables. Therefore the training is deterministio implying that the model will be not affected by the posterior collapse when the decoder is more complex or followed by additional losses such as the adversarial loss and the perceptual loss.
We compare LIA with state-of sfte-of-thheart generative models on inference and generation/recon- struction. The experimental results on FFHQ and LSUN datasets show the I.IA achieves superior performance on inference and generation.
LATENTLY INVERTIBLE AUTOENCODER
The neural architecture of LIA is designed such that the data distribution can be progressively unrolled from a complex or manifold-valued one to a given simple prior. We now describe the detail as follows.
2.1 NEURAL ARCHITECTURE OF LIA
Framework of LIA is based on the classic VAE and the realization of normalizing flow. As shown in Figure D we symmetrically embed an invertible neural network in the latent space of VAE, following the diagram of mapping process as
"The "deterministic" and "stochasticity-free" in this paper refer to the second-stage training, i.c. the learning of the inference process.
2

---

Under review as a conference paper at ICLR 2020
(())
oly)
s(z)
(a) data
(b) feature y
(c) prior
(d) feature y
(c) generation ă
c(2; #.
f(x)
9 y
exteactor
Alseriminator
invertible network
encoder
generator
(f) neural architecture of LIA
Figure I: Latently invertible autoencoder (LIA). LIA consists of five functional modules: an encoder to unfold the manifold fax). an invertible network 6 to reshape feature embeddings to match the prior distribution decoder to produce output æ g feature extractor to perform reconstruction measure, and discriminator e to distinguish real/fake distributions.
<(x) and Ø map latent variables to feature vectors y = (2),
decoler
encoder
2
x.
invertible network
invertible network
where
6h denotes the deep composite mapping of normalizing flow with depth k. LIA first performs nonlinear dimensionality reduction on the input data æ and transform them into the low-dimensional feature space Rdv, The role of f(x) for LIA can be regarded unfold the underlying data manifold, as illustrated in Figures [ and pp. Therefore, the Euclidean operations such as linear interpolation and vector arithmetic are reliable this feature space. Then we establish an invertible mapping o(y) from the feature to the latent variable z, opposed to the VAEs that directly map original data latent variables. The feature y can be exactly recovered via the invertibility of from z, which is the advantage of using invertible networks. The recovered feature y is then fed into a partial decoder g(y) to generate corresponding data æ. the maps Φ and ø-1 of the invertible network are bypassed, it comes back as a standard autoencoder, i.e. x 4 y - a.
In general, any invertible networks applicable in the LIA framework. We find in practice that simple invertible network is sufficiently capable of constructing the mapping from the fcature space Rd, to the latent space Rdd Let [xyi 2b) and fractions of æ and z, respectively. Then the invertible network (Dinh et al. 2015) can be built as follows,
[ #b] be the forms of the top and bottom
24 = 26 (xx:),
(3)
Zt.
3G 21 T(2)).
where 7 is the transformation that can be an arbitrary differentiable function. Alternatively, one can attempt to exploit the complex invertible network with affine coupling mappings for more challenging tasks (Dinh et al. 2017 Kingma& & Dhariwal 2018). As conducted in (Dinh et al. 2015), we set 7 as a multi-layer perceptron with the leaky ReL.U activation.
2.2 RECONSTRUCTION LOSS AND ADVERSARIAL LEARNING
To guarantee the precise reconstruction क, the conventional way by (variational) autoencoders is to use the distance || - 2 or the cross entropy directly between ø and æ. Here, we utilize the perceptual loss that is proven to be more robust variations of image details (Johnson et al.) 2016). Let e denote the feature extractor, e.g. VGG (Simonyan & Zisserman 2014). Then we can write the loss
(5) suffices to emphasize that the functionality of here is in essence to produce the representations of the input and the output á. The acquisition of fairly flexible. It can be attained by supervised
L(c,2, ||<(z)) - <())
3

---

Under review as a conference paper at ICLR 2020
22
e"
(2)
y
((Y)
f(x)
11
9 g'
C(E
3t
(a) Decoder training.
(b) Encoder training.
Figure 2: Detaching the functional modules of LIA for two-stage training.
unsupervised learning, meaning that can be trained with class labels or without class labels (van den Oord et al. 2018).
The norm-hased reconstruction constraints usually incur the blurry image generation in autoencoder- like architectures. This problem can be handled via the adversarial learning (Goodfellow al. 2014). To do so, a discriminator e is employed to balance the loss of the comparison between æ and æ. Using the Wasserstein GAN (Arjovsky al. 2017 Gulrajani et al. 2017). we can write the optimization
objective as
Lc,, x)
[c()]] T~~
E
[c(x)] ?
E [IV=c(=)] am-
where Pr and Ps denote the probability distributions of the real data and the generated data, respec- tively. y is the
of the regularization. The RS regularizer is formulated (Mescheder et al. 2018), which is proven more stable for training convergence.
TWO-STAGE STOCHASTICITY-EER TRAINING
Training deep models usually follows an end-to-end fashion for the whole architecture. To backprop- agate gradients through random variables for VAE, the reparameterization trick is harnessed (Kingma Welling 2013 i.e. x = + o * (O 1), where f is the mean and o the standard deviation. The regularization of coupling the prior and the posterior is the KL divergence used optimize the parameters of the encoder by backpropagation. For our framework, however, we find that this end- to-end learning strategy cannot lead the algorithm to converge we propose scheme of two-stage stochasticity-fie training, which decomposes the framework into two parts that can be well trained end-to-end respectively, as shown in Figure At the first step, the decoder of LIA is trained using adversarial learning with invertible network to acquire the ability of high-quality generation.
satisfactory optima. To proceed,
the second step, the invertible network that connects feature space and latent space is detached from the architecture, reducing the framework to standard autoencoder without variational inference. Thus this two-stage design prevents the posterior collapse issue while facilitates the decoder with more complex structure well as adversarial loss for high-resolution image generation.
3.1 DECODER TRAINING
ProGAN KKrras et al. 201 20)8 StyleGAN (Krrras et al. 2018b), and BigGAN (Brock et al. 2018) are capable of generating choto-realistic images from random noise sampled from some prior distribution. Then it naturally supposed that such GAN models are applicable to recover a precise ă we can find the latent variable for the given æ. Namely, we may train the associated GAN model separately in the LIA framework. To conduct this, we single out a standard GAN model for the first-stage training, as displayed in Figure 2h. Here is directly sampled from a prior distribution. According the principle of Wasserstein GAN, the optimization objective can be written as
{Ø", e²) = min max L(c, x 2), 6.g
where the superscript * denotes that the parameters of corresponding mappings have already been learned. It is worth noting that the role of the invertible network here is just its transformation invertibility. We do not pose any constraints normalizing flows. Our strategy of attaching an invertible network in front of the generator can be potentially applied any GAN models.
the probabilities of and d(y) in contrast to
4

---

Under review as a conference paper at ICLR 2020
3.2 ENCODER TRAINING
In the LIA architecture, the invertible network is embedded in the latent space in a symmetric fashion, in the sense that fz) = y = 6 (z). The unique characteristic of the invertible network allows us to detach the invertible network @ from the LIA framework. Thus we attain conventional autoencoder without stochastic variables, as shown Figure In practice, the feature extractor 6 in perceptual loss is the VGG weight up to conv4 pretrained on the ImageNet dataset. After the first-stage encoder training, the parameter of f is needed to be learned as
f = min BL x, .) + L(c", x; 3),
where / is the widely applied in computer vision. It is the backbone framework of various GANs for diverse image processing tasks (Isola et al. 2017 Zhu et al. 2017). For LIA, however, it is much simpler because we only need to learn the partial encoder J. This simplicity brought by the two-stage training is able to enforce the encoder to converge with more precise inference.
The above optimization serving to the architecture in Figure 2p is
4 RELATED WORK
Our LIA model is relevant to the works that solve the inference problem for VAEs with adversarial learning as well as the works that design encoders for GANs. The integration of GAN with VAE can be traced back to the work of VAE/GAN (Larsen et al.] (016) and implicit autoencoders (Makhzani (ct al., 2015 Makhani 2018). These methods encounter the difficulty of end-to-end training, because the gradients are prone becoming unstable after going through the latent space in deep complex architectures (Bowman et al. 2015 Kingma et al. 2016). Besides, there is an intriguing attempt of training VAE in the adversarial manner (Ulyanov et al.] [017] [Heljakka et al.] 2018). These approaches confront the trade-off between the roles of the encoder that performs inference and compares the real/fake distributions. This difficult tune. So we prefer the complete GAN with an indispensable discriminator.
The closely related works to LIA are the models of combining VAE and the inverse autoregressive flow (Kingma et al., (016) and the autent-flow-based VAE approach that are VAEs with latent variables conditioned by normalizing flows (Su & Wu 2018 Xiao al., 2019). These three models all need to optimize the posterior probability of normalizing flows, which is essentially different from our deterministic optimization in LIA. The stochasticity-ffer
training is directly derived from the symmetric design of the invertible network in the latent space, which is different from (Kingma et al., 2016) and (Su & Wu 2018 Xiao et al.] 2019). There are alternative attempts of specifying the generator of GAN with normalizing flow (Grover et al. 2017) or mapping images into feature space with partially invertible network (Lucas et al. 2019). These approach suffers from high complexity computation for high dimensions. The approach of two-stage training in (Luo et al. 2017) is incapable of solving the posterior estimation.
It worth noticing that the reconstruction task focus here is different the recent work of representation learning which learns features for recognition using adversarial inference (Dumoulin et al. 2017, Jeff Donahue, 2017 Donahue & Simonyan 2019). Our primary goal is to faithfully reconstruct real images from the latent code.
5 EXPERIMENTS
For experimental setup, we instantiate the decoder of LIA with the generator of StyleGAN (Karras ct al. 2018b). The difference is that we replace the mapping network (MLP) in StyleGAN with the invertible network. The layer number of the invertible network is 8. The paraam-rraseters for the discriminator are = 10 (equation (6)) and B = 0.001 (equation (8)). For perceptual loss in equation (5), we take conv4_3 3 from the VGG weight.
The generative models we compare are the MSE-based optimization methods (Radford et al. 2016
Berthelot et al. 2017 Lipton& & Tripathi 2 the adversarially learned inference (ALI) (Dumoulin et al. 2017), and the adversarial generator-encoota (AGE) network (Ulyanov et al. 2017) To evaluate
We use the code atttpp:://githuu..com//uusa
5

---

Under review as a conference paper at ICLR 2020
FFHQ
LIA (ours)
ALI
AGE
MSE-based
Encoded
Figure 3: Reconstructed faces by generative models on FFHQ database.
Table I: Quantitative comparison of image reconstruction.
Metric
LIA (ours)
ALI
AGE
MSE-based optimization
Encoded StyleGAN
FID
16.88
74.98
118.88
44.79
22.26
SWD
10.01
15.09
38.70
43.44
15.82
MSE
18.10
32.61
29.91
18.81
23.18
the necessity of the invertible network, we also train an encoder and StyleGAN with its original multi-layer perceptron, which is the last column in Figure 3 The two-stage training scheme is used as LIA does. The generator and discriminator of the StyleGAN is exactly same to that StyleGAN.
For quantitative evaluation metrics, use Fréchet inception distance (FID), sliced Wasserstein distance (SWD), and mean square error (MSE). These three metrics are commonly used measure numerical accuracy of generative algorithms (Ulyanov et al. 2017 Karras al., 2018a Jeff Donahue 2017: Karras et al.] (2) We directly use the code released by the authors of ProGAN (Karras al. 2018a). The prior for is Gaussian and the dimension is 512.
Figure 4: Manipulating reconstructed faces. The algorithm in Shen et al. 2019) is applied to manipulate the image reconstructed from the latent code w given by LIA. Each row shows the original image, the reconstruction, glass, pose, gender, smile, and age.
FFHQ FACE DATABASE
All models are first tested on the Flickr-Faces-HO (FFHQ) database created by the authors of StyleGAN the benchmark. FFHQ contains 70,000 high-quality face images. We take the first according to
65,000 faces as the training set and the remaining 5,000 faces as the reconstruction the exact order of the dataset. We do not split the dataset by random sampling for interested readers can precisely reproduce all the reported results with our experimental protocol.
/tt

---

Under review as a conference paper at ICLR 2020
Figure 5:
The exemplar real images of objects and scenes from LSUN database and their reconstructed images by LIA. Three categories are tested, i.e. cat, bedroom, and car. For each group, the first row is the original images, the second row shows the reconstructed images.
Figure shows the reconstructed faces of all models. It is clear that LIA significantly outperforms others. The reconstructed faces by ALI and AGE look correct, but the quality mediocre. The ideas of ALI and AGE are elegant. Their performance may be improved with the new techniques such as progressive growing of neural architecture or style-based one. The method of the MSE-based optimization produces facial parts of comparable quality with LIA when the faces are normal. But this approach fails when the variations faces become large. For example, the failure comes from the long fair, hats, beard, and large pose. The interesting phenomenon is that the StyleGAN with encoder only does not succeed recovering the target faces using the same training strategy as LIA, even though it is capable of generating photo-realistic faces in high quality due the StyleGAN generator. This indicates that the invertible network plays the crucial role to make the LIA work. The quantitative result in Table I shows the consistent superiority of LIA. More reconstruction results are included in Appendix A..5 The reconstruction from LIA facilitates semantic photo editing. Figure 4 shows the manipulation results on reconstructed faces. More results can be found in Appendix A.7 The interpolation and style mixing are also displayed for reference in Appendix A.2
Figure 6: Interpolation on objects in the LSUN database.
5.2 LSUN DATABASE
To further evaluate LIA on the data with large variations, we use the three categories from the large-scale LSUN database Yuu al., 2015). i.e. cat, car, and bedroom. For each category, the 0.1 million images are selected by ranking algorithm (Zhou et al. 2003) from the first 0.5 million images in the dataset. Each cat and bedroom images are resized to be 128 x 128 and the size of the car image is 128 x 96 for training. We take subsets because it does not take too long for training to converge while still maintains the data complexity. These subsets will be made available for evaluation.
Figure 5 shows that the reconstructed objects by LIA faithfully maintain the semantics as well as the appearance of the original ones. For example, the cats' whiskers are recovered, indicating that LIA is able recover very detailed information. More results are attached in Appendix A.6 We can see that LIA significantly improves the reconstruction quality. The improvement mainly comes from the two-stage training of LIA. The decoder trained with adversarial learning guarantees that the generated images are photo-realistic. The encoder deterministically trained with perceptual and adversarial losses ensures that latent feature vectors can be obtained more precisely. This two-stage training is enabled by the design that the invertible network detachs the encoder and decoder, thus avoiding the optimization of the posterior probability when leamnnng the encoder. Figure 6 shows the
7

---

Under review as a conference paper at ICLR 2020
30
0.05
1.4
20
1.2
LIA
Gradient
10
Gradient
lo5s
0.8
-10
0.6
-0.05
2
8
Iteration
Iteration
Iteraaioo
(a) The first layer
(b) The last layer
(c) Loss
Figure 7: Illustration of gradients and losses for training encoders on FFHQ dataset. For VAE, the decoder is exactly the generator of StyleGAN. The two-stage training of VAE is the same as that of LIA. The first and last layers in (a) and (b) refer the neural architecture of the encoder. The y-axis indicates the average value of the gradients. The ær-xxis is re-scaled by 10,000. The gradients for I.IA case are much more stable than the ones for VAE. The gradient of middle layer is also shown in Appendix
A.8 A.8
interpolation results on those reconstructed objects. More results are included in Appendix A.3 The experimental results on FFHQ and LSUN databases verify that the symmetric design of the invertible network and the two-stage training successfully handles the issue of inference.
5.3 WHY THE INVERTIBLE NETWORK MATTERS?
To further reveal the importance of the invertible network in LIA, we investigate the gradients of the encoders under two different cases of decoders when learning the encoder in the two-stage training. The first case is to directly apply the KL--divergence to optimize the random latent space
as VAE does. The mapping the encoder and the decoder is æ A f y S. ã. where op is the mapping network (MLP) in StyleGAN. This is the conventional way of learning encoders for GAN algorithms via variational inference. The second one is our architecture of LIA for the second-stage
training, i.e. x 4 y 2 ă. where the latent space is removed due the symmetric design of the invertible network. Figure 7 clearly illustrates the difference of gradients between these two cases. The gradient volatility for variational inference high and the associated loss is effectively reduced, meaning that the gradients during training are noisy and not always informative. This may indicate that the stochasticity in latent space causes problems for training encoder via variational inference. Instead, the encoder's gradients for LIA are rather stable across different layers and the loss decreases monotonically, showing the importance of the stochasticity-fre training and the invertible network.
CONCLUSION
new generative model, named Latently Invertible Autoencoder (LIA), has been proposed for generating image sample from probability prior and simultancously inferring accurate latent code for a given sample. The core idea of LIA is to symmetrically embed an invertible network in an autoencoder. Then the neural architecture is trained with adversarial learning as two decomposed modules. With the design of two-stage training, the decoder can be replaced with any GAN generator for high-resolution image generation. The role of the invertible network is to remove any probability optimization and bridge the prior with unfolded feature vectors. The effectiveness of LIA is validated with experiments of reconstruction (inference and generation) on FFHQ and LSUN datasets.
It is still challenging to faithfully recover all the image content especially when the objects or scenes have unusual parts. For example, LIA fails to recover the hand appeared at the top of the little girl (the second row in Figure 3 Besides, the Bombay cat's 's necklace (the second row in Figure 5) is missed the reconstructed image. These features belong to multiple unique parts of the objects or scenes, which are difficult for the generative model to capture. One possible solution is to raise the dimension of latent variables (c.g. using multiple latent vectors) or employ the attention mechanism to highlight such unusual structures in the decoder, which is left for future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. In arXiv: 1701007777 2017.
David Berthelot, Thomas Schumm, and Luke Metz. BEGAN: boundary equilibrium generative adversarial networks. arXiv:1703. 2017.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arxiv:1511.06334, 2015.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In arXiv: 1096, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. In International Conference on Learning Representations (ICLR), 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference Learning Representations (ICLR), 2017.
Jeff Donahue and Karen Simonyan. arXiv::907.02544 2019.
Large
scale adversarial representation learning.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. In International Conference on Learning Representations (ICLR), 2017.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor. David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 1204 1210, 2018.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN- Combining maximum likelihood and adversarial learning in generative models. In 1705. 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved 2017.
training of Wasserstein GANs. In arXiv:
Ari Heljakka, Arno Solin, and Juho Kannala. Pioneer networks: Progressively growing generative autoencoder. In 107.:v8887.0322. 2018
Phillip Isola, Jun-Yan Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Trevor Darrell Jeff Donahue, Philipp Krähenbühl. Adversarial feature learning. In International Conference on Learning Representations (ICLR), 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. arXiv: 1603.08:33.. 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. Proceedings the the 6th International Conference on Learning Representations (ICLR), 2018a.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv: 18/2.04948 2018b.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In arXiv:/807 /00,03399 2018.

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2th International Conference on Learning Representations (ICLR), 2013.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, Improving variational inference with inverse autoregressive flow. In arXiv: 160.04444 2016.
Anders Boesen Lindbo Larsen, Spren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. In International Conference Machine Learning (ICML), pp. 1558-1566, 2016.
Zachary C. Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adversarial networks. In International Conference on Learning Representations (ICLR), 2017.
Thomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, and Jakob Verbeek. Adver- sarial training of partially invertible variational autoencoders. arXiv:1901.0109) 2019.
Junyu Luo, Yong Xu, Chenwei Tang, and Jiancheng Lv. Learning inverse mapping by autoencoder based generative adversarial nets. In urXiv:1703.10099 2017.
Alireza Makhani. Implicit autoencoders. In arXiv: /805000400 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In arXivv 2015.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? arXiv:1801.04406 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the 4th International Conference Learning Representations (ICLR), 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning (ICML), pp. 1530-1538, 2015.
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290(5500))2323-22355 2000.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. a90iv: 10786, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, hos//109.1555 2014.
Jianlin Su and Guang Wu. f-VAEs: Improve VAEs with conditional flows. In 2018.
Esteban G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Communications in Mathematical Sciences, 66(2):145-16 2013.
Esteban G. Tabak and Eric Vanden-Eijnden. Density estimation by dual ascent of the logelikeli Communications Mathematical Sciences, 8(1):217-233, 2010.
Joshua B. Tenenbauml, Vin de Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500)2319-2323, 2000.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. takes (only) two: Adversarial generator-
encoder networks. In 000.:::1704.2..300 2017.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. In /60,00/0609 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In 1600 06759, 2016b.
10

---

Under review as a conference paper at ICLR 2020
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury. Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Hciga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast high-fidelity speech synthesis. In arXiv:/711 10433, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. TIX.::1807.03748 2018.
Jun Wang. Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. IRGAN: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, William Freeman, and Josh Tenenbaum, Learning probabilistic latent space of object shapes via 3D generative adversarial modeling. In Advances in Neural Information Processing Systems (NeurIPS), pp. 82-90, 2016.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv: 10485, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction a large-scale image dataset using deep learning with humans in the loop. arXiv:1506.03365. 2015.
Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf. Ranking on data manifolds. In Proceedings of the 16th International Conference on Neural Information Processing Systems (NeurIPS), 2003.
Bo Zhu, Jeremiah Z. Liu, Stephen F. Cauley, Bruce R. Rosen, and Matthew S. Rosen. Image reconstruction by domain-transform manifold learning. Nature, 555:487-492, 2018.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-imee ton--age translation using cycle-consistent adversarial networks. In International Conference Computer Vision (ICCV), 2017.
A APPENDIX
A.1 GENERATION RESULTS
We compare the generation results of StyleGAN and LIA. For the decoder (generator), the difference between these two models is the mapping network. To be specific, the architecture of StyleGAN generator of form
y 9 a.
(9)
where MLP is multi-layer perceptron. The L.A decoder consists of mappings
(10)
where Φ is the invertible network. We follow the experimental setting of StyleGAN (Karras et al. 2018b) compute the FIDs. We first generate 50,000 fake faces and then randomly sample another 50,000 real faces. The FID is computed with these two datasets. Table and Tigure&sshov that the performance of StyleGAN and LIA is comparably good. The benefit of using the invertible network is to facilitate the training of the encoder for inference.
Table 2: FIDs of generative results.
Algorithm
FFHQ
Cat
Car
Bedroom
StyleGAN
6.23
7.36
3.09
4.88
LIA
6.28
7.08
2.52
4.36
11

---

Under review as a conference paper at ICLR 2020
Figure 8: Generative results by StyleGAN and LIA. For each group of objects, the first row shows the generative results obtained by StyleGAN and the second row by LIA.
12

---

Under review as a conference paper at ICLR 2020
A.2 FACE INTERPOLATION AND STYLE MIXING
Examining the interpolation in the latent feature space is an effective way of visualizing the capability of generative models well measuring how well it fits the underlying data distribution. Here compare the three algorithms. As shown in Figure 9 while well preserve the facial properties. The interpolation quality of the MSE-based optimization is actually based on the reconstruction performance because it has a good generator (StyleGAN), The intermediate interpolation result from Glow deviates from real faces.
LIA achieves smoother interpolation result
Figure 9: Interpolation results by three generative models. The first row shows the result of LIA (ours), the second row that of the MSE-based optimization, and the last row that of Glow. The first and last faces in the Glow result are the real face images of FFHQ.
We further perform style mixing using small of reconstructed faces. Style mixing is conducted using the same approach presented in (Karras et al. (0188) The difference is that LIA uses the real faces thanks its encoding capability. Figure I0 shows that our algorithm can infer accurate latent codes and generate high-quality mixed faces.
Figure 10: Style mixing for real faces. The faces in the first row are the source faces where style comes from.
13

---

Under review as a conference paper at ICLR 2020
A.3 OBJECT INTERPOLATION
FFHQ Faces
Cat
Car
Bedroom
Figure 11: Interpolation on FFHQ faces and objects in the L.SUN database.
14

---

Under review as a conference paper at ICLR 2020
A.4 DISENTANGLEMENT ILLUSTRATION
2
2
20
Figure 12: Interpolation with the latent code and the feature For each group, the first row shows the result with z and the second row shows the result with w.
15

---

Under review as a conference paper at ICLR 2020
Figure 13: Generated faces along the optimization path with the latent code and the feature u. The path is formed by the optimization of the MSE loss with respect to and w, respectively. The faces in the first column are generated by the initial values of and w. Then the following faces sequentially generated with the optimization process until the optimization converges. For each group, the first row shows the result with z and the second row shows the result with w. The disentanglement effect of w is clearly shown. Instead, the latent code suffers from the entanglement of features.
16

---

Under review as a conference paper at ICLR 2020
A.5
RECONSTRUCTION FOR FFHQ DATABASE
FFHQ Faces
LIA (ours)
ALI
AGE
MSE-based optimization
Encoded StyleGAN
Figure 14: Real images from FFHQ dataset and their reconstructed faces by various generative models.
17

---

Under review as a conference paper at ICLR 2020
A.6 RECONSTRUCTION FOR LSUN DATABASE
NO
NO.
Figure 15: Real images from LSUN dataset and the reconstructed images by LIA.
Figure 16: Real images from LSUN dataset and the reconstructed images by LIA.
18

---

Under review as a conference paper at ICLR 2020
A.7 FACE MANIPULATION
Figure 17: Manipulating reconstructed faces. The algorithm in (Shen et al. 2019 is applied to manipulate faces reconstructed from the latent code ws given by LIA. Each row shows the original image, the reconstruction, glass, pose, gender, smile, and age.
19

---

Under review as a conference paper at ICLR 2020
A.8
GRADIENTS OF MIDDLE LAYER
1
0.5
Gradient
0
-0.5
-1
-1.5
2
4
8
Iteration
Figure 18: Illustration of gradients when training encoders on FFHQ dataset. For VAE, the decoder is exactly the generator of StyleGAN, The two-stage training of VAE is the same as that of LIA. The red and blue lines represent the gradients for VAE and LIA respectively. This figure shows the gradient of middle layer in the neural architecture of the encoder, which complementary Figure
20

---

Under review as a conference paper at ICLR 2020
A.9
THE ARCHITECTURAL DETAIL OF LIA
Input Image
Conv3x3
Reverse
BatchNorm
LeakyRelu
y1
y2
Downsample
F
LeakyRelu
Conv3x3
BatchNorm
FC LeakyRelu
LeakyRelu
y1
y2
Conv3x3
BatchNorm
Downsample
FC
BatchNorm
y
Z
Figure 19: The architectural detail of LIA for each block of the whole network. Here the encoder and the invertible network are shown. The synthesis network in the decoder is the same as StyleGAN.
21


Reviews:
[{"review_id": "rJxSeEOj5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work in the paper is interesting and seems to show some empirical progress on image generation. However, the work in this paper can be seen as adding the losses from GANs and VAEs together to help learn a better generative model, which is not very novel. The invertible part is help for training as well. Still more detail about how the method works would be very helpful. While there is lots of math in the paper it is difficult for the reader to follow and often not well motivated why these choices are made. For example, the optimization is split into two different updates steps because they can't be combined mathematically. Yet, performing two different types of update steps can often favour whichever is easier and the other is noise. More details on how this process was made successful are important to discuss in the paper.\n\n\nMore detailed comments:\n-\tThe conjecture that VAEs produce blurry images needs a reference.\n-\tI am not sure if GANs are limited by the lack of an encoder. It could be that the introduction of an encoder is exactly what makes it difficult for VAE to learn complex and detailed image generation.\n-\tThe second claim in the paper (not affected by posterior collapse) should be proved in the paper or at least illustrated in some way. Currently, this claim is not well backed up in the paper.\n-\tTHe claim that the method will have linear interpolation and vector arithmetic will need to be more rigorously proven. Right now it seems a little too much like proof by picture.\n-\tI do like figure 1 and 2. They help explain the method rather well.\n-\tIn the begining of the experiment section, there are a number of hyperparameter values defined yet what these hyper parameters are is not explained. More detail needs to be added in this area for the reader to understand the experiments.\n-\tThe latent encoding size use is rather large.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJeLsABqFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "LIA: Latently Invertible Autoencoder Review\n\nThis paper proposes a novel generative autoencoder, and a two-stage scheme for training it. A typical VAE is trained with a variational approximation: during training latents are sampled from mu(x) + sigma(x) * N(0,1), mu and sigma are regularized with KL div to match an isotropic normal, and the model minimizes a reconstruction loss. A LIA is instead first trained as a standard GAN, where an invertible model, Phi, (e.g. a normalizing flow) is hooked up to the Generator/Decoder, such that the output is G(Phi^{-1}(z)) with z~p(z), a simple distribution. In the second stage, the Generator/Decoder and Discriminator are frozen, and an Encoder is trained to minimize a reconstruction loss (in this paper, a pre-trained VGG network is used as a feature extractor to produce a perceptual loss) and maximize the “real” output of the frozen Discriminator on reconstructed samples.\n\nThe key advantage of this method is that during the second stage no stochasticity is injected into the network and Phi is not involved.This means that the encoder does not need to be regularized to produce a specific parametric form of its output (no KL(p || q))), instead implicitly learning to match the latent distribution expected by the generator through the reconstruction losses. Additionally, because the latent space is not e.g. an isotropic Gaussian, it can be more expressive and flexible, only being constrained to an invertible transformation of the distribution p(z) chosen during the first training stage.\n\nThe training procedure is evaluated using a StyleGAN architecture on several high-res, single-class datasets (faces and LSUN Cats/Cars/Bedrooms). The quality of the resulting reconstructions is compared against several methods which are also capable of inference (like ALI and post-training an encoder on a standard StyleGAN), and samples and interpolations are presented. There is also an experiment that compares gradients in the encoder when training using LIA against training using a more typical VAE setup. \n\nMy take: The key idea behind this paper is quite promising, and I believe this paper has tremendous potential. I agree with the authors that the usefulness of implicit generative models is limited by their typical lack of an encoder network, and that existing approaches have several design drawbacks, and incorporating invertible networks and advances in flow-based modeling looks like a fruitful avenue of research.\n\nHowever, I have a litany of concerns with the paper itself, concerning its high similarity with a paper published in May, its motivation, its presentation, its empirical evaluation, and the analysis presented within. While my concerns and suggestions are extensive, this paper is perhaps unusual in that all of the issues I have are quite fixable; the core idea is good, but its realization and presentation in the paper need a substantial amount of revision. I am currently giving this paper a weak reject as I do not believe it is ready for publication, but I believe that with an overhaul this paper could be a more clear accept.\n\nUpdate, post rebuttal:\n\nThanks to the authors for their response. While I appreciate their insistence that my issues with the paper likely stem from my simply not understanding it (or the underlying topics), I hope they can appreciate that such an appeal is unlikely to allay said concerns. Pointwise:\n\n1. While there is of course a difference between the LIA setup and the GLF setup, regardless of the two-stage training or the inclusion of an adversarial or perceptual loss or any other bells and whistles that get attached, the fact remains that the resulting architecture for both LIA and GLF is an autoencoder with an invertible model in the middle. Arguing that the LIA setup is somehow fundamentally different is akin to arguing that a VAE-GAN is an utterly different model from a VAE with a VGG perceptual loss. Yes, they're optimizing slightly different things (distribution-wise differences vs sample-wise differences) , they have different characteristics, etc., but at the end of the day they're still autoencoders with extra bits on the end. The same general principle holds here. As I stated in my original review, I consider the differences relatively minor and maintain my stance that comparison is warranted. \n\n1(a). While the authors may have completed the work over a year ago, the fact that the other work was made public multiple months before this work means that it does, in fact, count as prior work. There is plenty of precedent at ICLR for work which appeared on arXiv well before submission date to be considered as prior work. I understand that this can be frustrating if the authors have previously submitted to other conferences and wished to wait until acceptance before making the work public, but that is a personal choice that does not change the nature of the situation.\n\n2. If interfacing with image manipulation techniques is the motivation for improving reconstructions, this motivation should be clearly stated in the paper. After rebuttal there is still no mention of this motivation, which suggests to me that the authors expect all readers to consider \"reconstruction\" (which I again posit is not really a task) to matter intrinsically.\n\n3. I once again appreciate that the authors hold that this reviewer is incorrect about basic facts. The forward KL placed on the latent space of a VAE only encourages it to resemble a particular distribution (typically isotropic gaussian) but the information content passed through the bottleneck can indeed grow with the size of the latent space, as one can see experimentally by ablating the latent dimensionality. This general principle should also be reinforced when one considers that flow models with exact inference (i.e. perfect reconstructions) require dz==dx.\n\n4. This reviewer maintains that sampling a random input from a distribution during training involves stochasticity.\n\n5. This still does not address my concern that reconstruction is not a particularly relevant task.\n\n7-9: Thank you for modifying the caption in this figure, though I still hold that the y-axis should be correctly labeled. This still does not address my concern that this experiment does not actually show what the authors claim it does--the magnitude of the gradient noise is not by any measure a viable indication that the inclusion of phi is doing anything meaningful in place of a standard MLP  as the comparison is instead made to an entirely different training setup.\n\nI maintain my stance of rejection.\n\nOriginal review:\n\nFirst off, a paper published in May on arXiv titled “Generative Latent Flow” (GLF) proposes an idea which is in essence identical to the one proposed in this paper. In GLF, a VAE is trained, but rather than using the reparameterization trick, a true normalizing flow is trained to model the distribution of the output of the encoder (i.e. with NLL using the change of variables formula common to flow-based models), such that the training of the actual autoencoder is truly deterministic (in the sense that at no point is an epsilon~p(z) sampled like in a normal VAE. The core difference between LIA and GLF is that GLF learns to model the distribution of the encoder outputs to enable sampling, while LIA incorporates an invertible model into a generator which explicitly through sampling, and then fits an encoder post-hoc. There are other differences in implementation and the choice of datasets, but those are (IMO) minor details relative to the core similarity. Given that GLF was published 4 months before the ICLR2020 deadline, this paper absolutely must be cited, compared against, and discussed. I am somewhat inclined to argue that given the similarity, LIA is merely incremental relative to GLF, but for now I think it is sufficient to point out the existence and similarity.\n\nSecond, the stated motivation in this paper is, I think, misguided. The authors argue for the need of an inference network, but they explicitly make clear that their goal is to train this network to enable reconstruction of an x given a z, rather than e.g. to learn a “good representation” (bearing in mind that what constitutes a good representation is strongly subject to debate). The authors do not provide any motivation for why reconstruction matters. At no point is an application or downstream task presented or mentioned in which good reconstructions are relevant. One might argue that choosing reconstruction quality is as arbitrary as pursuing improved sample quality (as is in vogue in GAN papers)  but there is substantial evidence that improved sample quality correlates with improved representation learning (mode dropping in GANs notwithstanding); the case is more complex for high-quality reconstructions. \n\nReconstruction could perhaps be motivated from the point of view of compression, but this paper makes no attempt to examine compression: rate-distortion tradeoffs are not considered, nor are any empirical metrics of compression ratio or likelihood such as bits/dim presented. Given that one can produce a model which achieves arbitrarily high-quality reconstructions by simply increasing the dimensionality of the bottleneck, I do not find reconstruction to be a compelling problem.\n\nOne might also argue that improved reconstruction capacity is indicative of better ability to fit the distribution (i.e. less mode dropping), but in the LIA setup the generator is trained as a standard StyleGAN with the only modification being the replacement of the MLP with Phi, so there’s no reason to believe that the implicit model defined by G has been meaningfully affected by the inclusion of the post-hoc trained encoder.\n\nIf the authors wish to pursue “reconstruction” as the primary motivation for learning an encoder, I would suggest they spend more time discussing compression and the latent bottleneck, as well as performing more detailed empirical evaluations (explained below). Basically, *why* does reconstruction matter? Alternatively, the authors could demonstrate the usefulness of their learned encoders for downstream tasks to indicate that the representations they learn are of high quality and useful.\n\nThird, the presentation of this paper needs a lot of work. There are typos and confusing statements throughout, as well as several instances of overstatement. \n\nThe key insight of this paper appears to be that “having an invertible network at the input to the generator makes it more amenable to post-hoc learning an encoder.” If I understand correctly, the only difference between this method and Encoded StyleGAN is that this paper uses an invertible model in place of the StyleGAN MLP. If this is the case, then the paper needs to (a) make clear the minimality of this difference and (b) devote substantial exposition to exploring the difference and why this is important (see my comments in the experimental section).\n\nPhrases like “the two-stage training successfully handles the existing issues of generative models” suggests that this method has solved all of the problems in generative modeling, which the authors have by no means demonstrated to be the case. \n\nCalling the two stage training “Stochasticity free” is incorrect—if you’re training the model as a GAN, then (1) you’ll be sampling z’s in the first place so it already has a latent distribution defined and (2) the end result of training will be much more variable than, say, training with a likelihood metric. There is a *ton* of stochasticity in the first stage of training!\n\nThe paper states several times that the “critical limitation” of adversarial models is their lack of an encoder. While implicit generative models do not generally require an encoder, there are plenty of methods (BiGAN by Donahue and ALI by DuMoulin, along with all the VAE-GAN hybrids) that jointly learn encoders, and much work on training an encoder post-hoc. These methods are acknowledged in the related work, but I think they should be taken into consideration when describing this “critical limitation.” While not having an encoder does indeed hinder or prevent the use of an implicit model for inference, I think stability, mode dropping, and mode collapse are more prominent issues with GANs. I think the authors might do better to say something to indicate that the challenge is to train a model which both has sharp, high-quality samples (as with GANs) which is still capable of inference or explicit likelihood evaluation (VAEs, etc). \n\nIn general, I found the description of the model itself to be confusing, and needed several thorough read-throughs just to understand what was going on: what was being frozen when, the fact that the model is just a GAN with a post-hoc trained encoder--I felt that there was a lot of obfuscatory language obscuring the actual simplicity of the method (which might arguably be its strength).\n\nWhile I would generally like the paper’s exposition to be improved, I understand that saying “write it better” is unhelpful so please see my individual notes at the end of this review for additional specific points. \n\nFourth, I found the empirical evaluations to be somewhat weak. To be clear, the results appear to be very good-the model retains the sample quality of  StyleGAN (at least as far as can be seen from the presented samples) while achieving noticeably higher-quality reconstructions on all the tested datasets. However, the metrics used for evaluation are limited—while at least MSE is presented, I would again stress that reconstruction is an odd metric to use when other factors like compression rates are not considered. While it is interesting to note that in this exact setup (mostly dim_z=512) LIA outperforms the baselines wrt the chosen metrics, a more thorough evaluation would, for instance, sweep the choice of dim_z, and ideally present NLL results (which I think are possible to evaluate given that LIA has a flow model even if it’s not trained using NLL, but I’m not 100% sure on this front and am open to counterarguments on this front).\n\nWhat’s more, the datasets chosen are all single-class datasets with a massive amount of data—as far as generative modeling is concerned, these are very datasets with a minimal amount of variation. This is critical because the LIA method relies on pre-training a GAN, meaning that it does nothing to deal with problems like mode dropping and mode collapse. While we may not see much mode dropping on these very easy datasets (where there are, essentially, very few modes), this is still a substantial problem in the general case, as can be seen by results on e.g. ImageNet. If your GAN collapses or drops modes then post-training the encoder is not likely to be able to recover them. This is also arguably a weakness of this paper relative to GLF which incorporates the encoder into the training loop of the decoder and is likely to be better at covering modes.\n\nAccordingly, I have substantial concerns that this method will not work well on datasets outside of these highly-constrained, nearly-unimodal, single-object, very-high-data datasets. While I would of course prefer to see results on something massively multimodal like ImageNet (training on a 100-class subset @ 64x64 resolution would be about 100,000 images and should be even less hardware intensive than the already performed experiments) I am aware of how cliché it is for reviewers to ask for imagenet results. Auxiliary experiments on CIFAR-100 or something with more than one class would go a long way towards allaying my concerns on this front.\n\nNext, no error bars are presented; this is simply inexcusable. Given that no hardware requirements are presented it is difficult to judge if expecting multiple runs is unreasonable but unless each run requires weeks of the authors’ full hardware capacity, there is no reason for the authors not to include error bars or expected variances on the numbers for as many of their experiments as possible.\n\nFurther, I found the experiment in 5.3 to be confusing and the results irrelevant to the argument made by the authors. First of all, what does it mean that the “gradients” are plotted in the figures relating to this experiment? Are these gradient norms for a layer in the network, and if so, what type? Is the loss in Figure 5c the training loss or the test loss? I also disagree that the VAE “gradients” are “more unstable” than the LIA “gradients,” they are simply noisier. I do not see why the increased gradient noise relative to LIA is indicative of the superiority of the method, but is instead entirely expected given that noise is explicitly injected into a standard VAE—I would argue that the change in gradient noise is simply the result of removing the stochasticity, but it says nothing as to whether or not the LIA method is better than the VAE method. Again, I agree that using an invertible network in some capacity is preferable to using the reparameterization trick, but I found this specific experiment to be distracting.\n\nI think the paper would do better to explore the importance of the invertible network relative to the exact same procedure but with the invertible network replaced with an arbitrary MLP of similar capacity. This appears to be what the encoded styleGAN model is, but I think it would do more to elucidate the key insights of this paper if the analysis was to focus more on this front. Why is it helpful to have an invertible phi in place of the StyleGAN MLP? What happens as the capacity of this portion of the model is increased or decreased? What is the form of the distribution output by Phi (maybe just some histogram visualizations along different dims?), and how does it compare to that of the typical MLP? What is the form of the distribution output by the encoder, and how does it differ from (a) the analytical latent distribution in the case of encoded styleGAN and (b) the empirical latent distribution of LIA? There’s quite a bit to explore there but this paper doesn’t dig very deep on this topic.\n\nI recognize that the amount of suggestions and changes I have listed are exceptionally large (more than I’ve personally ever written before, for sure), and I want to make it clear that I don’t expect the authors to address them all in the limited timespan of the rebuttal period. While this unfortunately may mean that there is simply not enough time for my concerns to be addressed, if this is the case then I hope these suggestions prove useful for publication in the next conference cycle, where this paper could be very strong. As it is, given the extent of my concerns, this paper is currently sitting at about a 4/10 in my mind. \n\nMinor notes:\n\n“In the parlance of probability,” page 2. I liked this alliteration a lot. This paragraph as a whole was quite clear and well written.\n\n“But it requires the dimension dx of the data space to be identical to the dimension dz of the latent space” Minor nitpick, but I would swap “dx of the data” with “dz of the latent space” in this sentence, to make it clear that the model’s latent dimensionality is constrained by the dimensionality of the data. As written it makes it sound like it’s the other way around.\n\n“The prior distribution can be exactly fitted from an unfolded feature space.” While flows have exact inference, saying that you can exactly fit the distribution of the encoder is arguably inaccurate unless you can show perfect generalization. Specifically, if you attain 0 training loss for the flow, do you also have 0 test loss (i.e. the NLL of the flow on  the encoder’s output for test samples is also minimized). \n\nFurthermore, the phrasing “unfolded feature space” (used elsewhere in the paper) is confusing and not in common usage—does this mean the output of the encoder, or some sort of Taylor expansion? It’s not immediately clear, and I would recommend the authors find a different way to express what they mean.\n\n“Therefore the training is deterministic” Training is not deterministic if the first stage of training involves training a GAN. You are still sampling from a preselected prior in this stage.\n\n“As shown in Figure 1f, we symmetrically embed an invertible neural network in the latent space of VAE, following the diagram of mapping process as…” This sentence is confusing. The term “embed” has a specific meaning in the literature: you might use word embeddings, or embed a sample in a space, but to “embed a [model] in a latent space” doesn’t make sense to me. I think the authors would do well to use more standard terminology, and to reconsider their description of the model to be more concise and clear.\n\n“Our primal goal is to faithfully reconstruct real images from the latent code.” Primal should be primary. I would also like to see this motivated better—why do you care to exactly reconstruct real images? Is there a downstream task where this is relevant or an intrinsic reason why we should care about being able to attain exact reconstructions?\n\n“indispensable discriminator.” Indispensible means “something you can’t get rid of,” whereas it would appear the discriminator is not used after  training (and is frozen after the first stage)—do the authors perhaps mean “dispensable” or “disposable”?\n\n“The interesting phenomenon is that the StyleGAN with encoder only does not succeed in recovering the target faces using the same training strategy as LIA, even though it is capable of generating photo-realistic faces in high quality due to the StyleGAN generator” This sentence is confusingly written and poorly supported. While I do agree that the LIA reconstructions are superior to the encoded styleGAN reconstructions, exactly what do the authors mean that LIA “recovers” the target faces while StyleGAN does not? The LIA reconstructions are not identity preserving—while most of the semantic features are the same, and the model does do a good job of picking up on unusual details such as hats, the facial identities are definitely not preserved (i.e. for every face in row 1 and row 2, I would say that the two faces belong to different people with similar features, but they are still definitely different people) .\n\n“This indicates that the invertible network plays the crucial role to make the LIA work” This statement is unsupported. There are a number of differences in training setup, and the authors, in this reviewer’s opinion, have not presented evidence to indicate that the use of the flow model is specifically responsible for this. Specifically, what would happen if during the decoder training stage, the invertible network was not employed? While I do believe that the inclusion of the invertible network is important, the authors should go to greater lengths to elucidate exactly what it does (see my comments above in the experimental section re. the shape of the distribution and how the encoder ends up matched to the decoder depending on what the actual latent distribution is from the POV of the generator).\n\n“To further evaluate LIA on the data with large variations” The choice of three single-category, single-subject datasets for evaluation is strictly at odds with this statement. These are highly constrained, clean datasets with tremendous amounts of data per class, which are substantially less difficult to model than e.g. ImageNet \n\n“They will be made them available for evaluation.” -> “These subsets will be made available for evaluation”\n\n“The experimental results on FFHQ and LSUN databases verify that the symmetric design of the invertible network and the two-stage training successfully handles the existing issues of generative models.” This statement is far too strong—saying a method “successfully handles the existing issues of generative models” suggests that this method is the end-all be-all and has solved the problem of generative modeling entirely. I would suggest the authors dial back the strength of this claim.\n\n“Table 2: FID accuracy of generative results.” What is FID accuracy? Do the authors just mean FID?\n\nSpecify hardware used and training times, at least in the appendix.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByljTrZ0qB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nthe authors of this paper propose a two-stage model consisting of a Wasserstein GAN and an autoencoder. The goal is to learn an encoding of data within the GAN framework. \nThe idea is inspired by the concept of VAEs. However, instead of maximising the ELBO, the authors propose to learn/represent the generative model by a Wasserstein GAN (first stage). Here, the architecture is crucial: an invertible MLP_1 is used to map from a standard normal prior into the feature space; then, a classical MLP_2 maps into the data space\nIn the second stage, MLP_2 serves as decoder to train an encoder. By combining the latter with MLP_1, data can be encoded into the latent space.\n\nThe authors experimentally show that their method leads to improved reconstructions compared to previous GAN-based methods.\n\n\nIn the following, a few concerns:\n\n1) The authors motivate their approach with the goal of \"encoding real-world samples\" without accepting disadvantages of VAEs such as \"imprecise inference\" or \" posterior collapse\". However, the comparison to VAEs is difficult since the latent representation of the data learned by VAEs differs from the one of LIA.\nFore example, in contrast to VAEs, where similar data is clustered in the latent space, this is not necessarily the case for GANs (e.g. Mukherjee et al., 2019). Experiments regarding the learned latent representation are missing in the paper (the interpolation experiment in the appendix might be a starting point). \n\n2) The authors use posterior collapse in VAEs as a main argument for introducing LIA. However, it is easy to avoid as stated in e.g. Bowman et al. (2015) or Sønderby et al. (2016), and hence this argument doesn't make a strong case for LIA. \n\n3) It is difficult to interpret the experiments in Fig. 5: the first 10 iterations might not be very significant.\n\n4) Experimental details are missing. I would appreciate to have model architectures in the appendix (even if the authors are going to make the source code publicly available).\n\n5) How were the accuracies of the generations in Tab. 2 computed?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkxbdqGb5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a new generative model called latently invertible autodecoder (LIA). The major aim is to conduct variational inference for VAE and encoding real-world samples for GAN. My understanding is that the authors tried to achieve this by decomposing the framework into a wasserstein GAN and a standard autoencoder. I believe this paper contains promising idea.\n\nThe experiment is very thorough, and the results show that LIA achieves good empirical performance.\n\nThe method is not presented in a user-friendly fashion, and the presentation can be improved.\n\nI must admit that I do not work on this field, and cannot judge this paper with more details.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A nice idea: the latent prior is replaced by a GAN.  A general agreement between all four reviewers to reject the submission, based on a not thorough enough description of the approach, and possibly not being novel.
Decision: Reject