LIA: Latently Invertible Autoencoder with Adversarial Learning

1 INTRODUCTION . Deep generative models play a more and more important role in cracking challenges in computer vision as well as in other disciplines , such as high-quality image generation ( Isola et al. , 2017 ; Zhu et al. , 2017 ; Karras et al. , 2018a ; b ; Brock et al. , 2018 ) , text-to-speech transformation ( van den Oord et al. , 2016a ; 2017 ) , information retrieval ( Wang et al. , 2017 ) , 3D rendering ( Wu et al. , 2016 ; Eslami et al. , 2018 ) , and signal-to-image acquisition ( Zhu et al. , 2018 ) . Overall , the generative models fall into four categories : autoencoder and its most important variant of Variational AutoEncoder ( VAE ) ( Kingma & Welling , 2013 ) , auto-regressive models ( van den Oord et al. , 2016b ; a ) , Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , and normalizing flows ( NF ) ( Tabak & Vanden-Eijnden , 2010 ; Tabak & Turner , 2013 ; Rezende & Mohamed , 2015 ) . Here we compare these models through the perspective of data dimensionality reduction and reconstruction . To be formal , let x be a data point in the dx-dimensional observable space Rdx and y be its corresponding low-dimensional representation in the feature space Rdy . The general formulation of dimensionality reduction is f : Rdx → Rdy , x 7→ y = f ( x ) , where f ( · ) is the mapping function and dy dx . The manifold learning aims at requiring f under various constraints on y ( Tenenbaum1 et al. , 2000 ; Roweis & Saul , 2000 ) . However , the sparsity of data points in high-dimensional space often leads to model overfitting , thus necessitating research on opposite mapping from y to x , i.e . g : Rdy → Rdx , y 7→ x = g ( y ) , where g ( · ) is the opposite mapping function with respect to f ( · ) , to reconstruct the data . In general , the role of g ( · ) is a regularizer to f ( · ) or a generator to produce more data . The autoencoder is 1We will make source code of LIA publicly available of mapping x f7→ y g7→ x̃ . A common assumption in autoencoder is that the variables in lowdimensional space are usually sampled from a prior distribution P ( z ; θ ) such as uniform or Gaussian . To differentiate from y , we let z represent the low-dimensional vector following the prior distribution . Thus we can write g : Rdz → Rdx , z 7→ x = g ( z ) , z ∼ P ( z ; θ ) . It is crucial to establish such dual maps z = f ( x ) and x = g ( z ) . In the parlance of probability , the process of x 7→ z = f ( x ) is called inference , and the other procedure of z 7→ x = g ( z ) is called sampling or generation . VAE is capable of carrying out inference and generation in one framework by two collaborative functional modules . However , it is known that in many cases VAEs are only able to generate blurry images due to the imprecise variational inference . To see this , we write the approximation of the marginal log-likelihood log p ( x ) = log ∫ p ( x|z ) p ( z ) dz ≥ −KL [ q ( z|x ) ||p ( z ) ] + Eq [ log p ( x|z ) ] , ( 1 ) where KL [ q ( z|x ) ||p ( z ) ] is the Kullback-Leibler divergence with respect to posterior probability q ( z|x ) and prior p ( z ) . This lower-bound log-likelihood usually produces imprecise inference . Furthermore , the posterior collapse frequently occurs when using more sophisticated decoder models ( Bowman et al. , 2015 ; Kingma et al. , 2016 ) . These two issues greatly limit the generation capability of the VAE . On the other hand , GAN is able to achieve photo-realistic generation results ( Karras et al. , 2018a ; b ) . However , its critical limitation is the absence of the encoder f ( x ) for carrying inference on real images . Effort has been made on learning an encoder for GAN under the framework of VAE , however the previous two issues of learning VAE still exist . Normalizing flows can perform the exact inference and generation with one architecture by virtue of invertible networks ( Kingma & Dhariwal , 2018 ) . But it requires the dimension dx of the data space to be identical to the dimension dz of the latent space , thus posing computational issues due to high complexity of learning deep flows and computing the Jacobian matrices . Inspired by recent success of GANs ( Karras et al. , 2018a ; b ) and normalizing flows ( Kingma et al. , 2016 ; Kingma & Dhariwal , 2018 ) , we develop a new model called Latently Invertible Autoencoder ( LIA ) . LIA utilizes an invertible network to bridge the encoder and the decoder of VAE in a symmetric manner . We summarize its key advantages as follows : • The symmetric design of the invertible network brings two benefits . The prior distribution can be exactly fitted from an unfolded feature space , thus significantly easing the inference problem . Besides , since the latent space is detached , the autoencoder can be trained without variational optimization thus there is no approximation here . • The two-stage adversarial learning decomposes the LIA framework into a Wasserstein GAN ( only a prior needed ) and a standard autoencoder without stochastic variables . Therefore the training is deterministic2 , implying that the model will be not affected by the posterior collapse when the decoder is more complex or followed by additional losses such as the adversarial loss and the perceptual loss . • We compare LIA with state-of-the-art generative models on inference and generation/recon- struction . The experimental results on FFHQ and LSUN datasets show the LIA achieves superior performance on inference and generation . 2 LATENTLY INVERTIBLE AUTOENCODER . The neural architecture of LIA is designed such that the data distribution can be progressively unrolled from a complex or manifold-valued one to a given simple prior . We now describe the detail as follows . 2.1 NEURAL ARCHITECTURE OF LIA . Framework of LIA is based on the classic VAE and the realization of normalizing flow . As shown in Figure 1f , we symmetrically embed an invertible neural network in the latent space of VAE , following the diagram of mapping process as 2The “ deterministic ” and “ stochasticity-free ” in this paper refer to the second-stage training , i.e . the learning of the inference process . 1 encoder︷ ︸︸ ︷ x f7−→ y ↔ y φ17→ z1 · · · zk−1 φk7→︸ ︷︷ ︸ invertible network zk decoder︷ ︸︸ ︷ φ−1k7→ zk−1 · · · z1 φ−117→ y︸ ︷︷ ︸ invertible network ↔ y g7−→ x̃ , ( 2 ) where φ = φ1 ◦ · · · ◦ φk denotes the deep composite mapping of normalizing flow with depth k. LIA first performs nonlinear dimensionality reduction on the input data x and transform them into the low-dimensional feature space Rdy . The role of f ( x ) for LIA can be regarded to unfold the underlying data manifold , as illustrated in Figures 1a and 1b . Therefore , the Euclidean operations such as linear interpolation and vector arithmetic are reliable in this feature space . Then we establish an invertible mapping φ ( y ) from the feature y to the latent variable z , as opposed to the VAEs that directly map original data to latent variables . The feature y can be exactly recovered via the invertibility of φ from z , which is the advantage of using invertible networks . The recovered feature y is then fed into a partial decoder g ( y ) to generate the corresponding data x̃ . If the maps φ and φ−1 of the invertible network are bypassed , it comes back as a standard autoencoder , i.e . x f7→ y g7→ x̃ . In general , any invertible networks are applicable in the LIA framework . We find in practice that a simple invertible network is sufficiently capable of constructing the mapping from the feature space Rdy to the latent space Rdz . Let x = [ xt ; xb ] and z = [ zt ; zb ] be the forms of the top and bottom fractions of x and z , respectively . Then the invertible network ( Dinh et al. , 2015 ) can be built as follows , zt = xt , zb = xb + τ ( xt ) , ( 3 ) xt = zt , xb = zb − τ ( zt ) , ( 4 ) where τ is the transformation that can be an arbitrary differentiable function . Alternatively , one can attempt to exploit the complex invertible network with affine coupling mappings for more challenging tasks ( Dinh et al. , 2017 ; Kingma & Dhariwal , 2018 ) . As conducted in ( Dinh et al. , 2015 ) , we set τ as a multi-layer perceptron with the leaky ReLU activation . 2.2 RECONSTRUCTION LOSS AND ADVERSARIAL LEARNING . To guarantee the precise reconstruction x̃ , the conventional way by ( variational ) autoencoders is to use the distance ‖x − x̃‖ or the cross entropy directly between x and x̃ . Here , we utilize the perceptual loss that is proven to be more robust to variations of image details ( Johnson et al. , 2016 ) . Let denote the feature extractor , e.g . VGG ( Simonyan & Zisserman , 2014 ) . Then we can write the loss L ( , x , x̃ ) = ‖ ( x ) − ( x̃ ) ‖2 . ( 5 ) It suffices to emphasize that the functionality of here is in essence to produce the representations of the input x and the output x̃ . The acquisition of is fairly flexible . It can be attained by supervised or 1 1 unsupervised learning , meaning that can be trained with class labels or without class labels ( van den Oord et al. , 2018 ) . The norm-based reconstruction constraints usually incur the blurry image generation in autoencoderlike architectures . This problem can be handled via the adversarial learning ( Goodfellow et al. , 2014 ) . To do so , a discriminator c is employed to balance the loss of the comparison between x and x̃ . Using the Wasserstein GAN ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) , we can write the optimization objective as L ( c , x , x̃ ) = E x̃∼Px̃ [ c ( x̃ ) ] − E x∼Px [ c ( x ) ] + γ 2 E x∼Px [ ‖∇xc ( x ) ‖2 ` 2 ] , ( 6 ) where Px and Px̃ denote the probability distributions of the real data and the generated data , respectively . γ is the hyper-parameter of the regularization . The R1 regularizer is formulated in ( Mescheder et al. , 2018 ) , which is proven more stable for training convergence . 3 TWO-STAGE STOCHASTICITY-FREE TRAINING . Training deep models usually follows an end-to-end fashion for the whole architecture . To backpropagate gradients through random variables for VAE , the reparameterization trick is harnessed ( Kingma & Welling , 2013 ) , i.e . z = µ + σ ∗ N ( 0,1 ) , where µ is the mean and σ the standard deviation . The regularization of coupling the prior and the posterior is the KL divergence used to optimize the parameters of the encoder by backpropagation . For our framework , however , we find that this endto-end learning strategy can not lead the algorithm to converge to a satisfactory optima . To proceed , we propose a scheme of two-stage stochasticity-free training , which decomposes the framework into two parts that can be well trained end-to-end respectively , as shown in Figure 2 . At the first step , the decoder of LIA is trained using adversarial learning with invertible network to acquire the ability of high-quality generation . At the second step , the invertible network that connects feature space and latent space is detached from the architecture , reducing the framework to a standard autoencoder without variational inference . Thus this two-stage design prevents the posterior collapse issue while facilitates the decoder with more complex structure as well as adversarial loss for high-resolution image generation .

Summary: The work in the paper is interesting and seems to show some empirical progress on image generation. However, the work in this paper can be seen as adding the losses from GANs and VAEs together to help learn a better generative model, which is not very novel. The invertible part is help for training as well. Still more detail about how the method works would be very helpful. While there is lots of math in the paper it is difficult for the reader to follow and often not well motivated why these choices are made. For example, the optimization is split into two different updates steps because they can't be combined mathematically. Yet, performing two different types of update steps can often favour whichever is easier and the other is noise. More details on how this process was made successful are important to discuss in the paper.