Autoencoder Image Interpolation by Shaping the Latent Space

Autoencoders represent an effective approach for computing the underlying factors characterizing datasets of different types . The latent representation of autoencoders have been studied in the context of enabling interpolation between data points by decoding convex combinations of latent vectors . This interpolation , however , often leads to artifacts or produces unrealistic results during reconstruction . We argue that these incongruities are due to the structure of the latent space and because such naively interpolated latent vectors deviate from the data manifold . In this paper , we propose a regularization technique that shapes the latent representation to follow a manifold that is consistent with the training images and that drives the manifold to be smooth and locally convex . This regularization not only enables faithful interpolation between data points , as we show herein , but can also be used as a general regularization technique to avoid overfitting or to produce new samples for data augmentation . 1 INTRODUCTION . Given a set of data points , data interpolation or extrapolation aims at predicting novel data points between given samples ( interpolation ) or predicting novel data outside the sample range ( extrapolation ) . Faithful data interpolation between sampled data can be seen as a measure of the generalization capacity of a learning system ( Berthelot et al. , 2018 ) . In the context of computer vision and computer graphics , data interpolation may refer to generating novel views of an object between two given views or predicting in-between animated frames from key frames . Interpolation that produces novel views of a scene requires input such as the geometric and photometric parameters of existing objects , camera parameters and additional scene components , such as lighting and the reflective characteristics of nearby objects . Unfortunately , these characteristics are not always available or are difficult to extract in real-world scenarios . Thus , in such cases , we can apply data-driven interpolation that is deduced based on a sampled dataset drawn from the scene taken under various acquisition parameters . The task of data interpolation is to extract new samples ( possibly continuous ) between known data samples . Clearly , linear interpolation between two images in the input ( image ) domain does not work as it produces a cross-dissolve effect between the intensities of the two images . Adopting the manifold view of data ( Goodfellow et al. , 2016 ; Verma et al. , 2018 ; Bengio et al. , 2013 ) , this task can be seen as sampling new data points along the geodesic path between the given points . The problem is that this manifold is unknown in advance and one has to approximate it from the given data . Alternatively , adopting the probabilistic perspective , interpolation can be viewed as drawing samples from highly probable areas in the data space . One fascinating property of unsupervised learning is the network ’ s ability to reveal the underlying factors controlling a given dataset . Autoencoders ( Doersch , 2016 ; Kingma & Welling , 2013 ; Goodfellow et al. , 2016 ; Kramer , 1991 ; Vincent et al. , 2010 ) represent an effective approach for exposing these factors . Researchers have demonstrated the ability to interpolate between data points by decoding a convex sum of latent vectors ( Shu et al. , 2018 ; Mathieu et al. , 2016 ) ; however , this interpolation often incorporates visible artifacts during reconstruction . To illustrate the problem , consider the following example : A scene is composed of a vertical pole at the center of a flat plane ( Figure 1-left ) . A single light source illuminates the scene and accordingly , the pole projects a shadow onto the plane . The position of the light source can vary along the upper hemisphere . Hence , the underlying parameters controlling the generated scene are ( θ , φ ) , the elevation and azimuth , respectively . The interaction between the light and the pole produces a cast shadow whose direction and length are determined by the light direction . A set of images of this scene is acquired from a fixed viewing position ( from above ) with various lighting directions . Our goal in this example is to train a model that is capable of interpolating between two given images . Figure 1 , top row , depicts a set of interpolated images , between the source image ( left image ) and the target image ( right image ) , where the interpolation is performed in the input domain . As illustrated , the interpolation is not natural as it produces cross-dissolve effects in image intensities . Training a standard autoencoder and applying linear interpolation in its latent space generates images that are much more realistic ( Figure 1 , bottom row ) . Nevertheless , this interpolation is not perfect as visible artifacts occur in the interpolated images . The source of these artifacts can be investigated by closely inspecting the 2D manifold embedded in the latent space . Figure 2 shows two manifolds embedded in latent spaces , one with data embedded in 2D latent space ( left plot ) and one with data embedded in 3D latent space ( 2nd plot from the left ) . In both cases , the manifolds are 2D and are generated using vanilla autoencoders . The grid lines represent the ( θ , φ ) parameterization . It can be seen that the encoders produce non-smooth and non-convex surfaces in 2D as well as in 3D . Thus , linear interpolation between two data points inevitably produces in-between points outside of the manifold . In practice , the decoded images of such points are unpredictable and may produce non-realistic artifacts . This issue is demonstrated in the two right images in Figure 2 . When the interpolated point is on the manifold ( an empty circle denoted ‘ A ’ ) , a faithful image is generated by the decoder ( 2nd image from the right ) . When the interpolated point departs from the manifold ( the circle denoted ‘ B ’ ) , the resulting image is unpredictable ( right image ) . In this paper , we argue that the common statistical view of autoencoders is not appropriate when dealing with data that have been generated from continuous factors . Alternatively , the manifold structure of continuous data must be considered , taking into account the geometry and shape of the manifold . Accordingly , we propose a new interpolation regularization mechanism consisting of an adversarial loss , a cycle-consistency loss , and a smoothness loss . The adversarial loss drives the interpolated points to look realistic as it is optimized against a discriminator that learns to tell apart real from interpolated data points . The cycle-consistency and the smoothness losses encourage smooth interpolations between data points . We show empirically that these combined losses prompt the autoencoder to produce realistic and smooth interpolations while providing a convex latent manifold with a bijective mapping between the input and the latent manifolds . This regularization mechanism not only enables faithful interpolation between data points , but can also be used as a general regularization technique to avoid overfitting or to produce new samples for data augmentation , as suggested , among others , by Zhang et al . ( 2018 ) . To conclude , the contributions of the papers are : I . We define what constitutes an admissible interpolation between two data points on a continuous manifold . In particular we added the cycle-consistency and the smoothness terms and show their importance in generating admissible interpolations . II . We empirically demonstrate how the combination of the four losses ; the reconstruction , adversarial , cycle-consistency and the smoothness losses , contribute to admissible interpolations and produce state of the art results . 2 MANIFOLD DATA INTERPOLATION . Before presenting the proposed approach we would like to define what constitutes a proper interpolation between two data points . There are many possible paths between two points on the manifold . Even if we require the interpolations to be on a geodesic path , there might be infinitely many such paths between two points . Therefore , we relax the geodesic requirement and define less restrictive conditions . Formally , assume we are given a dataset sampled from a target domain X . We are interested in interpolating between two data points xi and xj from X . Let the interpolated points be x̂i→j ( α ) for α ∈ [ 0 , 1 ] and let P ( x ) be the probability that a data point x belongs to X . We define an interpolation to be an admissible interpolation if x̂i→j ( α ) satisfies the following conditions : 1 . Boundary conditions : x̂i→j ( 0 ) = xi and x̂i→j ( 1 ) = xj . 2 . Monotonicity : We require that under some defined distance on the manifold d ( x , x′ ) , the interpolated points will depart from xi and approach xj , as the parameterization α goes from 0 to 1 . Namely , ∀α′ ≥ α , d ( x̂i→j ( α ) , xi ) ≤ d ( x̂i→j ( α′ ) , xi ) and similarly : d ( x̂i→j ( α ′ ) , xj ) ≤ d ( x̂i→j ( α ) , xj ) 3 . Smoothness : The interpolation function x̂i→j ( α ) is Lipschitz continuous with a constant K : ‖x̂i→j ( α ) , x̂i→j ( α+ t ) ‖ ≤ K|t| 4 . Credibility : ∀α ∈ [ 0 , 1 ] We require that it is highly probable that interpolated images , x̂i→j ( α ) belong to X . Namely , P ( x̂i→j ( α ) ) ≥ 1− β , for some constant β ≥ 0 2.1 PROPOSED APPROACH . Following the above definitions for an admissible interpolation , we propose a new approach , called Autoencoder Adversarial Interpolation ( AEAI ) , which shapes the latent space according to the above requirements . The general architecture comprises a standard autoencoder with an encoder , z = f ( x ) , and a decoder x̂ = g ( z ) . We also train a discriminator D ( x ) to differentiate between real and interpolated data points . For pairs of input data points xi , xj , we linearly interpolate between them in the latent space : zi→j ( α ) = ( 1−α ) zi +αzj , where α ∈ [ 0 , 1 ] . The first requirement is that we would like x̂i→j ( α ) = g ( zi→j ( α ) ) to look real and fool the discriminator D. Additionally , we add a cycle-consistency loss that encourages the latent representation of x̂i→j ( α ) to be mapped back into zi→j ( α ) again ; namely , ẑi→j ( α ) = f ( g ( zi→j ( α ) ) ) should be similar to zi→j ( α ) . Finally , we add a smoothness loss that drives the linear parameterization to form a smooth interpolation . Putting everything together we define the loss Li→j between pairs xi and xj as follows : Li→j = Li→jR + λAL i→j A + λCL i→j C + λSL i→j S ( 1 ) where LR , LA , LC , LS are the reconstruction , adversarial , cycle , and smoothness losses , respectively . The first term LR is a standard reconstruction loss and is calculated for the two endpoints xi and xj : Li→jR = L ( xi , x̂i ) + L ( xj , x̂j ) where L ( · , · ) is some loss function between the two images ( we used the L2 distance or the perceptual loss ( Johnson et al. , 2016 ) ) and x̂k = g ( f ( xk ) ) . LA is the adversarial loss that encourages the network to fool the discriminator so that interpolated images are indistinguishable from the data in the target domain X : Li→jA = M∑ n=0 − logD ( x̂i→j ( n/M ) ) where D ( x ) ∈ [ 0 , 1 ] is a discriminator trying to distinguish between images in the training set and the interpolated images . The cycle-consistency loss LC encourages the encoder and the decoder to produce a bijective mapping : Li→jC = M∑ n=0 ‖zi→j ( n/M ) − ẑi→j ( n/M ) ‖2 where ẑi→j ( α ) = f ( g ( zi→j ( α ) ) ) . The last term LS is the smoothness loss encouraging x̂ ( α ) to produce smoothly varying interpolated points between xi and xj : Li→jS = M∑ n=0 ∥∥∥∥∂x̂i→j ( α ) ∂α ∥∥∥∥2 α=n/M where ‖∂x̂i→j ( α ) /∂α‖2α=α0 means that the derivative it taken at α = α0 . The three losses LA , LC and LS are accumulated over M + 1 sampled points , from α = 0/M up to α =M/M . Finally , we sum the Li→j loss over many sampled pairs . In the next section , we explain the motivation for each of the four losses comprising Li→j in Equation 1 and describe how these losses promote the four conditions defined in Section 2 .

Summary: This paper introduces several autoencoder (AE) regularization terms that aim at reproducing continuous realistic deformation by interpolating latent codes of images. The authors assume there is a continuous process generating the data and introduce three novel loss terms (in addition to the standard AE reconstruction loss). The first term is a GAN loss for decoded interpolated latents $\hat{x}(\alpha)$ where this terms makes sure the interpolated latents are decoded to images similar to the train images. The second term is called cycle-consistency and enforce injectivity of the decoder. The last term is enforcing smoothness of the decoded image as a function of the interpolated latent. Combining these three losses with the original loss leads to natural interpolations of latents that enjoy both smoothness and realism.  The method is tested on a synthetic "pole shadow" example, and COIL 100. The method seem to improve upon several baselines on these datasets.