Autoencoder Image Interpolation by Shaping the Latent Space

Autoencoders represent an effective approach for computing the underlying factors characterizing datasets of different types . The latent representation of autoencoders have been studied in the context of enabling interpolation between data points by decoding convex combinations of latent vectors . This interpolation , however , often leads to artifacts or produces unrealistic results during reconstruction . We argue that these incongruities are due to the structure of the latent space and because such naively interpolated latent vectors deviate from the data manifold . In this paper , we propose a regularization technique that shapes the latent representation to follow a manifold that is consistent with the training images and that drives the manifold to be smooth and locally convex . This regularization not only enables faithful interpolation between data points , as we show herein , but can also be used as a general regularization technique to avoid overfitting or to produce new samples for data augmentation . 1 INTRODUCTION . Given a set of data points , data interpolation or extrapolation aims at predicting novel data points between given samples ( interpolation ) or predicting novel data outside the sample range ( extrapolation ) . Faithful data interpolation between sampled data can be seen as a measure of the generalization capacity of a learning system ( Berthelot et al. , 2018 ) . In the context of computer vision and computer graphics , data interpolation may refer to generating novel views of an object between two given views or predicting in-between animated frames from key frames . Interpolation that produces novel views of a scene requires input such as the geometric and photometric parameters of existing objects , camera parameters and additional scene components , such as lighting and the reflective characteristics of nearby objects . Unfortunately , these characteristics are not always available or are difficult to extract in real-world scenarios . Thus , in such cases , we can apply data-driven interpolation that is deduced based on a sampled dataset drawn from the scene taken under various acquisition parameters . The task of data interpolation is to extract new samples ( possibly continuous ) between known data samples . Clearly , linear interpolation between two images in the input ( image ) domain does not work as it produces a cross-dissolve effect between the intensities of the two images . Adopting the manifold view of data ( Goodfellow et al. , 2016 ; Verma et al. , 2018 ; Bengio et al. , 2013 ) , this task can be seen as sampling new data points along the geodesic path between the given points . The problem is that this manifold is unknown in advance and one has to approximate it from the given data . Alternatively , adopting the probabilistic perspective , interpolation can be viewed as drawing samples from highly probable areas in the data space . One fascinating property of unsupervised learning is the network ’ s ability to reveal the underlying factors controlling a given dataset . Autoencoders ( Doersch , 2016 ; Kingma & Welling , 2013 ; Goodfellow et al. , 2016 ; Kramer , 1991 ; Vincent et al. , 2010 ) represent an effective approach for exposing these factors . Researchers have demonstrated the ability to interpolate between data points by decoding a convex sum of latent vectors ( Shu et al. , 2018 ; Mathieu et al. , 2016 ) ; however , this interpolation often incorporates visible artifacts during reconstruction . To illustrate the problem , consider the following example : A scene is composed of a vertical pole at the center of a flat plane ( Figure 1-left ) . A single light source illuminates the scene and accordingly , the pole projects a shadow onto the plane . The position of the light source can vary along the upper hemisphere . Hence , the underlying parameters controlling the generated scene are ( θ , φ ) , the elevation and azimuth , respectively . The interaction between the light and the pole produces a cast shadow whose direction and length are determined by the light direction . A set of images of this scene is acquired from a fixed viewing position ( from above ) with various lighting directions . Our goal in this example is to train a model that is capable of interpolating between two given images . Figure 1 , top row , depicts a set of interpolated images , between the source image ( left image ) and the target image ( right image ) , where the interpolation is performed in the input domain . As illustrated , the interpolation is not natural as it produces cross-dissolve effects in image intensities . Training a standard autoencoder and applying linear interpolation in its latent space generates images that are much more realistic ( Figure 1 , bottom row ) . Nevertheless , this interpolation is not perfect as visible artifacts occur in the interpolated images . The source of these artifacts can be investigated by closely inspecting the 2D manifold embedded in the latent space . Figure 2 shows two manifolds embedded in latent spaces , one with data embedded in 2D latent space ( left plot ) and one with data embedded in 3D latent space ( 2nd plot from the left ) . In both cases , the manifolds are 2D and are generated using vanilla autoencoders . The grid lines represent the ( θ , φ ) parameterization . It can be seen that the encoders produce non-smooth and non-convex surfaces in 2D as well as in 3D . Thus , linear interpolation between two data points inevitably produces in-between points outside of the manifold . In practice , the decoded images of such points are unpredictable and may produce non-realistic artifacts . This issue is demonstrated in the two right images in Figure 2 . When the interpolated point is on the manifold ( an empty circle denoted ‘ A ’ ) , a faithful image is generated by the decoder ( 2nd image from the right ) . When the interpolated point departs from the manifold ( the circle denoted ‘ B ’ ) , the resulting image is unpredictable ( right image ) . In this paper , we argue that the common statistical view of autoencoders is not appropriate when dealing with data that have been generated from continuous factors . Alternatively , the manifold structure of continuous data must be considered , taking into account the geometry and shape of the manifold . Accordingly , we propose a new interpolation regularization mechanism consisting of an adversarial loss , a cycle-consistency loss , and a smoothness loss . The adversarial loss drives the interpolated points to look realistic as it is optimized against a discriminator that learns to tell apart real from interpolated data points . The cycle-consistency and the smoothness losses encourage smooth interpolations between data points . We show empirically that these combined losses prompt the autoencoder to produce realistic and smooth interpolations while providing a convex latent manifold with a bijective mapping between the input and the latent manifolds . This regularization mechanism not only enables faithful interpolation between data points , but can also be used as a general regularization technique to avoid overfitting or to produce new samples for data augmentation , as suggested , among others , by Zhang et al . ( 2018 ) . To conclude , the contributions of the papers are : I . We define what constitutes an admissible interpolation between two data points on a continuous manifold . In particular we added the cycle-consistency and the smoothness terms and show their importance in generating admissible interpolations . II . We empirically demonstrate how the combination of the four losses ; the reconstruction , adversarial , cycle-consistency and the smoothness losses , contribute to admissible interpolations and produce state of the art results . 2 MANIFOLD DATA INTERPOLATION . Before presenting the proposed approach we would like to define what constitutes a proper interpolation between two data points . There are many possible paths between two points on the manifold . Even if we require the interpolations to be on a geodesic path , there might be infinitely many such paths between two points . Therefore , we relax the geodesic requirement and define less restrictive conditions . Formally , assume we are given a dataset sampled from a target domain X . We are interested in interpolating between two data points xi and xj from X . Let the interpolated points be x̂i→j ( α ) for α ∈ [ 0 , 1 ] and let P ( x ) be the probability that a data point x belongs to X . We define an interpolation to be an admissible interpolation if x̂i→j ( α ) satisfies the following conditions : 1 . Boundary conditions : x̂i→j ( 0 ) = xi and x̂i→j ( 1 ) = xj . 2 . Monotonicity : We require that under some defined distance on the manifold d ( x , x′ ) , the interpolated points will depart from xi and approach xj , as the parameterization α goes from 0 to 1 . Namely , ∀α′ ≥ α , d ( x̂i→j ( α ) , xi ) ≤ d ( x̂i→j ( α′ ) , xi ) and similarly : d ( x̂i→j ( α ′ ) , xj ) ≤ d ( x̂i→j ( α ) , xj ) 3 . Smoothness : The interpolation function x̂i→j ( α ) is Lipschitz continuous with a constant K : ‖x̂i→j ( α ) , x̂i→j ( α+ t ) ‖ ≤ K|t| 4 . Credibility : ∀α ∈ [ 0 , 1 ] We require that it is highly probable that interpolated images , x̂i→j ( α ) belong to X . Namely , P ( x̂i→j ( α ) ) ≥ 1− β , for some constant β ≥ 0 2.1 PROPOSED APPROACH . Following the above definitions for an admissible interpolation , we propose a new approach , called Autoencoder Adversarial Interpolation ( AEAI ) , which shapes the latent space according to the above requirements . The general architecture comprises a standard autoencoder with an encoder , z = f ( x ) , and a decoder x̂ = g ( z ) . We also train a discriminator D ( x ) to differentiate between real and interpolated data points . For pairs of input data points xi , xj , we linearly interpolate between them in the latent space : zi→j ( α ) = ( 1−α ) zi +αzj , where α ∈ [ 0 , 1 ] . The first requirement is that we would like x̂i→j ( α ) = g ( zi→j ( α ) ) to look real and fool the discriminator D. Additionally , we add a cycle-consistency loss that encourages the latent representation of x̂i→j ( α ) to be mapped back into zi→j ( α ) again ; namely , ẑi→j ( α ) = f ( g ( zi→j ( α ) ) ) should be similar to zi→j ( α ) . Finally , we add a smoothness loss that drives the linear parameterization to form a smooth interpolation . Putting everything together we define the loss Li→j between pairs xi and xj as follows : Li→j = Li→jR + λAL i→j A + λCL i→j C + λSL i→j S ( 1 ) where LR , LA , LC , LS are the reconstruction , adversarial , cycle , and smoothness losses , respectively . The first term LR is a standard reconstruction loss and is calculated for the two endpoints xi and xj : Li→jR = L ( xi , x̂i ) + L ( xj , x̂j ) where L ( · , · ) is some loss function between the two images ( we used the L2 distance or the perceptual loss ( Johnson et al. , 2016 ) ) and x̂k = g ( f ( xk ) ) . LA is the adversarial loss that encourages the network to fool the discriminator so that interpolated images are indistinguishable from the data in the target domain X : Li→jA = M∑ n=0 − logD ( x̂i→j ( n/M ) ) where D ( x ) ∈ [ 0 , 1 ] is a discriminator trying to distinguish between images in the training set and the interpolated images . The cycle-consistency loss LC encourages the encoder and the decoder to produce a bijective mapping : Li→jC = M∑ n=0 ‖zi→j ( n/M ) − ẑi→j ( n/M ) ‖2 where ẑi→j ( α ) = f ( g ( zi→j ( α ) ) ) . The last term LS is the smoothness loss encouraging x̂ ( α ) to produce smoothly varying interpolated points between xi and xj : Li→jS = M∑ n=0 ∥∥∥∥∂x̂i→j ( α ) ∂α ∥∥∥∥2 α=n/M where ‖∂x̂i→j ( α ) /∂α‖2α=α0 means that the derivative it taken at α = α0 . The three losses LA , LC and LS are accumulated over M + 1 sampled points , from α = 0/M up to α =M/M . Finally , we sum the Li→j loss over many sampled pairs . In the next section , we explain the motivation for each of the four losses comprising Li→j in Equation 1 and describe how these losses promote the four conditions defined in Section 2 .

Summary: This paper focused on developing a new regularization technique for autoencoders, which shapes the latent representation to follow a manifold that is consistent with the training images and that drives the manifold to be smooth and locally convex. The authors suggest that the manifold structure of continuous data must be considered to include the geometry and shape of the manifold. The new interpolation regularization mechanism consists of an adversarial loss, a cycle-consistency loss, and a smoothness loss. So the architecture of the proposed model includes a standard autoencoder, a discriminator and the loss mentioned above.