Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer

1 INTRODUCTION . One goal of unsupervised learning is to uncover the underlying structure of a dataset without using explicit labels . A common architecture used for this purpose is the autoencoder , which learns to map datapoints to a latent code from which the data can be recovered with minimal information loss . Typically , the latent code is lower dimensional than the data , which indicates that autoencoders can perform some form of dimensionality reduction . For certain architectures , the latent codes have been shown to disentangle important factors of variation in the dataset which makes such models useful for representation learning ( Chen et al. , 2016a ; Higgins et al. , 2017 ) . In the past , they were also used for pre-training other networks by being trained on unlabeled data and then being stacked to initialize a deep network ( Bengio et al. , 2007 ; Vincent et al. , 2010 ) . More recently , it was shown that imposing a prior on the latent space allows autoencoders to be used for probabilistic or generative modeling ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ; Makhzani et al. , 2015 ) . In some cases , autoencoders have shown the ability to interpolate . Specifically , by mixing codes in latent space and decoding the result , the autoencoder can produce a semantically meaningful combination of the corresponding datapoints . Interpolation has been frequently reported as a qualitative experimental result in studies about autoencoders ( Dumoulin et al. , 2016 ; Bowman et al. , 2015 ; Roberts et al. , 2018 ; Mescheder et al. , 2017 ; Mathieu et al. , 2016 ; Ha & Eck , 2018 ) and latent-variable generative models in general ( Dinh et al. , 2016 ; Radford et al. , 2015 ; van den Oord et al. , 2016 ) . The ability to interpolate can be useful in its own right e.g . for creative applications ( Carter & Nielsen , 2017 ) . However , it also indicates that the autoencoder can “ extrapolate ” beyond the training data and has learned a latent space with a particular structure . Specifically , if interpolating between two points in latent space produces a smooth semantic warping in data space , this suggests that nearby points in latent space are semantically similar . A visualization of this idea is shown in fig . 1 , where a smooth ∗Equal contribution . interpolation between a “ 2 ” and a “ 9 ” suggests that the 2 is surrounded by semantically similar points , i.e . other 2s . This property may suggest that an autoencoder which interpolates well could also provide a good learned representation for downstream tasks because similar points are clustered . If the interpolation is not smooth , there may be “ discontinuities ” in latent space which could result in the representation being less useful as a learned feature . This connection between interpolation and a “ flat ” data manifold has been explored in the context of unsupervised representation learning ( Bengio et al. , 2013b ) and regularization ( Verma et al. , 2018 ) . Given the widespread use of interpolation as a qualitative measure of autoencoder performance , we believe additional investigation into the connection between interpolation and representation learning is warranted . Our goal in this paper is threefold : First , we introduce a regularization strategy with the specific goal of encouraging improved interpolations in autoencoders ( section 2 ) ; second , we develop a synthetic benchmark where the slippery concept of a “ semantically meaningful interpolation ” is quantitatively measurable ( section 3.1 ) and evaluate common autoencoders on this task ( section 3.2 ) ; and third , we confirm the intuition that good interpolation can result in a useful representation by showing that the improved interpolation ability produced by our regularizer elicits improved representation learning performance on downstream tasks ( section 4 ) . We also make our codebase available1 which provides a unified implementation of many common autoencoders including our proposed regularizer . 2 AN ADVERSARIAL REGULARIZER FOR IMPROVING INTERPOLATIONS . Autoencoders , also called auto-associators ( Bourlard & Kamp , 1988 ) , consist of the following structure : First , an input x ∈ Rdx is passed through an “ encoder ” z = fθ ( x ) parametrized by θ to obtain a latent code z ∈ Rdz . The latent code is then passed through a “ decoder ” x̂ = gφ ( z ) parametrized by φ to produce an approximate reconstruction x̂ ∈ Rdx of the input x . We consider the case where fθ and gφ are implemented as multi-layer neural networks . The encoder and decoder are trained simultaneously ( i.e . with respect to θ and φ ) to minimize some notion of distance between the input x and the output x̂ , for example the squared L2 distance ‖x− x̂‖2 . Interpolating using an autoencoder describes the process of using the decoder gφ to decode a mixture of two latent codes . Typically , the latent codes are combined via a convex combination , so that interpolation amounts to computing x̂α = gφ ( αz1+ ( 1−α ) z2 ) for some α ∈ [ 0 , 1 ] where z1 = fθ ( x1 ) and z2 = fθ ( x2 ) are the latent codes corresponding to data points x1 and x2 . We also experimented with spherical interpolation which has been used in settings where the latent codes are expected to have spherical structure ( Huszár , 2017 ; White , 2016 ; Roberts et al. , 2018 ) , but found it made no discernible difference in practice for any autoencoder we studied . Ideally , adjusting α from 0 to 1 will produce a sequence of realistic datapoints where each subsequent x̂α is progressively less semantically similar to x1 and more semantically similar to x2 . The notion of “ semantic similarity ” is problem-dependent and ill-defined ; we discuss this further in section 3 . 2.1 ADVERSARIALLY CONSTRAINED AUTOENCODER INTERPOLATION ( ACAI ) . As mentioned above , a high-quality interpolation should have two characteristics : First , that intermediate points along the interpolation are indistinguishable from real data ; and second , that the intermediate points provide a semantically smooth morphing between the endpoints . The latter characteristic is hard to enforce because it requires defining a notion of semantic similarity for a given dataset , which is often hard to explicitly codify . So instead , we propose a regularizer which encourages interpolated datapoints to appear realistic , or more specifically , to appear indistinguishable from reconstructions of real datapoints . We find empirically that this constraint results in realistic and smooth interpolations in practice ( section 3.1 ) in addition to providing improved performance on downstream tasks ( section 4 ) . To enforce this constraint we introduce a critic network ( Goodfellow et al. , 2014 ) which is fed interpolations of existing datapoints ( i.e . x̂α as defined above ) . Its goal is to predict α from x̂α , i.e . to predict the mixing coefficient used to generate its input . When training the model , for each pair of training data points we randomly sample a value of α to produce x̂α . In order to resolve the ambiguity between predicting α and 1−α , we constrain α to the range [ 0 , 0.5 ] when feeding x̂α to the critic . In contrast , the autoencoder is trained to fool the critic to think that α is always zero . This is achieved by adding an additional term to the autoencoder ’ s loss to optimize its parameters to fool the critic . In a loose sense , the critic can be seen as approximating an “ adversarial divergence ” ( Liu et al. , 2017 ; Arora et al. , 2017 ) between reconstructions and interpolants which the autoencoder tries to minimize . Formally , let dω ( x ) be the critic network , which for a given input produces a scalar value . The critic is trained to minimize Ld = ‖dω ( x̂α ) − α‖2 + ‖dω ( γx+ ( 1− γ ) x̂ ) ‖2 ( 1 ) where , as above , x̂α = gφ ( αfθ ( x1 ) + ( 1− α ) fθ ( x2 ) ) , x̂ = gφ ( fθ ( x ) ) for some x ( not necessarily x1 or x2 ) , and γ is a scalar hyperparameter . The first term trains the critic to recover α from x̂α . The second term serves as a regularizer with two functions : First , it enforces that the critic consistently outputs 0 for non-interpolated inputs ; and second , by interpolating between x and x̂ ( the autoencoder ’ s reconstruction of x ) in data space it ensures the critic is exposed to realistic data even when the autoencoder ’ s reconstructions are poor . We found the second term was not crucial for our approach , but helped stabilize the convergence of the autoencoder and allowed us to use consistent hyperparameters and architectures across all datasets and experiments . The autoencoder ’ s loss function is modified by adding a regularization term : Lf , g = ‖x− gφ ( fθ ( x ) ) ‖2 + λ‖dω ( x̂α ) ‖2 ( 2 ) where λ is a scalar hyperparameter which controls the weight of the regularization term . Note that the regularization term is effectively trying to make the critic output 0 regardless of the value of α , thereby “ fooling ” the critic into thinking that an interpolated input is non-interpolated ( i.e. , having α = 0 ) . The parameters θ and φ are optimized with respect to Lf , g ( which gives the autoencoder access to the critic ’ s gradients ) and ω is optimized with respect to Ld . We refer to the use of this regularizer as Adversarially Constrained Autoencoder Interpolation ( ACAI ) . A diagram of the ACAI is shown in fig . 2 . Assuming an effective critic , the autoencoder successfully “ wins ” this adversarial game by producing interpolated points which are indistinguishable from reconstructed data . We find in practice that encouraging this behavior also produces semantically smooth interpolations and improved representation learning performance , which we demonstrate in the following sections . Our loss function is similar to the one used in the Least Squares Generative Adversarial Network ( Mao et al. , 2017 ) in the sense that they both measure the distance between a critic ’ s output and a scalar using a squared L2 loss . However , they are substantially different in that ours is used as a regularizer for autoencoders rather than for generative modeling and our critic attempts to regress the interpolation coefficient α instead of a fixed scalar hyperparameter . Note that the only thing ACAI encourages is that interpolated points appear realistic . The critic only ever sees a single reconstruction or interpolant at a time ; it is never fed real datapoints or latent vectors . It therefore will only be able to successfully recover α if the quality of the autoencoder ’ s output degrades consistently across an interpolation as a function of α ( as seen , for example , in fig . 3a where interpolated points become successively blurrier and darker ) . ACAI ’ s primary purpose is to discourage this behavior . In doing so , it may implicitly modify the structure of the latent space 1https : //github.com/anonymous-iclr-2019/acai-iclr-2019 learned by the autoencoder , but ACAI itself does not directly impose a specific structure . Our goal in introducing ACAI is to test whether simply encouraging better interpolation behavior produces a better representation for downstream tasks . Further , in contrast with the standard Generative Adversarial Network ( GAN ) setup ( Goodfellow et al. , 2014 ) , ACAI does not distinguish between “ real ” and “ fake ” data ; rather , it simply attempts to regress the interpolation coefficient α . Furthermore , GANs are a generative modeling technique , not a representation learning technique ; in this paper , we focus on autoencoders and their ability to learn useful representations .

Summary: The authors propose a new approach to encourage valid interpolation in Auto-Encoders (AE). It is based on a regularization procedure involving a critic network judging the realistic nature of reconstructed data point from its mixed latent representations by recovering the mixing coefficient. The authors show that this approach does indeed improve the quality of interpolated samples on few tasks. A synthetic tasks of lines interpolation (proposing new Mean Distance and Smoothness metric for this task), classification task (with a single-layer classifier) from the latent space representation and finally a clustering accuracy on the latent space. On the proposed regularization method seems to help significantly compared to commonly used AE architectures (Basic AE, Denoising AE, Variational AE, Adversarial AE and VQ-VAE).