Variation Network: Learning High-level Attributes for Controlled Input Manipulation

1 INTRODUCTION . We focus on the problem of generating variations of a given input in an intended way . This means that given some input element x , which can be considered as a template , we want to generate transformed versions of x with different high-level attributes . Such a mechanism is of great use in many domains such as image edition since it allows to edit images on a more abstract level and is of crucial importance for creative uses since it allows to generate new content . More precisely , given a dataset D = { ( x ( 1 ) , m ( 1 ) ) , . . . , ( x ( N ) , m ( N ) ) } of N labeled elements ( x , m ) ∈ X ×M , where X stands for the input space andM for the metadata space , we would like to obtain a model capable of learning a relevant attribute space Ψ ⊂ Rd for some integer d > 0 and meaningful attribute functions φ : X ×M→ Ψ that we can then use to control generation . In a great majority of the recent proposed methods Lample et al . ( 2017 ) ; Upchurch et al . ( 2016 ) , these attributes are assumed to be given . We identify two shortcomings : labeled data is not always available and this approach de facto excludes attributes that can be hard to formulate in an absolute way . The novelty of our approach is that these attributes can be either learned by the model ( we name them free attributes ) or imposed ( fixed attributes ) . This problem is an ill-posed one on many aspects . Firstly , in the case of fixed attribute functions φ , there is no ground truth for variations since there is no x with two different attributes . Secondly , it can be hard to determine if a learned free attribute is relevant . However , we provide empirical evidence that our general approach is capable of learning such relevant attributes and that they can be used for generating meaningful variations . In this paper , we introduce the Variation Network ( VarNet ) , a probabilistic neural network which provides means to manipulate an input by changing its high-level attributes . Our model has a sound probabilistic interpretation which makes the variations obtained by changing the attributes statistically meaningful . As a consequence , this probabilistic framework provides us with a novel mechanism to “ control ” or “ shape ” the learned free attributes which then gives interpretable controls over the variations . This architecture is general and provides a wide range of choices for the design of the attribute function φ : we can combine both free and fixed attributes and the fixed attributes can be either continuous or discrete . Our contributions are the following : • A widely applicable encoder-decoder architecture which generalizes existing approaches Kingma & Welling ( 2013 ) ; Rubenstein et al . ( 2018 ) ; Lample et al . ( 2017 ) • An easy-to-use framework : any encoder-decoder architecture can be easily transformed into a VarNet in order to provide it with controlled input manipulation capabilities , • A novel and statistically sound approach to navigate in the latent space , • Ways to control the behavior of the free learned attributes . The plan of this paper is the following : Sect . 2 presents the VarNet architecture together with its training algorithm . For better clarity , we introduce separately all the components featured in our model and postpone the discussion about their interplay and the motivation behind our modeling choices in Sect . 3 and Sect . 4 discusses about the related works . In particular , we show that VarNet provides an interesting solution to many constrained generation problems already considered in the literature . Finally , we illustrate in Appendix A the possibilities offered by our proposed model and show that its faculty to generate variations in an intended way is of particular interest . 2 PROPOSED MODEL . We now introduce our novel encoder-decoder architecture which we name Variation Network . Our architecture borrows principles from the traditional Variational AutoEncoder ( VAE ) architecture Kingma & Welling ( 2013 ) and from the Wasserstein AutoEncoder ( WAE ) architecture Tolstikhin et al . ( 2017 ) ; Rubenstein et al . ( 2018 ) . It uses an adversarially learned regularization Dumoulin et al . ( 2016 ) ; Lample et al . ( 2017 ) , introduces a separate latent space for templates Adel et al . ( 2018 ) and decomposes the attributes on an adaptive basis Wang et al . ( 2018 ) . It can be seen as a VAE with a particular decoder network or as a WAE with a particular encoder network . Our architecture is shown in Fig . 1 and our training algorithm is presented in Alg . 1 . We detail in the following sections the different parts involved in our model . In Sect . 2.1 , we focus on the encoder-decoder part of VarNet and explain Eq . ( 3 ) , ( 4 ) and ( 5 ) . In Sect . 2.2 , we introduce the adversarially-learned regularization whose aim is to disentangle attributes from templates ( Eq . ( 1 ) and ( 6 ) ) . Section 2.3 discusses the special parametrization that we adopted for the attribute space Ψ. Algorithm 1 Variation Network training procedure Require : Dataset D = { ( x ( i ) , m ( i ) } i=1 .. N , reconstruction cost c , reproducing kernel k , batch size n 1 : for Fixed number of iterations do 2 : Sample x : = ( x1 , . . . , xn ) and m : = ( m1 , . . . , mn ) where ( xi , mi ) i.i.d . samples from D 3 : Sample z∗i ∼ q∗ ( ·|xi ) 4 : Compute z : = { z1 , . . . , zn } where zi = fφ ( xi , mi ) ( z∗i ) 5 : Sample x̂ : = { x̂1 , . . . , x̂n } where x̂i ∼ p ( ·|zi ) , 6 : Sample random features { ψi } i=1 .. n from feature space Ψ using ν ( see Sect . 2.3 ) 7 : Let z̃ : = { z̃1 , . . . , z̃n } where z̃i ∼ p ( · ) 8 : Discriminator training phase 9 : Compute LDisc , n : = 1 n n∑ i=1 logD ( z∗i , ψi ) + log ( 1−D ( z∗i , φ ( xi , mi ) ) ) ( 1 ) 10 : Gradient ascent step on the discriminator parameters using∇LDisc 11 : Encoder-decoder training phase 12 : Compute LEncDec , n : = REn + βKL∗n + λMMDk , n + γRDisc , n ( 2 ) where REn : = 1 n n∑ i=1 c ( xi , x̂i ) , ( 3 ) KL∗n : = 1 n n∑ i=1 log q∗ ( z∗i |xi ) − log p∗ ( z∗i ) , ( 4 ) MMDk , n : = 1 n ( n− 1 ) ∑ l 6=k k ( zl , zj ) + 1 n ( n− 1 ) ∑ l 6=k k ( z̃l , z̃j ) − 2 n2 ∑ l , j k ( zl , z̃j ) , ( 5 ) RDisc , n = − 1 n n∑ i=1 logD ( z∗i , φ ( xi , mi ) ) . ( 6 ) 13 : Gradient ascent step on all parameters except the discriminator parameters ( encoder and decoder parameters , feature function parameters , features vectors and NAF f ) using ∇LEncDec 14 : end for 2.1 ENCODER-DECODER PART . Similar to the VAE architectures , we suppose that our data x ∈ X depends on some latent variable z ∈ Z through some decoder p ( x|z ) parametrized by a neural network . We introduce a prior p ( z ) over this latent space so that the joint probability distribution is expressed as p ( x , z ) = p ( x|z ) p ( z ) . Since the posterior distribution p ( z|x ) is usually intractable , an approximate posterior distribution q ( z|x ) parametrized by a neural network is usually introduced . The novelty of our approach is on how we write this encoder network . Firstly , we introduce an attribute space Ψ ⊂ Rd , where d is the dimension of the attribute space , on which we condition the encoder which we now denote as q ( ·|x , ψ ∈ Ψ ) . More details about the attribute space Ψ are given in Sect . 2.3 . For the moment , we can consider it to be a subspace of Rd from which we can sample from . The objective in doing so is that decoding z ∼ q ( ·|x , ψ ) using p ( x|z ) will result in a sample x̃ that is a variation of x but with features ψ. Secondly , in order to correctly reconstruct x , introduce an attribute function φ : X ×M → Ψ computed from x and its metadata m with values in the attribute space Ψ . This attribute function is a deterministic neural network that will be learned during training and whose aim is to compute attributes of x . For an input ( x , m ) ∈ D , we want to decouple a template obtained from x from its attributes φ ( x , m ) computed from x and ( possibly ) from its metadata m. This is done by introducing another latent space Z∗ that we term template space together with a approximated posterior distribution q∗ ( z∗|x ) parametrized by a neural network and a fixed prior p∗ ( z∗ ) . The idea is then to compute z from z∗ by applying a transformation parametrized only by the feature space Ψ . In practice , this is done by using a Neural Autoregressive Flow ( NAF ) Huang et al . ( 2018 ) fψ : Z∗ → Z parametrized by ψ ∈ Ψ. Neural autoregressive flows are universal density estimation models which are capable of sampling any random variable Y by applying a learned transformation over a base random variable X ( Thm . 1 in Huang et al . ( 2018 ) ) . Given a reconstruction loss c on X , we have the following mean reconstruction loss : RE : = E ( x , m ) ∼DEx̂∼p ( ·|z ) Ez∼q ( ·|x , φ ( x , m ) ) c ( x̂ , x ) . ( 7 ) We regularize the latent spaces Z∗ and Z by adding the usual KL term appearing in the VAE Evidence Lower Bound ( ELBO ) on Z∗ : KL∗ : = Ez∗∼q∗ ( .|x ) log q∗ ( z∗|x ) p∗ ( z∗ ) ( 8 ) and an MMD-based regularization on Z similar the one used in WAEs ( see Alg . 2 in Tolstikhin et al . ( 2017 ) ) : MMDk ( q ( ·|x , φ ( x , m ) ) , p ( · ) ) : = ∥∥∥∥∫ Z k ( z , · ) q ( z|x , φ ( x , m ) ) − ∫ Z k ( z , · ) p ( z ) ∥∥∥∥ Hk , ( 9 ) where k : Z×Z → R is an positive-definite reproducing kernel andHk the associated Reproducing Kernel Hilbert Space ( RKHS ) Berlinet & Thomas-Agnan ( 2011 ) . The equations ( 3 ) , ( 4 ) and ( 5 ) of Alg . 1 are estimators on a mini-batch of size n of equations ( 7 ) , ( 8 ) and ( 9 ) respectively , ( 5 ) being the unbiased U-statistic estimator of ( 9 ) Gretton et al . ( 2012 ) .

Summary: This paper introduces a new framework for learning an interpretable representation of images and their attributes. The authors suggest decomposing the representation into a set of 'template' latent features, and a set of attribute-based features. The attribute-based features can be either 'free', i.e. discovered from the data, or 'fixed', i.e. based on the ground truth attributes. The authors encourage the decomposition of the latent space into the 'template' and the 'attributes' features by training a discriminator network to predict whether the attributes and the template features come from the same image or not.