Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer

1 INTRODUCTION . In the problem of unsupervised domain translation , the algorithm receives two sets of samples , one from each domain , and learns a function that maps between a sample in one domain to the analogous sample in the other domain ( Zhu et al. , 2017a ; Kim et al. , 2017 ; Yi et al. , 2017 ; Benaim & Wolf , 2017 ; Liu & Tuzel , 2016 ; Liu et al. , 2017 ; Choi et al. , 2017 ; Conneau et al. , 2017 ; Zhang et al. , 2017a ; b ; Lample et al. , 2018 ) . The term unsupervised means , in this context , that the two sets are unpaired . In this paper , we consider the problem of domain B , which contains a type of content that is not present at A . As a running example , we consider the problem of mapping between a face without eyewear ( domain A ) to a face with glasses ( domain B ) . While most methods would map to a person with any glasses , our solution is guided and we attach to an image a ∈ A , the glasses that are present in a reference image b ∈ B . In comparison to other guided image to image translation methods , our method is considerably simpler . It relies on having a latent space with two parts : ( i ) a shared part that is common to both A and B , and ( ii ) a specific part that encodes the added content in B . By setting the second part to be the zero vector for all samples in A , a disentanglement emerges . Our analysis shows that this leads to the ability to train with a simple , straightforward domain confusion term , while enjoying the generalization guarantees that would otherwise require a more elaborate loss . 1.1 PREVIOUS WORK . In image to image translation , the transformations are often captured by multi-layered networks of an encoder-decoder architecture . The existing solutions often assume a one to one mapping between the domains , i.e. , that there exists a function y such that given a sample a in domain A , maps it to an analog sample in domain B . In fact , the circularity based constraints by Zhu et al . ( 2017a ) ; Kim et al . ( 2017 ) ; Yi et al . ( 2017 ) are based on this assumption , since going from one domain to the other and back , it is assumed that the original sample is obtained , which requires no loss of information . However , to employ an example made popular by Zhu et al . ( 2017a ) , when going from a zebra to a horse , the stripes are lost , which results in an ambiguity when mapping in the other direction . This shortcoming was identified by the literature , and a few contributions present many to many mappings . These include the augmented CycleGAN Almahairi et al . ( 2018 ) , which adds a random vector to each domain . A completely different approach is taken by the NAM method ( Hoshen & Wolf , 2018 ) , in which multiple solutions are obtained by considering multiple random initializations . In our work , multiplicity of outcomes arises from using a reference ( guide ) image . A powerful way to capture relations between the two domains , is by employing separate autoencoders for the two domains , but which share many of the weights ( Liu & Tuzel , 2016 ; Liu et al. , 2017 ) . This leads to a shared representation : while the low-level image properties , such as texture and color , are domain-specific and encoded/decoded separately , the mid- and top-level properties are common to both domains and are processed by identical replicas of the same layers . In our work , we rely on a shared encoder for both domains in order to enforce a shared representation . Our method performs one-sided mapping , from A to B , and does not learn the mapping in the other direction . Benaim & Wolf ( 2017 ) perform one-sided mapping using a distance based constraint , which we do not employ . That method is symmetric in the two domains and in many experiments , the distance constraint is used in tandem with the cycle constraint . In our case , the two domains are asymmetric and one domain contains an added content . Our method is inherently asymmetric , reflecting the asymmetry between the source image and the guide image , which contains the additional content . The work by Taigman et al . ( 2017 ) ; Hoshen & Wolf ( 2018 ) map in an asymmetric way , but rely on the existence of a perceptual distance , which we do not employ . Guided Translation The most relevant work contains very recent and concurrent methods in which the mapping between the domains employs two inputs : a source image a and a reference ( guide or attribute ) image b. Tab . 1 compares these methods to ours along two axes : ( i ) where sharing occurs in the architecture , and ( ii ) the number of trained sub-networks of each type . The sharing can be of layers between different encoders and decoders , following , e.g. , Liu & Tuzel ( 2016 ) ; sharing of a common part of a latent space ; and using the same encoder or decoder for multiple domains . The networks are of four types : encoders , which map images to a latent space , generators ( also known as decoders ) , which generate images from a latent representation , discriminators that are used as part of an adversarial loss , and other , less-standard , networks . It is apparent that our method is considerably simpler than the literature methods . The main reason is that our method is based on the emergence of disentanglement , as detailed in Sec . 4 . This allows us to to train with many less parameters and without the need to apply excessive tuning , in order to balance or calibrate the various components of the compound loss . The MUNIT architecture by Huang et al . ( 2018 ) , like our architecture , employs a shared latent space , in addition to a domain specific latent space . Their architecture is not limited to two domains1 and unlike ours , employs separate encoders and decoders for the various domains . The type of guiding that is obtained from the target domain in MUNIT is referred to as style , while in our case , the guidance provides content . Therefore , MUNIT , as can be seen in our experiments , can not add specific glasses , when shifting from the no-glasses domain to the faces with eyewear domain . The EG-UNIT architecture by Ma et al . ( 2018 ) presents a few novelties , including an adaptive method of masking-out a varying set of the features in the shared latent space . In our latent representation of domain A , some of the features are constantly zero , which is much simpler . This method also focuses on guiding for style and not for content , as is apparent form their experiments . The very recent DRIT work by Lee et al . ( 2018 ) learns to map between two domains using a disentangled representation . Unlike our work , this work seems to focus on style rather than content . The proposed solution differs from us in many ways : ( 1 ) it relies on two-way mapping , while we only map from A to B . ( 2 ) it relies on shared weights in order to ensure that the common representation is shared . ( 3 ) it adds a VAE-like ( Kingma & Welling , 2014 ) statistical characterization of the latent space , which results in the ability to sample random attributes . As can be seen in Tab . 1 , the solution of Lee et al . ( 2018 ) is considerably more involved than our solution . DRIT ( and also MUNIT ) employ two different types of encoders that enforce a separation of the latent space representations to either style or content vectors . For example , the style encoder , unlike the content encoder , employs spatial pooling and it also results in a smaller representation than the content one . This is important , in the context of these methods , in order to ensure that the two representations encode different aspects of the image . If DRIT or MUNIT were to use the same type of encoder twice , then one encoder could capture all the information , and the image-based guiding ( mixing representations from two images ) would become mute . In contrast , our method ( i ) does not separate style and content , and ( ii ) has a representation that is geared toward capturing the additional content . The work most similar to us in its goal , but not in method , is the PairedCycleGAN work by Chang et al . ( 2018 ) . This work explores the single application of applying the makeup of a reference face to a source face image . Unfortunately , the method was only demonstrated on a proprietary unshared dataset and the code is also not publicly available , making a direct comparison impossible at this time . The method itself is completely different from ours and does not employ disentanglement . Instead , a generator with two image inputs is used to produce an output image , where the makeup is transfered between the input images , and a second generator is trained to remove makeup . The generation is done separately to k = 5 pre-segmented facial regions , and the generators do not employ an encoder-decoder architecture . Lastly , there are guided methods , which are trained in the supervised domain , i.e. , when there are matches between domain A and B . Unlike the earlier one-to-one work , such as pix2pix Isola et al . ( 2017b ) , these methods produce multiple outputs based on a reference image in the target domain . Examples include the Bicycle GAN by Zhu et al . ( 2017b ) , who also applied , as baseline in their experiments , the methods of Bao et al . ( 2017 ) ; Gonzalez-Garcia et al . ( 2018 ) . Other Disentanglement Work InfoGAN ( Chen et al. , 2016 ) learns a representation in which , due to the statistical properties of the representations , specific classes are encoded as a one-hot encoding of part of the latent vector . In the work of Lample et al . ( 2017 ) ; Hadad et al . ( 2018 ) , the representation is disentangled by reducing the class based information within it . The separate class based information is different in nature from our multi-dimensional added content . Cao et al . ( 2018 ) , 1Ours method can be readily extended to multiple target domains B1 , . . . , Bk , but this is not explored here . which builds upon Hadad et al . ( 2018 ) , performs guided image to image translation , but assumes the availability of class based information , which we do not . 2 PROBLEM SETUP . We consider a setting with two domains A = ( XA , DA ) and B = ( XB , DB ) . Here , XA , XB ⊂ RM and DA , DB are distributions over them ( resp. ) . The algorithm is provided with two independent datasets SA = { ai } m1i=1 and SB = { bj } m2 j=1 of samples from the two domains that were sampled in the following manner : SA i.i.d∼ Dm1A and SB i.i.d∼ Dm2B ( 1 ) We denote , DA , B : = DA ×DB the distribution of sampling ( a , b ) for a ∼ DA and b ∼ DB independently . We assume a generative model , in which b is specified by a sample a and a specification c from a third unknown domainC = ( XC , DC ) of specifications whereDC is a distribution over the metric space XC ⊂ RN . Formally , there is an invertible function u ( b ) = ( u1 ( b ) , u2 ( b ) ) ∈ XA×XC that takes a sample b ∈ XB and returns the content u1 ( b ) of b and the specification u2 ( b ) of b . The goal is to learn a target function y : XA × XB → XB such that : y ( a , b ) ∼ DB where : a ∼ DA , b ∼ DB , a |= b and u ( y ( a , b ) ) = ( a , u2 ( b ) ) ( 2 ) Informally , the function y takes two samples a and b and returns the analog of a in B that has the specification of b . For example , A is the domain of images of persons , B is the domain of images of persons with sunglasses and C is the domain of images of sunglasses . The function y takes an image of a person and an image of a person with sunglasses and returns an image of the first person with the specified sunglasses . For simplicity , we assume that the target function is extended to inputs ( b1 , b2 ) ∈ X2B and u ( y ( b1 , b2 ) ) = ( u1 ( b1 ) , u2 ( b2 ) ) . In other words , b1 and b2 are mapped to a third b that has the content of b1 and the specification of b2 . In particular , u ( y ( b , b ) ) = ( u1 ( b ) , u2 ( b ) ) = u ( b ) and , therefore , y ( b , b ) = b . Note that within Eq . 2 , there is an assumption on the underlying distributionsDA andDB . Using the concrete example , our framework assumes that the distribution of images of persons with sunglasses and the distribution of images of persons without them is the same , except for the sunglasses . Otherwise , the distribution of the samples generated by y when resampling a would not be the same as DB . Note that we do not enforce this assumption on the data , and only employ it for our theoretical results , to avoid additional terms . For two functions f1 , f2 : X → R and a distribution D over X , we define the generalization risk between f1 and f2 as follows : RD [ f1 , f2 ] : = Ex∼D [ ` ( f1 ( x ) , f2 ( x ) ) ] ( 3 ) For a loss function ` : RM × RM → [ 0 , ∞ ) . Typically , we use the L1 ( u , v ) : = ‖u − v‖1 or L2 ( u , v ) : = ‖u− v‖22 losses . The goal of the algorithm is to return a hypothesis h ∈ H , such that h : RM × RM → RM , that minimizes the generalization risk , RDA , B [ h , y ] = E ( a , b ) ∼DA , B [ ` ( h ( a , b ) , y ( a , b ) ) ] ( 4 ) This quantity measures the expected loss of h in mapping two samples a ∼ DA and b ∼ DB to the analog y ( a , b ) of a , that has the specification u2 ( b ) of b . The main challenge is that the algorithm does not observes paired examples of the form ( ( a , b ) , y ( a , b ) ) as a direct supervision for learning the mapping y : XA × XB → XB .

Summary: The paper proposes an unsupervised approach for mapping two sets of objects, A and B, such that set B contains all the information that is in set A and some additional information. The paper learns a latent space which encodes: (a) information which is shared in both sets, and (b) the additional content present in B. This is done by employing a two-pathway encoder and a decoder for both the sets. Experiments on problems such as adding glasses or facial hair to faces shows that the proposed method performs better than existing disentanglement approaches.