Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer

1 INTRODUCTION . In the problem of unsupervised domain translation , the algorithm receives two sets of samples , one from each domain , and learns a function that maps between a sample in one domain to the analogous sample in the other domain ( Zhu et al. , 2017a ; Kim et al. , 2017 ; Yi et al. , 2017 ; Benaim & Wolf , 2017 ; Liu & Tuzel , 2016 ; Liu et al. , 2017 ; Choi et al. , 2017 ; Conneau et al. , 2017 ; Zhang et al. , 2017a ; b ; Lample et al. , 2018 ) . The term unsupervised means , in this context , that the two sets are unpaired . In this paper , we consider the problem of domain B , which contains a type of content that is not present at A . As a running example , we consider the problem of mapping between a face without eyewear ( domain A ) to a face with glasses ( domain B ) . While most methods would map to a person with any glasses , our solution is guided and we attach to an image a ∈ A , the glasses that are present in a reference image b ∈ B . In comparison to other guided image to image translation methods , our method is considerably simpler . It relies on having a latent space with two parts : ( i ) a shared part that is common to both A and B , and ( ii ) a specific part that encodes the added content in B . By setting the second part to be the zero vector for all samples in A , a disentanglement emerges . Our analysis shows that this leads to the ability to train with a simple , straightforward domain confusion term , while enjoying the generalization guarantees that would otherwise require a more elaborate loss . 1.1 PREVIOUS WORK . In image to image translation , the transformations are often captured by multi-layered networks of an encoder-decoder architecture . The existing solutions often assume a one to one mapping between the domains , i.e. , that there exists a function y such that given a sample a in domain A , maps it to an analog sample in domain B . In fact , the circularity based constraints by Zhu et al . ( 2017a ) ; Kim et al . ( 2017 ) ; Yi et al . ( 2017 ) are based on this assumption , since going from one domain to the other and back , it is assumed that the original sample is obtained , which requires no loss of information . However , to employ an example made popular by Zhu et al . ( 2017a ) , when going from a zebra to a horse , the stripes are lost , which results in an ambiguity when mapping in the other direction . This shortcoming was identified by the literature , and a few contributions present many to many mappings . These include the augmented CycleGAN Almahairi et al . ( 2018 ) , which adds a random vector to each domain . A completely different approach is taken by the NAM method ( Hoshen & Wolf , 2018 ) , in which multiple solutions are obtained by considering multiple random initializations . In our work , multiplicity of outcomes arises from using a reference ( guide ) image . A powerful way to capture relations between the two domains , is by employing separate autoencoders for the two domains , but which share many of the weights ( Liu & Tuzel , 2016 ; Liu et al. , 2017 ) . This leads to a shared representation : while the low-level image properties , such as texture and color , are domain-specific and encoded/decoded separately , the mid- and top-level properties are common to both domains and are processed by identical replicas of the same layers . In our work , we rely on a shared encoder for both domains in order to enforce a shared representation . Our method performs one-sided mapping , from A to B , and does not learn the mapping in the other direction . Benaim & Wolf ( 2017 ) perform one-sided mapping using a distance based constraint , which we do not employ . That method is symmetric in the two domains and in many experiments , the distance constraint is used in tandem with the cycle constraint . In our case , the two domains are asymmetric and one domain contains an added content . Our method is inherently asymmetric , reflecting the asymmetry between the source image and the guide image , which contains the additional content . The work by Taigman et al . ( 2017 ) ; Hoshen & Wolf ( 2018 ) map in an asymmetric way , but rely on the existence of a perceptual distance , which we do not employ . Guided Translation The most relevant work contains very recent and concurrent methods in which the mapping between the domains employs two inputs : a source image a and a reference ( guide or attribute ) image b. Tab . 1 compares these methods to ours along two axes : ( i ) where sharing occurs in the architecture , and ( ii ) the number of trained sub-networks of each type . The sharing can be of layers between different encoders and decoders , following , e.g. , Liu & Tuzel ( 2016 ) ; sharing of a common part of a latent space ; and using the same encoder or decoder for multiple domains . The networks are of four types : encoders , which map images to a latent space , generators ( also known as decoders ) , which generate images from a latent representation , discriminators that are used as part of an adversarial loss , and other , less-standard , networks . It is apparent that our method is considerably simpler than the literature methods . The main reason is that our method is based on the emergence of disentanglement , as detailed in Sec . 4 . This allows us to to train with many less parameters and without the need to apply excessive tuning , in order to balance or calibrate the various components of the compound loss . The MUNIT architecture by Huang et al . ( 2018 ) , like our architecture , employs a shared latent space , in addition to a domain specific latent space . Their architecture is not limited to two domains1 and unlike ours , employs separate encoders and decoders for the various domains . The type of guiding that is obtained from the target domain in MUNIT is referred to as style , while in our case , the guidance provides content . Therefore , MUNIT , as can be seen in our experiments , can not add specific glasses , when shifting from the no-glasses domain to the faces with eyewear domain . The EG-UNIT architecture by Ma et al . ( 2018 ) presents a few novelties , including an adaptive method of masking-out a varying set of the features in the shared latent space . In our latent representation of domain A , some of the features are constantly zero , which is much simpler . This method also focuses on guiding for style and not for content , as is apparent form their experiments . The very recent DRIT work by Lee et al . ( 2018 ) learns to map between two domains using a disentangled representation . Unlike our work , this work seems to focus on style rather than content . The proposed solution differs from us in many ways : ( 1 ) it relies on two-way mapping , while we only map from A to B . ( 2 ) it relies on shared weights in order to ensure that the common representation is shared . ( 3 ) it adds a VAE-like ( Kingma & Welling , 2014 ) statistical characterization of the latent space , which results in the ability to sample random attributes . As can be seen in Tab . 1 , the solution of Lee et al . ( 2018 ) is considerably more involved than our solution . DRIT ( and also MUNIT ) employ two different types of encoders that enforce a separation of the latent space representations to either style or content vectors . For example , the style encoder , unlike the content encoder , employs spatial pooling and it also results in a smaller representation than the content one . This is important , in the context of these methods , in order to ensure that the two representations encode different aspects of the image . If DRIT or MUNIT were to use the same type of encoder twice , then one encoder could capture all the information , and the image-based guiding ( mixing representations from two images ) would become mute . In contrast , our method ( i ) does not separate style and content , and ( ii ) has a representation that is geared toward capturing the additional content . The work most similar to us in its goal , but not in method , is the PairedCycleGAN work by Chang et al . ( 2018 ) . This work explores the single application of applying the makeup of a reference face to a source face image . Unfortunately , the method was only demonstrated on a proprietary unshared dataset and the code is also not publicly available , making a direct comparison impossible at this time . The method itself is completely different from ours and does not employ disentanglement . Instead , a generator with two image inputs is used to produce an output image , where the makeup is transfered between the input images , and a second generator is trained to remove makeup . The generation is done separately to k = 5 pre-segmented facial regions , and the generators do not employ an encoder-decoder architecture . Lastly , there are guided methods , which are trained in the supervised domain , i.e. , when there are matches between domain A and B . Unlike the earlier one-to-one work , such as pix2pix Isola et al . ( 2017b ) , these methods produce multiple outputs based on a reference image in the target domain . Examples include the Bicycle GAN by Zhu et al . ( 2017b ) , who also applied , as baseline in their experiments , the methods of Bao et al . ( 2017 ) ; Gonzalez-Garcia et al . ( 2018 ) . Other Disentanglement Work InfoGAN ( Chen et al. , 2016 ) learns a representation in which , due to the statistical properties of the representations , specific classes are encoded as a one-hot encoding of part of the latent vector . In the work of Lample et al . ( 2017 ) ; Hadad et al . ( 2018 ) , the representation is disentangled by reducing the class based information within it . The separate class based information is different in nature from our multi-dimensional added content . Cao et al . ( 2018 ) , 1Ours method can be readily extended to multiple target domains B1 , . . . , Bk , but this is not explored here . which builds upon Hadad et al . ( 2018 ) , performs guided image to image translation , but assumes the availability of class based information , which we do not . 2 PROBLEM SETUP . We consider a setting with two domains A = ( XA , DA ) and B = ( XB , DB ) . Here , XA , XB ⊂ RM and DA , DB are distributions over them ( resp. ) . The algorithm is provided with two independent datasets SA = { ai } m1i=1 and SB = { bj } m2 j=1 of samples from the two domains that were sampled in the following manner : SA i.i.d∼ Dm1A and SB i.i.d∼ Dm2B ( 1 ) We denote , DA , B : = DA ×DB the distribution of sampling ( a , b ) for a ∼ DA and b ∼ DB independently . We assume a generative model , in which b is specified by a sample a and a specification c from a third unknown domainC = ( XC , DC ) of specifications whereDC is a distribution over the metric space XC ⊂ RN . Formally , there is an invertible function u ( b ) = ( u1 ( b ) , u2 ( b ) ) ∈ XA×XC that takes a sample b ∈ XB and returns the content u1 ( b ) of b and the specification u2 ( b ) of b . The goal is to learn a target function y : XA × XB → XB such that : y ( a , b ) ∼ DB where : a ∼ DA , b ∼ DB , a |= b and u ( y ( a , b ) ) = ( a , u2 ( b ) ) ( 2 ) Informally , the function y takes two samples a and b and returns the analog of a in B that has the specification of b . For example , A is the domain of images of persons , B is the domain of images of persons with sunglasses and C is the domain of images of sunglasses . The function y takes an image of a person and an image of a person with sunglasses and returns an image of the first person with the specified sunglasses . For simplicity , we assume that the target function is extended to inputs ( b1 , b2 ) ∈ X2B and u ( y ( b1 , b2 ) ) = ( u1 ( b1 ) , u2 ( b2 ) ) . In other words , b1 and b2 are mapped to a third b that has the content of b1 and the specification of b2 . In particular , u ( y ( b , b ) ) = ( u1 ( b ) , u2 ( b ) ) = u ( b ) and , therefore , y ( b , b ) = b . Note that within Eq . 2 , there is an assumption on the underlying distributionsDA andDB . Using the concrete example , our framework assumes that the distribution of images of persons with sunglasses and the distribution of images of persons without them is the same , except for the sunglasses . Otherwise , the distribution of the samples generated by y when resampling a would not be the same as DB . Note that we do not enforce this assumption on the data , and only employ it for our theoretical results , to avoid additional terms . For two functions f1 , f2 : X → R and a distribution D over X , we define the generalization risk between f1 and f2 as follows : RD [ f1 , f2 ] : = Ex∼D [ ` ( f1 ( x ) , f2 ( x ) ) ] ( 3 ) For a loss function ` : RM × RM → [ 0 , ∞ ) . Typically , we use the L1 ( u , v ) : = ‖u − v‖1 or L2 ( u , v ) : = ‖u− v‖22 losses . The goal of the algorithm is to return a hypothesis h ∈ H , such that h : RM × RM → RM , that minimizes the generalization risk , RDA , B [ h , y ] = E ( a , b ) ∼DA , B [ ` ( h ( a , b ) , y ( a , b ) ) ] ( 4 ) This quantity measures the expected loss of h in mapping two samples a ∼ DA and b ∼ DB to the analog y ( a , b ) of a , that has the specification u2 ( b ) of b . The main challenge is that the algorithm does not observes paired examples of the form ( ( a , b ) , y ( a , b ) ) as a direct supervision for learning the mapping y : XA × XB → XB .

Summary: This paper tackles the task of content transfer. For a given type of images (frontal face shots), the goal is to transfer a particular localized property (e.g. glasses or facial hair) extracted from one image to another image of the same type (difference face). This is also known as the problem of guided image-to-image translation.