Masked Based Unsupervised Content Transfer

We consider the problem of translating , in an unsupervised manner , between two domains where one contains some additional information compared to the other . The proposed method disentangles the common and separate parts of these domains and , through the generation of a mask , focuses the attention of the underlying network to the desired augmentation alone , without wastefully reconstructing the entire target . This enables state-of-the-art quality and variety of content translation , as demonstrated through extensive quantitative and qualitative evaluation . Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content . Furthermore , our method enables weakly-supervised semantic segmentation of the separate part of each domain , where only class labels are provided . Our code is available at https : //github.com/rmokady/mbu-content-tansfer . 1 INTRODUCTION . The task of content transfer , as depicted in Fig . 1 , involves identifying the component of interest ( for example , glasses ) in a given input ( for example , an image of a face ) , adapting it , and adding it to a second given input ( for example , another image of a face , without glasses ) , hopefully in the semantically correct manner . Such an operation can be used to prototype or demonstrate changes in appearance Gatys et al . ( 2016 ) , augment music Grinstein et al . ( 2018 ) , compose text Prabhumoye et al . ( 2019 ) , generate data for training purposes Mueller et al . ( 2018 ) , etc . Recent advancements ( Huang et al . ( 2018 ) ; Lee et al . ( 2019 ) ) translate one domain to another with varying styles , but not content . Others ( Lample et al . ( 2017 ) ; He et al . ( 2019 ) ; Liu et al . ( 2019 ) ) produce images in a target domain with a given attribute ( e.g glasses ) , but such attribute is unique and not varying or controlled ( e.g. , not allowing the specification of a specific pair of glasses ) . A recent advancement in the realm of attribute transfer has been presented by Press et al . ( 2019 ) . In this work , the input is two domains of images , such that the images in one domain , B , contain a specific class ( e.g . faces with facial hair ) , while in the other domain , A , the images do not ( e.g. , faces without facial hair ) . Training on this input , the method learns to transfer only the specific class information from an unseen image in the domain B to an unseen one in the domain A , while preserving all other details . The proposed architecture yields a simple network which is able to perform the required disentanglement through an emergence effect . In this setting , the content to be added by the system is not explicitly marked in the target domain , nor does it have an equivalent counterpart in the source domain for training ( e.g . an image with and without glasses of the same person ) . A form of weak supervision can be provided by a simple annotation of whether the relevant content exists or not in every example . However , this method , and other typical ones addressing similar tasks , generate the details for the entire image , by using auto-encoder or GAN-based architectures , resulting in a degradation of details and quality . In this paper , we build upon the emerging disentanglement idea , but also adopt the growing understanding that one should minimize redundant use of computational resources and model parameters ( Chen et al . ( 2016 ) ; Mejjati et al . ( 2018 ) ; Chen et al . ( 2018 ) ) , to the aforementioned task . In other words , using a mask , we focus the attention of the baseline network to the desired augmentation alone , without asking it to wastefully reconstruct the entire target . As can be seen in Fig 2 , the method consists of two main steps . The first is the disentanglement step , which encodes the domain specific and the domain invariant contents separately , and is inspired by the work of Press et al . ( 2019 ) . The second step is the key insight of our proposal . It locates the part of the target that should be changed and generates relevant augmentation content to go with it . This allows keeping the unrelated details intact , facilitating a great improvement in generation quality ; The augmentation focuses on the relevant part , leaving all other details to be taken from the target image directly , without going through the bottleneck of an auto-encoder-like module . By applying this simple yet effective principle and a novel regularization scheme , our method preserves target details which are irrelevant to the augmentation and is able to improve upon the state-of-the-art in terms of quality and variety of the content transferred . Furthermore , we demonstrate how the method performs well , even when presented with images outside the domain trained on , and that the aforementioned mask , generated by the system to mark the regions of augmentation , is accurate enough to provide a semantic segmentation of the content transferred . Lastly , our method can also be used in the opposite direction — to remove the domain specific attribute , and thus translating from B to A . Despite not being our main focus , we outperform the various literature methods that address this task ( Lample et al . ( 2017 ) ; Liu et al . ( 2019 ) ; He et al . ( 2019 ) ; Press et al . ( 2019 ) ) . Our advantage is also evident in that once an object is removed , we can then add a different attribute to the resulting image , thus allowing the translation from any two domains , each having a domain specific attribute ( e.g removing glasses and adding the smile of a guide image ) . A related unique ability is that of removing and adding the same attribute , e.g . replacing one ’ s facial hair with a different facial hair . 2 PREVIOUS WORK . In the unsupervised image to image translation , the learner is given two unpaired domains of visual samples , A and B , and is asked , given image a ∈ A to generate an analogue image in the domain B . This problem is inherently ill-posed , as multiple analogous solutions may exist . In several of different approaches ( Zhu et al . ( 2017a ) ; Kim et al . ( 2017 ) ; Yi et al . ( 2017 ) ) a circularity constraint is used to reduce this ambiguity . COGAN Liu & Tuzel ( 2016 ) and UNIT Liu et al . ( 2017 ) enforce a shared latent representation between the two domains . Unlike our method , these methods produce a single solution per input image a . Moving from one to one mappings , multiple approaches provide many to many mappings . Supervised multimodal approaches , where paired samples are provided , include BicycleGAN Zhu et al . ( 2017b ) , which injects random noise z in a generator and enforces an encoder to recover z from the target translation , and MAD-GAN Ghosh et al . ( 2018 ) . The latter trains multiple generators to produce aligned mappings , which are distant from each other . These methods require paired samples from both domains — a costly supervision , which we do not require . Guided Multimodal Approaches MUNIT Huang et al . ( 2018 ) , DRIT Lee et al . ( 2018 ) , and DRIT++ Lee et al . ( 2019 ) are trained on unmatched images . MUNIT trains two encoders ; one captures the content of an image , and another its style , inducing disentanglement . During inference , multiple solutions are produced using the style of a guide image in the target domain . In DRIT ( and DRIT++ ) , a cycle constraint is employed , in a setting where the generator of each domain consists of two disentangled encoders , one of which encodes the content and the second the style of the image . For all these methods , different non symmetrical architectures of encoders are used to capture the style and content . In MUNIT , for example , residual connections are used for the content encoder , and global pooling and adaptive instance normalization are used for the style encoder . Hence , the style code is significantly smaller in dimensions than the content one . In our method we employ two encoders as well , but the architecture of these encoders is symmetric , allowing both encoders to capture content in both . The most relevant work to ours is that of Press et al . ( 2019 ) , which also uses the setting in which the samples in domain B contain all the information in domain A and some additional information . Two encoders are used — the first captures the information that is common between the two domains and the second encodes the unique information of domain B . The decoder maps the concatenation of the two encodings into an image in the domain B , or , in the case that the second encoding is set to zero , to an image in A . Content is transferred between images by mixing the encoding of the former type of one image with the encoding of the latter type of a different image . In many cases , however , only a local area in the image needs to change during translation . Consider the case where A is images of faces and B is faces with facial hair . For a ∈ A , only the location in which the facial hair is placed in a needs to change . In the method of Press et al . ( 2019 ) , the entire image , including other facial features , is generated from scratch , and as a result , many low level details are lost and the quality of generation is reduced . This is not the case for our method , where outside the generated mask , which denotes the location of the facial hair , the content of the generated image is taken from the input image a . This is achieved by employing two decoders , one for domain A and one , with two outputs ( raw image and mask ) in domain B , and by a new set of loss terms . Mask Based Approaches The use of masks is prevalent in a variety of visual tasks . For example , for virtual try on , Han et al . ( 2018 ) uses a supervised human parser network to transfer the desired clothing to a target person . Unlike our method , this method can not perform general content transfer as it crucially relies on a supervised pose estimator . In the context of style transfer , Ma et al . ( 2018 ) uses masks to transfer the style of semantically similar regions from the target to the source image . In the context of image to image translation , the one to one case was addressed by Chen et al . ( 2018 ) ; Mejjati et al . ( 2018 ) , where in addition to mapping to the target domain , a mask is learned , to cover only the relevant area in the translation . For example , in the case of mapping from horses to zebras , the mask learns to cover the area of the zebra , which allows the background to be taken from the source image , thus allowing for a much better quality of generation . In our method , we extend this masking ( or attention ) approach to the one to many ( guided ) case . Note that while Chen et al . ( 2018 ) ; Mejjati et al . ( 2018 ) learn a mask and then employ it directly to the image , this does not allowthe mask to adapt both target and guide image . Weakly supervised semantic segmentation methods can be stratified based on the type of supervision used . In the first set of methods ( Song et al . ( 2019 ) ; Hu et al . ( 2018 ) ; Zhao et al . ( 2018 ) ) a bounding box is used . Other approaches ( Cheng et al . ( 2018 ) ; Caelles et al . ( 2017 ) ) use the entire supervision of the fully supervised approach , but are required to find a segmentation in a single shot . Our approach belongs to a set of methods Zhang et al . ( 2018 ) ; Zhou et al . ( 2018 ) ; Wei et al . ( 2018 ) ; Ahn & Kwak ( 2018 ) that use only the class label information to find a segmentation . Zhou et al . ( 2018 ) use the visual cues arising from peaks in class response maps ( local maxima ) to generate highly informative regions . Zhou et al . ( 2016 ) use Class Activation Maps ( CAMs ) extracted from a classifier to obtain discriminative localization . Wei et al . ( 2018 ) use varying dilation rates to transfer surrounding discriminative information to non-discriminative object locations . Ahn & Kwak ( 2018 ) propagate local discriminative parts to nearby regions that belong to the same semantic entity . Other methods ( Tang et al . ( 2018b ) ; Kervadec et al . ( 2019 ) ; Tang et al . ( 2018a ) ) use regularized losses with different levels of supervision . Unlike these methods , our main focus is on generating the added part in a way that it is adapted to the placement context for the domain specific information and yet , our segmentation results are competitive with such methods on this task .

Summary: This paper proposes a method for unpaired image-to-image translation, where the target domain explicitly contains some additional information than the source domain. The authors use auto-encoders to separate the common and specific representations and to generate masks, which seems to be related to [1]. The authors empirically show the proposed method can be used for image translation, attribute editing.