Masked Based Unsupervised Content Transfer

We consider the problem of translating , in an unsupervised manner , between two domains where one contains some additional information compared to the other . The proposed method disentangles the common and separate parts of these domains and , through the generation of a mask , focuses the attention of the underlying network to the desired augmentation alone , without wastefully reconstructing the entire target . This enables state-of-the-art quality and variety of content translation , as demonstrated through extensive quantitative and qualitative evaluation . Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content . Furthermore , our method enables weakly-supervised semantic segmentation of the separate part of each domain , where only class labels are provided . Our code is available at https : //github.com/rmokady/mbu-content-tansfer . 1 INTRODUCTION . The task of content transfer , as depicted in Fig . 1 , involves identifying the component of interest ( for example , glasses ) in a given input ( for example , an image of a face ) , adapting it , and adding it to a second given input ( for example , another image of a face , without glasses ) , hopefully in the semantically correct manner . Such an operation can be used to prototype or demonstrate changes in appearance Gatys et al . ( 2016 ) , augment music Grinstein et al . ( 2018 ) , compose text Prabhumoye et al . ( 2019 ) , generate data for training purposes Mueller et al . ( 2018 ) , etc . Recent advancements ( Huang et al . ( 2018 ) ; Lee et al . ( 2019 ) ) translate one domain to another with varying styles , but not content . Others ( Lample et al . ( 2017 ) ; He et al . ( 2019 ) ; Liu et al . ( 2019 ) ) produce images in a target domain with a given attribute ( e.g glasses ) , but such attribute is unique and not varying or controlled ( e.g. , not allowing the specification of a specific pair of glasses ) . A recent advancement in the realm of attribute transfer has been presented by Press et al . ( 2019 ) . In this work , the input is two domains of images , such that the images in one domain , B , contain a specific class ( e.g . faces with facial hair ) , while in the other domain , A , the images do not ( e.g. , faces without facial hair ) . Training on this input , the method learns to transfer only the specific class information from an unseen image in the domain B to an unseen one in the domain A , while preserving all other details . The proposed architecture yields a simple network which is able to perform the required disentanglement through an emergence effect . In this setting , the content to be added by the system is not explicitly marked in the target domain , nor does it have an equivalent counterpart in the source domain for training ( e.g . an image with and without glasses of the same person ) . A form of weak supervision can be provided by a simple annotation of whether the relevant content exists or not in every example . However , this method , and other typical ones addressing similar tasks , generate the details for the entire image , by using auto-encoder or GAN-based architectures , resulting in a degradation of details and quality . In this paper , we build upon the emerging disentanglement idea , but also adopt the growing understanding that one should minimize redundant use of computational resources and model parameters ( Chen et al . ( 2016 ) ; Mejjati et al . ( 2018 ) ; Chen et al . ( 2018 ) ) , to the aforementioned task . In other words , using a mask , we focus the attention of the baseline network to the desired augmentation alone , without asking it to wastefully reconstruct the entire target . As can be seen in Fig 2 , the method consists of two main steps . The first is the disentanglement step , which encodes the domain specific and the domain invariant contents separately , and is inspired by the work of Press et al . ( 2019 ) . The second step is the key insight of our proposal . It locates the part of the target that should be changed and generates relevant augmentation content to go with it . This allows keeping the unrelated details intact , facilitating a great improvement in generation quality ; The augmentation focuses on the relevant part , leaving all other details to be taken from the target image directly , without going through the bottleneck of an auto-encoder-like module . By applying this simple yet effective principle and a novel regularization scheme , our method preserves target details which are irrelevant to the augmentation and is able to improve upon the state-of-the-art in terms of quality and variety of the content transferred . Furthermore , we demonstrate how the method performs well , even when presented with images outside the domain trained on , and that the aforementioned mask , generated by the system to mark the regions of augmentation , is accurate enough to provide a semantic segmentation of the content transferred . Lastly , our method can also be used in the opposite direction — to remove the domain specific attribute , and thus translating from B to A . Despite not being our main focus , we outperform the various literature methods that address this task ( Lample et al . ( 2017 ) ; Liu et al . ( 2019 ) ; He et al . ( 2019 ) ; Press et al . ( 2019 ) ) . Our advantage is also evident in that once an object is removed , we can then add a different attribute to the resulting image , thus allowing the translation from any two domains , each having a domain specific attribute ( e.g removing glasses and adding the smile of a guide image ) . A related unique ability is that of removing and adding the same attribute , e.g . replacing one ’ s facial hair with a different facial hair . 2 PREVIOUS WORK . In the unsupervised image to image translation , the learner is given two unpaired domains of visual samples , A and B , and is asked , given image a ∈ A to generate an analogue image in the domain B . This problem is inherently ill-posed , as multiple analogous solutions may exist . In several of different approaches ( Zhu et al . ( 2017a ) ; Kim et al . ( 2017 ) ; Yi et al . ( 2017 ) ) a circularity constraint is used to reduce this ambiguity . COGAN Liu & Tuzel ( 2016 ) and UNIT Liu et al . ( 2017 ) enforce a shared latent representation between the two domains . Unlike our method , these methods produce a single solution per input image a . Moving from one to one mappings , multiple approaches provide many to many mappings . Supervised multimodal approaches , where paired samples are provided , include BicycleGAN Zhu et al . ( 2017b ) , which injects random noise z in a generator and enforces an encoder to recover z from the target translation , and MAD-GAN Ghosh et al . ( 2018 ) . The latter trains multiple generators to produce aligned mappings , which are distant from each other . These methods require paired samples from both domains — a costly supervision , which we do not require . Guided Multimodal Approaches MUNIT Huang et al . ( 2018 ) , DRIT Lee et al . ( 2018 ) , and DRIT++ Lee et al . ( 2019 ) are trained on unmatched images . MUNIT trains two encoders ; one captures the content of an image , and another its style , inducing disentanglement . During inference , multiple solutions are produced using the style of a guide image in the target domain . In DRIT ( and DRIT++ ) , a cycle constraint is employed , in a setting where the generator of each domain consists of two disentangled encoders , one of which encodes the content and the second the style of the image . For all these methods , different non symmetrical architectures of encoders are used to capture the style and content . In MUNIT , for example , residual connections are used for the content encoder , and global pooling and adaptive instance normalization are used for the style encoder . Hence , the style code is significantly smaller in dimensions than the content one . In our method we employ two encoders as well , but the architecture of these encoders is symmetric , allowing both encoders to capture content in both . The most relevant work to ours is that of Press et al . ( 2019 ) , which also uses the setting in which the samples in domain B contain all the information in domain A and some additional information . Two encoders are used — the first captures the information that is common between the two domains and the second encodes the unique information of domain B . The decoder maps the concatenation of the two encodings into an image in the domain B , or , in the case that the second encoding is set to zero , to an image in A . Content is transferred between images by mixing the encoding of the former type of one image with the encoding of the latter type of a different image . In many cases , however , only a local area in the image needs to change during translation . Consider the case where A is images of faces and B is faces with facial hair . For a ∈ A , only the location in which the facial hair is placed in a needs to change . In the method of Press et al . ( 2019 ) , the entire image , including other facial features , is generated from scratch , and as a result , many low level details are lost and the quality of generation is reduced . This is not the case for our method , where outside the generated mask , which denotes the location of the facial hair , the content of the generated image is taken from the input image a . This is achieved by employing two decoders , one for domain A and one , with two outputs ( raw image and mask ) in domain B , and by a new set of loss terms . Mask Based Approaches The use of masks is prevalent in a variety of visual tasks . For example , for virtual try on , Han et al . ( 2018 ) uses a supervised human parser network to transfer the desired clothing to a target person . Unlike our method , this method can not perform general content transfer as it crucially relies on a supervised pose estimator . In the context of style transfer , Ma et al . ( 2018 ) uses masks to transfer the style of semantically similar regions from the target to the source image . In the context of image to image translation , the one to one case was addressed by Chen et al . ( 2018 ) ; Mejjati et al . ( 2018 ) , where in addition to mapping to the target domain , a mask is learned , to cover only the relevant area in the translation . For example , in the case of mapping from horses to zebras , the mask learns to cover the area of the zebra , which allows the background to be taken from the source image , thus allowing for a much better quality of generation . In our method , we extend this masking ( or attention ) approach to the one to many ( guided ) case . Note that while Chen et al . ( 2018 ) ; Mejjati et al . ( 2018 ) learn a mask and then employ it directly to the image , this does not allowthe mask to adapt both target and guide image . Weakly supervised semantic segmentation methods can be stratified based on the type of supervision used . In the first set of methods ( Song et al . ( 2019 ) ; Hu et al . ( 2018 ) ; Zhao et al . ( 2018 ) ) a bounding box is used . Other approaches ( Cheng et al . ( 2018 ) ; Caelles et al . ( 2017 ) ) use the entire supervision of the fully supervised approach , but are required to find a segmentation in a single shot . Our approach belongs to a set of methods Zhang et al . ( 2018 ) ; Zhou et al . ( 2018 ) ; Wei et al . ( 2018 ) ; Ahn & Kwak ( 2018 ) that use only the class label information to find a segmentation . Zhou et al . ( 2018 ) use the visual cues arising from peaks in class response maps ( local maxima ) to generate highly informative regions . Zhou et al . ( 2016 ) use Class Activation Maps ( CAMs ) extracted from a classifier to obtain discriminative localization . Wei et al . ( 2018 ) use varying dilation rates to transfer surrounding discriminative information to non-discriminative object locations . Ahn & Kwak ( 2018 ) propagate local discriminative parts to nearby regions that belong to the same semantic entity . Other methods ( Tang et al . ( 2018b ) ; Kervadec et al . ( 2019 ) ; Tang et al . ( 2018a ) ) use regularized losses with different levels of supervision . Unlike these methods , our main focus is on generating the added part in a way that it is adapted to the placement context for the domain specific information and yet , our segmentation results are competitive with such methods on this task .

Summary: This work proposed a mask based approach for instance-level unsupervised content transfer, which is an extension of the disentanglement work in (Press et al., 2019) and the attention guided translation (Chen et al., 2018, Mejjati et al., 2018). Unlike the disentanglement work, the introduced mask allows the adaptation to focus on the relevant content which substantially reduce the complexity of the generation. On the other hand, the proposed method extends the attention guided translation from the domain level to the instance level which allows more specific and diverse translations. Experiments on benchmark data shows both improved qualitative and quantitative results comparing to existing methods. It is really nice that the authors also considered the situation of generalization to out of domain images.