Masked Based Unsupervised Content Transfer
Authors: Ron Mokady
Sagie Benaim
Lior Wolf
Amit Bermano
Venue:  (2020)

Abstract: We consider the problem of translating, in an unsupervised manner, between two domains where one contains some additional information compared to the other. The proposed method  disentangles the common and separate parts of these domains and, through the generation of a mask, focuses the attention of the underlying network to the desired augmentation alone, without wastefully reconstructing the entire target. This enables state-of-the-art quality and variety of content translation, as demonstrated through extensive quantitative and qualitative evaluation. Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content. Furthermore, our method enables weakly-supervised semantic segmentation of the separate part of each domain, where only class labels are provided. Our code is available at https://github.com/rmokady/mbu-content-tansfer.


Published as a conference paper at ICLR 2020
MASK BASED UNSUPERVISED CONTENT TRANSFER
Ron Mokady', Sagie Benaim', Lior Wolf4., and Amit Bermano'
The School of Computer Science, Tel Aviv University FFacebook Al Research
ABSTRACT
We consider the problem of translating. in an unsupervised manner, between two domains where one contains some additional information compared to the other. The proposed method disentangles the common and separate parts of these domains and, through the generation of a mask, focuses the attention of the underlying network the desired augmentation alone, without wastefully reconstructing the entire target. This enables state-of-the-art quality and variety of content translation, demonstrated through extensive quantitative and qualitative evaluation. Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content. Furthermore, our method enables reakry-superrvied semantic segmentation of the separate part of each domain, where only class labels are provided. Our code is available at httpps http://w...ooooooooa////oooaoaoa -onttetttttstfer
INTRODUCTION
The task of content transfer, as depicted in Fig. I involves identifying the component of interest (for example, glasses) in given input (for example, an image of a face), adapting it, and adding it to a second given input (for example, another image of face, without glasses), hopefully in the semantically correct manner. Such an operation can be used to prototype or demonstrate changes in appearance Gatys et al. (2016), augment music Grinstein et (2018). compose text [Prabhumoye et al. (019)), generate data for training purposes Mueller et al. (0118) etc.
Recent advancements (Huang et al. (2018); Lee et al. (2019)) translate one domain to another with varying styles, but not content. Others (Lample et al. (2017): He et al. (2019): Liu et al. (2019) produce images in a target domain with a given attribute (e.g glasses), but such attribute is unique and not varying controlled (e.g.. not allowing the specification of a specific pair of glasses).
recent advancement in the realm attribute transfer has been presented by Press et al. (2019). In this work, the input is two domains of images, such that the images in one domain, B, contain specific class (e.g. faces with facial hair), while in the other domain. A, the images do not (c.g., faces without facial hair). Training on this input, the method learns to transfer only the specific class information from an unseen image in the domain B to an unseen one in the domain A, while preserving all other details. The proposed architecture yields a simple network which is able to perform the required disentanglement through an emergence effect. In this setting, the content to be added by the system is not explicitly marked in the target domain, nor does have equivalent counterpart in the source domain for training (e.g. an image with and without glasses of the same person). A form of weak supervision can be provided by simple annotation of whether the relevant content exists or not in every example. However, this method, and other typical ones addressing similar tasks, generate the details for the entire image, by using auto-encoder or GAN-based architectures, resulting in a degradation of details and quality.
In this paper, we build upon the emerging disentanglement idea, but also adopt the growing under- standing that one should minimize redundant use of computational resources and model parameters (Chen et al. (016)): Mejjati et al. (018)): Chen et al. (2018). ), to the aforementioned task. In other words, using mask, we focus the attention of the baseline network to the desired augmentation alone, without asking it to wastefully reconstruct the entire target. As can be seen in Fig the method consists of two main steps. The first is the disentanglement step, which encodes the domain specific

---

Published as a conference paper at ICLR 2020
Figure I: Content transfer example. Given an image of face with glasses (left), and another image of face without ones (top), the proposed method successfully identifies and translates the specified glasses from the former domain to the latter one.
and the domain invariant contents separately, and is inspired by the work of Press et al. (2019). The second step is the key insight of our proposal. It locates the part of the target that should be changed and generates relevant augmentation content to go with it. This allows keeping the unrelated details intact, facilitating great improvement in generation quality; The augmentation focuses the relevant part, leaving all other details to be taken from the target image directly, without going through the bottleneck of an auto-encoder-like module.
By applying this simple yet effective principle and a novel regularization scheme, our method preserves target details which are irrelevant the augmentation and is able to improve upon the state-of-the-ard in terms of quality and variety of the content transferred. Furthermore, we demonstrate how the method performs well, even when presented with images outside the domain trained on, and that the aforementioned mask, generated by the system to mark the regions of augmentation, is accurate enough provide a semantic segmentation of the content transferred.
Lastly, our method can also be used in the opposite direction - to remove the domain specific attribute, and thus translating from to A. Despite not being our main focus, we outperform the various literature methods that address this task (Lample et al. (2017); Liu et al. (2019): : He et al.) (2019); Press et al. (2019)). Our advantage is also evident in that once an object is removed, we can then add different attribute to the resulting image, thus allowing the translation from any two
domains, each having a domain specific attribute (e.g removing glasses and adding the smile of guide image). A related unique ability is that of removing and adding the same attribute, e.g. replacing one's 's facial hair with a different facial hair.
2 PREVIOUS WORK
In the unsupervised image image translation, the learner is given two unpaired domains of visual samples, and B, and is asked, given image a E A to generate an analogue image in the domain B. This problem is inherently ios-po multiple analogous solutions may exist. In several of different approaches (Zhu et al. (2017a); Kim et al. (2017) Yi et al. (2017) a circularity constraint is used to reduce this ambiguity. COGAN & (2ze) and UNIT LiTT'Luu et al.) (2017) enforce shared latent representation between the two domains. Unlike our method, these methods produce a single solution per input image a.
Moving from one to one mappings, multiple approaches provide many to many mappings. Supervised multimodal approaches, where paired samples are provided, include BicycleGAN Zhu et al. (2017)) which injects random noise in generator and enforces an encoder to recover z from the target translation, and MAD-GAN Ghosh et al. (2018). The latter trains multiple generators to produce aligned mappings, which are distant from each other. These methods require paired samples from both domains - costly supervision, which we do not require.
MUNIT Huang et al.] (2018). DRIT Lee et al. (2018), and DRIT+ Lee et al.] (2019) are trained on unmatched images. MUNIT trains two encoders; one captures the content of an image, and another its style, inducing disentanglement. During inference, multiple solutions are produced using the style a guide image in the target domain. In DRIT (and DRIT++), cycle constraint is employed, in setting where the generator of each domain consists of two disentangled encoders, one of which encodes the content and the second the style of the image. For all these methods, different non symmetrical architectures of encoders used to capture the
Guided Multimodal Approaches
2

---

Published as a conference paper at ICLR 2020
style and content. In MUNIT, for example, residual connections are used for the content encoder, and global pooling and adaptive instance normalization are used for the style encoder. Hence, the style code is significantly smaller in dimensions than the content one. In our method we employ two encoders as well, but the architecture of these encoders is symmetric, allowing both encoders to capture content in both.
The most relevant work to ours is that of Press et al. (2019) which also uses the setting in which the samples in domain B contain all the information in domain A and some additional information. Two encoders are used - the first captures the information that is common between the two domains and the second encodes the unique information of domain B. The decoder maps the concatenation of the two encodings into an image in the domain B, or, in the case that the second encoding is set to zero, to an image A. Content is transferred between images by mixing the encoding of the former type of one image with the encoding of the latter type of a different image.
In many cases, however, only a local area in the image needs to change during translation. Consider the case where A is images of faces and B is faces with facial hair. For E A, only the location in which the facial hair is placed in a needs to change. In the method of Press et al. (2019), the entire image, including other facial features, is generated from scratch, and as result, many low level details are lost and the quality of generation is reduced. This is not the case for our method, where outside the generated mask, which denotes the location of the facial hair, the content of the generated image is taken from the input image a. This is achieved by employing two decoders, one for domain A and one, with two outputs (raw image and mask) in domain B, and by new set of loss terms.
The use of masks is prevalent in a variety of visual tasks. For example, for virtual try on. Han et al. (2018) uses a supervised human parser network to transfer the desired clothing to a target person. Unlike our method, this method cannot perform general content transfer as it crucially relies on a supervised pose estimator. In the context of style transfer, Ma et al. (2018) uses masks to transfer the style of semantically similar regions from the target to the source image. In the context of image to image translation, the one to one case was addressed by Chen et al. (2018); Mejjati et al. (2018), where in addition to mapping to the target domain, a mask is learned, to cover only the relevant area in the translation, For example, in the case of mapping from horses to zebras, the mask learns to cover the area of the zebra, which allows the background to be taken from the source image, thus allowing for a much better quality of generation. In our method, we extend this masking (or attention) approach to the one to many (guided) case. Note that while Chen al. (2018); Mejjati et al. (2018) learn a mask and then employ it directly to the image, this does not allowthe mask to adapt both target and guide image.
Mask Based Approaches
Weakly supervised semantic segmentation methods can be stratified based on the type of super- vision used. In the first set of methods (Song et al. (2019): y |Hu et al. (2018); : Zhao et al. (2018)) bounding box is used. Other approaches (Cheng et al. (2018); Caelles et al. (017)) use the entire | supervision of the fully supervised approach, but are required to find segmentation in a single shot.
Our approach belongs to a set of methods ZZhng el al. (2018) Zhou et al. (2018); Weii el al. (2018); Ahn & Kwak (2018) that use only the class label information to find a segmentation. Zhou al.) (2018) use the visual cues arising from peaks in class response maps (local maxima) to generate highly informative regions. //hou et al. (2016) use Class Activation Maps (CAMs) extracted from a classifier to obtain discriminative localization. Wei et al. (2018) use varying dilation rates to transfer surrounding discriminative information to non-discriminative object locations. Ahnn & Kwak (2018) propagate local discriminative parts to nearby regions that belong to the same semantic entity. Other methods (Tang et al. (2018b); Kervadec et al. (2019): Tang et al. (2118a) use regularized losses with different levels of supervision. Unlike these methods, in a way that it is adapted to the placement context for the domain specific information and yet, our
main focus is generating the added part
segmentation results are competitive with such methods on this task.
3 METHOD
We transfer content that exists in a sample b in domain B onto a sample a from a similar domain A, in which this domain specific content is not found. In addition, also consider the task of weakly supervised semantic segmentation of the domain specific content. That is, given unpaired samples from domains A and B, we wish to label (or generate a segmentation mask for) the domain specific
3

---

Published as a conference paper at ICLR 2020
E
DA
Eg
Dg
Sepurata
Figure 2: An illustration of the inference procedure. The discriminator C' and decoder DA are not used during inference but are included for illustrative purposes.
part an image E B. We also consider attribute removal. That is, given the domain specific part of b.
B, we wish to remove
Our method consists of five different networks: the common encoder, Ee, aims to capture the common (or domain invariant) information between domains A and B. The separate encoder, Eg, aims to capture the separate (or domain specific) information in domain B. The domain confusion network, C, is used make the encodings generated by E. for images from both domains indistinguishabl.. The decoder, DA generates samples in domain A, given representation that is obtained by the common encoder E.. If that sample comes from domain B, the domain-specific content is removed.
The generation of the image that combines the content of and the domain specific content of b is done by the decoder Dg, which returns image-sized outputs: zaw and m.
m(a.b). =ma (a, = Dn (E)(aa), E.(b)
where m(a, b) is a soft mask with values between and and n is an image. is important to note that the mask and the generated image both depend the content in b well as the image which determines the placement and other appearance modifications.
The final output is combination of these outputs and image a,
za, b) ma, b) raw (a, b) (1 -m(a, ()) 2
where @ stands for an element wise multiplication. Fig. illustrates the inference step as well as the five networks.
Domain Confusion Loss. We seek to ensure that the common encoding, generated by Eg, contains only information that is common to both domains. This is done by combining reconstruction losses with domain confusion loss. The latter employs discriminator network, C, that encourages the encodings of the two domains to statistically match Ganin et al. (2016).
LDC = SSA
1
Σ aESA
((C(E(())))) + [SB beSs
Σ ((C(E.(b))
(3)
where SA and SB are the training sets sampled from the two domains and 1(p, q) - (glo(()
(1 9) loggl11 p)) is the binary cross entropy loss for p [0, 1] and q E {0. 1}.
Our formulation of the domain confusion loss similar to that of Tzeng et al. 20117 except where for LDC, Eg attempts to fool the discriminator C, that the encodings of both domain A and domain B would classified as 1. Namely, C tries to distinguish between encodings of domain and B, while E. attempts to produce encoding which is indistinguishable for C.
While E. attempts to make the two distributions indistinguishable, C trained in an adversarial
manner to minimize the following objective:
Lc
Σ (CC(E(a)).
Σ ((C(E(()). bESB
(4)
SA
[Sp
aESA
Reconstruction Loss encode any separate information from domain B. For samples A, we also need verify that the
The domain confusion loss ensures that the common encoder, E,,, does not
4

---

Published as a conference paper at ICLR 2020
information in (a) is sufficient to reconstruct it, ensuring that all the information of domain A is encoded by Ee. We use
C Recon1
1
Σ
[Da(E.(a)) all2
(5)
SAA
aES.
where |||| the LI loss directly applied RGB image values.
Similarly, we wish to verify that the information encoded by F. is sufficient for reconstructing the separate details, so that (B) contains the domain specific information of domain B. Given an image b E B, we do this by removing the separate information from it, using D (b)), and adding back:
L Reconl
Σ
I>((D((E(0)))bb b. bl12.
(6)
SS
where 2' is defined as:
(c, a, m(a, & nn (a, b) + (1 - ma, b)) 0
For Eq. 6 use z' instead of z. This is so Eg applied (b))). but directly on b. In both cases, one recovers the common information of b, but when using 2, additional error is introduced thought the use of D Ee.
Finally, we reinforce the roles of the two domains by encouraging the mask to be minimal. In our experiments, we saw that explicitly penalizing the mask size, or using other traditional regularization terms, yielded inferior results, as shown in Sec. 4.1 Instead, we achieve this goal in softer way, by running samples from each domains through both inputs of our transfer pipeline and favouring successful reconstruction:
L Recon2 =
Σ ||=(0,a) - all1 aESA
C Recon2 = S
Σ bESB
|= b) bll1
(8)
JSA
The first term of the loss introduced in Eq. (ecoom)) encourages a minimal distance between z(a, a) and a, where z(a, a) = 20ow(aa a) @ m(a, + a @ (1 m(a, a)). Ideally, z would be equal to a, but since we use an encoder and a decoder which cannot auto-encode perfectly, we get that there is some distance between zrew and Hence, in order to minimize the distance between z(a, a) and a, the network minimizes the size of the mask. Similar argument holds for LB R
Cycle Consistency Losses to encourage disentanglement. Specifically, we have:
Cycle consistency in the latent spaces is used as additional constraint
LCycle = SA
Σ
|| b)) (a(alla + | 6)) (b) (9)
aESA,bESH
where |||| the MSE loss.
The overall loss term we minimize is:
L = LDC + A& + -e + Xa >a + X4 A4 + \ A5 Recon2
where Ag...., As are positive constants. We train discriminator C separately to minimize Lc.
Inference The network's 's architecture is provided in appendix Once trained, the networks can be used for unsupervised content transfer and weakly supervised segmentation of the domain specific information. In the first case, generate examples 2(a, b) for A. E B. In the second, consider the mask generated by feeding an image from domain B to both inputs m(b, b). then apply threshold to get binary mask. As shown in appendix Fig. 37 the method is largely insensitive to the exact value of the threshold.
The network
also be used for attribute removal by generating Zunmasked D (b)). Fummasked is with its separate part removed. In order to avoid missing reconstructed facial features the generated output calculated as:
m(b, 6), (b), Eremoved = (1 ((b b)) 3 M(, Zunmasked-
where M is the binarized mask of the soft mask

---

Published as a conference paper at ICLR 2020
NNaH my
NNaHH m
mmmT LIC
INNAH my
IR
RMAH LiT
NNaH my
Figure 3: Glasses from guide images in domain (left) augment the glasses-less source images from domain A (top). The content transfer of Press et al. (2019) (middle) is compared to results (right).
4 EXPERIMENTS
We evaluate our method for guided content transfer, out of domain manipulation, attribute removal, sequential content transfer, sequential attribute removal and content addition, and weakly supervised segmentation of the domain specific content.
Guided Content Transfer We employ three attributes that are expressed locally in the images of the celebA dataset Yang al. (2015): smile, facial hair, and glasses. In each case, we consider B to be the domain of images with the attribute, and A to be the domain without it.
We first consider the ability add the separate part of an image b B the common part of a E A. This is shown for the domain of glasses, in Fig. 3 compared to the baseline method of Press et al. (2019). As can seen, only the local structure of the glasses is changed, whereas the baseline many low level details are lost (for example, the background writing) and unnecessary changes made (for example, an open mouth is replaced with a closed one, or facial hair is added, changing the identity of the source image). Furthermore, Fig. demonstrates the ability of our method to accommodate for different orientations of the source image a, and to properly adapt the glasses from b to the correct orientation. Please refer the appendix B for more examples.
To assess the quality of the domain translation, we conduct a handful of quantitative evaluations. In Tab. we consider the Frechet Inception Distance (FID) Heusel et al. (2017) and Kernel Inception Distance (KID) Binkowski et al. (2018) scores of images with the common part of and separate part of b over test set of images from domains and B. The FID score is commonly used metric to evaluate the quality and diversity of produced images; KID is recently proposed alternative for FID. We note that these values should only be used comparatively, as the size of the test set used affects the score magnitude. As can be seen, our method scores significantly better.
(FID)Heusell
We also consider the ability our method to transfer the separate part to the target image. To do so, we use a pretrained classifier to distinguish between domains A and B (on the respective training sets) and consider the score the translated images. These results are reported in Tab. and show clear advantage to method. As expected, the MUNIT and DRIT methods Huang et al. (2018) Lee cct al. (2018) are not competitive in this metric, since they transfer style and not content. Additionally, in contrast to method, Fader networks (Lample et al. (207)) transfer to B without the of specific guide image from this domain.
To evaluate if the source identity is preserved, we compute the cosine similarity of the pretrained VGG-face network (CooookCCooooo et al. (2017). High values indicate preserved identity. Tab, [3 indiicates that results exhibit much better similarity source images than baseline methods.

---

Published as a conference paper at ICLR 2020
Table 1: FID and KID scores (lower is better) for generated images using the common part of A and the separate part of E B. As real images, we consider the images in A. For KID we used y 0.01, kemne kx,, y) (yæ' y + 1)9.
Table 2: The accuracy of generated images according to a pretrained clas- sifier distinguishing between A and B.
Smile Glasses Beard
Facial hair
Glasses
Fader
93.9 % 93.6%
81.8%
Method
FID
KID
FID
KID
Press et al.
98.9%
94.8%
88.1%
Real images
85.4:2.9
2.5+0.2 115.5#3.8 0.1
MUNIT
8.5%
8.3%
7.2%
Ours
90.7+t..8
3.5
134.944.8 5.2+00.8
DRIT 9.2%
7.4%
6.5%
Press et al.
139.4 1.9 16.8 178.5++3.2 14.6
Ours
99.2%
96.2%
88.0%
Table 3: An evaluation of the cosine similarity (higher is better) before and after translation between the VGG-face descriptors. Shown are average results over 100 random images created by sampling and from the test sets.
A to B mapping
Transfer A' B'
Facial hair Glasses male to male all genders all genders male to female train women, test men
Smile
Facial hair Glasses
Ours
0.89
0.84
0.94
0,90
0.82
Press et al. (019
0.73
0.68
0.73
0.64
0.59
Table 4: User study (questions (1), (2) and (3)) showing preference our method vs. Press al. (011)), see text.
A to B mapping
A' to B' shift
Facial hair male to male
Glasses all genders
Smile all genders
Hand- bags
Two Attrs
Remove Smile Add Glasses
Facial hair swap
Facial hair
Glasses
train women
female A', B'
test men
(1)
96%
95%
55%
87%
91%
83%
93%
91%
93%
84%
82%
43%
72%
93%
90%
83%
70%
86%
97%
95 %
95%
90%
95%
91 %
91%
91%
97%
To evaluate the interpretability of the latent space, we interpolate between the latent code of the separate parts of images 62 E B and b2 E B with the common latent code of an image a E A. This is shown in Fig. and appendix
C
Note the mask changes throughout the interpolation.
User study To further strengthen the evaluation, we conduct a user study. We randomly sample 20 images from E A and b E B and consider the translated image of our method vs. that of Press et al. (2019). We conduct three experiments where the user is asked to select: (1) the translated image that matches the distribution of B more closely, (2) Given the guide image b, in which translated image, the separated part of b is better transferred, and (3) Given the source image a, which translated image better preserves the facial features of a. Average scores are reported in Tab, 4 For the tasks of facial hair and glasses, we score consistently higher than the baseline method. For smile, our ability to produce realistic smiling faces is slightly higher, the ability transfer the smile from the source image is slightly worse, while our ability to preserve the identity of the source image is significantly higher. This probably stems from the smile taking place not only in the specific mouth region.
Out of domain manipulations We also consider the ability of the learned model handle domain shift, i.e. to perform a translation from a domain which was not seen during training. For example, we train on female faces without glasses as domain A, and female faces with glasses for domain B. At test time, is replaced with a domain A' of male faces, and we are asked to transfer the glasses onto the male's 's face, generating domain B' from which we no train or test samples. Quantitative evaluation is provided in Tab. showing a negligible difference quality for our method, and a significant one for the baseline method. Visual results can be found in appendix C where also consider out of domain LFW dataset dataset/Huang et al. (2007) as well as extremely out of domain images, which our method successfully handles.

---

Published as a conference paper at ICLR 2020
Figure 4: Interpolation between (bi) (bottom left) and (b2) (bottom right) for bj, by E B, while fixing the source image A (top ends). The generated images (top) and masks (bottom) are shown.
Table 5: Mean and SD IoU for the two hair seg- mentation benchmarks.
Method
Women's 's hair
Men's hair
Ours
0.77 0.15
0.77 = 0.13
Press et al.
0.67 0.13
0.58 = 0.11
Ahn & Kwak. CAM
0.54 0.10
0.52 = 0.10
0.43: 0.09
0.56 = 0.07
Figure 5: Adding handle to handbag,
Handbags We also consider the domain of handbags Zhu et al. (2016), where we split this domain into images with handle (B) and those without (A).. The transfer results are illustrated in Fig. 5 The generated mask and raw outputs are clearly adapted to the bag on which the handle content is placed. The user study in Tab. evaluates these results. Please refer appendix D for visual comparison.
Attribute removal While our method is more general than attribute transfer methods, it can be used to remove given attribute shown in Fig see appendix [or full qualitative comparison to the literature methods. A quantitative evaluation is given in Tab 6 For generation quality we use KID and FID; for successful attribute removal, a pretrained classifier used to measure the percentage of test images without the attribute, and for similarity with the source image, perceptual loss is used (using the features of VGG-face Cao al. (2017) network). Our method is significantly superior in terms of generation quality over all baseline methods for all tasks and presents a good tradeoff between fidelity and transformation success. For facial hair removal, |Press et al.) (2019) and Lample et al. (2017) are superior in terms of classifier accuracy. yet their generation quality is far worse (blurry images) and the similarity to the source image is significantly impaired. As the comparison images in appendix [ show, Lample et al. (2017) achieves higher accuracy by producing female images while Press et al. (2019) makes the persons younger looking. He et al. (2019) has slightly superior similarity score, but worse removing the facial hair and has worse generation quality. For smile, Liu et al. (2019) is slightly superior in removing the smile in terms of accuracy, yet worse on generation quality and similarity to the source image. For glasses, all classifier scores close 100% meaning an almost perfect glasses removal, yet our similarity score and generation quality is higher.
Sequential content transfer Our method enables sequential addition of guided content from different guide images and from different domains by applying our method sequentially. Fig 28 considers this case for glasses and facial hair addition. Our method significantly outperforms Press al. (2019), as it does not wastefully reconstruct the facial features twice as shown in Tab (ttwo attributes") for adding facial hair and glasses.
Attribute removal and content addition Given the ability our method to remove a specific attribute, one can perform guided content transfer between any given domain, A and B, each with separate domain specific information. First, we remove the domain specific attribute of domain A and then perform guided content addition for domain B. For example, in Fig, 4 smile is removed and glasses are then added, see appendix for more results, as well as facial hair swap in appendix Fig As for sequential content transfer, do not wastefully reconstruct the facial features and so sigmificantly outperform Press et al. (2019) as can be seen in Tab for the task of smile removal and glasses addition as well facial hair swap (removing and adding facial hair).
8

---

Published as a conference paper at ICLR 2020
Figure 6: Attr removal.
Table 6: Attribute removal for the task of Smile, Facial hair and Glasses.
Facial Hair
Task
Method
KID
FID
Class.
Sim.
Smile
Ours
2.6 0.4
120.0 + 2.6
96.9%
0.96
Press et al.
15.0 +: 0.6
167.7 +: 0.3
96.9%
0.81
He al.
4.1 0.4
127.7
4.5
96.9%
0.95
Liu et al.
0.3
129.0 3
98.4%
0,92
Fader
11.3
155.6 4.7
93.7
0.89
Mustache
Ours
1.5
119.0
0.8
95.3
0.95
Smile
Press al.
16.6
175.9 + 1.4
100.0%
0.80
He al.
0.5
130.0
87.5%
0.96
Liu et al.
0..0
3.3
87.5%
0.85
Fader
1..1
162.6 1.5
98.4
0.76
Glasses
Ours
5.2+
136.5 2.6
99.2%
0.87
Glasses
Press al.
0.5
172.0
100.0%
0.73
He et al.
8.3
141.4=6.8
100.0%
0.84
Liu et al.
+.3
141.8 4.8
98.4%
0.86
Fader
12.5 0.3
137.7 4.2
100.0%
0.76
Figure 7:
Removal of smile and addition of glasses according to the guided image on the left. In the middle, the translation of Press al. (2019) and on the right, our result.
Weakly supervised segmentation We consider the task of segmenting women's and men's hair. For men, consists of bald men, while contains men with dark hair. For women, consists of women with blond hair, while B contains women with black hair. We evaluate our method using the labels given in Borza al. [Borza et al. (2018).
We generate the segmentation using the method described in Sec. 3 We compare our method to Press al. (2019), where we take the translated image and subtract, in pixel space, the source image from We also compare to the results obtained by the recent weakly supervised segmentation method of Ahn and Kwak Ahn Kwak (2018), which performs segmentation using the same level supervision we employ, using published code. In addition, compare to CAM Zhou et al. (2016), where train an Inception- network to classify between the domain and extract localization from the classifier, which we then binarize to get segmentation mask.
As can be seen in Fig. our results provide smooth labeling of the hair, while Press et al. (2019) yield broken with unnecessary details. The result of Ahn & Kwak (2018) also lacks in comparison. CAM is unable to generate the required shape, as the classifier always focuses on the same place. Similar results shown for man's hair in the appendix G Our results also superior quantitatively, as shown in Tab. 5 for the Intersection over Union (ToU) measure. We also perform semantic segmentation for both glasses and facial hair, refer to appendix G The success our method stems from requiring the network to minimally add the separate content in the correct location to reconstruct b E B, thus localizing the separate content.
4.1 ABLATION ANALYSIS
An ablation analysis is performed and reported quantitatively Tab. and visually in appendix for the task of facial hair content transfer. Without LB ftecon1 and LA Ricconl the masks produced are

---

Published as a conference paper at ICLR 2020
Table 7: The effect of removing losses. Shown are classifier accuracy, cosine similarity, KID, and percentage mask from the total size of the face.
Class. Sim.
KID
Perc.
C
88.1%
0.89 3.5 0.1 23%
w/o L
88.5%
0.85
4.1 0.5 34%
Recon2
w/o L Recon2
88.1%
0.87
4.2+0.44 65%
w/o Cycle
67.1%
0.95
4.1 0.9 29%
Figure 8: Segmentation of women's 's hair. (a) orig-
w/o L
Recon1
9.4%
1.0
4.3
0%
inal image, (b) ground truth segmentation, (c) our
w/o LA
Rocoml
9.7%
1.0
4.6
0%
results, (d) the results of Press et al. (2019), (e)
w/o LDC
9.5%
1.0
5.0
0%
the results of Ahn & Kwak (2018). 8), (f) the results of CAM.
1.2 reg
88.0%
0.82
4.6
33%
1.2 recon #1
87.7%
0.89
3.3 + 0.5
22%
L2 recon #2
74.2%
0.93
4.2 + 0.6
30%
empty and hence facial hair is not transferred to the target image, indicated by the high cosine similarity values but low classifier scores (i.e., the classifier labels the output as belonging to domain A). Similarly, without LDC, masks produced are empty as no disentanglement is possible.
Without Lyyele the masks produced include larger portions of the face, which also maintains similarity but hurts the classification score. L Recon2 and L Recon2 play lesser role for the mask. Without Recon2 the mask is less smooth, and without L Recon2 the mask still captures additional objects (e.g eyes). In fact, Liccoo is a way to enforce the mask to capture the relevant content in self- regularizing way. Recon2 and LB Recon2 are dependent on z, which is semantically aware of the domain specific content, while L2 equally penalizes any region of the image regardless of its content. When trying to use L2 norm, the mask had to be carefully adjusted to each experiment and resulted in a non-smooth mask which covers unnecessary parts of the face. This can be seen visually in appendix Fig 36 and from the "L2 reg" entry of Tab. where L2 regularization is used instead of C Recom2 and L Recoa2 We note that Chen et al. (018)) uses sparsity regularization on the masks and Mcjjati
B
et al. 2018 uses early stopping, which we do not require due to the regularization of Recor2 and C33
Refer to appendix H for further discussion.
Recon2
CONCLUSIONS
When transferring content between two images, we need to know what transfer, where to transfer it to, and how to transfer it. Previous work in guided transfer either transferred global style properties or neglected the "wheee" aspect, which ultimately lead to an ineffective generation that lacks attention.
In our work, the "what" aspect captured by E., and D captures both the "where" and the "how". Our results demonstrate that the context (image a) in which the content is placed determines not just the location of the inserted content but also the form in which it is presented, where both aspects can vary dramatically, even for a fixed content-gad -uide image b. The comprehensive modeling of the guided content transfer problem leads to results that are far superior to the current state of the art. In addition, the modelling of ""where" allows supervised way, remove content, swap content between images, and add multiple contents without suffering gradual degradation in quality.
obtain accurate segmentation masks in weakly
ACKNOWLEDGEMENTS
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant ERC CoG 725974).
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-Ievel semantic affinity with image-level supervision for weakly supervised semantic segmentation. In The IEFF Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Mikotaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: (801,11400) 2018.
Diana Borza, Tudor Ileni, and Adrian Darabant. A deep learning approach hair segmentation and color extraction from facial images. In Interatiional Conference on Advanced Concepts for Intelligent Vision Systems, pp. 438 449. Springer, 2018.
S. Caelles, K.-K. Maninis, J. Pont-Tusee L. Leal-Taixé, D. Cremers, and L. Van Gool. One-shot video object segmentation. In Computer Vision and Pattern Recognition (CVPR), 2017.
Qiong Cao, Shen, Weidi Xie, Omkar Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. arXiv preprint arXiv:1710.08092, 2017
Liang-Chieh Chich Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale: Scale-aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3640-3649, 2016.
Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao. Attention-gan for object transfiguration in wild images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 164-180, 2018.
J. Cheng. Y.-H. Tsai, W.-C. Hung. S. Wang, and M.-H. Yang. Fast and accurate online video object segmentation via tracking parts. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35, 2016.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks, In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2414-2423, 2016.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi- agent diverse generative adversarial networks. Proceedings of the IEEE Conference on Computer 2018.
Vision and Pattern Recognition, pp.
E. Grinstein, N. Q. K. Duong, A. Ozerov, and P. Pérez. Audio style transfer. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 586-590, April 2018. doi: 10.1 10.1009/CAASSP22018...60 88.66711...
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In CVPR, 2018.
Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen. Attgan: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 28(1 1):5464-5478, Nov 2019. ISSN 1057-7149. doi: 10. 11110/TTT220192211755..
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4233 4241, 2018.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Lealed-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
11

---

Published as a conference paper at ICLR 2020
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image- to-image translation. ECCV, 2018.
Hoel Kervadec, Jose Dolz, Meng Tang, Eric Granger, Yuri Boykov, and Ismail Ben Ayed, Constrained- enn losses for weakly supervised segmentation. Medical Image Analysis, 54, 02 2019. doi: 1..10166..eddia.211999
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. arXiv preprint arXiv::703.0000 2017.
Sebastian Korb, Stéphane With, Paula M. Niedenthal, Susanne Kaiser, and Didier Grandjean. The perception and mimicry of facial movements predict judgments of smile authenticity. PloS one, 2014.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, et al. Fader networks: Manipulating images by sliding attributes. In NIPS, pp. 5967-5976, 2017.
Hsin-Ying Lee, Hung-Yo Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In The European Conference on Computer Vision (ECCV), September 2018.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Kumar Singh, and Ming-Hsuan Yang. Drit++: Diverse image-to-image translation viadisentangled representations. arXiv preprint 190.vv1905.01222 2019.
Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen. Stgan: A unified selective transfer network for arbitrary image attribute editing. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In NIPS, pp. 469 469-4777 477, 2016.
Ming-Yu Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NIPS, 2017.
Liqian Ma, Xu Jía, Stamatios Georgoulis, Tinne Tuytelaars, and Lue Van Gool. Exemplar guided unsupervised image-to-image translation. arXiv preprint arXiv: / /455, 2018.
Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim.
Unsupervised iitention-guided image-to-image translation. In Advances in Neural Information Processing Systems, pp. 3693-3703, 2018.
Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. Ganerated hands for real-time 3d hand tracking from monocular rgb. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 49-59, 2018
Shrimai Prabhumoye, Chris Quirk, and Michel Galley, Towards content transfer through grounded text generation. In Proc. of NAACL, 2019.
Ori Press, Tomer Galanti, Sagie Benaim, and Lior Wolf. Emerging disentanglement in auto-encoder based unsupervised image content transfer. International Conference on Learning Representa- tions, 2019. URL htttts::/cpenrevieeeet net wtt:://:mmmidi11111eeme1ee
Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven class-wise region mask- ing and filling rate guided loss for weakly supervised semantic segmentation. arXiv preprint arXiv:I904. 1693, 04 2019.
Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Normal- ized cut loss for weakly-supervise enn segmentation. 2018 JEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 1818-1827, 2018a.
12

---

Published as a conference paper at ICLR 2020
Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. On regularized losses for superv-seperviied cnn segmentation. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp.
524-540, Cham, 201 Springer International Publishing. ISBN 978-3-030-0117000
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2962-2971, 2017.
Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S. Huang. Revisiting dilated convolution: A simple approach for weakly- and semi-supervised semantic segmentation. In CVPR, 2018.
Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. From facial parts responses to face detection: A deep learning approach. In ICCV, pp. 3676-3684, 2015.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. DualGAN: Unsupervised dual learning for image-to-image translation. arXiv preprint arXiv: 1704.00500. 2017.
Xiaolin Zhang, Yunchao Wei, Guoliang Kang, Yi Yang, and Thomas Huang. Self-produced guidance for veakly-supervised object localization. European Conference on Computer Vision. Springer, 2018.
Xiangyun Zhao, Shuang Liang, and Yichen Wei. Pseudo mask augmented object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4061-4070, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Weakly supervised instance segmentation using class peak response. In CVPR, 2018.
Jun-Yan Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexci A. Efros. Generative visual manipula- tion on the natural image manifold. In ECCV, 2016.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-mmage translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017a.
Jun Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NIPS, 2017b.
A ARCHITECTURE AND HYPERPARAMETERS
We consider samples in A and B to be images in 128x128 The encoders F, and F,, each consist of convolutional blocks. Similarly, D and D3 consist of 6 de-convolutional blocks.
A convolutional block de consisting of: (a) 4 x 4 convolutional layer with stride 2, pad 1 and k filters (b) a batch normalization layer (c) a Leaky ReLU activation with slope 0.2. Similarly, a de-convolutional block чy consists of: (a) 4 x 4 de-convolutional layer with stride 2, pad and k filters (b)& a batch normalization layer (c) a ReLU activation.
The structure of the encoders and decoders is then: Egi da2, der d12s. d256, d512-eep 2- Eg: daz, dos, d128, d128, d128, deeps DA U512 U256. U128; uss, U32 Ug and DB: U512, u255. U128. U64, U32, u4
The last layer of DA (ug) differs in that it does not contain batch normalization and tanh activation is applied, instead of ReLU. Dg 's last layer (ut) similarly does not contain batch normalization. The output is of size 4 x 128 x 128. We split the output to a mask (first channel) and raw output (other three channels). We apply sigmoid activation the mask to get values between 0 and and tanh
13

---

Published as a conference paper at ICLR 2020
activation for the raw output and Tanh activation for the raw output. sep is the dimension of the separate encoders, set to be 100 for all datasets.
The discriminator C' consists of fully connected layer of 512 filters, Leaky ReLU activation with slope 0.2, a second fully connected layer of one filter and a final sigmoid activation.
We the Adam optimizer with 31 0.5, 52 0.999, and learning rate of 0.0002. We batch size of size 32 in training.
We constructed the train/test sets using 90%-95% split. This consists of about 7,200- 18,000 examples for train and about 800-2,000 examples for test for each attribute.
B ADDITIONAL CONTENT TRANSFER RESULTS
Additional results to the ones presented in the main text are provided here.
Fig. 9 gives a comparison of our method to the state-of-the-art for the transfer of facial hair. Fig. T0 provides additional interpolation results for this task while Fig. TT provides additional content transfer results. Fig. 12 shows the masks generated for this content transfer. Fig. 13 gives an example of the raw output given by our method for this task.
Fig. 14 gives additional results for the task of glasses transfer, while Fig. 15 shows the masks generated for this content transfer. Fig. 16 provides additional comparison to the baseline method.
Fig. 17 and Fig. 18 provide additional content transfer results and the generated masks for the task of smile transfer. It is well known that smile includes not only the mouth but also other facial features such eyebrows and checks Korb et al. (2014), thus when our method transfer the smile, it transfer all the relevant facial features for the smile as can be seen in the generated masks in Fig. 18 Fig. 19 provide interpolation results and Fig. 20 gives comparison of our method for this task.
gives Fig gives a comparison of our method to Mejjati et al. (2018). While our method uses guidance image which allows one many translation, Mejjati et al. 2018 translate only one image.
C ADDITIONAL OUT OF DOMAIN MANIPULATIONS RESULTS
Fig. 24 shows sample results where the mapping of facial hair is applied to female faces. Out of distribution translation where the train domains are different from the inference domains are shown for glasses in Fig 23 We further consider the ability of our method to perform translation on images from the out-of-din of-distributioon LFW dataset Huang et al. (2007) (Fig(2(aa). as well as images of an alien and baby (Fig (b))) which are extremely of distribution, all not present during training. As can be seen, even in these cases, our method successfully transfers the desired content.
D COMPARATIVE RESULTS FOR THE HANDBAG DATASET
Fig. 25 gives a comparison our method for the handle transfer for handbags, while Fig. 26 provides more results for this task (on top of Fig. 5).
E A QUALITATIVE COMPARISON OF OUR METHOD TO LITERATURE METHODS ON THE ATTRIBUTE REMOVAL TASK
Fig. 27 gives a comparison of our method on the task of attribute removal. The translation Press et al. (2019) is blurry and suffers from many of the facial features being lost. For example, for glasses removal, the men on the right have facial hair which lost in the translation. Lample et al. (017)) completely changes the facial features. For example, for facial hair removal, the gender seems to change from men to women. He et al. (2019) and Lid et al. (2019) are unable to remove the mustache and the translation is of lower quality in general. For example, for the glasses removal, for the in the middle, the translation is unnatural around the eyes. Our translation is of consistently higher quality for all tasks and successfully removes the desired attribute.
14

---

Published as a conference paper at ICLR 2020
pink
(a)
(c)
Figure 9: (a) Guide images in domain B (faces with facial hair). (b) Results by the method of Press) et al. 2219): the top row is the source images in domain A. The others incorporate the facial hair from the corresponding row of (a). (c) Same mapping for our method.
IHU
Figure 10: Facial Hair Interpolation
F ADDITIONAL SEQUENTIAL CONTENT TRANSFER AND ATTRIBUTE REMOVAL RESULTS
We provide additional images produced by our method as well as by the baseline method. As can be in order perform the guided content transfer of two attributes from different domain, Press et al. (2019) passes the source input image into the network twice which wastefully reconstructs static facial features twice.
For example, for sequential addition of glasses and facial hair, as seen in Fig 28 our method successfully transfers the attributes, while Press al. (2019) not only produces blurry images,
15

---

Published as a conference paper at ICLR 2020
M
Figure 11: Additional results for the guided transfer of facial hair.
but is much worse at transferring the content from both attributes. This observation is also supported by the user study performed in Tab 4
Fig. 29 show the comparison to the baseline model for the task of closing the mouth and adding glasses; Fig. 30 shows additional results from our method for this task. Fig, 3 provides the comparison for the task of replacing facial hair; while Fig. 32 presents additional results for this task.
G ADDITIONAL WEAKLY SUPERVISED SEGMENTATION RESULTS
Fig 33 gives comparison method for the task of men's hair segmentation given in section 4.2 of the main text, while Fig 34 gives additional results for the segmentation of woman's 's hair.
Additional segmentation results shown in Fig. 35 for the domain of glasses and facial hair. In this domain quantitative results cannot be obtained due to lack of ground truth segmentations.
H ADDITIONAL ABLATION STUDY DISCUSSION
While the loss in Eq. directly affects the mask generation, the losses in Eq. 3 (Cpc) and Eq. Aecom) affect it indirectly. Without the loss in 3 (Cpc), no disentanglement possible, and the common encoder would contain all of the image information including the separate information.
16

---

Furthermore, without the loss of Eq. (C Recoma) we empirically observe that D (b)(b) outputs the image with the specific part intact (for example, the facial hair is not removed). This indirect effect on the disentanglement probably stems from the fact that without this loss, there is reconstruction only on faces with facial hair (running example for the specific part). Thus, Eg can encode generic facial Published as a conference paper at ICLR 2020
Figure 12: Masks generated for the guided transfer of facial hair experiment. Masks generated are for the translated images Fig
This means that the image produced by (x(E((b) is close to and, therefore, the generated mask is empty.
hair information for shaved faces and have (b) and (a) still indistinguishable. Eq. 5 (C Recon1) makes sure that E, won't t encode facial hair for shaved faces, since it requires reconstruction of an image without facial hair.
We further consider the effect of replacing the norm used for reconstruction losses from LI to L.2 When using L2 norm for LB Recoon and L Recon1 the result is comparable both numerically ('L.2 recon #1" in Tab. 7b and visually (see appendix Fig 36). When using L2 norm for LB ('L2 recon #2" Tab. 7 the results change more significantly: the size of the mask is larger and the classifier score significantly lower. We attribute this to the sparsity inducing effect of the I.1
Recon2 and A Recon2
17

---

Published as a conference paper at ICLR 2020
Figure 13: Raw outputs generated by Dü for the task of facial hair content transfer.
I VISUAL RESULTS OF THE ABLATION STUDY
Fig 36 shows the masks generated when different losses removed as discussed in the ablation analysis of section 4.3 of the main text.
J HYPERPARAMETIE SENSITIVITY
The À coefficients were set in a way that reflects their relative importance and were observed. For example, if the mask obtained was too large we would increase L Recon2 and L Recon2- As illustrated in Fig. 38 and Fig. 39 our network is not overly sensitive to the choice of these values. For example, for L Recoom: each value in the range 0.4-1.0 results similar output and for A each value in the range 3.0-7.0 results in similar output.
Fig. 37 shows the affect of the threshold used to binarized the mask for the task of segmentation.
Fig. 38 and Fig. 39 show that the network is not overly sensitive to the choice of A and x respectively.
18

---

Published as a conference paper at ICLR 2020
Figure 14:
Additional content transfer example. Given an image with glasses (left), and another image of face with no glasses (top), the proposed method identifies and translates the specified glasses from the former domain to the latter.
19

---

Published as a conference paper at ICLR 2020
g
Figure 15: Masks generated for the guided transfer of glasses experiment. Masks generated are for the translated images in Fig [4
20

---

Published as a conference paper at ICLR 2020
NNaH my
NNNa m
EP
INNaH my
IR
RIIH LOT
NNNAH m
LAT
NNNaH m
ap.
HT
amaa LO
NNNAH m
(a)
(b)
(c)
Figure 16: (a) Guide images in domain B (glasses). (b) Press et al. (2019) method: the top row is the source images in domain A. The others incorporate the glasses from the corresponding row of (a). (c) Same mapping for our method.
21

---

Published as a conference paper at ICLR 2020
LAG
ST
Sm
Sr
Figure 17: Additional content transfer example. Given image of smiling face (left), and another image a non-smiling face (top), the proposed method identifies and translates the specified smiles from the former domain the latter.
22

---

Published as a conference paper at ICLR 2020
Sm
UN
Figure 18: Masks generated for the guided transfer of smile experiment. Masks generated are for the translated images in Fig 17
Figure 19: Smile Interpolation
23

---

Published as a conference paper at ICLR 2020
NON
MO
(a)
(b)
Figure 20: (a) Guide images in domain B (faces with smile). (b) Press et al. (2019): the top row is the source images in domain A. The others incorporate the smile from the corresponding row of (a). (c) Same mapping for our method.
24

---

Published as a conference paper at ICLR 2020
IR
R
tc mj
Figure 21: Our method (left) compared to Mejjati et al. (2018) (right) on the task of adding glasses to the original image (top). We show the generated masks and the final result for both methods. For our method we also show the guidance images.
25

---

Published as a conference paper at ICLR 2020
Figure 22: Out of domain translation. (a) Results extremely out of domain images. (b) Results
obtained by manipulating LFW images.
IILL
(a)
(b)
(c)
Figure 23: Out of distribution translation. The mapping between faces without and with glasses is trained only women and applied to men. (a) Guide images in domain B' (men with glasses) and top row is the source images in domain A' (men without glasses), both not given during training. (b) The remaining rows translated images by the method of Press et al. (2019). (c) Same mapping for our results.
26

---

Published as a conference paper at ICLR 2020
A
.
AM
it
IG
A
Figure 24: Additional out of distribution translation results. We train mapping facial hair from male faces (left) to male faces, and apply this to women's s faces (top) during inference time. Our domain translation results (right) compared to those of Press et al. 2019 (middle).
27

---

Published as a conference paper at ICLR 2020
(b)
(c)
Figure 25: (a) Guide images in domain B (handbags with handles). (b) Press et al. (2019) the top row is the source images in domain A. The others incorporate the handles from the corresponding row of (a). (c) Same mapping for our method.
28

---

Published as a conference paper at ICLR 2020
Figure 26: Additional content transfer example. Given image of bag with a handle (left), and another image a handbag (top), the proposed method identifies and translates the specified handbag from the former domain the latter.
29

---

Published as a conference paper at ICLR 2020
Original
Press
Fader
DIGOSTORE
AttGAN
STGAN
Ours
Original
Press
Fader
AttGAN
STGAN
Ours
Figure 27: Attribute removal for the task of mustache (top left), smile (top right) and glasses (bottom left). The result of our method is shown alongside the baseline methods, Press al. (011). Fader (Lample et al. (2017)), AllGAN (He et al. (2019) and STGAN (Liu et al. (2019)).
30

---

Published as a conference paper at ICLR 2020
IR IR
R
R
IR
Figure 28: First two images on the left are the content donors applied sequentially, either with facial hair glasses. The top is the input source images. The results the bottom the translation of Press et al. (2019) while on the top are our results.
31

---

Published as a conference paper at ICLR 2020
68
ATOGRAFICA
Figure 29: Additional removal and content transfer results. Given image with glasses (left), and another image of a face with no glasses and a smile (top), the proposed method removes the smile and identifies and translates the specified glasses from the former domain the latter. In the middle are the translated examples of Press et al. (2019) while on the right are our translated results.
32

---

Published as a conference paper at ICLR 2020
IR
TH
R
tc mp
ing
Figure 30: Additional removal and content transfer results for smile removal and glasses addition. Given an image with glasses (left). and another image of face with no glasses and smile (top), the proposed method removes the smile and translates the specified glasses from the former domain to the latter.
33

---

Published as a conference paper at ICLR 2020
Figure 31: Facial hair swap results. Our method first removes the facial hair and then adds the facial hair of the guided image on the left. In the middle are the translated examples of Press et al. (019
while on the right are our translated results.
34

---

Published as a conference paper at ICLR 2020
in
an
i
ii
ii
in
iina
nn
iio
iir
in
intk
i
in
Figure 32: Additional facial hair swap results. Our method first removes the facial hair and then adds the facial hair of the guided image on the left
35

---

Published as a conference paper at ICLR 2020
(a)
(b)
(d)
Figure 33: Segmentation of men's hair. (a) original image, (b) ground truth segmentation, (c) our
results, (d) the results of Press et al. 2019), (e) the results of Ahn & Kwak (2018). (f) results of CAM.
36

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(c)
Figure 34: Additional Segmentation results for women's 's hair. (a) original image, (b) ground truth segmentation, (c) our results, (d) the results of Press al. (2019), (e) the results of Ahn & Kwak (2018). (f) results of CAM.
Figure 35: Additional segmentation results for the domain of glasses and facial hair.
37

---

Published as a conference paper at ICLR 2020
11
1
1
1
Figure 36: Ablation analysis. The first row is images without facial hair on which we want to transfer the facial hair of the image in the first column. Second row: All losses L, third row: without CA A fourth row: without C
Recon2: fifth row: without LCydte sixth row: without LB Recomm seventh row: without CA Recor1 eighth row: without LDC. The ninth row shows the translation where L Recon2 and fitccmm are replaced by L2 regularization of the mask. The tenth row: LI norm is replaced with L2 norm for LA Reconz and C Recong. The last row: LI norm replaced with L2 norm for CA Cecona and
CB Recon1
38

---

Published as a conference paper at ICLR 2020
83
00
8
5C
8
00
0
c
CKD
mp
Figure 37: The effect of the threshold used binarized the mask. The left column is the original image and rest of the column are the segmentation mask created using the thresholds: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 (from left to right).
Figure 38: Sensitivity changes in \4. Given image with glasses (left), and another image of face with no glasses (top), the proposed method translates the specified glasses using different values of A4; 0.4, 0.5, 0.7, 0.9, 1.0 (from left right). All other hyperparameters remain fixed.
39

---

Published as a conference paper at ICLR 2020
Figure 39: Sensitivity changes in λ₁ Given image with glasses (left), and another image of face with no glasses (top), the proposed method translates the specified glasses using different values of A 3.0, 4.0, 5.0, 6.0, 7.0 (from left right). All other hyperparameters remain fixed.
40


Reviews:
[{"review_id": "S1xJ_D2jtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed a mask based approach for instance-level unsupervised content transfer, which is an extension of the disentanglement work in (Press et al., 2019) and the attention guided translation (Chen et al., 2018, Mejjati et al., 2018). Unlike the disentanglement work, the introduced mask allows the adaptation to focus on the relevant content which substantially reduce the complexity of the generation. On the other hand, the proposed method extends the attention guided translation from the domain level to the instance level which allows more specific and diverse translations. Experiments on benchmark data shows both improved qualitative and quantitative results comparing to existing methods. It is really nice that the authors also considered the situation of generalization to out of domain images.\n\nHowever, I would encourage the authors to spend more discussion on the \"Method\" and \"Ablation Analysis\" sections to give a better illustration. First is the choice of the L2 norm in all the reconstruction losses, which is different from L1 norm used in both (Press et al., 2019) and (Mejjati et al., 2018). What is the advantage of using L2 instead of L1 norm here? Does it work better with the mask generation? Second, the domain confusion loss. The presence of both equation (3) and (4) are quite confusing and the domain confusion loss (3) seems different from traditional ones. In Table 7, it shows that the learned mask is empty without any of the losses (3), (5), (7). But only loss (7) is directly related to the mask generation. How does the loss (3) or (5) impact the mask learning? It is also unclear why the losses introduced in (8) would encourage the mask to be minimal despite the quantitative results shown in Table 7. Actually, I am very curious about the performance of the loss introduced in (Press et al., 2019) on top of the network introduced in Figure 2.\n\nOther comments:\n- It would be nice to see the out of domain transfer in the \"attribute\" domain. Ideally, the network should be able to detect \"difference\" in the image from domain B and apply it to the image from domain A. For example, the model is trained on faces without and with glasses, but applied to faces without and with facial hair. Indeed, the introduction of mask alleviates the decoder to learn the attribute itself, and provides the ability to locate the place of difference.\n- Please unify the citation style: there are both Press et al. (2019) and (Press et al., 2019) used.\n- In Section 2 under \"Mask Based Approaches\", the authors argued that the existing attention guided translation \"does not allow for the adaptation of the image information in the masked area\". I do not think this is the case. The existing work also introduced adaptation of the image information in the masked area. For example, the equation (1) in (Mejjati et al., 2018).\n- How is the binarized mask generated in inference? Specifically, how to determine the threshold?\n- In Section 4.1, the authors argued that \"without L_{Cycle} the masks produced include larger portions of the face\". But this actually produces the second smallest mask in Table 7.\n- What is the \"L2 reg\" in Table 7?\n- It would be good to show the sensitivity of the lambdas in the overall loss.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1eBm1OYOB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for unpaired image-to-image translation, where the target domain explicitly contains some additional information than the source domain. The authors use auto-encoders to separate the common and specific representations and to generate masks, which seems to be related to [1]. The authors empirically show the proposed method can be used for image translation, attribute editing.\n\nA small citation error:\n\"Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networkss. arXiv preprint arXiv:1703.10593, 2017a.\"\nnetworkss -> networks\nIt is a published paper at ICCV, not just on arxiv.\n\n[1] Domain Separation Networks, Bousmalis et.al, NIPS 2016", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BylwOzYpqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to disentangle the common and separate parts of two domains and to focus the attention of the underlying network to the desired part only, without reconstructing the entire target. The proposed method is also able to add or remove separate contents, and to enable weakly-supervised semantic segmentation of the separate part of each domain.\nThis work relates to the problem of content transfer between images. The proposed method consists of five networks: one encoder for common domain invariant features, one encoder for separate domain specific information, one network for mapping encodings from common features from both domains undistinguishable, a decoder that generates sample in the origin domain and a decoder that generates the image that combines content from the origin image and domain specific content from the target image. This last decoder also outputs a mask that focuses the attention of the model to the specific part.\nThe proposed model is trained using a combination of different losses: domain confusion loss, reconstruction losses, and cycle consistency losses. Ablation studies reported in the paper nicely show the contribution of each loss. The final loss is obtained by a weighted sum of the losses: how are the lambda coefficient chosen/learned?\nThe proposed method is evaluated on guided content transfer, out of domain manipulation, attribute removal, sequential content transfer, sequential attribute removal and content addition, weakly supervised segmentation of the domain specific content. Experimental results are clear, thorough and satisfactory, both quantitative and qualitative results are reported, as well as a user study. Presented results demonstrate the strengths and limitations of the proposed approach, and the analysis of the results helps understanding and emphasizing the contribution of the paper.\nInformation in appendix also enable reproducibility by providing parameters and architecture structure. \nOther comments: did you observe overfit for some choice of parameter? is your validation/test set large enough for evaluating results? did you observe biases in your method (e.g. to specific features/domain specific information)?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the prior work on disentanglement and attention guided translation to instance-based unsupervised content transfer. The method is somewhat complicated, with five different networks and a multi-component loss function, however the importance of each component appears to be well justified in the ablation study. Overall the reviewers agree that the experimental section is solid and supports the proposed method well. It demonstrates good performance across a number of transfer tasks, including transfer to out-of-domain images, and that the method outperforms the baselines. For these reasons, I recommend the acceptance of this paper.
Decision: Accept (Poster)