LOMIT: LOCAL MASK-BASED IMAGE-TO-IMAGE TRANSLATION VIA PIXEL-WISE HIGHWAY ADAPTIVE INSTANCE NORMALIZATION
Year: 2018

Abstract:
Recently, image-to-image translation has seen a significant success. Among many approaches, image translation based on an exemplar image, which contains the target style information, has been popular, owing to its capability to handle multimodality as well as its suitability for practical use. However, most of the existing methods extract the style information from an entire exemplar and apply it to the entire input image, which introduces excessive image translation in irrelevant image regions. In response, this paper proposes a novel approach that jointly extracts out the local masks of the input image and the exemplar as targeted regions to be involved for image translation. In particular, the main novelty of our model lies in (1) co-segmentation networks for local mask generation and (2) the local mask-based highway adaptive instance normalization technique. We demonstrate the quantitative and the qualitative evaluation results to show the advantages of our proposed approach. Finally, the code is available at https://github.com/AnonymousIclrAuthor/ Highway-Adaptive-Instance-Normalization.

heading: 1 INTRODUCTION
text: Unpaired image-to-image translation (or in short, image translation) based on generative adversarial networks (Goodfellow et al., 2014) aims to transform an input image from one domain to another, without using paired data between different domains (Zhu et al., 2017a; Liu et al., 2017; Kim et al., 2017; Choi et al., 2018; Liu et al., 2017). An unpaired setting, however, is inherently multimodal, denoting a single input image can be mapped to multiple different outputs within a target domain. For example, when translating the hair color of a given image into a blonde, the detailed region (e.g., upper vs. lower, and partial vs. entire) and color (e.g., golden, platinum, and silver) may vary.
Previous studies have achieved such multimodal outputs by adding a random noise sampled from a pre-defined prior distribution (Zhu et al., 2017b) or taking a user-selected exemplar image as additional input, which contains the detailed information of an intended target style (Chang et al., 2018). Recent studies (Lin et al., 2018; Ma et al., 2018) including MUNIT (Huang et al., 2018) and DRIT (Lee et al., 2018) combine those two approaches, showing the state-of-the-art performance by separating (i.e., disentangling) content and style information of a given image through two different encoder networks. However, existing exemplar-based image translation method has several limitations as follows. First, the style information is typically extracted and encoded from the entire region of a given exemplar, thus being potentially noisy due to those regions involved with respect to the target attribute to transfer. Suppose we translate the hair color of an image using an exemplar image. Since the hair color information is available only in the hair region of an image, the style information extracted from the entire region of the exemplar may contain the irrelevant information (e.g., color of the wall and edge pattern of the floor), which should not be reflected in the intended image translation.
On the other hand, the extracted style is then applied to the entire region of the target image, even though particular regions should be kept as it is. Due to this limitation, some of the previous approaches (Huang et al., 2018; Lee et al., 2018) often distort irrelevant regions of an input image such as the background.
Furthermore, when multiple attributes are involved in an exemplar image, one has no choice but to impose all of them when translating a given image. For example, in a person’s facial image translation, if the exemplar image has two attributes, (1) a smiling expression and (2) a blonde hair, then both attributes have to be transferred with no other options.
To tackle these issues, we propose a novel, LOcal Mask-based Image Translation approach, called LOMIT, which jointly generates a local, pixel-wise soft binary mask of an exemplar (i.e., the source region from which to extract out the style information) and that of an input image to translate (i.e., the target region to which to apply the extracted style). This approach has something in common with those recent approaches that have attempted to leverage an attention mask in image translation (Pumarola et al., 2018; Chen et al., 2018; Yang et al., 2018; Ma et al., 2018; Mejjati et al., 2018). In most approaches, the attention mask (extracted from an input image) plays a role of determining the target region to apply a translation. Note that we expand those approaches by jointly extracting two masks, from an input and an exemplar image, respectively, acting as the attention mask of an input and a relevant region (foreground) extractor of an exemplar.
The main novelty of LOMIT is that to jointly obtain the local masks of two images, we utilize the cosegmentation networks (Rother et al., 2006), which aim (1) to extract the targeted style information without noise introduced from irrelevant regions and (2) to translate only the necessary region of a target image while minimizing its distortion. While co-segmentation approaches were originally proposed to capture the regions of a common object existing in multiple input images (Rother et al., 2006; Li et al., 2018), we adopt and train co-segmentation networks for our own purpose.
Once obtained local masks, LOMIT extends a recently proposed technique for image translation, called adaptive instance normalization, using highway networks (Srivastava et al., 2015), which computes the weighted average of the input and the translated pixel values using the abovementioned pixel-wise local mask values as different linear combination weights per pixel location. LOMIT has an additional advantage of being able to manipulate the computed masks to selectively transfer an intended style, e.g., choosing either a hair region (to transfer the hair color) or a facial region (to transfer the facial expression).
The effectiveness of LOMIT is evaluated on two facial datasets, via a user study and other quantitative measures such as the inception score and the classification accuracy.

heading: 2 BASIC SETUP
text: We define “content” as common features (an underlying structure) across all domains (e.g., the pose of a face, the location and the shape of eyes, a nose, a mouth, and hair), and “style” as a representation of the structure (e.g., background color, facial expression, skin tone, and hair color).
As shown in Fig. 1, we assume that an image x can be represented as x = c⊕ s, where c is a content code in a content space, and s is a style code in a style space. The operator⊕ combines and converts the content code c and the style code s into a complete image x.
By considering the local mask indicating the relevant region (or simply, the foreground) to extract the style from or to apply it to, we further assume that s is decomposed into s = sf ⊕ sb, where sf is the style code extracted from the foreground region and sb is that from the background region. Separating an integrated style space S into a foreground style space Sf and a background style space Sb play a role of disentangling style feature representation1. The pixel-wise soft binary mask m of an image x is represented as a matrix with the same spatial resolution of x. Each entry of m lies between 0 and 1, which indicates the degree of the corresponding pixel belonging to the foreground. Then, the local foreground/background regions xf /xb of x is obtained as
xf = m x, xb = (1−m) x, (1)
where is an element-wise multiplication. Finally, our assumption is extended to x = c⊕ sf ⊕ sb, where c, sf , and sb are obtained by the content encoder Ec:, the foreground style encoder Efs , and the background style encoder Ebs , respectively, which are all shared across multiple domains in LOMIT, i.e.,
{cx, sfx, sbx} = {Ec(x), Efs (xf ), Ebs(xb)} cx ∈ C, sfx ∈ Sf , sbx ∈ Sb (2)
It is critical in LOMIT to properly learn to generate the local mask involved in image translation. To this end, we propose to combine the mask generation networks with our novel highway adaptive instance normalization, as will be described in Section 3.2.
1 To verify the representations are properly disentangled, we refer the readers to the 2D embedding visualization of each space (C,Sf ,Sb) in Fig. 8 in in the appendix.

heading: 3 LOCAL IMAGE TRANSLATION MODEL
text: We first denote x1 ∈ X1 and x2 ∈ X2 as images from domains X1 and X2, respectively. As shown in Fig. 2, LOMIT converts a given image x1 to X2 and vice versa, i.e., x1→2 = G(h(Ec(x1), E f s (x f 2 ), E b s(x b 1))), and x2→1 =G(h(Ec(x2), E f s (x f 1 ), E b s(x b 2))), where G is decoder networks and h is our proposed, local mask-based highway adaptive instance normalization (or in short, HAdaIN), as will be described in detail in Section 3.2.
For a brevity purpose, we omit the domain index notation in, say,m = {m1,m2} and x = {x1, x2}, unless needed for clarification.

heading: 3.1 LOCAL MASK EXTRACTION
text: LOMIT utilizes the local mask m to separate the image x into the foreground and the background regions, xf and xb. That is, we jointly extract the local masks of the input and the exemplar images, as those effectively involved in image translation, via co-segmentation networks. For example, given the input image and the exemplar, if LOMIT identifies the the hair color difference of a facial image, e.g., blonde vs. black, then, our local masks should be obtained as the hair regions from two images.
As shown in Fig. 3, given two images x1 and x2, the co-segmentation networks first encode the content of each as c1 and c2 via the content encoder Ec. Next, in the case of computing the segmentation of x2, after average-pooling c1 globally, we forward it to an MLP to obtain the channel-wise soft binary mask cattn1 , which is then multiplied with c2 in a channel-wise manner, i.e., c attn 1 c2, where cattn1 = σ(MLP(c1)). This step works as transferring the objects information from x1 to x2. Finally, we forward-propagate this output into the attention network A to obtain the local mask m2 of x2, i.e., m2 = A(cattn1 c2). The same process applies to the opposite case in a similar manner, resulting in m1 = A(cattn2 c1). Note that our co-segmentation networks are trained in an end-to-end manner with no direct supervision.

heading: 3.2 HIGHWAY ADAPTIVE INSTANCE NORMALIZATION
text: Adaptive instance normalization is an effective style transfer technique (Huang & Belongie, 2017). Generally, it matches the channel-wise statistics, e.g., the mean and the variance, of the activation map of an input image with those of a style image. In the context of image translation, MUNIT (Huang et al., 2018) extends AdaIN in a way that the target mean and the variance are computed
as the outputs of the trainable functions β and γ of a given style code, i.e., AdaINx1→x2(c1, s2) = γ(s2) ( c1 − µ(c1) σ(c1) ) + β(s2), (3)
As we pointed out earlier, such a transformation is applied globally over the entire region of an image, which may unnecessarily distort irrelevant regions. Hence, we formulate our local maskbased highway AdaIN (HAdaIN) as
HAdaINx1→x2(m1, c1, s f 2 , s b 1) = m1 AdaINx1→x2(c1, s f 2 ) + (1−m1) AdaINx1→x1(c1, sb1), (4) where each of β and γ in Eq. (3) is defined as a multi-layer perceptron (MLP), i.e., [β(sf ); γ(sf )] = MLPf (sf ) and [β(sb); γ(sb)] = MLPb(sb). Note that we use different MLPs for the foreground and the background style code inputs. The first term of Eq. (4) corresponds to the local region of an input image translated by the foreground style, while the second corresponds to the complementary region where the original style of the input is kept as it is.

heading: 4 TRAINING OBJECTIVES
text: This section describes each of our loss terms in the objective function used for training our model.

heading: 4.1 STYLE AND CONTENT RECONSTRUCTION LOSS
text: The foreground style of the translated output should be close to that of the exemplar, while the background style of the translated output should be close to that of the original input image. We formulate this criteria as the following style reconstruction loss terms:
L1→2sf = Exf1→2,xf2 [‖E f s (x f 1→2)− Efs (x f 2 )‖1] (5) L1→2sb = Exb1→2,xb1 [‖E b s(x b 1→2)− Ebs(xb1)‖1]. (6)
From the perspective of content information, the content feature of an input image should be consistent with its translated output, which is represented as the content reconstruction loss as L1→2c = Ex1→2,x1 [‖Ec(x1→2)− Ec(x1)‖1]. (7) Note that the content reconstruction is imposed across the entire region of the input image, regardless of the local mask.

heading: 4.2 IMAGE RECONSTRUCTION LOSS
text: As an effective supervision approach in an unpaired image translation setting, we adopt the imagelevel cyclic consistency loss (Zhu et al., 2017a) between an input image and its output through two consecutive image translations of X1 → X2 → X1 (or X2 → X1 → X2), i.e.,
L1→2→1cyc = Ex1 [‖x1→2→1 − x1‖1] . (8)
Meanwhile, similar to previous studies (Huang et al., 2018; Lee et al., 2018), we translate not only (x1 → x1→2) but also (x1 → x1→1). This intra-domain translation (x1 → x1→1) should work similarly to auto-encoder (Larsen et al., 2016), and the corresponding loss term is written as
L1→1x = Ex1 [‖x1→1 − x1‖1] (9)

heading: 4.3 DOMAIN ADVERSARIAL LOSS
text: To approximate the real-data distribution via our model, we adopt the domain adversarial loss by introducing the discriminator networks Dsrc. Among the loss terms proposed in the original GAN(Goodfellow et al., 2014), LSGAN(Mao et al., 2017), and WGAN-GP(Arjovsky et al., 2017; Gulrajani et al., 2017), we chose WGAN-GP, which is shown to empirically work best, as an adversarial method. That is, our adversarial loss is written as
L1→2adv = Ex1 [Dsrc(x1)]− Ex1→2 [Dsrc(x1→2)]− λgp Ex̂[(‖∇x̂Dsrc(x̂)‖2 − 1)2], (10) where x1→2 = G(h(c1, s f 2 , s b 1)), x̂ is a sampled value from the uniform distribution, and λgp = 10. Also, we apply the loss proposed in patchGAN (Isola et al., 2017; Zhu et al., 2017a).

heading: 4.4 MULTI-ATTRIBUTE TRANSLATION LOSS
text: we use an auxiliary classifier (Odena et al., 2016) to cover multi-attribute translation with a single shared model, similar to StarGAN (Choi et al., 2018). The auxiliary classifier Dcls, which shares the parameters with the discriminator Dsrc except for the last layer, classifies the domain of a given image. In detail, its loss term is defined as
L1→2clsr = Ex1 [− logDcls(yx1 |x1)] (11) L1→2clsf = Ex1→2 [− logDcls(yx2 |x1→2)] , (12)
where yx is the domain label of an input image x. Similar to the concept of weakly supervised learning (Zhou et al., 2016; Selvaraju et al., 2017), This loss term plays a role of supervising the local mask m to point out the proper region of the corresponding domain y through the HAdaIN module, allowing our model to extract out the style from its proper region of the exemplar.

heading: 4.5 MASK REGULARIZATION LOSSES
text: We impose several additional regularization losses on local mask generation to improve the overall image generation performance as well as the interpretability of the generated mask.
The first regularization is to minimize the difference of the mask values of those pixels that have similar content information. This helps the local mask consistently capture a semantically meaningful region as a whole, e.g., capturing the entire hair region even when the lighting conditions and the hair color vary significantly within the exemplar. In detail, we design this regularization as minimizing
R1 = E  ∑ i=1,··· ,W,j=1,··· ,H [ |(m ·~1T )− (~1 ·mT )| (ĉ · ĉT ) ] ij  (13) where ~1 is a vector whose elements are all ones, {~1,m} ∈ RWH×1, and ĉ ∈ RWH×C where ĉ = c ‖c‖ . The first term is the distance matrix of all the pairs of pixel-wise mask values in m, and the second term is the cosine similarity matrix of all the pairs of C-dimensional pixel-wise content vectors. Note that we backpropagate the gradients generated by this regularization term only through m to train the co-segmentation networks, but not through ĉ, which does not affect the encoder E.
The second regularization is to make the local masks of the two images capture only those regions having contrasting styles. This regularization is useful especially when multiple attributes are involved in image translation. For example, if the two facial images have different hair colors but common facial expressions, then the local mask should indicate only the hair region. We formulate this regularization by maximizing the style difference between the local mask of two images, which is written as
R2 = −E [ ‖sf1 − s f 2‖1 ]
(14)
The third regularization is simply to minimize the local mask region (Chen et al., 2018; Pumarola et al., 2018) to encourage the model to focus only on a necessary region involved in image translation, by minimizing
R3 = E ‖m‖1 (15)

heading: 4.6 FULL LOSS
text: Finally, our full loss is defined as
LD = −Ladv + λclsLclsr , LG = Ladv + λclsLclsf + λs,c(Lsf + Lsb + Lc) + λx(Lcyc + L1→1x + L2→2x )
+λ1R1 + λ2R2 + λ3R3, (16)
where L without superscript denotes (1→ 2, 2→ 1), λcls = 1, λs,c = 1, λx = 10, λ1 = 0.1, λ2 = 0.01, and λ3 = 0.0001. Note that our training process contains both intra-domain translation, (x1 → x1→1 and x2 → x2→2), and inter-domain translation, (x1 → x1→2 and x2 → x2→1).

heading: 5 EXPERIMENTS
text: We conduct LOMIT and other baseline models on two facial datasets, CelebA (Liu et al., 2015) and EmotioNet (Fabian Benitez-Quiroz et al., 2016). We first describe the datasets and the baseline models in subsection 5.1, 5.2. Second, we present the qualitative comparisons of both multi- and single-attribute translation results against baseline methods in subsection 5.3. Third, we report the user study results to validate the human-perceived quality of the translated results in subsection 5.4. Lastly, we evaluate the performances of LOMIT using the inception score (Salimans et al., 2016) and the classification accuracy. The model architecture and training details we set in the experiments are described in appendix (subsection 7.1, 7.2).

heading: 5.1 DATASETS
text: CelebA. The CelebA (Liu et al., 2015) dataset consists of 202,599 face images of celebrities and 40 attribute annotations per image. We pick 10 attributes (i.e., black hair, blond hair, brown hair, smiling, goatee, mustache, no beard, male, heavy makeup, wearing lipstick) that would convey meaningful local masks. We randomly select 2,000 images for testing and use the others for training. Images are center-cropped and scaled down to 128×128.
EmotioNet. The EmotioNet (Fabian Benitez-Quiroz et al., 2016) dataset contains 975,000 images of facial expressions in the wild, each annotated with 12 action units (AUs). Each AU denotes an activation of a specific facial muscle (e.g., jaw drop, nose wrinkler). We crop each image using a face detector 2 and resize them to 128×128. We use 2,000 images for testing and 200,000 images for training.

heading: 5.2 BASELINE METHODS
text: MUNIT. MUNIT (Huang et al., 2018) decomposes an image into the domain-invariant content code and the domain-specific style code. Involving random sampling for latent style codes while
2https://github.com/ageitgey/face_recognition
training, MUNIT attempts to reflect the multimodal nature of various style domains. We implement MUNIT to be trained on CelebA (Liu et al., 2015) dataset and report results for our comparison.
DRIT. DRIT (Lee et al., 2018) employs two encoders of which each extracts the domain-invariant content information and domain-specific style information, respectively. The model is trained leveraging a content discriminator that ensures the content space to be shared. Loss functions and training processes are similar to MUNIT.

heading: 5.3 QUALITATIVE RESULTS
text: CelebA. As shown in Fig. 4, we compare our model with the baseline models using CelebA dataset (Liu et al., 2015). The baseline models are trained with a dataset corresponding to each target attribute at the topmost column (e.g., in the gender case, a dataset is divided into male/female). On the other hand, when training LOMIT, we construct a set of multiple attributes by combining a few different attributes and train the model for multi-attribute translation (i.e., the columns in a black border, (a), (b), and (c) in Fig. 4). Meanwhile, in order to conduct the single-attribute translation, we interactively modify the output masks of the co-segmentation module and forward the manipulated masks into the networks. That is, as illustrated in Figs. 4(a’), (b’), and (c’), we manually remove the mask for both an input and an exemplar and obtain the result for single-attribute translation. Note that the area marked as a red rectangle in the mask indicates the removed area.
The first four rows correspond to the input images, their generated masks, the exemplars, and their generated masks. Each of the last three rows provides comparisons of our model against the baselines. The topmost row indicates the target attribute for each column. Those in black denote multiple attributes while those in red represent a single attribute after removing the mask subregion (a red rectangle) corresponding to the other attribute. We denote Facial Hair when belonging to any of the three classes, Beard, Goatee or Mustache. LOMIT tends to keep the background intact across various classes and apply the style to the appropriate region, while transferring the properly extracted style (attribute) from the exemplar. When compared to LOMIT, the two other models suffer from undesirable distortion in the background as shown in the first and the second rows from the bottom of (a), (a’). Meanwhile, as can be seen in the bottommost row of (e), MUNIT fails to apply the style to the suitable region due to the lack of an attention mask (through the highway networks). The images of DRIT in the columns (e’) show a translation through the improperly extracted style because the hair region on the images contains a white color which seems to be referenced from the shoulder of a person in the exemplar. It indicates that a mask for the exemplar should be properly incorporated in the process. From the comparison with the baseline models, we justify the needs of the local masks and the HAdaIN module of LOMIT. EmotioNet. Fig. 5 shows the results for AU translation. For the training, we use each AU (1, 2, 4, 5, 6, 9, 12, 17, 20, 25, 26, and 43) as a label for the multi-attribute translation loss, so that the model can be trained for translating multi-AUs from the exemplar. Each section is composed of an input image, its mask, an exemplar, its mask, and a translated output. For example, the left top part, the input containing AUs 1, 4, 25 (Expressionless) takes the exemplar whose AUs are 6, 12, 25 (Happy).

heading: 5.4 QUANTITATIVE EVALUATION
text: DRIT MUNIT LOMIT Class IS CA IS CA IS CA
Mean Std (%) Mean Std (%) Mean Std (%)
Facial Hair 0.3295 0.24 23.8 0.2808 0.25 60.0 0.3105 0.27 71.4 Gender 0.2703 0.21 21.1 0.1368 0.19 53.0 0.2348 0.22 83.9
Wearing Lipstick 0.2685 0.20 19.9 0.1751 0.20 57.1 0.2528 0.20 73.7 Facial Hair + Gender 0.3805 0.24 14.3 0.1972 0.25 34.9 0.2069 0.23 68.1
Makeup + Wearing Lipstick 0.2853 0.21 16.1 0.2472 0.24 27.3 0.2834 0.22 72.8
Inception score and classification accuracy. We compare LOMIT with the baselines using inception score (Salimans et al., 2016) (IS) and classification accuracy (CA). IS is high if translated images are diverse and high quality. We follow the procedure in MUNIT (Huang et al., 2018) to obtain IS. For the classification, we use the pretrained Inception-v3 (Szegedy et al., 2016) and finetune with CelebA (Liu et al., 2015) dataset. To be classified well with high accuracy, a translated image must have appropriate attribute in the exemplar. Table 1 lists up the resulting scores and accuracies. In terms of the CA, LOMIT achieves the highest accuracy across all classes evaluated by large margins. DRIT achieves slightly higher IS than LOMIT, but in the cost of the CA. It indicates that DRIT produces diverse outputs, however with less recognizable image outputs.

heading: 6 CONCLUSIONS
text: In this work, we proposed a local mask-based image-to-image translation model called LOMIT. The co-segmentation networks jointly generate the mask of an input image and that of an exemplar, respectively. A mask of the exemplar exclude an irrelevant region to extract the style information from a relevant region. The other mask of an input captures the region to apply the style of an exemplar while maintaining an original style in the rest (through our highway adaptive instance normalization). LOMIT achieves outstanding results compared with the state-of-the-art methods (Huang et al., 2018; Lee et al., 2018). As future work, we will extend our approach as a general normalization method which can be used in other computer vision tasks.

heading: 7 APPENDIX

heading: 7.1 MODEL ARCHITECTURES
text: Content Encoder. Similar to MUNIT (Huang et al., 2018), the content encoderEc is composed of two strided-convolutional layers and four residual blocks (He et al., 2016). Following the previous approaches (Huang & Belongie, 2017; Nam & Kim, 2018), IN is used across all the layers in the content encoder.
Style Encoders. The style encoders Efs , Ebs have the same architecture but with different parameters. They consist of four strided-convolutional layers, a global average pooling layer, and a fullyconnected layer. The style codes sf , sb are eight-dimensional vectors. Also, style encoders Efs , E b s share first few layers as the first few layers detect low-level feature. To maintain the style information, we do not use IN in the style encoders.
Co-segmentation Networks. Co-segmentation networks are composed of six convolutional layers with a batch normalization (Ioffe & Szegedy, 2015). MLP in Fig. 3 has two linear layers with tanh and sigmoid activation functions, respectively.
Decoder. Decoder G has four residual blocks and two convolutional layers with an upsampling layer each. Because the layer normalization (LN) (Ba et al., 2016) normalizes the entire feature map, maintaining the differences between the channels, we use LN in the residual blocks for stable training.
Discriminator. Following StarGAN (Choi et al., 2018), the discriminator D is composed of six strided-convolutional layers, followed by the standard discriminator and the auxiliary classifier.

heading: 7.2 TRAINING DETAILS
text: We utilize the Adam optimizer (Kingma & Ba, 2015) with β1 = 0.5 and β2 = 0.999. Following the state-of-the-art approach (Choi et al., 2018) in multi-attribute translation, we load the data with a horizontal flip with 0.5 percent. For stable training, we update {Ec, Efs , Ebs, G} in every five updates of D (Gulrajani et al., 2017). We initialize the weights of D from a normal distribution and apply the initialization (He et al., 2015) on others. Also, we use a batch size of eight and the learning rate of 0.0001. We linearly decay the learning rate by half in every 10,000 iterations from 100,000 iterations. All the models used in the experiments are trained for 200,000 iterations using a single NVIDIA TITAN Xp GPU for 30 hours each.

Review summaries:
The paper tries to address an issue existing in current image-to-image translation at the point that different regions of the image should be treated differently. In other word, background should not be transferred while only foreground of interest should be transferred. The paper propose to use co-segmentation to find the common areas to for image translation. It reports the proposed method works through experiments.