U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
Authors: Junho Kim
Minjae Kim
Hyeonwoo Kang
Kwang Hee Lee
Venue:  (2020)
Keywords: Image-to-Image Translation
Generative Attentional Networks
Adaptive Layer-Instance Normalization

Abstract: We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model to focus on more important regions distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adaptive Layer-Instance Normalization) function helps our attention-guided model to flexibly control the amount of change in shape and texture by learned parameters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with a fixed network architecture and hyper-parameters. 

Published as a conference paper at ICLR 2020
U-GAT-IT: UNSUPERVISED GENERATIVE ATTEN- TIONAL NETWORKS WITH ADAPTIVE LAYER- INSTANCE NORMALIZATION FOR IMAGE-TO-IMAGE TRANSLATION
Junho Kim Minjae Kim", Hyeonwoo Kang", Kwang Hee Lee
Clova Al Research, NAVER Corp, "NCSOFT, BBoeing Korea Engineering and Technology Center jhkim.al a anavercorpp com, (minjaekim, hakang01 31 com, kwanghee, lee2@boeing.com
ABSTRACT
We propose a novel method for unsupervised image-to-image translation, which incorporates new attention module and new learnable normalization fune- tion in an end-to-end manner. The attention module guides our model to fo- more important regions distinguishing between source and target do- mains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes be- tween domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adap- tive Layer-Instance Normalization) function helps our inientiongguider model to flexibly control the amount of change in shape and texture by learned param- eters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with fixed network architecture and hyper-parameters. Our code and datasets are available at https::/github.com/aaaa 12/UGATIT or pytorch
INTRODUCTION
Image-to-image translation aims learn function that maps images within two different domains. This topic has gained lot of attention from researchers in the fields of machine learning and com- puter vision because of its wide range of applications including image inpainting (Pathak et al.
2014); ia et al. (2017)), super resolution (Dong et al. (): Kim et al. (2016)), colorization Zhang et al. (2016) 2017)) and style transfer (Gatys al. (T,((201)) Huang & Belongie (207)) When paired samples are given, the mapping model can be trained in a supervised manner using a con-
ditional generative model (Isola et al. 2017); Li et al. (2017a); WWan et al. (2018) or a simple
regression model (Larsson et al. (2016) Long et al, (2015): Zhang et al. ()). In unsupervised
settings where no paired data is available, multiple works (Anoosheh et al. (2018) Choi et al. (2018);
Huang et al. (2018); Kim et al. 2017); Liu et al. 2017);; Royer et al. (2017 Taigman et al. 2017);
Yi al. 201 /hu Zhu et al. (2017] successfully have translated images using shared latent space (Liu
al." 2017)) and cycle consistency assumptions (Kim et al. (217)) Zhuu et al. (2017). These works
have been further developed to handle the multi-modality of the task (Huang et (2018))
Despite these advances, previous methods show performance differences depending the amount of change in both shape and texture between domains. For example, they are successful for the style transfer tasks mapping local texture (e.g., photo2vangogh and photo2portrait) but typically unsuccessful for image translation tasks with larger shape change (e.g.. selfie2anime and cat2dog) in wild images. Therefore, the pre-processing steps such as image cropping and alignment are often required to avoid these problems by limiting the complexity of the data distributions (Huang et al. 2018 Liu al. (2017). addition, existing methods such as DRIT (Lee et al. (2018)) cannot
Most work was done in NCSOFT.
corresponding author

---

Published as a conference paper at ICLR 2020
Decoder
Generator Targe geverator
Fuly Y
Adaptive Blocks
Up sumpling
Ausiliary clossifiar f
Fully connected
Recdial Blocks
M
samplin
Escadlr B
Encoder Festur map
Discriminator
Classifier Co.
Target decriminstor
Logit
sigmoid conv
Ausiltary classi/ter Do.
Fully connected
sampling
Escoder Egy
Encoder
Feature
Festure map
Figure 1: The model architecture of U-GAT-IT. The detailed notations are described in Section Model
acquire the desired results for both image translation preserving the shape (e.g., horse2zebra) and image translation changing the shape (e-g.. car2dog) with the fixed network architecture and hyper- parameters. The network structure or hyper-parameter setting needs to be adjusted for the specific dataset.
In this work, we propose novel method for unsupervised image-to-image translation, which in- corporates a new attention module and a new learnable normalization function an end-to-end manner. Our model guides the translation to focus on more important regions and ignore minor regions by distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. These attention maps embedded into the generator and discrimina- tor to focus on semantically important areas, thus facilitating the shape transformation. While the attention map in the generator induces the focus areas that specifically distinguish between the two domains, the attention map in the discriminator helps fine-tuning by focusing on the difference between real image and fake image in target domain. In addition the attentional mechanism, have found that the choice of the normalization function has a significant impact on the quality of the transformed results for various datasets with different amounts of change shape and texture. Inspired by Batch-Instance Normalization(BIl (BIN) (Nam & Kim (2018), we propose Adaptive Layer- Instance Normalization (AdaLINN, whose parameters are learned from datasets during training time by adaptively selecting proper ratio between Instance normalization (IN) and Layer Normaliza- tion (LN). The AdaLIN function helps our attention-guided model to flexibly control the amount of change in shape and texture. As result, can perform image translation tasks not only requiring holistic changes but also requiring large shape changes. In the experiments, method compared to the existing state-of-the-a models on not only style transfer but also object transfiguration. The main contribution of the proposed work can be summarized follows:
model, without modifying the model architecture or
the ryper-parametto
show the superiority of the proposed
2

---

Published as a conference paper at ICLR 2020
We propose novel method for unsupervised image-to-imane translation with atten- tion module and new normalization function, AdaLIN.
Our attention module helps the model to know where to transform intensively by distin- guishing between source and target domains based the attention map obtained by the auxiliary classifier.
AdaLIN function helps our attention- guided model to flexibly control the amount of change in shape and texture without modifying the model architecture the hyper-paane pper-ppaameters.
UNSUPERVISED GENERATIVE ATTENTIONAL NETWORKS WITH ADAPTIVE LAYER-INSTANCE NORMALIZATION
Our goal is to train function G.-+e that maps images from a source domain X, to a target domain X, using only unpaired samples drawn from each domain. Our framework consists of two generators G.--t and G.-ss and two discriminators D, and D,. We integrate the attention module into both generator and discriminator. The attention module in the discriminator guides the generator focus on regions that are critical to generate realistic image. The attention module in the generator gives attention to the region distinguished from the other domain. Here, we only explain G.-at and D. (See Fig as the vice versa should be sorwight-forward.
2.1 MODEL
2.1.1
GENERATOR
Let E (X,,Xi X2) represent a sample from the source and the target domain. Our translation model Ga-st consists of an encoder E,, decoder Gt, and an auxiliary classifier 7/s where Me (x))) represents the probability that comes from X,. Let E (x) be the k-th activation map of the encoder and
Ehi (x) be the value at (i,j)- Inspired by CAM (Zoou et al. (0066). ) the auxiliary classifier is trained learn the weight of the k-th feature map for the source domain, w by using the global
average pooling and global max pooling, i.e., (x()) = (I Eig Frits (s:)) By exploiting w we can calculate set of domain specific attention feature map (x) = W. Ex(x) = {w Ex [kkkn where is the number of encoded feature maps. Then, our translation model G. becomes equal to G, (x)))). Inspired by recent works that use affine transformation parameters in normalization layers and combine normalization functions (Huang & Belongie (2017): Nam & Kim
(0188) ) we equip the residual blocks with AdaLIN whose parameters, and 3 are dynamically
computed by fully connected layer from the attention map.
p) diz) + + AdaLIN(a,7,3) 3) -7. (p (1 p)
1T
PL di Vol +e
áj Vot + p cp - AA)
(I)
where μr, μr and az are chsnnel-wise, layer-wise mean and standard deviation respectively, and B are parameters generated by the fully connected layer, 7 is the learning rate and Ap indicates the parameter update vector (e.g.. the gradient) determined by the optimizer. The values of are constrained the range of [0. simply by imposing bounds the parameter update step. Generator adjusts the value so that the value of is close to 1 in the task where the instance normalization is important and the value of p close to in the task where the LN important. The value of p initialized to in the residual blocks of the decoder and 0 in the ua-sampling blocks of the decoder.
An optimal method to transfer the content features onto the style features to apply Whitening and Coloring Transform (WCT) (Li et al. (2017))), but the computational cost is high due to the calculation of the covariance matrix and matrix inverse. Although, the AdaIN (Huang & Belongie 2017 is much faster than the WCT, it is sub-optimal to WCT assumes uncorrelation between feature channels. Thus the transferred features contain slightly more patterns of the content. On the other hand, the LN (Ba et al. (2016)) does not assume uncorrelation between channels, but
3

---

Published as a conference paper at ICLR 2020
sometimes it does not keep the content structure of the original domain well because it considers global statistics only for the feature maps. To overcome this, our proposed normalization technique AdaLIN combines the advantages of AdaIN and LN by selectively keeping changing the content information, which helps to solve a wide range of image-to-image translation problems.
2.1.2 DISCRIMINATOR
Let x (X,, GX,()) represent a sample from the target domain and the translated source domain. Similar to other translation models, the discriminator D. which is a multi-scale model consists of an encoder ED., a classifier CD. and an auxiliary classifier T]D,- Unlike the other translation models, both n1, (a) and (z:) are trained discriminate whether comes from X₁ or G,-r(X,). Given a sample z, Dz(x)) exploits the attention feature maps ap, (x) = wp, ED, (x) using Wp, the encoded feature maps Ep, (x) that is trained by M), (z:.. Then, our discriminator (x(r) becomes equal to CD, (a (x)).
2.2
LOSS FUNCTION
The full objective of our model comprises four loss functions. Here, instead of using the vanilla GAN objective, we used the Least Squares GAN (Mao et al. (2017)) objective for stable training.
Adversarial loss An adversarial loss is employed match the distribution of the translated images to the target image distribution:
-- Lsgan -E ((D,,=))((((())))1 E (D((--))(()((()))))
(2)
Cycle loss To alleviate the mode collapse problem, we apply a cycle consistency constraint to the generator. Given an image a E X,, after the sequential translations of from X, X, and from X, to Xs, the image should be successfully translated back to the original domain:
L Leyele E, (
(3)
Identity loss To ensure that the color distributions of input image and output image similar, we apply an identity consistency constraint to the generator. Given an image x E X,, after the translation of z using G5-tt, the image should not change.
L g-8 Aidentity = =E Gs-re())1)-
(4)
CAM loss By exploiting the information from the auxiliary classifiers 7) and np, given image E {X,,, X.). G.-zt and D, get to know where they need to improve or what makes the most difference between two domains the current state:
- - (E (lgg( + Ee (oogl m7 (x)))),
(5)
De EE-X, +
Exxx, 7D, (G.r(G.)()))))
(6)
Full objective Finally, we jointly train the encoders, decoders, discriminators, and auxiliary classi- fiers to optimize the final objective:
min G Gr D.. D9D,
max
Ligaan + Ax + Ag + AsLeams
where > 1, A2 10, 10, 1000. Here, Lisgan Lisgen Lisgan and the other losses defined in the similar way (Treyetes Tiieentity, and Tecam)
4

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(e)
Figure 2: Visualization of the attention maps and their effects shown in the ablation experiments: (a) Source images, (b) Attention map of the generator, (c-d) Local and global attention maps of the discriminator, respectively. (e) Our results with CAM, (f) Results without CAM.
3 EXPERIMENTS
3.1 BASELINE MODEL
We have compared our method with various models including CycleGAN Zhu et al. (017)), UNIT [Luu et al. (2017). ). MUNIT (Huang et (2018), DRIT (Lee al. (0118) AGGAN (Mejjati al. 2018 ) and CartoonGAN (Chen et al. (2018)). All the bascline methods are implemented using the
author's 's code.
3.2 DATASET
We have evaluated the performance of each method with five unpaired image datasets including four representative image translation datasets and newly created dataset consisting of real photos and animation artworks, i.c., selfic2anime. All images are resized to 256 256 for training. See Appendix C for each dataset for our experiments.
3.3 EXPERIMENT RESULTS
We first analyze the effects of attention module and AdaLIN in the proposed model. We then com- pare the performance our model against the other unsupervised image translation models listed in the previous section. To evaluate, the visual quality of translated images, have conducted user study. Users are asked to select the best image among the images generated from five different methods. More examples the results comparing our model with other models included in the supplementary materials.
3.3. CAM ANALYSIS
First, we conduct ablation study to confirm the benefit from the attention modules used in both generator and discriminator. As shown Fig 2 focus on the source image regions that more discriminative from the target domain, such eyes and mouth. Meanwhile, we can see the regions where the discriminator concentrates its attention to determine whether the target image is real or fake by visualizing local and global attention maps of
(b), the attention feature map helps the generator to
5

---

Published as a conference paper at ICLR 2020
(b)
(c)
(d)
(e)
(I)
Figure 3: Comparison of the results using each normalization function: (a) Source images, (b) Our results, (c) Results only using IN in decoder with CAM, (d) Results only using LN in decoder with CAM, (e) Results only using AdaIN in decoder with CAM, (f) Results only using GN in decoder with CAM.
the discriminator as shown in Fig 2 (c) and (d), respectively. The generator can fine-tune the area where the discriminator focuses on with those attention maps. Note that we incorporate both global and local attention maps from two discriminators having different size of receptive field. Those maps can help the generator to capture the global structure (e.g., face area and near of eyes) as well as the local regions. With this information some regions are translated with more care. The results with the attention module shown in Fig (e) verify the advantageous effect of exploiting attention feature map in an image translation task. On the other hand, one can see that the eyes are misaligned, or the translation is not done at all in the results without using attention module as shown in Fig (f).
3.3.2 ADALIN ANALYSIS
As described in Appendix B we have applied the AdaLIN only to the decoder of the generator. The role of the residual blocks in the decoder is to embed features, and the role of the up-sampling convolution blocks in the decoder is to generate target domain images from the embedded features. If the learned value of the gate parameter P is closer to I, it means that the corresponding layers rely more on IN than LN. Likewise, if the learned value of p is closer to O, it means that the corresponding layers rely more on LN than IN. As shown in Fig 3 (c), in the case of using only IN in the decoder, the well preserved due to channel-wise normalized feature statistics used in the residual blocks. However, the amount of translation to target domain style is somewhat insufficient since the global style cannot be captured by IN of the up-sampling convolution blocks. On the other hand, As shown in Fig 3 (d), if we use only LN in the decoder, target domain style can be transferred sufficiently by virtue of layer- wise normalized feature statistics used in the up-sampling convolution. But the features of the source domain image are less preserved by using LNN the residual blocks. This analysis of two extreme cases tells us that it is beneficial to rely more on IN than LN in the feature representation layers to preserve semantic characteristics of source domain, and the opposite is true for the up- sampling layers that actually generate images from the feature embedding. Therefore, the proposed AdaLIN which adjusts the ratio of IN and LN in the decoder according to source and target domain distributions is more preferable in unsupervised image-to to-image translation tasks. Additionally, the (e). (f) are the results of using the AdaIN and Group Normalization (GN) (Wu & He (2018) respectively, and our methods showing better results compared to these.
features of the source domain (e.g., earrings and shades around cheekbones)

---

Published as a conference paper at ICLR 2020
(b)
(c)
(d)
(f)
Figure 4: Visual comparisons on the five datasets. From top to bottom: selfie2anime, horse2zebra, cat2dog, photo2portrait, and photo2vangogh. (a)Source images, (b)U-GAT-IT, (c)CycleGAN, (d)UNIT, (e)MUNIT, (DDRIT, (g)AGGAN
Table 1: Kernel Inception Distance x 100 100 for ablation our model. Lower is better. There are some notations; GN: Group Normalization, G CAM: CAM of generator, D CAM: CAM of discriminator
Model
selfie2anime
anime2selfie
U-GAT-IT
11.61 0.57
11.52 0.57
U-GAT-IT w/ IN
13.64 + 0.76
13.58 + 0.8
U-GAT-IT w/ LN
12.39 0.61
13.17 = 0.8
U-GAT-IT w/ AdaIN
12.29 0.78
11.81 = 0.77
U-GAT-IT w/ GN
12.76
12.30 = 0.77
U-GAT-IT w/o CAM
12.85 0.82
14.06 = 0.75
U-GAT-IT w/o G.CAM
12.33 # 0.68
13.86 0.75
U-GAT-IT w/o D.CAM
12.49 + 0.74
13.33 + 0.89
Also, as shown in Table 1 we demonstrate the performance of the attention module and AdaLIN in the selfie2anime dataset through an ablation study using Kernel Inception Distance (KID) (Binkowski al. (2018). ) Our model achieves the lowest KID values. Even if the attention mod- ule and AdaLIN are used separately, we can see that our models perform better than the others. However, when used together, the performance is even better.
3.3.3 QUALITATIVE EVALUATION
For qualitative evaluation, we have also conducted a perceptual study. 135 participants are shown translated results from different methods including the proposed method with source image, and asked to select the best translated image to target domain. We inform only the name of target domain, i.e., animation, dog, and zebra to the participants. But, some example images of target domain are provided for the portrait and Van Gogh datasets minimum information ensure proper judgments. Table 2 shows that the proposed method achieved significantly higher score except for photo2vangogh but comparable in human perceptual study compared to other methods. In Fig we present the image translation results from each method for performance comparisons. U-GAT-IT can generate undistorted image by focusing more the distinct regions between source
7

---

Published as a conference paper at ICLR 2020
Table 2: Preference translated images by user study.
Model
selfie2anime
horse2zebra
cat2dog
photo2portrait
photo2vangogh
U-GAT-IT
73.15
73.56
58.22
30.59
48.96
CycleGAN
20.07
23.07
6.19
26.59
27.33
UNIT
1.48
0.85
18.63
32.11
11.93
MUNIT
3.41
1.04
14.48
8.22
2.07
DRIT
1.89
1.48
2.48
2.48
9.70
Table 3: Kernel Inception Distance x 100 for difference image translation mode. Lower better.
Model
selfie2anime
horse2zebra
cat2dog
photo2portrait
photo2vangogh
U-GAT-IT
11.61 0.57
7.06 + 0.8
7.07 + 0.65
1.79 = 0.34
4.28 = 0.33
CycleGAN
13.08 0.49
8.05 + 0.72
8.92 + 0.69
1.84 + 0.34
5.46 + 0.33
UNIT
14.71 0.59
10.44 0.67
8.15 0.48
1.20 + 0.31
4.26
0.29
MUNIT
13.85 - 0.41
11.41 + 0.83
10.13 + 0.27
4.75 = 0.52
13.08 = 0.34
DRIT
15.08 0.62
9.79 + 0.62
10.92 + 0.33
5,85 - 0.54
12.65 + 0.35
AGGAN
14.63 0.55
7.58 0.71
9.84 0.79
2.33 = 0.36
6.95 = 0.33
CartoonGAN
15.85 0.69
Model
anime2selfie
zebra2horse
dog2cat
portrait2photo
vangogh2photo
U-GAT-IT
11,52 0.57
7.47 0.71
8.15 0.66
1.69 + 0.53
5.61 + 0.32
CycleGAN
11.84 0.74
8.0 0.66
9.94 0.36
1.82 + 0.36
4.68
0.36
UNIT
26.32 + 0.92
14.93 + 0.75
9.81 + 0.34
1.42 = 0.24
9.72 = 0.33
MUNIT
13.94 + 0.72
16.47 + 1.04
10.39 1 0.25
3.30 + 0.47
9.53 + 0.35
DRIT
14.85 0.60
10.98 0.55
10.86 0.24
4.76 = 0.72
7.72 = 0.34
AGGAN
12.72 = 1.03
8.80 = 0.66
9.45 = 0.64
2.19 = 0.40
5.85 = 0.31
and target domain by exploiting the attention modules. Note that the regions around heads of two zebras or eyes of dog are distorted in the results from CycleGAN. Moreover, translated results using U-GAT-IT
visually superior other methods while preserving semantic features of source domain. It is worth noting that the results from MUNIT and DRIT are much dissimilar to the source images since they generate images with random style codes for diversity. Furthermore, it should be emphasized that U-GAT-IT have applied with the same network architecture and hyper-parameter for all of the five different datasets, while the other algorithms are trained with preset networks or hyper- hyper-paramene parameters. Through the results of user study, we show that the combination of our attention module and AdaLIN makes our model more flexible.
3.3.4 QUANTITATIVE EVALUATION
For quantitative evaluation, the recently proposed KID, which computes the squared Max- imum Mean Discrepancy between the feature representations of real and generated images. The feature representations are extracted from the Inception network (Szegedy al. (2016)). In trast to the Fréchet Inception Distance (Heusel et al. (2017). KID has an unbiased estimator, which makes more reliable, especially when there are fewer test images than the dimensionality of the inception features. The lower KID indicates that the more shared visual similarities between real and generated images (Mejjati et al.) (2018). Therefore, if well translated, the KID will have small value in several datasets. Table 3 shows that the proposed method achieved the lowest KID scores except for the style transfer tasks like photo2vangogh and photo2portrait. However, there is no big difference from the lowest score. Also, unlike UNIT and MUNIT, we can see that the source - target, target - source translations are both stable. U-GAT-IT shows even lower KID than the recent attention- based method, AGGAN. AGGAN yields poor performance for the transformation with shape change such as dog2cat and anime2selfie unlike the U-GAT-IT, the attention module of which focuses distinguishing not between background and foreground but differences between
8

---

Published as a conference paper at ICLR 2020
two domains. CartoonGAN, as shown in the supplementary materials, has only changed the over- all color of the image to an animated style, but compared to selfie, the eye, which is the biggest characteristic of animation, has not changed at all. Therefore, CartoonGAN has the higher KID.
CONCLUSIONS
In this paper, we have proposed unsupervised image-to-image translation (U-GAT-IT), with the attention module and AdaLIN which can produce more visually pleasing results in various datasets with a fixed network architecture and hyper-parameter. Detailed analysis of various experimental results supports our assumption that attention maps obtained by an auxiliary classifier can guide generator to focus more on distinct regions between source and target domain. In addition, we have found that the Adaptive Layer-Instance Normalization (AdaLIN) is essential for translating various datasets that contains different amount of geometry and style changes. Through experiments, we have shown that the superiority of the proposed method compared the existing state-of-the GAN-based models for unsupervised image-to-ing -mage translation tasks.
REFERENCES
Asha Anoosheh, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Combogan: Unrestrained scalability for image domain translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 783-790, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
David Berthelot, Tom Schumm, and Luke Metz. Began: Boundary equilibrium generative adversar- ial networks. arXiv preprint arXiv: 1703:1007..77 2017.
Mikolaj Binkkoski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https:
Yang Chen, Yu-Kun Lai, and Yong-Jin Liu. Cartoongan: Generative adversarial networks for photo cartoonization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 9065-94477779 2018
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789- 8797, 2018.
Chao Dong. Chen Change Loy. Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307, 2016
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. learned representation for artis- tic style. In International Conference on Learning Representations, 2017. URL https:
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 2414-2423, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.

---

Published as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision. pp. 1501- 1510, 2017.
Xun Huang, Ming-Ya Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision, pp. 172-189, 2018.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions Graphics, 36(4):107, 2017.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1125-1134, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL httpttt:: www.o/peerrvviiieeee ew net. forum? d= 992CeAb
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1646-1654, 2016.
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In Proceedings of the International Conference Machine Learning, pp. 1857-1865, 2017. URL/itttp://proceeddng mmlr.
press/v70/EB https///////mmmmmmm wtw.mmm7a.mmmlmm..mmmmmmmmmmmm
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimizziio. In International Conference on Learning Representations, 2015. URL http:://aaxiv..rg/abbb////
6980
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In Proceedings of the European Conference Computer Vision, pp. 577-593. Springer, 2016.
Hsin-Yine Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision, pp. 35-51, 2018.
Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. Alice: Towards understanding adversarial learning for joint distribution matching. In Advances in Neural Information Processing Systems, pp. 5501-5509, 2017a
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Hsuan Yang. Universal style transfer via feature transforms. In Advances in Neural Information Processing Systems, pp. 386- 396, 2017b.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz, Unsupervised image-to tongee translation networks. In Advances Neural Information Processing Systems, pp. 700-708, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic seg- mentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3431-3440, 2015.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- on Computer Vision, pp. 2813-2821. IEEE, 2017.
10

---

Published as a conference paper at ICLR 2020
Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. Unsupervised aatention-guident image-to-ing -o-eage translation. In Advances in Neural Information Processing Systems, pp. 3697-3707, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. [RRL:ttts///ppeeeeee/e
net/ F.t.
Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. In Advances in Neural Information Processing Systems, pp. 2563-2572, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2534, 2014.
Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Moressi, Forrester Cole, and Kevin Murphy. Xgan: Unsupervised image-to-image translation for many-to-many mappings. arXiv preprint 17..51/71..05139, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and 7higniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation, In International Conference Learning Representations, 2017. URL /tttttps:///pperevieee
net / Eorum id Sk2 Im59ex
Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798-8807, 2018.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference Computer Vision, September 2018.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2849-2857, 2017.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proceedings of the European Conference on Computer Vision, pp. 649-666. Springer, 2016.
Richard Zhang, Jun- Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros. Real-time user-guided image colorization with learned deep priors. ACM Transactions
Graphics,
2017. doi: 10. 01553072959.30737777 URL htttp:://ddd org/10..1455 3072291/////99911 3073703]
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial net- works. In International Conference on Learning Representations, 2017. URL https://
openreview www.:fouummmeeeeee wttp://oooommlldrr.npppepppe
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2921-2929 IEFE, 2016.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
11

---

Published as a conference paper at ICLR 2020
A RELATED WORKS
A.1 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GAN) Goodfellow et al. (2114) have achieved impressive results on wide variety of image generation (Arjovsky et al. (201 7h: Berthelot et al. (2017) Karras et al. Zhao et al.] (2017)) image inpainting(lizuts lizuka et al. (2017)) image translation Choi et al.
2018;;
2018 Huang et al. 2018): Isola et al. (0017): 7): Liu et al. 2: Wang et al. (2018); 8 Zhu et al.
2017 tasks. In training, a generator aims to generate realistic images to fool a discriminator while
the discriminator tries to distinguish the generated images from real images. Various multi-stage generative models(Karras
al. (2018); 8); Wang et al. (2018) and better training Sojjettive(Arjoovaa et al. (2017): Berthelot et al. (2017): Mao et al. (2017), 7); Zhao et al. (2017)) have been proposed to generate realistic images. In this paper, our model uses GAN to learn the transformation from a source domain to significantly different target domain, given unpaired training data.
A.2 IMAGE-TO-IMAN TRANSLATION
Isola et al. Isola et al. (2017) have proposed conditional GAN-based unified framework for sange-to-iaage translation. High-resolution version of the pix2pix have been proposed by Wang @tal(Wang et al. (2018) Recently, there have been various attempts (Huang et al. (2018); Kim al.
(217) Liu et al. (2017): (Taigman et al.] (2017); /Zhu et al. (201)) ) to learn image translation from an unpaired dataset. CycleGAN (Zhu et al. (2017)) have proposed a cyclic consistence loss for the first time to enforce one-to-one mapping. UNIT (Liu et al. (017)) assumed a shared-latent space to tackle unsupervised image translation. However, this approach performs well only when the two domains have similar patterns. MUNIT (Huang et al. (2018)) makes it possible to extend to many- to-many mapping by decomposing the image into content code that is domain-invarian and a style code that captures domain-specifer properties. MUNIT synthesizes the separated content and style to generate the final image, where the image quality is improved by using adaptive instance normal- ization (Huang & Belongie (2177). With the same purpose MUNIT, DRIT (Lee al. (2018) decomposes images into content and style, so that many-to-man
mapping is possible. The only dif- ference is that content space is shared between the two domains using the weight sharing and content discriminator which is auxiliary classifier. Nevertheless, the performance of these methods (Huang al. 2018); Liu et al. (2017): Lee et al. (2118) are limited to the dataset that contains well-aligned images between source and target domains. In addition, AGGAN (Mejjati et al. (2018) improved distinguish between fore- ground and background. However, the attention module in AGGAN cannot help to transform the object's shape in the image. Although, CartoonGAN (Chen al. (2088) shows good performance for animation style translation, it changes only the color, tone, and thickness of line in the image. Therefore it not suitable for the shape change in the image.
the performance of image translation by using attention mechanism
A.3 CLASS ACTIVATION MAP
Zhou et al. (Zouu et al. 2016 have proposed Class Activation Map (CAM) using global average pooling CNN. The CAM for particular class shows the discriminative image regions by the CNN to determine that class. In this work, our model leads to intensively change discriminative image regions provided by distinguishing two domains using the CAM approach. However, not only global average pooling is used, but global max pooling is also used to make the results better.
A.4 NORMALIZATION
Recent neural style transfer researches have shown that CNN feature statisties (e.g., Gram matrix (Gatys et al. (2016)), mean and variance (Huang & Belongie 2017)) can be used as direct descriptors for image styles. In particular, Instance Normalization (IN) has the effect of removing the style vari- ation by directly normalizing the feature statistics of the image and is used more often than Batch Normalization (BN) or Layer Normalization (LN) in style transfer. However, when normalizing images, recent studies Adaptive Instance Normalization (AdalN) (Huang & Belongie (017)), Conditional Instance Normalization (CIN) (Dumoulin et al. (017)). and Batch-Instance Normaliza- tion (BIN) (Nam & Kim (2018) instead of using IN alone. In work, we propose Adaptive
12

---

Published as a conference paper at ICLR 2020
Inster-nstance Normalization (AdaLIN) function to adaptively select a proper ratio between IN and LN. Through the AdaLIN, our attention-guian guudded model can flexibly control the amount of change in shape and texture.
B IMPLEMENTATION DETAILS
B. NETWORK ARCHITECTURE
The network architectures of U-GAT-IT are shown in Table 45 and 6 The encoder of the gener- ator composed of two convolution layers with the stride size of for down-sampling and four residual blocks. The decoder of the generator consists of four residual blocks and two up-sampling convolution layers with the stride size of one. Note that we use the instance normalization for the en- coder and AdaLIN for the decoder, respectively. In general. LN does not perform better than batch normalization in classification problems (Wu & He (2018). Since the auxiliary classifier con- nected from the encoder in the generator, to increase the accuracy of the auxiliary classifier we use the instance normalization (batch normalization with a mini-batch size of 1) instead of the AdaLIN. Spectral normalization (Miyato et al. (2018) is used for the discriminator. We employ two different scales of PatchGAN (Isola et al. (2)17)) for the discriminator network, which classifies whether local (70 70) and global (286 x 286) image patches are real or fake. For the activation function, we use ReLU in the generator and leaky-ReLJ. with a slope of 0.2 in the discriminator.
B.2 TRAINING
All models are trained using Adam (Kingma & Ba 2015)) with 31=0.5 and (3=00.999. For data augmentation, we flipped the images horizontally with probability of 0.5, resized them to 286 x 286, and random cropped them 256 x 256. The batch size is set one for all experiments. We train all models with fixed learning rate of 0.0001 until 500,000 iterations and linearly decayed up 1,000,000 iterations. We also use weight decay rate of 0.0001. The weights are initialized from a zero-centered normal distribution with a standard deviation of 0.02.
C DATASET DETAILS
selfie2anime The selfie dataset contains 46,836 selfie images annotated with 36 different attributes. We only use photos of females as training data and test data. The size of the training dataset is 3400, and that of the test dataset is 100, with the image size of 256 256. For the anime dataset, have firstly retrieved 69,926 animation character images from Anime- Anine-Planef]] Among those images, 27,023 face images are extracted by using anime face detecto After selecting only female character images and removing monochrome images manually, we have collected two datasets of female anime face images, with the sizes of 3400 and 100 for training and test data respectively, which is the same numbers as the selfie dataset. Finally, all anime face images are resized to 256 x 256 by applying a CNN-based image super-resolution algorithm
horse2zebra and photo2vangogh These datasets are used in CycleGAN (Zuu et al. (017)) The training dataset size of each class: 1.067 (horse), 1,334 (zebra), 6,287 (photo), and 400 (vangogh). The test datasets consist of 120 (horse), 140 (zebra), 751 (photo), and 400 (vangogh). Note that the training data and the test data of vangogh class are the same.
cat2dog and photo2portrait These datasets are used in DRIT Lee et al. (0018). The numbers of data for each class are 871 (cat), 1,364 (zebra), 6,452 (photo), and 1,811 (vangogh). We use 120 (horse), 140 (zebra), 751 (photo), and 400 (vangogh) randomly selected images as test data, respectively.
http::///gtuub..ommmaaaaa/////// animeface htaps//githhh.ccm/ggaammi/i/iiii
13

---

Published as a conference paper at ICLR 2020
D ADDITIONAL EXPERIMENTAL RESULTS
In addition to the results presented in the paper, we show supplement generation results for the five
datasets in Figs 5
10
and 12
Table 4: The detail of generator architecture.
Part
Input - Output Shape
Layer Information CONV-(N64, K7, S1, P3), IN, ReLU
Encoder samplingg
(h, w, 3) - (h, w, 64)
(h, w, 64) + (.. 2. 128)
CONV-(N128, K3, S2, PI), IN, ReLU
G 2. 128)
+ + 4. 256)
CONV-(N256, K3, S2, PI), IN, ReLU
Encoder Bottleneck
4 w. 256)
- -(", w.
+ ( 4: 256)
ResBlock-NN25,, K3, S1, PT), IN, ReLU
( 4:
1' 4: 256)
256)
ResBlock-NN256, K3, S1, PT), IN, ReLU
4' 1' 256)
+ +(1 256)
ResBlock-(N256, K3, S1, PI), IN, ReLU
L 4: 256)
+ G E. 256)
ResBlock-(N256, K3, SI, PI), IN, ReLU
CAM of Generator
Global Average & Max Pooling, MLP-(NI), Multiply the weights of MLP
à à T 256)
+ ('. T.. 512)
" w T: 512)
- w.
J. 256)
CONV-(N256, K1, S1), ReLU
%/s B
4 w 45266)
> (1, 2566)
MLP-(N256). ReLU MILP-(2266,, ReLU
1. (1, 1,26)
- 1,, 256)
- (,,6)
(1,,
MLP-(N236), ReLU
Decoder Bottleneck
T17:220
(h w 256)
+ +4 T: 256)
w
ANaResBlock-(N256, K3, S1, P1), AdaILN, ReLU
n w 4: 256)
+ + 1
4: 256)
AdaResBlock-<N2556, K33, S1, P1), AdalLN, ReLU
h w
4' 4: 256)
+ + w
4: 256)
AdaRessBoock<NN256, K3, S1, P1), AdaILN, ReLU
b w.
4' 4: 256)
+ + w.
4: 256)
(NaRssBlock--N256, K3, S1, P1), AdaILN, ReLU
Decoder Up-sampling
w 41 4: 256)
+ +(2 W
2: 128)
Up-CONV-(N128, K3, S1, P1), LIN, ReLU Up-CONV-(N64, K3, S1, PI), LIN, ReLU
( w.
7, 128)
+ (h, W, 64)
(h, 64)
(h, W,
CONV-(N3, K7, S1, P3). Tanh
Table 5: The detail of local discriminator.
Part
Input - Output Shape
Layer Information
Encoder Down- samplsanpling
(h, w, 3)
+ (
64)
CONV-(N64, K4, S2 P1), SN, Leaky- ReLU
2.
( w
2, 64)
+ 4
4 128)
CONV-(N128, K4, S2, P1), SN, Lea.L-Keelll ReLL
(4, m
T 128)
+ [
R. 256)
CONV-(N256, K4, S2, PI), SN, Leaky-ReLU
A d
(: g. 256) + ( 3. 512)
CONV-(N512, K4, S1, P1), SN, Leaky-Rel.U
CAM of Discriminator
Global Average Max Pooling, MLP-(N1), Multiply the weights of MI. CONV-N512, K1, S1), Leaky-ReLLa CONV-(N1, K4, S1, P1), SN
( 5. 512) + ( x. 1024)
h (g. 8. 1024) - (,. w 512)
Classifier
($ g 512) - (5. 8 1)
14

---

Published as a conference paper at ICLR 2020
Table 6: The detail of global discriminator.
Part
Input - Output Shape
Layer Information
Encoder Down-sampling
(h, w, 3) + ( 7. 64)
CONV-(N64, K4, S2, P1), SN, Reak
(2. 7. 64)
- (7 W
T: 128)
CONV-NN128, K4, S2, P1), SN, Leaky-ReLL
(1 oi
4: 128)
- (g. K. 256)
CONV-(N256, K4, S2, PI), SN, Leaky-ReI.U
(g w 256)
+ +(1 16.
w
512)
CONV-(N512, K4, S2, P1), SN, Leaky-ReLU
8
16. 16. 512)
At
+ ( [gg
32 1024)
w
CONV-(N1024, K4, S2, P1), SN, Leaky-ReLJU
1024) -> ( A
In 2048)
CONV-(N2048, K4, S1, PI), SN, Leaky-Rel.U
17. 32
37
32
CAM of Diseriminator
Global Average Max Pooling, MLP-(NI), Multiply the weights of MLP CONV-(N2048, KI, SI), Seaky-ReLL CONV-(N1, K4, S1, P1), SN
hh
32' 32 2048)
- 32
4096)
32'
( h
@
4096)
+ (
ht
32 2048)
w
32. 32
32-
Classifier
32 32
h w
2048)
+
32
h
32 1)
(a)
(b)
(c)
(d)
(e)
(I)
(g)
(h)
(k)
Figure 5: Visual comparisons of the selmie2anime with attention features maps. (a) Source images, (b) Attention map of the generator, (c-d) Local and global attention maps of the discriminators, (e) Our results, (f) CycleGAN (Zhu et al. (2017). (g) UNIT (Liu et al. (2017)) D, (h) MUNIT (Huang
et al. (2018). (i) DRIT (Lee et al. (2018)) (j) AGGAN (Mejjati et al. (018)). (k) CartoonGAN (Chen et al. (2018).
15

---

Published as a conference paper at ICLR 2020
(b)
(d)
(f)
(g)
(h)
i
Figure 6: Visual comparisons of the anime2selfie with attention features maps. (a) Source images, (b) Attention map of the generator, (c-d) Local and global attention maps of the discriminators, (e)
Our results, (f) CycleGAN (Zhu et al. (2017), (g) UNIT (Liu et al. (2017)), (h) MUNIT (Huang ct al. (018)), (i) DRIT (ee et al. (2018). (i) AGGAN (Mejaai et al. (011)).
16

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(c)
(1)
g
(h)
Figure 7: Visual comparisons of the horse2zebra with attention features maps. (a) Source images, (b) Attention map of the generator, (c-d) Local and global attention maps of the discriminators, (c) Our results, (f) CycleGAN (Zhu et al. (017)). (g) UNIT (Liu al. (2017)) ), (h) MUNIT (Huang et al. (2018), (i) DRIT (Lee et al. (2018). ), (i) AGGAN (Mejjati et al. (2018))
(a)
(b)
(c)
(d)
(c)
(f)
(h)
Figure 8: Visual comparisons of the zebra2horse with attention features maps. (a) Source images, (b) Attention map of the generator, (c-d) Local and global attention maps of the discriminators. (e) Our results, CycleGAN (Zhu et 2017)) (g) UNIT (Liu al. (2017)). ), (h) MUNIT (Huang
et al. (2018)), (i) DRIT (Lee et al (0118) ), (j) AGGAN (Mejjati et al. (0188)
17

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(c)
(f)
(g)
(h)
Figure 9: Visual comparisons of the cat2dog with attention features maps. (a) Source images, (b) Attention map of the generation, (c-d) Local and global attention maps of the discriminators, (c) Our results, (f) CycleGAN (Zhu al. (2017) D. (g) UNIT (Liu al. (2017), (h) MUNIT (Huang et al.
(218 (i) DRIT (Lee et al. (2018)), B),, (i) AGGAN (Mejaati et al. (2018))
to
(a)
(b)
(c)
(d)
(c)
(f)
gg)
(h)
Figure 10: Visual comparisons of the dog2cat with attention features maps. (a) Source images, (b) Attention map of the generation, (c-d) Local and global attention maps of the discriminators, (e) Our results, (f) CycleGAN (Zhu al. (2017 ). (g) UNIT (Liu al. (017)). (h) MUNIT (Huang et al.
(2118) ) (i) DRIT (Lee et al. (2018), 1), (j) AGGAN Mejjati et al. (2018)).
18

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(c)
(1)
g
(h)
Figure 11:
Visual comparisons of the photo2vangogh with attention features maps. (a) Source
images, (b) Attention map of the generation, (c-d) Local and global attention maps of the discrimi- nators, respectively, (e) Our results, (f) CycleGAN (Zhu al. (2017). (g) UNIT (Liu et (2017)
(h) MUNIT (Huang et al. (018)), (i) DRIT (Lee et al. (2018). (i) AGGAN (Mejjati et al. (2018)
(a)
(b)
(c)
(d)
(c)
(f)
(g)
(h)
(i)
Figure 12: Visual comparisons of the photo2portrait with attention features maps. (a) Source images, (b) Attention map of the generator, (c-d) Local and global attention maps of the discriminators, respectively, (e) Our results.(f) CycleGAN (Zhu al. (2017). (g) UNIT (Liu al. (017 ). (h) MUNIT (Huang et al. (2018)). (i) DRIT (Lee et al. (2018), (i) AGGAN (Mejjati et al. (0018)
19


Reviews:
[{"review_id": "rJeTLRLCFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have read the authors' rebuttal and satisfied with their response. Novelty is a little on the lower side, but thorough writing, results, and insightful comparisons make up for this in my opinion. I have updated my score to 8: Accept.\n\n=====\n\nThis paper proposes an approach to perform image translation called U-GAT-IT. In image translation, the goal is to learn a mapping from images in a source domain to corresponding images in a target domain. Contemporary image translation approaches are able to transfer local texture but struggle to handle shape transfer. To address this concern, the authors introduce an attention mechanism based on CAM [1] and an adaptive normalization layer into a GAN-based image translation framework. Results indicate favorable quantitative and qualitative performance relative to a number of baselines.\n\nSpecific contributions include:\n* Introduction of a normalization layer called AdaLIN that can interpolate between instance normalization and layer normalization based on the input.\n* Introduction of an attention mechanism based on CAM [1] that allows the model to focus on specific parts of the image when either generating or discriminating.\n* Collection and release of a selfie-to-anime dataset.\n* Release of U-GAT-IT code.\n  \nIn my opinion this paper is borderline, leaning towards weak accept. The experiments are thorough and the paper is well-written. I have concerns about the novelty and significance of the work, but overall the paper feels very close to being a finished piece of work in spite of its (relatively minor) flaws.\n  \nStrong points of this work include the writing and experiments. The paper is clearly organized and feels polished. It cites many relevant works, giving the reader a sense of the contemporary approaches for image translation. There is a thorough description of model architecture, dataset and tuning parameters in the appendix. In addition, code and the selfie-to-anime dataset have been released by the authors. In terms of experiments, the authors provide many qualitative visualizations comparing the proposed model to baselines on various datasets. Quantitative evaluation includes KID and a perceptual evaluation on human subjects.\n\nWeak points include novelty and significance. The proposed approach combines two ideas already applied to image translation (adaptive normalization [3] and attention [4]).  It therefore synthesizes these ideas into an effective algorithm rather than directly adding something new. It is unclear to me how others can build on top of this work to further advance state-of-the-art in image translation. Are more sophisticated normalization and attention mechanisms truly the key to improving image translation in the future?\n\nSpecific comments:\n* The formulation of AdaLIN in Equation (1) is vague. The text states \"parameters are dynamically computed by a fully connected layer from the attention map\", but it's not clear what those parameters are in the equation. Explicitly writing \\gamma and \\beta as functions of the fully-connected layer and \\mu_I, \\sigma_I, \\mu_L, \\sigma_L as the corresponding mean and standard deviation expressions would make things more clear.\n* The motivation for using layer normalization was discussed in 2.1.1 but I still do not understand why it is beneficial.\n* The term \"importance weights\" has a specific meaning in the context of Monte Carlo methods. I would suggest choosing a different term here.\n\nQuestions for the authors:\n* How does U-GAT-IT compare to TransGaGa [2]? One of the stated goals of U-GAT-IT is to better handle shape when performing image translation. TransGaGa has a similar motivation and so I would have liked to see an experimental comparison or at the very least a description of how U-GAT-IT differs. What sorts of shape transfer could U-GAT-IT handle that TransGaGa couldn't and vice versa?\n* What are the shortcomings of the model and how could they possibly be addressed? \n  \n[1] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A., 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2921-2929).\n[2] Wu, W., Cao, K., Li, C., Qian, C. and Loy, C.C., 2019. Transgaga: Geometry-aware unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8012-8021).\n[3] Huang, X. and Belongie, S., 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1501-1510).\n[4] Mejjati, Y.A., Richardt, C., Tompkin, J., Cosker, D. and Kim, K.I., 2018. Unsupervised attention-guided image-to-image translation. In Advances in Neural Information Processing Systems (pp. 3693-3703).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syg1daPM9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to do/contribute.\n* The paper proposes a new image-to-image GAN-based translator that uses attention and a new normalization that learns a proper ratio between instance and layer normalization. Experiments benchmark the new method against multiple prior ones, and on a number of dataset pairs.\n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\nWeak Accept\n\n* The paper was well-written and the method and contributions are clearly explained.\n* There is clear novelty in this paper, even if slightly limited. However, the newly proposed normalization seems to work quite well.\n* The results look good, however it is hard to compare methods quantitatively with only few samples. (Nothing that the authors could have done: there are many samples in the supplementary material and results seem consistent.) Qualitative measures like FID and KID should be taken with a grain of salt also. It is a big plus that a user study was conducted! (However, details of how these subjects were selected would be useful)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g6gslaFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new attention mechanism for unsupervised image-to-image translation task. The proposed attention mechanism consists of an attention module and a learnable normalization function. Sufficient experiments and analysis are done on five datasets.  \n\nPros:\n1. The proposed method seems to generalize well to the different datasets with the same network architecture and hyper-parameters compared to previous works. This could benefit other researchers who want to apply the method to other data or tasks.\n2. The translated results seem more semantic consistent with the source image compared to other methods, although the sores are not the top on photo2portrait and photo2vangogh. The results also look more pleasing.\n\nCons:\n1. The CAM loss is one of the key components in the proposed method. However, there is only the reference and no detailed description in the paper. More intuitive descriptions are necessary for easy understanding.\n2. The local and global discriminators are not explained until the result analysis. It’s a bit confusing when I see the local and global attention maps visualization results. It’s better to mention it in the method section.\n3. I wonder why some translations are not done at all in the results without CAM in Figure 2(f). Because without CAM, the framework would be somehow similar to MUNIT or DRIT. I suppose the hyper-parameters are not suitable for this setting.\n4. The generator model architecture in Figure 1 is confusing. The adaptive residual blocks only receive the gamma and beta parameters. I suppose that the encoder feature maps are also fed into the adaptive residual blocks.\n5. In Figure 3, the comparison of the results using each normalization function is reported. While in my view, the results only using GN in decoder with CAM looks more natural. I wonder why the proposed method only consists of instance norm and layer norm? I suppose the group norm might help with the predefined group.\n6. In the ablation study, the CAM is evaluated for generator and discriminator together. I would recommend doing this ablation study for generator and discriminator separately to see if it’s necessary for generator or discriminator.\n7. It would be good to see some discussion on the attention mechanism compared with other related works. For example,  [a,b] predict the attention masks for unsupervised I2I, but applies them on the pixel/feature spatial level to keep the semantic consistency.\n[a] Unsupervised-Attention-guided-Image-to-Image-Translation. NIPS’18\n[a] Exemplar guided unsupervised image-to-image translation with semantic consistency. ICLR’19\n\nMy initial rating is above boardline.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new architecture for unsupervised image2image translation.
Following the revision/discussion, all reviewers agree that the proposed ideas are reasonable, well described, convincingly validated, and of clear though limited novelty. Accept.
Decision: Accept (Poster)