Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency

1 INTRODUCTION . Image-to-image ( I2I ) translation refers to the task of mapping an image from a source domain to a target domain , e.g . semantic maps to real images , gray-scale to color images , low-resolution to high-resolution images , and so on . The recent advances in deep learning have greatly improved the quality of I2I translation methods for a number of applications , including super-resolution ( Dong et al. , 2014 ) , colorization ( Zhang et al. , 2016 ) , inpainting ( Pathak et al. , 2016 ) , attribute transfer ( Lee et al. , 2018 ) , style transfer ( Gatys et al. , 2016 ) , and domain adaptation ( Hoffman et al. , 2018 ; Liu et al. , 2017 ) . Most of these works ( Isola et al. , 2017 ; Wang et al. , 2018 ; Zhu et al. , 2017b ) have been very successful in these cross-domain I2I translation tasks because they rely on large datasets of paired training data as supervision . However , for many tasks it is not easy , or even possible , to obtain such paired data that show how an image in the source domain should be translated to an image in the target domain , e.g . in cross-city street view translation or male-female face translation . For this ∗Part of this work was done when Xu Jia was in KU Leuven Published as a conference paper at ICLR 2019 Transfer xB Z EA GB Exemplar-specific branch A ( ESA ) Exemplar-specific branch B ( ESB ) Domain A Domain B Transfer Feature Mask Instance Normalization content style semantic xA xAB Gaussian prior γB , βB xB Z EA GB Guide Branch A ( GBA ) Guide Branch B ( GBB ) Domain A Domain B Transfer Featuremask AdaIN param content style semantic xA xAB Gaussian prior γB , βB AdaIN Transfer Weights sharing unsupervised setting , Zhu et al . ( 2017a ) proposed to use a cycle-consistency loss , which assumes that a mapping from domain A to B , followed by its reverse operation approximately yields an identity function , that is , F ( G ( xA ) ) ≈ xA . Liu et al . ( 2017 ) further proposed a shared-latent space constraint , which assumes that a pair of corresponding images ( xA , xB ) from domains A and B respectively can be mapped to the same representation z in a shared latent space Z . Note that , all the aforeme tioned methods assume that there is a deterministic one-to-one mapping between the two domains , i.e . each image in A is translated to only a single image in B . By doing so , they fail to capture the multimodal nature of the image distribution within the target domain , e.g . different color and style of shoes in sketch-to-image translation and different seasons in synthetic-to-real street view translation . In this work , we propose Exemplar Guided & Semantically Consistent I2I Translation ( EGSCIT ) to explicitly address this issue . As shown in concurrent works ( Gonzalez-Garcia et al. , 2018 ; Huang et al. , 2018 ; Lee et al. , 2018 ) , we assume that an image is composed of two disentangled representations . In our case , first a domain-shared representation that models the content in the image , and second a domain-specific representation that contains the style information . However , for a multimodal domain with complex inner-variations , as the ones we target in this paper , e.g . street views of day-and-night or different seasons , it is difficult to have a single static representation which covers all variations in that domain . Moreover , it is unclear which style ( time-of-day/season ) to pick during the image translation process . To handle such multimodal I2I translations , some approaches ( Almahairi et al. , 2018 ; Gonzalez-Garcia et al. , 2018 ; Lee et al. , 2018 ) incorporate noise vectors as additional inputs to the generator , but as shown in ( Isola et al. , 2017 ; Zhu et al. , 2017b ) this could lead to mode collapsing issues . Instead , we propose to condition the image translation process on an arbitrary image in the target domain , i.e . an exemplar . By doing so , EGSC-IT does not only enable multimodal ( i.e . many-to-many ) image translations , but also allows for explicit control over the translation process , since by using different exemplars as guidance we are able to translate an input image into images of different styles within the target domain – see Fig . 1 . To instantiate this idea , we adopt the weight sharing architecture proposed in UNIT ( Liu et al. , 2017 ) , but instead of having a single latent space shared by both domains , we propose to decompose the latent space into two components according to the two disentangled representations presented above . That is , a domain-shared component that focuses on the image content , and a domain-specific component that captures the style information associated with the exemplar . In our particular case , the domain-shared content component contains semantic information , such as the objects ’ category , shape and spatial layout , while the domain-specific style component contains the style information , such as the color and texture , to be translated from a target domain exemplar to an image in the source domain . To realize this translation , we apply adaptive instance normalization ( AdaIN ) ( Huang & Belongie , 2017 ) to the shared content component of the source domain image using the AdaIN parameters computed from the target domain exemplar . However , directly applying AdaIN to the feature maps of the shared content component would mix up all objects and scenes in the image , Published as a conference paper at ICLR 2019 making the image translation prone to failure when an image contains diverse objects and scenes . To tackle this problem , existing works ( Gatys et al. , 2017 ; Hoffman et al. , 2018 ; Li et al. , 2018 ; Luan et al. , 2017 ) use semantic labels as an additional form of supervision . However , ground-truth semantic labels are not easy to obtain for most tasks as they require labor-intensive annotations . Instead , to maintain the semantic consistency during image translation without using any semantic labels we propose to compute feature masks . One can think of feature masks as attention modules that approximately decouple different semantic categories in an unsupervised way under the guidance of perceptual losses and adversarial loss . In particular , one feature mask corresponding to a certain semantic category is applied to one feature map of the shared content component , and consequently the AdaIN for that channel is only required to capture and model the style difference for that category , e.g . sky ’ s style in two domains . To the best of our knowledge , this is the first line of work that addresses the semantic consistency issue under this setting . See Fig . 2 for an overview of EGSC-IT . Our contribution is three-fold . i ) We propose a novel approach for the I2I translation task , which enables multimodal ( i.e . many-to-many ) mappings and allows for explicit style control over the translation process . ii ) We introduce the concept of feature masks for the unsupervised , multimodal I2I translation task , which provides coarse semantic guidance without using any semantic labels . iii ) Evaluation on different datasets show that our method is robust to mode collapse and can generate results with semantic consistency , conditioned on a given exemplar image . 2 RELATED WORK . I2I translation . I2I translation is used to learn a mapping from one image ( i.e . source domain ) to another ( i.e . target domain ) . Recently , with the advent of generative models ( Goodfellow et al. , 2014 ; Kingma & Welling , 2013 ) , there have been a lot of works on this topic . Isola et al . ( 2017 ) proposed pix2pix to learn the mapping from input images to output images using a U-Net neural network in an adversarial way . Wang et al . ( 2018 ) extended the method to pix2pixHD , to turn semantic label maps into high-resolution photo-realistic images . Zhu et al . ( 2017b ) extended pix2pix to BicycleGAN , which can model multimodal distributions and produce both diverse and realistic results . All these methods , however , require paired training data as supervision which may be difficult or even impossible to collect in many scenarios , such as synthetic-to-real street view translation or face-to-cartoon translation ( Royer et al. , 2017 ) . Recently , several unsupervised methods have been proposed to learn the mappings between two image collections without paired training data . Note that , this is an ill-posed problem since there are infinitely many mappings existing between two unpaired image domains . To address this ill-posed problem , different constraints have been added to the network to regularize the learning process ( Kim et al. , 2017 ; Liu et al. , 2017 ; Royer et al. , 2017 ; Yi et al. , 2017 ; Zhu et al. , 2017a ) . One popular constraint is cycle-consistency , which enforces the network to learn deterministic mappings for various applications . Going one step further , Liu et al . ( 2017 ) proposed a shared-latent space constraint which encourages a pair of images from different domains to be mapped to the same representation in the latent space . In a similar vein , Royer et al . ( 2017 ) proposed to enforce a feature-level constraint with a latent embedding reconstruction loss . However , we argue that these constraints are not well suited for complex domains with large inner-domain variations , as also mentioned in ( Almahairi et al. , 2018 ; Gonzalez-Garcia et al. , 2018 ; Lee et al. , 2018 ; Lin et al. , 2018 ) . Unlike these methods , to address this problem we propose to add a target domain exemplar as guidance during image translation through AdaIN ( Huang & Belongie , 2017 ) . As explained in the previous section , the AdaIN technique is utilized to transfer the style component from the target domain exemplar to the shared content component of the source domain image . This allows multimodal ( i.e . many-to-many ) translations and can produce images of desired styles with explicit control over the translation process . Concurrent to our work , MUNIT ( Huang et al. , 2018 ) , also proposed to use AdaIN to transfer style information from the target domain to the source domain . Unlike MUNIT , before applying AdaIN to the shared content component we compute feature masks to decouple different semantic categories and preserve the semantic consistency during the translation process . In particular , by applying feature masks to the feature maps of the shared content component , each channel can specialize and model the style difference only for a single semantic category , which is crucial when handling domains with complex scenes . Style transfer . Style transfer aims at transferring the style information from an exemplar image to a content image , while preserving the content information . The seminal work by Gatys et al . ( 2016 ) Published as a conference paper at ICLR 2019 proposed to transfer style information by matching the feature correlations , i.e . Gram matrices , in the convolutional layers of a deep neural network ( DNN ) following an iterative optimization process . In order to improve the speed and flexibility , several feed-forward neural networks have been proposed . Huang & Belongie ( 2017 ) proposed a simple but effective method , called AdaIN , which aligns the mean and variance of the content image features with those of the style image features . Li et al . ( 2017 ) proposed the whitening and coloring transform ( WCT ) algorithm , which directly matches the features ’ covariance in the content image to those in the given style image . However , due to the lack of semantic consistency during translation , these stylization methods usually generate non-photorealistic images , suffering from the “ spills over ” problem ( Luan et al. , 2017 ) . To address this , semantic label maps are used as additional supervision to help style transfer between corresponding semantic regions ( Gatys et al. , 2017 ; Li et al. , 2018 ; Luan et al. , 2017 ) . Unlike these works , we propose to compute feature masks to approximately model such semantic information without using any semantic labels that are very hard to collect . Table 1 summarizes the features of the most related works . As can be seen , our method using the combination of AdaIN and feature masks under the guidance of perceptual loss is , to the best of our knowledge , the first to achieve multimodal I2I translations in the unsupervised setting with high semantic consistency , without requiring any ground-truth semantic labels .

Summary: The introduction is written to perfection. The paper discusses a core failing and need for I2I translation models. The one-to-one mapping assumption does not apply to most tasks. While the approach seems novel the analysis of the results are insufficient to convince me that the method is really working. This should be a workshop paper.