CROSS-DOMAIN CASCADED DEEP TRANSLATION

In recent years we have witnessed tremendous progress in unpaired image-toimage translation methods , propelled by the emergence of DNNs and adversarial training strategies . However , most existing methods focus on transfer of style and appearance , rather than on shape translation . The latter task is challenging , due to its intricate non-local nature , which calls for additional supervision . We mitigate this by descending the deep layers of a pre-trained network , where the deep features contain more semantics , and applying the translation between these deep features . Specifically , we leverage VGG , which is a classification network , pre-trained on ImageNet with large-scale semantic supervision . Our translation is performed in a cascaded , deep-to-shallow , fashion , along the deep feature hierarchy : we first translate between the deepest layers that encode the higher-level semantic content of the image , proceeding to translate the shallower layers , conditioned on the deeper ones . We show that our method is able to translate between different domains , which exhibit significantly different shapes . We evaluate our method both qualitatively and quantitatively and compare it to state-of-the-art image-to-image translation methods . Our code and trained models will be made available . 1 INTRODUCTION . In recent years , neural networks have significantly advanced generative image modeling . Following the emergence of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , image-to-image translation methods have dramatically progressed , revolutionizing applications such as inpainting ( Yu et al. , 2018 ) , super resolution ( Wang et al. , 2018 ) , domain adaptation ( Hoffman et al. , 2017 ) , and more . In particular , there have been intriguing advances in the setting of unpaired image-to-image translation through the use of cycle-consistency ( Yi et al. , 2017 ; Zhu et al. , 2017 ) as well as other approaches ( Benaim & Wolf , 2017 ; Huang et al. , 2018 ; Lee et al. , 2018 ; Ma et al. , 2018 ) . However , most existing methods acknowledge the difficulty in translating shapes from one domain to another , as this might entail highly non-trivial geometric deformations . Consider , for example , translating between elephants and giraffes , where we would expect the neck of an elephant to be extended , while the elephant ’ s head should shrink . The challenge is compounded by the fact that , even within the same domain , images might exhibit extreme variations in object shape and pose , partial occlusions , and contain multiple instances of the object of interest . One might even argue that this translation task is ill-posed to begin with , and at the very least , requires high-level semantics to be accounted for . Nonetheless , several works do address shape deformation in the context of image-to-image translation by requiring supervision in the form of a foreground mask ( Liang et al. , 2018 ; Mo et al. , 2018 ) . In contrast , GANimorph ( Gokaslan et al. , 2018 ) and the recently proposed TransGaGa ( Wu et al. , 2019 ) show remarkable translation without requiring additional supervision for several datasets . However , these techniques excel in controlled setting only , where the images are controlled , and the foreground separation is rather simple . In this paper , we address the problem of unpaired image-to-image translation , without requiring foreground masks , between two different domains , where the objects of interest share some semantic similarity ( e.g. , four-legged mammals ) , whose shapes and appearances may , nevertheless , be drastically different . Our key idea is to accomplish the translation task by learning to translate between deep feature maps . Rather than learning to extract the relevant higher-level semantic information for the specific pair of domains at hand , we leverage deep features extracted by a network pre-trained for image classification , thereby benefiting from its large-scale fully supervised training . Our work is motivated by the well-known observation that neurons in the deeper layers of pre-trained classification networks represent larger receptive fields in image space , and encode higher-level semantic content Zeiler & Fergus ( 2014 ) . In other words , local activation patterns in the deeper layers may encode very different shapes in size and structure . Furthermore , Aberman et al . ( 2018 ) showed that semantically similar regions from different domains , e.g . dog and cat , have similar activations . That is , the encoding of a cat ’ s eye resembles that of a dog ’ s eye more than that of its tail . From the point of view of the translation task , these properties are attractive , since they suggest that it might be possible to learn a semantically consistent translation between activation patterns produced by images from different domains , and that the resulting ( reconstructed ) image would be able to change drastically , hopefully bypassing the common difficulties in image-to-image translation methods . More specifically , we learn to translate between several layers of deep feature maps , extracted from two domains by a pre-trained classification network , namely VGG-19 ( Simonyan & Zisserman , 2014 ) . The translation is carried out one layer at a time in a deep-to-shallow ( coarse-to-fine ) cascaded manner . For each layer , we adversarially train a dedicated translator that acts in the features space of that layer . The deepest layer translator effectively learns to translate between semantically similar global structures , such as body shape or head position , as demonstrated by the middle pair of images in Figure 1 . The translator of each shallower layer is conditioned on the translation result of the previous layer , and learns to add fine scale and appearance details , such as texture . At every layer , in order to visualize the generated deep features , we use a network pre-trained for inverting the deep features of VGG-19 , following the method of Dosovitskiy & Brox ( 2016 ) . The images shown in Figure 1 were generated in this manner . Our conceptual novelty , can be viewed as a transfer learning for image translation as we are translating high level semantics , encoded in the deeper layers of a pre-trained classification network , a.k.a deep features . This is in contrast to existing methods Gokaslan et al . ( 2018 ) ; Wu et al . ( 2019 ) , which learn to translate the images directly . We compare our method with several state-of-the-art image translation methods . To demonstrate the effectiveness of our approach , we present results for several challenging pairs of domains that exhibit drastically different shapes and appearances , but share some high-level semantics . Our translations are semantically consistent , typically preserving pose and number of instances of objects of interest , and reproducing their partial occlusion or cropping , as may be seen in Figure 5 . 2 RELATED WORK . Zhu et al . ( 2017 ) ( simultaneously with Yi et al . ( 2017 ) and Kim et al . ( 2017 ) ) have presented remarkable unpaired image-to-image translations , using a framework known as CycleGAN . The key idea is that the ill-posed conditional generative process can be regularized by a cycle-consistency constraint , which enforces the translation to perform a bijective mapping . The cycle constraint has become a popular regularization technique for unpaired image-to-image translation . For example , the UNIT framework ( Liu et al. , 2017 ) assumes a shared latent space between the domains and enforces the cycle constraint in the shared latent space . Several works were developed to extend the one-to-one mapping to many-to-many mapping ( Ma et al. , 2018 ; Huang et al. , 2018 ; Lee et al. , 2018 ; Almahairi et al. , 2018 ) . These methods decompose the encoding space to shared latent space , representing the domain invariant content space , and domain specific style space . Therefore , many translations can be achieved from a single content code by changing the style code of the input image . Many translation methods share the inability to translate high-level semantics , including different shape geometry . This type of translation is usually necessary in the case of transfiguration , where one aims to transform a specific type of objects without changing the background . Both Lee et al . ( 2018 ) and Mejjati et al . ( 2018 ) learn an attention map and apply translation only on the the foreground object . However , both methods only improve translations that do not deform shapes . Gokaslan et al . ( 2018 ) succeed in preforming several shape-deforming translations by several modifications to the CycleGAN framework , including using dilated convolutions in the discriminator . However , they have not demonstrated strong shape deformations , such as zebras to elephants or giraffes , as we show in Section 4 . Liang et al . ( 2018 ) and Mo et al . ( 2018 ) assume some kind of segmentation is given , and use this segmentation to guide shape deformation translation . However , such segmentation is hard to achieve . In a recent work , Wu et al . ( 2019 ) disentangle the input images to geometry and appearance spaces , relying on high intra-consistency , and learn to translate each of the two domains separately . However , the variation of geometry and appearance of in-the-wild images is too large to be disentangled successfully1 . Contrary to the above works , our work leverages a pre-trained network and the translation is applied directly on deep feature maps , thus being guided by high-level semantics . Several image-to-image methods , such as Xu et al . ( 2018 ) ; Di et al . ( 2018 ) ; Ignatov et al . ( 2018 ) , also incorporate such pretrained networks , though usually , only as perceptual loss , constraining the translated image to remain semantically close to the input image . Differently , Sungatullina et al . ( 2018 ) incorporate pre-trained VGG features into the discriminator architecture , to assist in the discrimination phase . Wu et al . ( 2018 ) use VGG-19 as a fixed encoder in the translation , where only the decoder is learned . Upchurch et al . ( 2017 ) present the only method , to our knowledge , that actually translates deep features between two domains . However , the translation is not learned , but defined by simply interpolating between the deep features , which restricts the scope of method to highly aligned domains . For completeness , we also mention that Yin et al . ( 2019 ) train an autoencoder to embed point clouds , and perform translation in the learned embedding . In contrast , we utilize semantics to preform the translation in the much more difficult scenario of images . Our work shares some similarities with the early work of Huang et al . ( 2017 ) , which suggests using a generative adversarial model ( Goodfellow et al. , 2014 ) in a coarse-to-fine manner with respect to a pre-trained encoder . The generation process begins from the deepest features and then recursively synthesizes shallower layers conditioned on the deeper layer , until generating the final image . This method was only applied on small encoders and low resolution images and was not explored for very deep and semantic encoding neural networks such as VGG-19 ( Simonyan & Zisserman , 2014 ) . Deep image analogies ( Liao et al. , 2017 ) transfer visual attributes between semantically similar images , by feed-forwarding them through a pre-trained network . Their work does not train a generative model ; nonetheless , they create new deep features by fusing content features from one image with style features of another . Similarly , Aberman et al . ( 2018 ) synthesize hybrid images from two aligned images by selecting the dominant deep features response .

Summary: The paper proposes a new method for image-to-image translation. The problem of most existing methods is that they are good in translating style  (e.g from photo to Van Gogh) but do not allow for significant changes in shape (e.g. from zebra to giraffe). The authors address this by performing the translation in a cascaded fashion starting from a semantic (deep) level (fifth layer of VGG). The underlying idea is that translating at this more semantic level puts less spatial constraints on the final resulting images (making translations from zebra to giraffe possible). After the fifth layer is translated other layers are translated conditioned on the previous translation results.