Title: CROSS-DOMAIN CASCADED DEEP TRANSLATION

Abstract: In recent years we have witnessed tremendous progress in unpaired image-toimage translation methods, propelled by the emergence of DNNs and adversarial training strategies. However, most existing methods focus on transfer of style and appearance, rather than on shape translation. The latter task is challenging, due to its intricate non-local nature, which calls for additional supervision. We mitigate this by descending the deep layers of a pre-trained network, where the deep features contain more semantics, and applying the translation between these deep features. Specifically, we leverage VGG, which is a classification network, pre-trained on ImageNet with large-scale semantic supervision. Our translation is performed in a cascaded, deep-to-shallow, fashion, along the deep feature hierarchy: we first translate between the deepest layers that encode the higher-level semantic content of the image, proceeding to translate the shallower layers, conditioned on the deeper ones. We show that our method is able to translate between different domains, which exhibit significantly different shapes. We evaluate our method both qualitatively and quantitatively and compare it to state-of-the-art image-to-image translation methods. Our code and trained models will be made available.

1. INTRODUCTION:

In recent years, neural networks have significantly advanced generative image modeling. Following the emergence of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) , image-to-image translation methods have dramatically progressed, revolutionizing applications such as inpainting (Yu et al., 2018) , super resolution (Wang et al., 2018) , domain adaptation (Hoffman et al., 2017) , and more. In particular, there have been intriguing advances in the setting of unpaired image-to-image translation through the use of cycle-consistency (Yi et al., 2017; Zhu et al., 2017) as well as other approaches (Benaim & Wolf, 2017; Huang et al., 2018; Lee et al., 2018; Ma et al., 2018) . However, most existing methods acknowledge the difficulty in translating shapes from one domain to another, as this might entail highly non-trivial geometric deformations. Consider, for example, translating between elephants and giraffes, where we would expect the neck of an elephant to be extended, while the elephant's head should shrink. The challenge is compounded by the fact that, even within the same domain, images might exhibit extreme variations in object shape and pose, partial occlusions, and contain multiple instances of the object of interest. One might even argue that this translation task is ill-posed to begin with, and at the very least, requires high-level semantics to be accounted for.

Nonetheless, several works do address shape deformation in the context of image-to-image translation by requiring supervision in the form of a foreground mask (Liang et al., 2018; Mo et al., 2018) . In contrast, GANimorph (Gokaslan et al., 2018) and the recently proposed TransGaGa (Wu et al., 2019) show remarkable translation without requiring additional supervision for several datasets. However, these techniques excel in controlled setting only, where the images are controlled, and the foreground separation is rather simple.

In this paper, we address the problem of unpaired image-to-image translation, without requiring foreground masks, between two different domains, where the objects of interest share some semantic similarity (e.g., four-legged mammals), whose shapes and appearances may, nevertheless, be drastically different. Our key idea is to accomplish the translation task by learning to translate between deep feature maps. Rather than learning to extract the relevant higher-level semantic information for the specific pair of domains at hand, we leverage deep features extracted by a network pre-trained for image classification, thereby benefiting from its large-scale fully supervised training. Our work is motivated by the well-known observation that neurons in the deeper layers of pre-trained classification networks represent larger receptive fields in image space, and encode higher-level semantic content Zeiler & Fergus (2014) . In other words, local activation patterns in the deeper layers may encode very different shapes in size and structure. Furthermore, Aberman et al. (2018) showed that semantically similar regions from different domains, e.g. dog and cat, have similar activations. That is, the encoding of a cat's eye resembles that of a dog's eye more than that of its tail. From the point of view of the translation task, these properties are attractive, since they suggest that it might be possible to learn a semantically consistent translation between activation patterns produced by images from different domains, and that the resulting (reconstructed) image would be able to change drastically, hopefully bypassing the common difficulties in image-to-image translation methods.

More specifically, we learn to translate between several layers of deep feature maps, extracted from two domains by a pre-trained classification network, namely VGG-19 (Simonyan & Zisserman, 2014) . The translation is carried out one layer at a time in a deep-to-shallow (coarse-to-fine) cascaded manner. For each layer, we adversarially train a dedicated translator that acts in the features space of that layer. The deepest layer translator effectively learns to translate between semantically similar global structures, such as body shape or head position, as demonstrated by the middle pair of images in Figure 1 . The translator of each shallower layer is conditioned on the translation result of the previous layer, and learns to add fine scale and appearance details, such as texture. At every layer, in order to visualize the generated deep features, we use a network pre-trained for inverting the deep features of VGG-19, following the method of Dosovitskiy & Brox (2016) . The images shown in Figure 1 were generated in this manner.

Our conceptual novelty, can be viewed as a transfer learning for image translation as we are translating high level semantics, encoded in the deeper layers of a pre-trained classification network, a.k.a deep features. This is in contrast to existing methods Gokaslan et al. (2018) ; Wu et al. (2019) , which learn to translate the images directly. We compare our method with several state-of-the-art image translation methods. To demonstrate the effectiveness of our approach, we present results for several challenging pairs of domains that exhibit drastically different shapes and appearances, but share some high-level semantics. Our translations are semantically consistent, typically preserving pose and number of instances of objects of interest, and reproducing their partial occlusion or cropping, as may be seen in Figure 5 . Zhu et al. (2017) (simultaneously with Yi et al. (2017) and Kim et al. (2017) ) have presented remarkable unpaired image-to-image translations, using a framework known as CycleGAN. The key idea is that the ill-posed conditional generative process can be regularized by a cycle-consistency constraint, which enforces the translation to perform a bijective mapping. The cycle constraint has become a popular regularization technique for unpaired image-to-image translation. For example, the UNIT framework (Liu et al., 2017) assumes a shared latent space between the domains and enforces the cycle constraint in the shared latent space. Several works were developed to extend the one-to-one mapping to many-to-many mapping (Ma et al., 2018; Huang et al., 2018; Lee et al., 2018; Almahairi et al., 2018) . These methods decompose the encoding space to shared latent space, representing the domain invariant content space, and domain specific style space. Therefore, many translations can be achieved from a single content code by changing the style code of the input image.

2. RELATED WORK:

Many translation methods share the inability to translate high-level semantics, including different shape geometry. This type of translation is usually necessary in the case of transfiguration, where one aims to transform a specific type of objects without changing the background. Both Lee et al. (2018) and Mejjati et al. (2018) learn an attention map and apply translation only on the the foreground object. However, both methods only improve translations that do not deform shapes. Gokaslan et al. (2018) succeed in preforming several shape-deforming translations by several modifications to the CycleGAN framework, including using dilated convolutions in the discriminator. However, they have not demonstrated strong shape deformations, such as zebras to elephants or giraffes, as we show in Section 4. Liang et al. (2018) and Mo et al. (2018) assume some kind of segmentation is given, and use this segmentation to guide shape deformation translation. However, such segmentation is hard to achieve. In a recent work, Wu et al. (2019) disentangle the input images to geometry and appearance spaces, relying on high intra-consistency, and learn to translate each of the two domains separately. However, the variation of geometry and appearance of in-the-wild images is too large to be disentangled successfully1 . Contrary to the above works, our work leverages a pre-trained network and the translation is applied directly on deep feature maps, thus being guided by high-level semantics. Several image-to-image methods, such as Xu et al. (2018) ; Di et al. (2018) ; Ignatov et al. (2018) , also incorporate such pretrained networks, though usually, only as perceptual loss, constraining the translated image to remain semantically close to the input image. Differently, Sungatullina et al. (2018) incorporate pre-trained VGG features into the discriminator architecture, to assist in the discrimination phase. Wu et al. (2018) use VGG-19 as a fixed encoder in the translation, where only the decoder is learned. Upchurch et al. (2017) present the only method, to our knowledge, that actually translates deep features between two domains. However, the translation is not learned, but defined by simply interpolating between the deep features, which restricts the scope of method to highly aligned domains. For completeness, we also mention that Yin et al. (2019) train an autoencoder to embed point clouds, and perform translation in the learned embedding. In contrast, we utilize semantics to preform the translation in the much more difficult scenario of images.

Our work shares some similarities with the early work of Huang et al. (2017) , which suggests using a generative adversarial model (Goodfellow et al., 2014) in a coarse-to-fine manner with respect to a pre-trained encoder. The generation process begins from the deepest features and then recursively synthesizes shallower layers conditioned on the deeper layer, until generating the final image. This method was only applied on small encoders and low resolution images and was not explored for very deep and semantic encoding neural networks such as VGG-19 (Simonyan & Zisserman, 2014) .

Deep image analogies (Liao et al., 2017) transfer visual attributes between semantically similar images, by feed-forwarding them through a pre-trained network. Their work does not train a generative model; nonetheless, they create new deep features by fusing content features from one image with style features of another. Similarly, Aberman et al. ( 2018) synthesize hybrid images from two aligned images by selecting the dominant deep features response.

3. METHOD:

Our general setting is similar to that of previous unpaired image-to-image translation methods. Given images from two domains, A and B, our goal is to learn to translate between them. However, unlike other image-to-image translation methods, we perform the translation on the deep features extracted by a pre-trained classification network, specifically VGG-19 (Simonyan & Zisserman, 2014) .

The translation is carried out in a deep-to-shallow (coarse-to-fine) manner, using a cascade of pairs of translators, one pair per layer. The entire architecture used to train the translators is shown schematically in Figure 2 , while Figure 3 illustrates the test-time translation (inference) process. Once the deepest feature map has been translated, we translate the next (shallower and less semantic feature map), conditioned on the translated deeper layer. In this manner, the translation of the shallower map preserves the general structure of the translated deeper one, but adds finer details, which are not encoded in the deeper feature maps. We repeat this procedure until the original image level is reached. Below we describe the training and the inference processes in more detail.

G 0 B G 1 B G 2 B G 3 B G 4 B G 5 B

Pre-processing: We extract high-level semantic features from input images from both domains, A and B, by feed-forwarding the images through the pre-trained VGG-19 (Simonyan & Zisserman, 2014) network. Next, we sample five of the resulting deep feature maps, specifically conv_i_1 (i = 1, 2, 3, 4, 5), where each map has progressively coarser spatial resolution, but a larger number channels. We denote the i-th sampled feature map for image a ∈ A as a i . Since propagation through the pre-trained VGG-19 network may yield features in any range, while the range of the synthesized features is usually known, to ease the translation, we first normalize each channel, of every layer i, by calculating its mean and standard deviation across the domain and then we clamp the normalized feature values to the range of [-1, 1]. While the clamping is a potentially harmful irreversible operation, we did not observe any adverse effect on the results. We use A i (B i ) to denote set of all normalized deep feature maps of level i, extracted from images in domain A (B).

Inference: We perform the translation in a coarse-to-fine fashion. Thus, the translator from domain A to B, actually consists of a sequence of translators G 5 B , G 4 B , . . . , G 1 B , where each translator is responsible for translating the i-th feature map layer a i , from A i to B i conditioned on the previously translated deeper layer bi+1 (except for the deepest layer translator G 5 B , which is unconditioned).

Finally, G 0 B uses feature inversion to convert b1 to obtain the translated image. The translators G i A from domain B i to A i are defined symmetrically. The entire inference pipeline is shown in Figure 3 .

Feature inversion::

In all the results we show, e.g., Figure 1 , we visualize the output of the various translators by pre-training a deep feature inversion network (per domain), for each layer i = 1, . . . , 5, following Dosovitskiy & Brox (2016) . The network aims to reconstruct the original image given the feature map of a specific layer, regularized by adversarial loss so that the reconstructed image would lie in the manifold of natural images. For more details we refer the reader to Dosovitskiy & Brox (2016) . The specific settings used in our implementation are elaborated in Appendix A.1.

Deepest layer translation: We begin by translating the deepest feature maps, encoding the highestlevel semantic features, i.e., A 5 and B 5 , hence, our problem is reduced to translating high-dimensional tensors. Our solution builds upon the commonly used CycleGAN framework (Zhu et al., 2017) . Specifically, we use the three losses proposed in Zhu et al. (2017) . First, in order to generate deep features in the appropriate domain, we utilize an adversarial domain loss L adv . We simultaneously train two translators G 5 A , G 5 B which try to fool domain-specific discriminators, D 5 A , D 5 B (for domains A 5 , B 5 , respectively). However, differently from Zhu et al. (2017) and other image translation methods (Huang et al., 2018; Mo et al., 2018) , we have found LSGAN (Mao et al., 2017) not to be well-suited for our task, leading to mode collapse or convergence failures. Instead, we found WGAN-GP (Gulrajani et al., 2017) more effective, thus, the adversarial loss for translation from X to Y is defined as

L adv (G Y , D Y , X, Y ) = E x∼P X [D Y (G Y (x))] -E y∼P Y [D Y (y)] + λ gp E ŷ∼P Ŷ ( ∇D Y (ŷ) -1) 2 , (1)

where G Y : X → Y is the translator, D Y is the target domain discriminator, λ gp = 10 in all our experiments, and P Ŷ is defined by uniformly sampling along straight lines between ỹ ∼ G (P X ) and y ∼ P Y . For more details we refer the reader to Gulrajani et al. (2017) .

Second, for regularizing the translation to one-to-one mapping, we add the cycle constraint,

EQUATION

where • stands for the L 1 norm.

Finally, as in Zhu et al. (2017) , we have found it helpful to use an identity loss, which guides the networks to preserve common high level features,

EQUATION

The entire loss combines these components as follows

EQUATION

where λ cyc and λ idty were set to 100 in all our experiments.

Coarse to fine conditional translation: Consider two successive layers, a i ∈ A i and a i+1 ∈ A i+1 , where the latter has already been translated, yielding bi+1 as the translation outcome (see Figure 3 ). We next perform the translation of the layer a i to yield bi , using the translator G i B , conditioned on bi+1 . Note that G i B is effectively a function of all the previously translated layers. The architecture of G i B is schematically shown in Figure 4 . Since shallower layers encode less of the semantic content of the image, it is more difficult to learn how they should be deformed, and thus they are used to transfer "style", while the "content" comes from the already translated deeper layer. Inspired by Huang et al. (2018) , we add an adaptive instance normalization (AdaIN) (Huang & Belongie, 2017) component, whose parameters are learned from the current layer. Thus, several layers of G i B are normalized according to the AdaIN component. G i A , which is designed symmetrically, is learned simultaneously with G i B , as shown in Fig. 4 (a). The loss for training these shallower translators is roughly the same as that used for training the deepest translation: it consists of adversarial domain loss, cycle constraint loss, and identity loss. While we formalize the adversarial loss unconditionally, similarly to equation 1, the cyclic loss is now conditioned: 

G i A G i B a i , bi+1 , a i+1 -a i + G i B G i A (b i , ãi+1 ) , b i+1 -b i ,

and the same is true for the identity loss:

G i A (a i , a i+1 ) -a i + G i B (b i , b i+1 ) -b i .

4. EXPERIMENTS:

We evaluate our methods on several publicly available datasets: (1) Cat ↔ dog faces (Lee et al., 2018) , which contains 871 cats images and 1364 dogs images and does not require high shape deformation;

(2) Kaggle Cat ↔ dog (Elson et al., 2007) dataset with over 12, 500 images in each domain (the images may contain part of the object or several instances); (3) Challenging MSCOCO dataset (Lin et al., 2014) , specifically, zebra ↔ elephant and zebra ↔ giraffe (the number of images is each category is reported in (overall there are 1917 zebras, 2547 giraffes and 2144 elephants). We note that no previous method has used MSCOCO, without supervision in the form of segmentation. →/← We compare the FID scores on three datasets, measured both directions per dataset. The two directions appear side-by-side, →/←, at each cell.

Our deepest translators, i.e., G 5 A , G 5 B , consist of encoder-decoder structure with several strided convolutional layers followed by symmetric transpose convolutional layers. We use group normalization (Wu & He, 2018) and ReLU activation function (except the last layer, which is tanh). The conditional generators, consist of learned AdaIN layer, achieved by several strided convolutional layers followed by fully connected layers. The content generator has also several convolutional layers and one single transpose convolutional layer which doubles the spatial resolution (Figure 4 (right)). In practice we only train G 5 , G 4 , G 3 , and apply feature inversion directly on the output of the latter, with negligible degradation. For the exact layer's specifics we refer the reader to Appendix A.1 and to our (soon to be published) code. We run each layer for 400 epochs with fixed learning rate of 0.0001 and Adam optimizer (Kingma & Ba, 2014) . On a single RTX 2080 reaching the final image takes around 2.5 days, including the final inversion network training.

Several translation examples are presented in Figure 5 . Our translation is able to achieve high shape deformation. Note that our translations are semantically consistent, in the sense that they preserve the pose of the object of interest, as well the number of instances is mostly preserved. Furthermore, partial occlusions of such objects, or their cropping by the image boundaries are correctly reproduced. See for example, the translations of the pairs of animals in columns 5-6. More results are provided in Appendix A.2.

4.1. ABLATION STUDY:

We analyze two main elements of our method. Loss components First, we validate the use of CycleGAN loss components. As shown in Figure 6 , we translate the 5th (deepest) layer with and without cycle, identity and adversarial losses. The best approach is achieved by using all of the losses, which balance each other.

VGG layers In addition, In Figure 7 we compare translation of different VGG-19 layers. Evidently, shallower layers introduce spatial constraints, thus, limiting the translation in the sense of shape's changes. The shallowest layer can hardly change the shape of the input image, which may explain the failure of traditional image translation methods. In Table 1 , we use the common FID score (Szegedy et al., 2015) to show that the cascaded translation achieves better translation compared to individual layer translation. Additional results are shown in Appendix A.4

Type of pre-trained network While our method is conceptually agnostic to the type of features extraction network, we still require the extracted features to represent high-level semantic. Therefore, VGG-19 layers, which are the most common features used for image generation tasks (Aberman et al. . We fine-tuned VGG to classify between zebras and giraffes and trained our translation networks using the resulting features. As can be seen, the translation results are inferior to the results achieved by the standard VGG-19. This maybe attributed to VGG-19 fixating on the unique differences between zebras and giraffes, unrelated to the translation, such as background. For more about the extracted features, we refer the reader to Appendix A.1.3. In addition, in Figure 9 , we examine a different network, AlexNet, also pretrained on ImageNet. We have observed that the deepest image translation was not able to generate a valid shape of zebra or giraffe. AlexNet is composed of 5 convolutional layers, where the first one uses a stride of 4. Therefore, the resulting features have less spatial encoding, especially at the deeper layers, which may explain the difficulty to invert and translate these features.

Under review as a conference paper at ICLR 2020 Figure 10 : Comparison of the deepest latent spaces (5th layer), projected using t-SNE. The latent space of the source domain is in blue, and the target domain is in red. The distribution of the translation results (in cyan) is most similar to that of the target domain when using our method.

4.2. COMPARISON TO OTHER METHODS:

We compare our result with leading image translation methods, i.e. CycleGAN (Zhu et al., 2017) , MUNIT (Huang et al., 2018) , DRIT (Lee et al., 2018) and GANimorph (Gokaslan et al., 2018) .

Quantitative comparison: In order to perform a quantitative comparison, we use the common FID score (Szegedy et al., 2015) , as reported in Table 2 . Our method achieved the best FID score on five out of the six cross-domain translations for which this score was measured.

Qualitative comparison::

In Figure 8 we show several challenging translation examples. While traditional image translation methods struggle to preform translations with such drastic shape deformation, our method is able to do so thanks to its use of the pre-trained VGG-19 network.

The success of our method can also be explained and visualized by examining the translated deep features. We feed forward every image, original and translated, through the entire VGG network, extracting the last fully-connected layer (before the classification layer). We project this vector (of size 4096) to 2D, using t-SNE, as shown in Figure 9 . It may be seen that the distribution of the translated vectors (in cyan) is closest to that of the target domain (in red) when using our method.

Limitations Our method achieves translations with significant shape deformation in many previously unattainable scenarios, yet, a few limitations remain. First, the background of the object is not preserved, as the background is encoded in the deep features along with the semantic parts. Also, in some cases the translated deep features may be missing small instances or parts of the object. This may be attributed to the fact that VGG-19 is generally not invertible and was trained to classify finite set of classes. In addition, since we translate deep features, small errors in the deep translation may be amplified to large errors in the image, while for image-to-image translation method that operate on the image directly, small translation errors would typically be more local. Please note that, similarly to CycleGAN and GANimorph, our translation is deterministic.

5. CONCLUSIONS:

Translating between image domains that differ not only changes in appearance, but also exhibit significant geometric deformations, is a highly challenging task. We have presented a novel image-toimage translation scheme that operates directly on pre-trained deep features, where local activation patterns provide a rich semantic encoding of large image regions. Thus, translating between such patterns is capable of generating significant, yet semantically consistent, shape deformations. In a sense, this solution may be thought of as transfer learning, since we make use of features that were trained for a classification task for an unpaired translation task. In the future, we would like to continue exploring the applications of powerful pre-trained deep features for other challenging tasks, possibly in different domains, such as videos, sketches or 3D shapes.

A.2 MORE COMPARISON RESULTS:

In this section we show more results, not presented in the paper, for zebra↔giraffe, zebra↔elephant and cat↔dog translations.

Original:

Cycle GAN MUNIT DRIT GANimorph Ours Our method is also suited to none-shape deformation tasks, as in the case of dataset(1) (Lee et al., 2018) . 

:

Unfortunately, at the time of this submission the authors ofWu et al. (2019) were unable to release their code or train their network on the datasets presented in our paper.

Figures and Tables:

Figure 1: Given an image from domain A (zebras), we extract its deep features using a network pre-trained for classification, specifically VGG-19 pre-trained on ImageNet, and translate them into deep features of domain B (giraffes). We first translate high level semantics encoded in conv_5_1 of the zebra to those of a giraffe, represented by the inner images. Then, we use a cascade of deep-toshallow adversarially trained translators, one for each deep feature layer, to translate shallower layers, i.e. conv_4_1 (middle sized zebra and giraffe) and then conv_3_1 (outer images) . The presented images were obtained by training feature inversion networks, from deep features to image space.

Figure 2: Translation architecture. We translate between domains A and B starting from the deepest feature maps A 5 and B 5 , which encode the highest level semantic content of the images. Translation proceeds from deeper to shallower feature maps until reaching the image itself. The feature maps are extracted by feed forwarding every image through the pre-trained VGG-19 network and sampling five of its layers. Every layer's translation is learned individually, conditioned on the translation result of the next deeper layer (except the deepest layer, whose translation is unconditional).

Figure 3: Translation of the top left image at test time. The input image (left) is fed forward through VGG-19 layers, as indicated by the right purple arrows. Then, starting from the deepest layer a 5 we translate each layer. The final result is obtained from the shallowest layer using feature inversion.

Figure 4: Translation of layer i is conditioned on the previously translated layer i + 1. The two translators G i A and G i B are trained simultaneously (see left figure), where i + 1, . . . , 5 translators are fixed. On the right we show the schematic architecture of G i B which has two inputs: a i ∈ A i and bi+1 . a i is fed-forward through several layers to yield AdaIN parameters which control the generation of bi . Since bi has twice the spatial size of bi+1 , we add an upsampling layer marked by ↑.

We train the pairs of translators one layer at a time, starting from G 5 A and G 5 B . More details regarding the implementation and the training of the translators are included in Appendix A.1.

Figure 6: Translation of the 5th (deepest) layer with different loss combinations. Using all three components yields the best result.

Figure 8: Comparison to other image-to-image translation methods. The unpaired translations, from left to right, are zebra ↔ giraffe, elephant ↔ zebra and dog ↔ cat, where every translation has four examples, two in each direction. While previous translation methods struggle to deform the geometry of the source images, our method is able to preform drastic geometric deformation, while preserving the poses of the subjects and the overall composition of the image.

Figure 11: Qualitative comparisons. MSCOCO zebra to giraffe.

Figure 23: Translation results from cats to dogs (faces).

Figure 26: Coarse to fine translation of giraffe to zebra. Two different examples are shown in each row. The original image (left) is translated by the deepest translator (second left) and then in coarse to fine manner, shallower layers are translated (second right and most right).

Dog 125.75/94.27 159.57/108.51 153.94/139.17 139.17/134.14 67.58/46.02 FID score comparison. We compare FID scores on three datasets, measured for both translation directions per dataset. The two directions appear side-by-side, →/←, at each cell.

Table content: <table><tr><td>→/←</td><td>Cycle GAN</td><td>MUNIT</td><td>DRIT</td><td>GANimorph</td><td>Ours</td></tr><tr><td>Cat ↔ Zebra ↔ Giraffe</td><td>55.65/58.93</td><td>238.06/60.78</td><td>59.75/54.06</td><td colspan="2">98.25/120.05 67.41/39.38</td></tr><tr><td>Zebra ↔ Elephant</td><td>86.55/68.44</td><td>109.56/80.1</td><td>78.01/56.39</td><td>99.98/89.74</td><td>68.45/47.86</td></tr><tr><td colspan="2">CycleGAN</td><td>MUNIT</td><td>DRIT</td><td>GANimorph</td><td>Ours</td></tr><tr><td>Zebra Elephant !</td><td/><td/><td/><td/><td/></tr><tr><td>Zebra ! Gira↵e</td><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
* Summarize what the paper claims to do/contribute. This paper claims to extend existing image translation works, like CycleGAN, to domain pairs that are not similar in shape. It is proposed to do so by using a VGG network trained on classification (I assume on Imagenet), extracting features from the two domains and learn 5 CycleGANs to translate for each level of the feature hierarchy. At each level of the hierarchy the translation from the previous level is used to condition the translation for the current level. During inference, the final image translation is done by *feature inversion* (a technique proposed in Dosovitsikiy and Brox, 2016) from the final feature layer. The technique is show on example from a number of pairs of domains like Zebra-to-Elephant (and back), Giraffe-to-Zebra (and back), Dog-to-Cat (and back) and is compared with a number of baselines qualitatively and quantitatively with the FID score. * Clearly state your decision (accept or reject) with one or two key reasons for this choice. Weak Reject. Major reasons: - The problem itself, as stated in the introduction, seems ill-posed to me. One of the struggles I had while looking through the results was to understand what the images should be looking like. ie What should a zebra translated to a giraffe look like? The motivation for such a problem is also not immediately clear either. - Most of the resulting images do not seem *translated* to me. As stated in the paper (end of p.2) *one aims to transform a specific type of object without changing the background.* As one can see in eg Fig. 1 the resulting translations are completely different images with the foreground object of the new domain in roughly similar poses. The background in most cases does not persist. What I suspect is actually happening here is that the high-level semantics from the first image are used as some sort of noise to generate new images from the new domain. One question I had, for example: could we be getting similar results if we used the VGG bottleneck as the noise vector in an InfoGAN? Since the VGG network is pretrained and used in the same way in both domains, I imagine we would be seeing something very similar. (and it would be def. preferrable to tuning 10 GANs!) * Provide supporting arguments for the reasons for the decision. Some of the decisions made in the paper were unclear and not supported adequately. The questions (in rough order of importance) that made some of the contributions unclear to me: - Why wasn*t a final translator used for the final image, conditioned on the final 	ilde{b}_1? - Is the VGG network pretrained on ImageNet? Why wasn*t another task used that could be retaining more of the relevant features? eg on semantic segmentation - Could this be used for networks pretrained on other datasets? Presumably ImageNet has information about the animals translated in this paper. Even better, could we somehow learn these features for the domain pairs automatically somehow? - How meaningful is the FID score really in this case? - How were the 10 GANs tuned? * Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment. - It is mentioned on p.4 that *clamping is potentially a harmful irreversible operation* but that harmful results were not observed. As I was reading that I was wondering how these results would actually look like. - On p. 6 it is mentioned that the number of images for 2 categories are reported in another paper. I think it*d take less space to actually report the number of images here. - On p.7 it is mentioned that the number of instances is preserved, however it should be made clear that it*s is perserved in some (or most if that is what was observed) of the examples.