CROSS-DOMAIN CASCADED DEEP TRANSLATION
Authors: Oren Katzir
Dani Lischinski
Daniel Cohen-Or
Venue:  (2020)
Keywords: computer vision
image translation
generative adversarial networks

Abstract: In recent years we have witnessed tremendous progress in unpaired image-to-image translation methods, propelled by the emergence of DNNs and adversarial training strategies. However, most existing methods focus on transfer of style and appearance, rather than on shape translation. The latter task is challenging, due to its intricate non-local nature, which calls for additional supervision. We mitigate this by descending the deep layers of a pre-trained network, where the deep features contain more semantics, and applying the translation between these deep features. Specifically, we leverage VGG, which is a classification network, pre-trained with large-scale semantic supervision. Our translation is performed in a cascaded, deep-to-shallow, fashion, along the deep feature hierarchy: we first translate between the deepest layers that encode the higher-level semantic content of the image, proceeding to translate the shallower layers, conditioned on the deeper ones. We show that our method is able to translate between different domains, which exhibit significantly different shapes. We evaluate our method both qualitatively and quantitatively and compare it to state-of-the-art image-to-image translation methods. Our code and trained models will be made available.

Under review as a conference paper at ICLR 2020
CROSS-DOMAIN CASCADED DEEP TRANSLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
In recent years we have witnessed tremendous progress in unpaired image-to- image translation methods, propelled by the emergence of DNNs and adversarial training strategies. However, most existing methods focus on transfer of style and appearance, rather than on shape translation. The latter task is challenging, due to its intricate non-local nature, which calls for additional supervision. We mitigate this by descending the deep layers of pre-trained network, where the deep features contain more semantics, and applying the translation between these deep features. Specifically, we leverage VGG, which is a classification network, pre-trained on ImageNet with large-scale semantic supervision. Our translation is performed in a cascaded, deep-to- thep-to-shallow fashion, along the deep feature hierarchy: we first translate between the deepest layers that encode the higher-level semantic content of the image, proceeding to translate the shallower layers, conditioned on the deeper ones. We show that our method is able to translate between different domains, which exhibit significantly different shapes. We evaluate method both qualitatively and quantitatively and compare it to state-of oo-thh--rr image-to toaggett--imaan -mage translation methods. Our code and trained models will be made available.
INTRODUCTION
In recent years, neural networks have significantly advanced generative image modeling. Following the emergence of Generative Adversarial Networks (GANs) (Goodfellow et [2014). mmage-to-image translation methods have dramatically progressed, revolutionizing applications such as inpainting (Yu)
et al. 2018), super resolution (Wang et al. 2018), domain adaptation (Hoffman et al. 2017). and more. particular, there have been intriguing advances in the setting of unpaired image-to-image translation through the use of cycle-consistency (Yi et al., 2017: Zhu et al. (017) well as other approaches (Benaim & Wolf, 2017 Huang et al. 20188] Lee et al. 2018 Ma et al. 2018). However, most existing methods acknow ledge the difficulty in translating shapes from one domain to another, as this might entail highly non- trivial geometric deformations. Consider, for example, translating between elephants and giraffes, where would expect the neck of an elephant to be extended, while the elephant's 's head should shrink. The challenge is compounded by the fact that, even within the same domain, images might exhibit extreme variations in object shape and pose, partial occlusions, and contain multiple instances of the object of interest. One might even argue that this translation task ill- ill-posed begin with, and at the very least, requires high-level semantics be accounted for.
Nonetheless, several works do address shape deformation in the context of image-to-image translation by requiring supervision in the form of foreground mask (Liang et al.) 2018 Mo al. 2018). In contrast, GANimorph (Gokaslan et al. (018)) and the recently proposed TransGaGa (Wu et al, 2019) show remarkable translation without requiring additional supervision for several datasets, However, these techniques excel in controlled setting only, where the images are controlled, and the foreground separation rather simple.
In this paper, we address the problem of unpaired image-to-imant translation, without requiring foreground masks, between two different domains, where the objects of interest share some semantic similarity (e.g., four-legged mammals), whose shapes and appearances may, nevertheless, be drasti- cally different. Our key idea to accomplish the translation task by learning to translate between deep feature maps. Rather than learning extract the relevant higher-level semantic information for the specific pair of domains at hand, we leverage deep features extracted by a network pre-trained for image classification, thereby benefiting from its large-scale fully supervised training.

---

Under review as a conference paper at ICLR 2020
Figure 1: Given an image from domain A (zebras), we extract its deep features using a network pre-trained for classification, specifically VGG-19 19 pre-trained on ImageNet, and translate them into deep features of domain B (giraffes). We first translate high level semantics encoded in conv_ __v____1 of the zebra to those of a giraffe, represented by the inner images. Then, we use a cascade of deep-to- shallow adversarially trained translators, one for each deep feature layer, to translate shallower layers, iee conv_4 4_1 (middle sized zebra and giraffe) and then conv_ 3_1 (outer images). The presented images were obtained by training feature inversion networks, from deep features to image space.
Our work is motivated by the well-known observation that neurons in the deeper layers of pre-trained classification networks represent larger receptive fields in image space, and encode higher-level semantic content /Zeiler & Fergus (0044). In other words, local activation patterns in the deeper layers may encode very different shapes size and structure. Furthermore, Aberman AAberma al. (2018) showed that semantically similar regions from different domains, e.g. dog and cat, have similar activations. That is, the encoding of cat's eye resembles that of dog's eye more than that of its tail. From the point of view of the translation task, these properties are attractive, since they suggest that it might be possible learn semantically consistent translation between activation patterns produced by images from different domains, and that the resulting (reconstructed) image would be able to change drastically, hopefully bypassing the common difficulties in image-to-image translation methods.
More specifically, we learn to translate between several layers of deep feature maps, extracted from two domains by a pre-trained classification network, namely VGG-19 (Simonyan & Zisserman 2014). The translation is carried out one layer at time in a deep-to-shallow (coarse-to-fine) cascaded manner. For each layer, we adversarially train a dedicated translator that acts in the features space of that layer. The deepest layer translator effectively learns to translate between semantically similar global structures, such as body shape or head position, as demonstrated by the middle pair of images in Figure [ The translator of each shallower layer is conditioned the translation result of the previous layer, and learns to add fine scale and appearance details, such as texture. At every layer, in order visualize the generated deep features, use a network pre-trained for inverting the deep features of VGG-19, following the method of Dosovitskiy & Brox (2016). The images shown in Figure were generated in this manner.
Our conceptual novelty, can be viewed as transfer learning for image translation as we are translating high level semanties, encoded in the deeper layers of a pre-trained classification network, a.k.a deep features. This is in contrast to existing methods Gokaslan et al. (2018): Wu et al. (2019) which learn to translate the images directly. We compare our method with several state-of state-of-the-art image translation methods. To demonstrate the effectiieness of our approach, we present results for several challenging pairs of domains that exhibit drastically different shapes and appearances, but share some high-level semantics. Our translations are semantically consistent, typically preserving pose and number of instances of objects of interest, and reproducing their partial occlusion or cropping, as may be seen in Figure
RELATED WORK
Zhu et al. (017) (simultaneously with Yi al. (2017) and Kim al. (007)) have presented remarkable unpaired image-to-image translations, using framework known as CycleGAN. The key idea is that the ill-posed conditional generative process can be regularized by a cycle-consistency constraint, which enforces the translation to perform bijective mapping. The cycle constraint has become a popular regularization technique for unpaired image-to-image translation. For example, the UNIT framework (Liu et al.) (217)) assumes a shared latent space between the domains and enforces the cycle constraint in the shared latent space. Several works were developed to extend the one-to-one
mapping mony-to-many mapping (Ma et al. 2018 Huang al. 2018
Lee et al. 2018 Almahairi
2

---

Under review as a conference paper at ICLR 2020
et al.
2018). These methods decompose the encoding space to shared latent space, representing the domain invariant content space, and domain specific style space. Therefore, many translations can be achieved from single content code by changing the style code of the input image.
Many translation methods share the inability to translate high-level semantics, including different shape geometry. This type of translation is usually necessary in the case of transfiguration, where one aims to transform a specific type of objects without changing the background. Both Lee et al. (2018) and Mejjati et al. (2018) learn an attention map and apply translation only on the the foreground object. However, both methods only improve translations that do not deform shapes. Gokaslan et al. (2018) succeed in preforming several shape-deforming translations by several modifications to the CycleGAN framework, including using dilated convolutions in the discriminator. However, they have not demonstrated strong shape deformations, such as zebras to elephants or giraffes, as we show in Section
4
Liang et al. (2018) and Mo et al. (2018) assume some kind of segmentation is given, and use this segmentation to guide shape deformation translation. However, such segmentation is hard to achieve. In a recent work, Wu et al. (2019) disentangle the input images to geometry and appearance spaces, relying on high intra-consistency, and learn to translate each of the two domains separately. However, the variation of geometry and appearance of in-the-wild images is too large to be disentangled successfully
Contrary to the above works, our work leverages a pre-trained network and the tramslation is applicd directly on deep feature maps, thus being guided by high-level semantics. Several image-to-image methods, such as Xu et al. : Di et al. (2018); Ignatov et al. (0118), also incorporate such pre- trained networks, though usually, only as perceptual loss, constraining the translated image remain semantically close to the input image. Differently, /Sungatullina et al. (2018) incorporate pre-trained VGG features into the discriminator architecture, to assist in the discrimination phase. Wu al. (2018) use VGG-19 as a fixed encoder in the translation, where only the decoder is learned. Upehurch al. 20177) present the only method, to our knowledge, that actually translates deep features between two domains. However, the translation is not learned, but defined by simply interpolating between the deep features, which restricts the scope of method highly aligned domains. For completeness, we also mention that Yin et al. (2019) train an autoencoder to embed point clouds, and perform translation in the learned embedding. In contrast, we utilize semantics to preform the translation in the much more difficult scenario of images.
Our work shares some similarities with the early work of Huang et al. (2017), which suggests using a generative adversarial model (Goodfellow et al. (014)) in course-to-fing manner with respect to pre-trained encoder. The generation process begins from the deepest features and then recursively synthesizes shallower layers conditioned on the deeper layer, until generating the final image. This method was only applied on small encoders and low resolution images and was not explored for very deep and semantic encoding neural networks such as VGG-19 (Simonyan & Zisserman 2014).
Deep image analogies (Liao et al., 2017) transfer visual attributes between semantically similar images, by feed-forwarding them through a pre-trained network. Their work does not train generative model; nonetheless, they create new deep features by fusing content features from one image with style features of another. Similarly, Abberman al. (2018) synthesize hybrid images from two aligned images by selecting the dominant deep features response.
METHOD
Our general setting is similar to that of previous unpaired image-to-image translation methods. Given images from two domains, A and B, our goal is to learn to translate between them. However, unlike other image-to-image translation methods. we perform the translation on the deep features extracted by a pre-trained classification network, specifically VGG-19 (Simonyan & Zisserman 2014).
The translation is carried out in deep-to-shallow (coarse-to-fine) manner, using a cascade of pairs of translators, one pair per layer. The entire architecture used to train the translators is shown schematically in Figure 2 while Figure iillsstrats the pest-time translation (inference) process. Once
Unnorrunately, at the time of this submission the authors of Wu et al. 2019) were unable to release their code or train their network the datasets presented in our paper.
3

---

Under review as a conference paper at ICLR 2020
VGG
VGG
DA
Layer
G
Lager 5
D7
D4
Layer 5
As
B2
G
Figure 2: Translation architecture. We translate between domains A and starting from the deepest feature maps A5 and Bs. which encode the highest level semantic content of the images. Translation proceeds from deeper to shallower feature maps until reaching the image itself. The feature maps are extracted by feed forwarding every image through the pre-trained VGG-19 network and sampling five of its layers. Every layer's 's translation is learned individually, conditioned on the translation result of the next deeper layer (except the deepest layer, whose translation is unconditional).
@1
42
0.1
025
G
52
5
Figure 3: Translation of the top-to image time. The input image (left) is fed forward through VGG-19 layers, as indicated by the right purple arrows. Then, starting from the deepest layer as we translate each layer. The final result is obtained from the shallowest layer using feature inversion.
the deepest feature map has been translated, we translate the next (shallower and less semantic feature map), conditioned on the translated deeper layer. In this manner, the translation of the shallower map preserves the general structure of the translated deeper one, but adds finer details, which are encoded the deeper feature maps. We repeat this procedure until the original image level is reached. Below we describe the training and the inference processes in more detail.
Pre-processing: We extract high-level semantic features from input images from both domains, A and B, by ford-forwarding the images through the pre-trained VGG-19 (Simonyan Zisserman 2014) network. Next, we sample five of the resulting deep feature maps, specifically conv_i_ (i 1,2,3, 4, 5), where each map has progressively coarser spatial resolution, but a larger number channels. We denote the i-th sampled feature map for image E A as Since propagation through the pre-trained VGG-19 network may yield features any range, while the range of the synthesized features is usually known, to case the translation,
first normalize each channel, of every layer i, by calculating its mean and standard deviation across the domain and then we clamp the normalized feature values the range of []. While the clamping is potentially harmful irreversible operation, we did not observe any adverse effect on the results. We use A, (B;) to denote set of all normalized deep feature maps of level i, extracted from images in domain (B).
Inference: We perform the translation in coarse-to-fine fashion. Thus, the translator from domain to B, actually consists of sequence of translators {G , ....... G 1. where each translator is responsible for translating the i-th feature map layer a,, from A, B, conditioned on the previously translated deeper layer bi+1 (except for the deepest layer translator G5 which is unconditioned).
4

---

Under review as a conference paper at ICLR 2020
Finally, G% uses feature inversion to convert bj obtain the translated image. The translators G from domain B; to A; are defined symmetrically. The entire inference pipeline is shown in Figure
visualize the output of the various translators by pre- pre-training a deep feature inversion network (per domain), for each layer i 1,... 5, following Dosovitskiy & Brox (016). The network aims reconstruct the original image given the feature map of a specific layer, regularized by adversarial loss so that the reconstructed image would lie in the manifold of natural images. For more details we refer the reader to Dosovitskiy & Brox (2016). The specific settings used in our implementation are elaborated in Appendix A.1
Feature inversion: In all the results show, e.g., Figure
Deepest layer translation: We begin by translating the deepest feature maps, encoding the highest- level semantic features, i.e., As and Bs, hence, our problem is reduced to translating high-dimensional tensors. Our solution builds upon the commonly used CycleGAN framework (Zhu et al., (017). Specifically, use the three losses proposed in Zhu al. (2017). First, in order to generate deep features in the appropriate domain, we utilize an adversarial domain loss Ladv. We simultaneously train two translators G -², G which try to fool domain-specific discriminators, D D (for domains As. Bs, respectively). However, differently from Zhhu et al.] (2017) and other image translation methods (Huang al.) (2018) Mo et al.,, 2018),
have found LSGAN (Mao et al. (017) not to be well-suited for our task, leading to mode collapse or convergence failures. Instead, we found WGAN-GP (Gulrajani al. 2017) more effective, thus, the adversarial loss for translation from X to Y is defined as
Lado (Gy, Dr,,,,,YY, X,Y)
E (Dy (Gy (x)) Dy (V) +>gp E (IVDy (7) - -1)² (1) youPy
P~Fx
where Gy X Y is the translator, Dy is the target domain discriminator, App 10 in all our experiments, and Py is defined by uniformly sampling along straight lines between ÿ ~ G (IP (Px)) and y Py. For more details we refer the reader to Gulrajaai al. (2017).
Second, for regularizing the translation to one-to-one mapping, we add the cycle constraint,
GGy Gy, ,r,
|Gx (Gy (x:) æ=1 + E a~Px
E
||Gy (Gx (v)) srl.
(2)
g-Py
where || stands for the L norm.
Finally, in Zhu et al. (2017), have found it helpful to an identity loss, which guides the networks to preserve common high level features,
Lidty(Gx, Gy, = E x>Px
||x (x) x|ll y~Fy
E
||y (v) - -yl--
(3)
The entire loss combines these components follows
L =aadv (G D. As. Bs) Ladv (G., D3 Bs, As)
+ >eeyeeoye S 5 As, Bb) + Autry Cudty (G GS As, Bs),
(4)
where Aege and Xidty were set to 100 in all our experiments.
Coarse to fine conditional translation: Consider two successive layers, a; E A; and 0:+1 A
where the latter has already been translated, yielding 6x41 as the translation outcome (see Figure 3 We next perform the translation of the layer a; to yield b,, using the translator G conditioned on 7++1, Note that Gjp is effectively a function of all the previously translated layers.
The architecture of G is schematically shown in Figure Since shallower layers encode less of the semantic content of the image, it is more difficult to learn how they should be deformed, and thus they are used to transfer "style", while the "content" comes from the already translated deeper layer. Inspired by Huang et (2018), Belongic 2017) component, whose parameters are learned from the current layer. Thus, several layers of G are normalized according to the AdaIN component. G",, which designed symmetrically, is learned simultaneously with G'g, shown in Fig a).
add an adaptive instance normalization (AdalN) (Huang &
The loss for training these shallower translators is roughly the same as that used for training the deepest translation: it consists of adversarial domain loss, cycle constraint loss, and identity loss.
5

---

Under review as a conference paper at ICLR 2020
D
Db
a;
AdaIN
A
A1+1
B
+
14
b
b
Figure 4: Translation of layer is conditioned on the previously translated layer + The two translators G¹ and G are trained simultaneously (see left figure), where i + 1,...,5 5 translators are fixed. On the right we show the schematic architecture of G which has two inputs: a, E A, and b O2 is fed-forward through several layers yield AdaIN parameters which control the generation of 65. Since by has twice the spatial size of b,+1, add upsampling layer marked by 1.
MSCOCO
Zebra I Elephant
MSCOCO
Zebra 1 Elephant
MSCOCO
Giraffe Zebra.
+
MSCOCO
Giraffe T Zebra
Kaggle
Dog J Cat
Kaggle
Dog
Cat
Figure 5: Examples of challenging translation results, featuring significant shape deformations.
While we formalize the adversarial loss unconditionally, similarly to equation
D the cyclic loss is now conditioned: || G² (G's ai bu-1) a.t1) a. + |G' (G*s (b,, ā++1) bu+1) 6./]. and the same is true for the identity loss ||G' (a;, a++1) - a. +I|G% (b,, b++1) - bell-
We train the pairs of translators one layer at time, starting from G3 and G More details regarding the implementation and the training of the translators are included in Appendix A.I
EXPERIMENTS
We evaluate our methods on several publicly available datasets: (1) Cat <>> dog faces (Lee et al. 2018), which contains 871 cats images and 1364 dogs images and does not require high shape deformation; (2) Kaggle Cat <> dog (Elson et al. 2007) dataset with over 12, 500 images in each domain (the images may contain part of the object or several instances); (3) Challenging MSCOCO dataset (Lin) et al.) 2014), specifically. zebra < elephant and zebra giraffe (the number of images es-h category is reportetd (overall there are 1917 zebras, 2547 giraffes and 2144 elephants). We note that previous method has used MSCOCO, without supervision in the form of segmentation.

---

Under review as a conference paper at ICLR 2020
Original
lidty &eyc
adv
adv
cyc
adv & idiy
aiv kycyidty
Original
Level
Level
Level
Level 4
Figure 6: Translation of the 5th (deepest) layer with different loss combinations. Using all three components yields the best result.
Figure 7: Translation of different VGG layers, separately. Low level semantics translation fails to deform the geometry of the object.
Layer
Layer 4
Layer
Cascaded (ours)
Cat <> Dog
126.93/127.53 167.62/184.37
181.90/164.42
178.13/91.77
67.58/46.02
Zebra + Giraffe
103.41/53.36
112.43 /68.62 62.62222
67.41/39.38 68.45/47.86
Zebra <> Elephant
101.26/76
105.58/57.34
166.32/113228
Table 1:
FID score comparison of different layer translation. Each translation was trained indepen- dently. We compare the FID scores on three datasets, measured both directions per dataset. The two directions appear side-by-side, - at each cell.
Our deepest translators, i.c. G GB consist of encoder-decoder structure with several strided convolutional layers followed by symmetric transpose convolutional layers. We use group normal- ization (Wu & He (018) and ReLU activation function (except the last layer, which is tanh). The conditional generators, consist of learned AdaIN layer, achieved by several strided convolutional layers followed by fully connected layers. The content generator has also several convolutional layers and one single transpose convolutional layer which doubles the spatial resolution (Figure 4 right)). In practice only train GO, G G and apply feature inversion directly the output of the latter, with negligible degradation. For the exact layer's specifics we refer the reader to Appendix | and and to (soon be published) code. We each layer for 400 epochs with fixed learning rate of 0.0001 and Adam optimizer (Kingma & Ba 2014). On single RTX 2080 reaching the final image takes around 2.5 days, including the final inversion network training.
Several translation examples are presented in Figure 5 Our translation is able to achieve high shape deformation. Note that our translations are semantically consistent, in the sense that they preserve the pose of the object of interest, as well the number of instances is mostly preserved. Furthermore, partial occlusions of such objects, or their cropping by the image boundaries are correctly reproduced. See for example, the translations of the pairs of animals in columns 5-6. More results are provided in Appendix
A.2 A.2
4.1 ABLATION STUDY
We analyze two main elements of our method.
Loss components First, we validate the use of CycleGAN loss components. As shown in Figure 6 we translate the 5th (deepest) layer with and without cycle, identity and adversarial losses. The best approach is achieved by using of the losses, which balance each other.
VGG layers Ia-adiition In Figure we compare translation of different VGG-19 layers. Evidently, shallower layers introduce spatial constraints, thus, limiting the translation in the sense of shape's changes. The shallowest layer can hardly change the shape of the input image, which may explain the failure of traditional image translation methods. In Table [ we use the common FID score (Szegedy
al. 2015 to show that the cascaded translation achieves better translation compared to individual layer translation. Additional results are shown in Appendix A..4
Type of pre-trained network While method is conceptually agnostic to the type of features extraction network, we still require the extracted features to represent high-level semantic. Therefore, VGG-19 layers, which the most common features used for image generation tasks (Aberman et al.
7

---

Under review as a conference paper at ICLR 2020
Original
Cycle
GAN
MUNIT
DRIL
CHA
Ours
Figure 8: Comparison to other image-to-image translation methods. The unpaired translations, from left to right, are zebra ++ giraffe, elephant zebra and dog ++ cat, where every translation has four examples, two in each direction. While previous translation methods struggle to deform the geometry of the source images, the poses of the subjects and the overall composition the image.
method is able to preform drastic geometric deformation, while preserving
Original
Alexnet
VGG Fine-tuned
VGG
Original
A Alexnet
VGG Fine-tuned
VGG
Figure 9: Translation with different pre-trained networks. All the networks were pre- pre-trained on ImageNet. VGG fine-tuned was further trained to classify between zebra and giraffe. Evidently, using fine-tuned VGG-19 does not assist the translation process. In addition, translation of AlexNet features doesn't preserve the shape of the input image.
(2018 Dosovitskiy Brox 20166; Gatys et al. (2016) Liao al. (011)) was chosen. Nonetheless, we experiment with a fine-tuned version of VGG-19 and different network architecture as shown We fine-tuned VGG classify between zebras and giraffes and trained our translation networks using the resulting features. As can be seen, the translation results are inferior to the results achieved by the standard VGG-19. This maybe attributed to VGG-19 fixating on the unique differences between zebras and giraffes, unrelated to the translation, such as background. For more about the extracted features, we refer the reader to Appendix A.1.3 In addition, in Figure 9 we examine different network, AlexNet, also pretrained on ImageNet. We have observed that the deepest image translation was not able to generate valid shape of zebra or giraffe. AlexNet is stride of 4. Therefore, the resulting
Figure g
composed of 5 convolutional layers, where the first features have less spatial encoding, especially at the deeper layers, which may explain the difficulty to invert and translate these features.
8

---

Under review as a conference paper at ICLR 2020
-/
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Cat 4+ Dug
125.75/94.27
159.57/108.55
153.94/139.11 17
139.17/134.11
67.58/46.02
Zebra Giraffe
55.65/58.92
238.06660..7
59.75/54.06
98.25/120.05
67.41/39.35
Zebra ++ Elephant
86.55/68.44
109.56/80.1
78.01/56.39
99.98/89.74
68.45/47.86
Table 2: FII score comparison. We compare FII) scores on three datasets, measured for both
translation directions per dataset. The two directions appear side-by-side, - at each cell.
CyclGAN
MUNIT
DRIT
GANimorph
Ous
7ebra
Elephant
Zehra
>
Giraffe
Figure 10: Comparison of the deepest latent spaces (5th layer), projected using t-SNE. The latent space of the source domain in blue, and the target domain is red. The distribution of the translation results (in cyan) is most similar to that of the target domain when using our method.
4.2 COMPARISON OTHER METHODS
We compare our result with leading image translation methods. i.e. CycleGAN (Zhu et al. 2017). MUNIT (Huang al. 2018), DRIT (Lee al. 2018) and GANimorph (Gokaslan et al. 2018).
Quantitative comparison: In order to perform quantitative comparison, use the common FID score (Szegedy et al. (2015), as reported in Table Our method achieved the best FID score on five out of the six cross-domain translations for which this score measured.
Qualitative comparison: In Figure 8 we show several challenging translation examples. While traditional image translation methods struggle preform translations with such drastic shape defor- mation, our method is able to do so thanks to its use of the pre-trained VGG-19 network.
The success of our method can also be explained and visualized by examining the translated deep features. We feed forward every image, original and translated, through the entire VGG network, extracting the last fully-connected layer (before the classification layer). size 4096) to 2D, using t-SNE, as shown in Figure 9. It may be seen that the distribution of the translated vectors (in cyan) is closest to that of the target domain (in red) when using our method.
project this vector (of
Limitations Our method achieves translations with significant shape deformation in many previ- ously unattainable scenarios, yet, a few limitations remain. First, the background of the object is not preserved, as the background encoded in the deep features along with the semantic parts. Also, in some cases the translated deep features may be missing small instances or parts of the object. This may be attributed the fact that VGG-19 is generally not invertible and was trained classify finite set of classes. In addition, since we translate deep features, small errors in the deep translation may be amplified to large errors the image, while for image-to-image translation method that operate on the image directly, small translation errors would typically be more local. Please note that, similarly to CycleGAN and GANimorph, our translation is deterministic.
CONCLUSIONS
Translating between image domains that differ not only changes in appearance, but also exhibit significant geometric deformations, is highly challenging task. We have presented novel image-to- image translation scheme that operates directly pre-trained deep features, where local activation

---

Under review as a conference paper at ICLR 2020
patterns provide rich semantic encoding of large image regions. Thus, translating between such patterns is capable of generating significant, yet semantically consistent, shape deformations. In sense, this solution may be thought of as transfer learning, since we make use of features that were trained for classification task for an unpaired translation task. In the future, we would like to continue exploring the applications of powerful pre-trained deep features for other challenging tasks, possibly in different domains, such as videos, sketches or 3D shapes.
REFERENCES
Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-On Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics (TOG), 37(4): 69, 2018.
Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Aug- mented CycleGAN: Learning many-to-many mappings from unpaired data. arXiv preprint arXiv:1802.10133 10151, 2018.
Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. In Advances in neural information processing systems, pp. 752-762, 2017.
Xing Di, Vishwanath Sindagi, and Vishal M Patel. GP-GAN: Gender preserving GAN for synthesizing faces from landmarks. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1079-1084. IEEE, 2018
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in neural information processing systems, pp. 658-666, 2016.
Jeremy Elson, John JD Douceur, Jon Howell, and Jared Saul. Asirra: a CAPTCHA that exploits interest-aligned manual image categorization. In ACM Conference on Computer and Communica- tions Security, 2007.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414-2423, 2016.
Aaron Gokaslan, Vivek Ramanujan. Daniel Ritchie, Kwang In Kim, and James Tompkin. Improving shape deformation in unsupervised image-to-image translation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 649-665, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. Advances in Neural Information Processing Systems, pp. 5767-5777, 2017. 2017.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:/71
2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017.
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie. Stacked generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5077-5086, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- -t-iimage translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 182-189, 2018.
10

---

Under review as a conference paper at ICLR 2020
Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. Wespe: weakly supervised photo enhancer for digital cameras. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 691-700, 2018.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning dis- cover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1857-1865. JMLR. org. 2017.
Diederik Kingma and Jimmy Ba. abs/1412.6980, 2014.
Adam: A method for stochastic optimization. CoRR,
Hsin-Yre Ying Lee, Hung-Yt Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuar Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference Computer Vision (ECCV), pp. 35-51, 2018.
Xiaodan Liang, Hao Zhang, Liang Lin, and Eric Xing. Generative semantic manipulation with mask-contrasting gan. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 558-573, 2018.
Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. arXiv preprint arXiv:1705.00 1705.0008. 2017.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised soage-to-image translation networks. In Advances in Neural Information Processing Systems, pp. 700-708, 2017.
Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, and Luc Van Gool. Exemplar guided unsupervised image-to-image translation. arXiv preprint 1rXiv:1805.11144 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang. and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2017.
Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. Unsupervised attention-guided image-to-image translation. In Advances in Neural Information Processing Systems, pp. 3693-3703, 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instagan: Instance-aware image-to-image translation. arXiv preprint arXiv:1812. 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for targe-scale image recognition. arXiv preprint arXivv1409.1556 2014.
Diana Sungatullina, Egor Zakharov, Dmitry Ulyanov, and Victor Lempitsky. Image manipulation with perceptual discriminators. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 579-595, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Paul Upchurch, Jacob Gardner, Gcoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064-7073, 7077,00711 2017.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pp. 0-0, 2018.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. TransGaGa: Geometry- unsupervised image-to-image translation. arXiv preprint arXiv: /xXiv:990400957/ 2019.
11

---

Under review as a conference paper at ICLR 2020
Xuehui Wu, Jie Shao, Lianli Gao, and Heng Tao Shen. Unpaired image-to to-image translation from shared deep space. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp- 2127-2131. IEEE, 2018.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Jingjin Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. Unpaired celpiimmeen-tt--setinne translation: A cycled reinforcement learning approach. arXiv preprint arXiv:/805.0518/ 2018.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proc. IEEE ICCV, pp. 2849-28495222 2017.
Kangxue Yin, Zhigin Chen, Hui Huang, Daniel Cohen- Or, and Hao Zhang. Logan: Unpaired shape transform in latent overcomplete space. arXiv preprint 10X10: 2019.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas $ Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv: arXiv:/806. 085660358,, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 118-833. Springer, 2014.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc, IEEE ICCV, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 TRAINING DETAILS
A.1.1 Hyper parameters
In all our experiments, unless stated otherwise, we use Adam optimizer [16] with 31 = 0.5, 8g 0.999. The learning rate was set to 0.0001 and the batch size to 10. During training, random crop and image mirroring is applied. Our training methodology follows WGAN-GP (Gulrajani et al. 2017). thus for generator update we update the discriminator four times.
A.1.2 NETWORK ARCHITECTURE
Feature Inversion Our implementation is similar to Dosovitskiy & Brox (016). We train an individual feature inversion network for VGG layer, where each layer has different channels (512, 512, 256, 128, 64). All layers utilize Leaky ReLU nonlinearity (0.2) and employ no normalization. The last layer utilizes Tanh. All inversion networks, first apply three non-strided convolutional layers, with N input number of channels, equal to the number of channels each deep layer has. Next, several transpose convolutional layers are applied, each doubles the resolution of the image and decreases the channel resolution (by factor of 2) until the image resolution 224 is achieved (thus, different amount of Conv Transpoee layers per layer). The final layer is a non-strided convolutional layer followed by Tanh layer. Together they project the features back to the original image dimensions and range (number of output channels is 3). For the discriminator we have used Patch GAN discriminator, with four strided convolutional layers, each utilizes batch normalization (except the first one) and Leaky ReLU. For the adversarial metric, only here, we have used LS-GAN. Here we also set the batch size 25.
Deepest layer translation The input to the deepest translation network is conv_5_1. thus, the input size 14 x x 512 (recall the input image size is 224 x 224). The identity and cycle losses are multiplied by Xisty Aeye 100. The architecture is reported in Table The networks is relatively small and achieve good results in a few hours on a single GPU (RTX2080), We use WGAN-GP optimization method, updating the generator once for every four discriminator updates.
Name Input ch.
Output ch.
Kernel s7.
Stride
GN
cony
512
512
3
conv
512
256
3
yes
relu
conv
256
512
3
2
yes
relu
convT
512
256
3
relu
convT
256
256
3
yes
reiu
conv
256
512
3
yes
relu
conv
512
512
3
tanh
Table Deepest layer translation architecture.
Coarse to fine conditional translation The coarse-to-fine generator, for generating level i, has two inputs: the current source VGG level and the previous translated VGG features (i + 1). An AdaIN component, acts the current deep features and normalizes several layers in the translator
13

---

Under review as a conference paper at ICLR 2020
itself. We report the AdalN component structure for generating layer four in Table The architecture can be extended easily to other layers. The core components of the translator, which takes as input the previous translated layer, are reported in T
Name
Input ch.
Output ch.
Kernel sz.
Stride
GN
conv
512
512
Irelu
conv
512
512
no
Irelu
conv
512
512
no
Irelu
linear
x 512
1000
no
Irelu
linear
1000
no
Table 4: AdaIN component for the second deepest layer. The output x is equal to the number of parameters the AdaIN normalizes. AdaIN for different VGG layer's translation are defined similarly, where we simply add more conv layer for each shallower VGG layer.
Name Input
Output ch.
Kernel sz.
Stride
AdaIN
conv
2
yes
Irelu
conv 2
3
ycs
Irelu
convT
Irelu x
/
yes
conv x/2
2/2
3
no
tanh
Table 5: Coarse to fine translator. VGG layer.
The input number of channels, v, varies according the current
A.1.3 DIFFERENT NETWORKS
VGG-19 fine-tuned We fine-tuned VGG-19 by fixing all layers but the fully connected layers and conv___i (i = 1, 2, 3, 4). We replaced the final fully connected layer with new fully connected layer of size 4096 x 2 corresponding the two domains. We trained the classifier for 50 epochs, with batch size of 25 and learning rate of 0.01.
AlexNet We extract each of the 5 convolution of AlexNet as different layers. Each layer was normalized, in similar manner was described for VGG-19.
14

---

Under review as a conference paper at ICLR 2020
A.2 MORE COMPARISON RESULTS
In this section we show more results, not presented in the paper, for zebrae-giraffe, zebra<>elephant and cat+>dog translations.
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 11: Qualitative comparisons. MSCOCO zebra to giraffe.
15

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 12: Qualitative comparisons. MSCOCO zebra to giraffe.
16

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 13: Qualitative comparisons. MSCOCO giraffe to zebra.
17

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 14: Qualitative comparisons. MSCOCO giraffe zebra.
18

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 15: Qualitative comparisons. zebra to elephant.
19

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 16: Qualitative comparisons. zebra elephant.
20

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
KICG
Figure 17: Qualitative comparisons. MSCOCO elephant to zebra.
21

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Ty Souce
Figure 18: Qualitative comparisons. MSCOCO elephant to zebra.
22

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Jon
Figure 19: Qualitative comparisons. Kaggle cat to dog.
23

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 20: Qualitative comparisons. Kaggle cat to dog.
24

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 21: Qualitative comparisons. Kaggle dog to cat.
25

---

Under review as a conference paper at ICLR 2020
Original
Cycle GAN
MUNIT
DRIT
GANimorph
Ours
Figure 22: Qualitative comparisons. Kaggle dog cat.
26

---

Under review as a conference paper at ICLR 2020
A.3
NON-SHAPE DEFORMATION TRANSLATION
Our method is also suited to none-shape deformation tasks, as in the case of dataset(1 Lee et al.
2018
Original
Ours
Original
Ours
Original
Ours
shutterst
Figure 23: Translation results from cats to dogs (faces).
27

---

Under review as a conference paper at ICLR 2020
Original
Ours
Original
Ours
Original
Ours
Figure 24: Translation results from dogs to cats (faces).
28

---

Under review as a conference paper at ICLR 2020
A.4 COARSE TO FINE TRANSLATION
We here present the translation of each layer. The translation of each shallower layer is conditioned the translation result of the previous layer, and learns to add fine scale and appearance, such as texture. At every layer, in order to visualize the generated deep features, we use a network pre-trained for inverting the deep features of VGG-19, following the method Dosoviiitk & Brox (2016).
Figure 25: Coarse to fine translation of zebra to giraffe. Two different examples are shown in each row. The original image (left) is translated by the deepest translator (second left) and then in coarse to fine manner, shallower layers are translated (second right and most right).
29

---

Under review as a conference paper at ICLR 2020
Figure 26: Coarse to fine translation of giraffe to zebra. Two different examples are shown in each row. The original image (left) is translated by the deepest translator (second left) and then in coarse to fine manner, shallower layers are translated (second right and most right).
30

---

Under review as a conference paper at ICLR 2020
A.5 NEAREST NEIGHBOR COMPARISON
In this section we show side by side, source images, our translation and the three nearest neighbors in the target domain. We use the LPIPS metric, presented in Zhang et al. (2018). This metrie is based on Le distance of deep features extracted from pre-trained network. In our case we use the default settings proposed by Zhang et (i.e. alex net). As we show, the closest image in the target dataset vary in pose, scale and content (i.e. different parts of the objects).
Original
NN 1
NN 2
NN 3
Ours
Figure 27: Nearest neighbor comparison result for zebra to giraffe translation. The NNs were found by exhaustive search on all the giraffe dataset using perceptual metric (LPIPS). The closest giraffe to the source zebra vary in scale, position and content
31

---

Under review as a conference paper at ICLR 2020
Original
NN 1
NN 2
NN 3
Ours
Figure 28: Nearest neighhor comparison result for giraffe zebra translation. The NNs were found by exhaustive search on all the giraffe dataset using perceptual metric (LPIPS). The closest zebra the source giraffe vary in scale, position and content.
32


Reviews:
[{"review_id": "rJljMjEctH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method for image-to-image translation. The problem of most existing methods is that they are good in translating style  (e.g from photo to Van Gogh) but do not allow for significant changes in shape (e.g. from zebra to giraffe). The authors address this by performing the translation in a cascaded fashion starting from a semantic (deep) level (fifth layer of VGG). The underlying idea is that translating at this more semantic level puts less spatial constraints on the final resulting images (making translations from zebra to giraffe possible). After the fifth layer is translated other layers are translated conditioned on the previous translation results. \n\nThe method is compared to other translation methods including DRIT, MUNIT, GANimorph. Both visually and quantitatively as measured by FID. The results of the proposed method are superior. No comparison to TransGaGa is provided (but I could not find code for this method).\n\nMy recommendation is borderline accept. The proposed method is simple. The experimental results are limited but show both visually and quantitatively superior results. Especially FID scores are much better. I think the paper could be improved in explaining the conceptual novelty of the paper (especially with respect to GANimorph and TransGaGa).\n\n1. I like the idea of applying the cycle consistency to the deeper layers rather than at the pixel level. Are there other methods which  do this ? It could be highlighted more as part of the contribution.\n2. An ablation study should be added (in FID scores). I would like to see the necessity of the cascade (which is in the title) confirmed: results for only translating a single layer (3,4, or 5)  should be compared to translating 3,4,5 together as in proposed method. \n3. Would it be possible to not use pretrained feature from VGG-19 ? This might also be a limitation. In principle, I guess you could train everything end-to-end, or is this impractical because of the feature inversion. \n4.The authors could add some text on the lack of diversity for the translations in the limitation section. I understand there is no diversity and the translation is deterministic. \n \nIn general, I think the paper clearly explains what it does, and it also shows cases that it performs better than state-of-the-art. The paper could be much improved in its analysis of the reasons for its better performance, analyzing key aspects of its design like cycle GAN on features, pretrained VGG features and the use of cascaded generation of the final image. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skl0SYi6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new cascaded image-to-image translation method to address the I2I tasks where the domains have exhibit significantly different shapes. The proposed method train cycle GAN on different levels of feature extracted by pre-trained VGG and combine the futures with the AdaIN layer to keep the correct shape from the deep features. \n\nPros:\n1. The proposed method seems to work well on different shape I2I datasets without using semantic masks compared to previous works.\n2. The idea of cascaded translators sounds simple and reasonable which can probably benefit other related tasks. The way of applying AdaIn to combine features of different levels is also a nice trick to keep the correct shape from deep features.\n3. The paper writing is OK, but some explanation and organization should be improved as mention in cons.\n\nCons:\n1. Some figures are hard to understand without looking at the text. For example, in Figure 1, the caption does not explain the figure well. What does each image, the order, and the different sizes mean?  As to Figure 3, the words “top left image”, “right purple arrows” are a bit confusing.\n2. The “Coarse to fine conditional translation” section describes the conditional translation in the shallow layers. I suggest mentioning it in previous sections for easy understanding.\n3. As to the t-SNE visualization in Figure 9, different methods seem to use different N-D to 2-D mapping functions. This may lead to an unfair comparison.\n\nSuggestions:\n1. The authors use the pre-trained classification network VGG for feature extraction and then train dedicated translators based on these features. I wonder if the authors also tried finetuning VGG on the two domains or training an auto-encoder on the two domains. The domain-specific knowledge may help to improve the results and alleviate the limitations presented in the paper, e.g. background of the object is not preserved, missing small instances or parts of the object due to invertible VGG-19.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgbvGF-cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Summarize what the paper claims to do/contribute.\nThis paper claims to extend existing image translation works, like CycleGAN, to domain pairs that are not similar in shape. It is proposed to do so by using a VGG network trained on classification (I assume on Imagenet), extracting features from the two domains and learn 5 CycleGANs to translate for each level of the feature hierarchy. At each level of the hierarchy the translation from the previous level is used to condition the translation for the current level. During inference, the final image translation is done by \"feature inversion\" (a technique proposed in Dosovitsikiy and Brox, 2016) from the final feature layer. The technique is show on example from a number of pairs of domains like Zebra-to-Elephant (and back), Giraffe-to-Zebra (and back), Dog-to-Cat (and back) and is compared with a number of baselines qualitatively and quantitatively with the FID score.  \n\n* Clearly state your decision (accept or reject) with one or two key reasons for this choice.\nWeak Reject.\n\nMajor reasons:\n- The problem itself, as stated in the introduction, seems ill-posed to me. One of the struggles I had while looking through the results was to understand what the images should be looking like. ie What should a zebra translated to a giraffe look like? The motivation for such a problem is also not immediately clear either. \n- Most of the resulting images do not seem \"translated\" to me. As stated in the paper (end of p.2) \"one aims to transform a specific type of object without changing the background.\" As one can see in eg Fig. 1 the resulting translations are completely different images with the foreground object of the new domain in roughly similar poses. The background in most cases does not persist. What I suspect is actually happening here is that the high-level semantics from the first image are used as some sort of noise to generate new images from the new domain. One question I had, for example: could we be getting similar results if we used the VGG bottleneck as the noise vector in an InfoGAN? Since the VGG network is pretrained and used in the same way in both domains, I imagine we would be seeing something very similar. (and it would be def. preferrable to tuning 10 GANs!)\n\n* Provide supporting arguments for the reasons for the decision.\nSome of the decisions made in the paper were unclear and not supported adequately. The questions (in rough order of importance) that made some of the contributions unclear to me:\n- Why wasn't a final translator used for the final image, conditioned on the final \\tilde{b}_1? \n- Is the VGG network pretrained on ImageNet? Why wasn't another task used that could be retaining more of the relevant features? eg on semantic segmentation\n- Could this be used for networks pretrained on other datasets? Presumably ImageNet has information about the animals translated in this paper. Even better, could we somehow learn these features for the domain pairs automatically somehow?\n- How meaningful is the FID score really in this case?\n- How were the 10 GANs tuned?\n\n* Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n- It is mentioned on p.4 that \"clamping is potentially a harmful irreversible operation\" but that harmful results were not observed. As I was reading that I was wondering how these results would actually look like. \n- On p. 6 it is mentioned that the number of images for 2 categories are reported in another paper. I think it'd take less space to actually report the number of images here.\n- On p.7 it is mentioned that the number of instances is preserved, however it should be made clear that it's is perserved in some (or most if that is what was observed) of the examples.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses image translation by extending prior models, e.g. CycleGAN, to domain pairs that have significantly different shape variations. The main technical idea is to apply the translation directly on the deep feature maps (instead of on the pixel level). 
While acknowledging that the proposed model is potentially useful, the reviewers raised several important concerns:
(1) ill-posed formulation of the problem and what is desirable, (2) using fine-tuned/pre-trained VGG features, (3) computational cost of the proposed approach, i.e. training a cascade of pairs of translators (one pair per layer). 
AC can confirm that all three reviewers have read the author responses. AC suggests, in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject