CROSS-DOMAIN CASCADED DEEP TRANSLATION
Year: 2019

Abstract:
In recent years we have witnessed tremendous progress in unpaired image-toimage translation methods, propelled by the emergence of DNNs and adversarial training strategies. However, most existing methods focus on transfer of style and appearance, rather than on shape translation. The latter task is challenging, due to its intricate non-local nature, which calls for additional supervision. We mitigate this by descending the deep layers of a pre-trained network, where the deep features contain more semantics, and applying the translation between these deep features. Specifically, we leverage VGG, which is a classification network, pre-trained on ImageNet with large-scale semantic supervision. Our translation is performed in a cascaded, deep-to-shallow, fashion, along the deep feature hierarchy: we first translate between the deepest layers that encode the higher-level semantic content of the image, proceeding to translate the shallower layers, conditioned on the deeper ones. We show that our method is able to translate between different domains, which exhibit significantly different shapes. We evaluate our method both qualitatively and quantitatively and compare it to state-of-the-art image-to-image translation methods. Our code and trained models will be made available.

text: In recent years we have witnessed tremendous progress in unpaired image-toimage translation methods, propelled by the emergence of DNNs and adversarial training strategies. However, most existing methods focus on transfer of style and appearance, rather than on shape translation. The latter task is challenging, due to its intricate non-local nature, which calls for additional supervision. We mitigate this by descending the deep layers of a pre-trained network, where the deep features contain more semantics, and applying the translation between these deep features. Specifically, we leverage VGG, which is a classification network, pre-trained on ImageNet with large-scale semantic supervision. Our translation is performed in a cascaded, deep-to-shallow, fashion, along the deep feature hierarchy: we first translate between the deepest layers that encode the higher-level semantic content of the image, proceeding to translate the shallower layers, conditioned on the deeper ones. We show that our method is able to translate between different domains, which exhibit significantly different shapes. We evaluate our method both qualitatively and quantitatively and compare it to state-of-the-art image-to-image translation methods. Our code and trained models will be made available.

heading: 1 INTRODUCTION
text: In recent years, neural networks have significantly advanced generative image modeling. Following the emergence of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), image-to-image translation methods have dramatically progressed, revolutionizing applications such as inpainting (Yu et al., 2018), super resolution (Wang et al., 2018), domain adaptation (Hoffman et al., 2017), and more. In particular, there have been intriguing advances in the setting of unpaired image-to-image translation through the use of cycle-consistency (Yi et al., 2017; Zhu et al., 2017) as well as other approaches (Benaim & Wolf, 2017; Huang et al., 2018; Lee et al., 2018; Ma et al., 2018). However, most existing methods acknowledge the difficulty in translating shapes from one domain to another, as this might entail highly non-trivial geometric deformations. Consider, for example, translating between elephants and giraffes, where we would expect the neck of an elephant to be extended, while the elephant’s head should shrink. The challenge is compounded by the fact that, even within the same domain, images might exhibit extreme variations in object shape and pose, partial occlusions, and contain multiple instances of the object of interest. One might even argue that this translation task is ill-posed to begin with, and at the very least, requires high-level semantics to be accounted for.
Nonetheless, several works do address shape deformation in the context of image-to-image translation by requiring supervision in the form of a foreground mask (Liang et al., 2018; Mo et al., 2018). In contrast, GANimorph (Gokaslan et al., 2018) and the recently proposed TransGaGa (Wu et al., 2019) show remarkable translation without requiring additional supervision for several datasets. However, these techniques excel in controlled setting only, where the images are controlled, and the foreground separation is rather simple.
In this paper, we address the problem of unpaired image-to-image translation, without requiring foreground masks, between two different domains, where the objects of interest share some semantic similarity (e.g., four-legged mammals), whose shapes and appearances may, nevertheless, be drastically different. Our key idea is to accomplish the translation task by learning to translate between deep feature maps. Rather than learning to extract the relevant higher-level semantic information for the specific pair of domains at hand, we leverage deep features extracted by a network pre-trained for image classification, thereby benefiting from its large-scale fully supervised training.
Our work is motivated by the well-known observation that neurons in the deeper layers of pre-trained classification networks represent larger receptive fields in image space, and encode higher-level semantic content Zeiler & Fergus (2014). In other words, local activation patterns in the deeper layers may encode very different shapes in size and structure. Furthermore, Aberman et al. (2018) showed that semantically similar regions from different domains, e.g. dog and cat, have similar activations. That is, the encoding of a cat’s eye resembles that of a dog’s eye more than that of its tail. From the point of view of the translation task, these properties are attractive, since they suggest that it might be possible to learn a semantically consistent translation between activation patterns produced by images from different domains, and that the resulting (reconstructed) image would be able to change drastically, hopefully bypassing the common difficulties in image-to-image translation methods.
More specifically, we learn to translate between several layers of deep feature maps, extracted from two domains by a pre-trained classification network, namely VGG-19 (Simonyan & Zisserman, 2014). The translation is carried out one layer at a time in a deep-to-shallow (coarse-to-fine) cascaded manner. For each layer, we adversarially train a dedicated translator that acts in the features space of that layer. The deepest layer translator effectively learns to translate between semantically similar global structures, such as body shape or head position, as demonstrated by the middle pair of images in Figure 1. The translator of each shallower layer is conditioned on the translation result of the previous layer, and learns to add fine scale and appearance details, such as texture. At every layer, in order to visualize the generated deep features, we use a network pre-trained for inverting the deep features of VGG-19, following the method of Dosovitskiy & Brox (2016). The images shown in Figure 1 were generated in this manner.
Our conceptual novelty, can be viewed as a transfer learning for image translation as we are translating high level semantics, encoded in the deeper layers of a pre-trained classification network, a.k.a deep features. This is in contrast to existing methods Gokaslan et al. (2018); Wu et al. (2019), which learn to translate the images directly. We compare our method with several state-of-the-art image translation methods. To demonstrate the effectiveness of our approach, we present results for several challenging pairs of domains that exhibit drastically different shapes and appearances, but share some high-level semantics. Our translations are semantically consistent, typically preserving pose and number of instances of objects of interest, and reproducing their partial occlusion or cropping, as may be seen in Figure 5.

heading: 2 RELATED WORK
text: Zhu et al. (2017) (simultaneously with Yi et al. (2017) and Kim et al. (2017)) have presented remarkable unpaired image-to-image translations, using a framework known as CycleGAN. The key idea is that the ill-posed conditional generative process can be regularized by a cycle-consistency constraint, which enforces the translation to perform a bijective mapping. The cycle constraint has become a popular regularization technique for unpaired image-to-image translation. For example, the UNIT framework (Liu et al., 2017) assumes a shared latent space between the domains and enforces the cycle constraint in the shared latent space. Several works were developed to extend the one-to-one mapping to many-to-many mapping (Ma et al., 2018; Huang et al., 2018; Lee et al., 2018; Almahairi
et al., 2018). These methods decompose the encoding space to shared latent space, representing the domain invariant content space, and domain specific style space. Therefore, many translations can be achieved from a single content code by changing the style code of the input image.
Many translation methods share the inability to translate high-level semantics, including different shape geometry. This type of translation is usually necessary in the case of transfiguration, where one aims to transform a specific type of objects without changing the background. Both Lee et al. (2018) and Mejjati et al. (2018) learn an attention map and apply translation only on the the foreground object. However, both methods only improve translations that do not deform shapes. Gokaslan et al. (2018) succeed in preforming several shape-deforming translations by several modifications to the CycleGAN framework, including using dilated convolutions in the discriminator. However, they have not demonstrated strong shape deformations, such as zebras to elephants or giraffes, as we show in Section 4.
Liang et al. (2018) and Mo et al. (2018) assume some kind of segmentation is given, and use this segmentation to guide shape deformation translation. However, such segmentation is hard to achieve. In a recent work, Wu et al. (2019) disentangle the input images to geometry and appearance spaces, relying on high intra-consistency, and learn to translate each of the two domains separately. However, the variation of geometry and appearance of in-the-wild images is too large to be disentangled successfully1.
Contrary to the above works, our work leverages a pre-trained network and the translation is applied directly on deep feature maps, thus being guided by high-level semantics. Several image-to-image methods, such as Xu et al. (2018); Di et al. (2018); Ignatov et al. (2018), also incorporate such pretrained networks, though usually, only as perceptual loss, constraining the translated image to remain semantically close to the input image. Differently, Sungatullina et al. (2018) incorporate pre-trained VGG features into the discriminator architecture, to assist in the discrimination phase. Wu et al. (2018) use VGG-19 as a fixed encoder in the translation, where only the decoder is learned. Upchurch et al. (2017) present the only method, to our knowledge, that actually translates deep features between two domains. However, the translation is not learned, but defined by simply interpolating between the deep features, which restricts the scope of method to highly aligned domains. For completeness, we also mention that Yin et al. (2019) train an autoencoder to embed point clouds, and perform translation in the learned embedding. In contrast, we utilize semantics to preform the translation in the much more difficult scenario of images.
Our work shares some similarities with the early work of Huang et al. (2017), which suggests using a generative adversarial model (Goodfellow et al., 2014) in a coarse-to-fine manner with respect to a pre-trained encoder. The generation process begins from the deepest features and then recursively synthesizes shallower layers conditioned on the deeper layer, until generating the final image. This method was only applied on small encoders and low resolution images and was not explored for very deep and semantic encoding neural networks such as VGG-19 (Simonyan & Zisserman, 2014).
Deep image analogies (Liao et al., 2017) transfer visual attributes between semantically similar images, by feed-forwarding them through a pre-trained network. Their work does not train a generative model; nonetheless, they create new deep features by fusing content features from one image with style features of another. Similarly, Aberman et al. (2018) synthesize hybrid images from two aligned images by selecting the dominant deep features response.

heading: 3 METHOD
text: Our general setting is similar to that of previous unpaired image-to-image translation methods. Given images from two domains, A and B, our goal is to learn to translate between them. However, unlike other image-to-image translation methods, we perform the translation on the deep features extracted by a pre-trained classification network, specifically VGG-19 (Simonyan & Zisserman, 2014).
The translation is carried out in a deep-to-shallow (coarse-to-fine) manner, using a cascade of pairs of translators, one pair per layer. The entire architecture used to train the translators is shown schematically in Figure 2, while Figure 3 illustrates the test-time translation (inference) process. Once
1Unfortunately, at the time of this submission the authors of Wu et al. (2019) were unable to release their code or train their network on the datasets presented in our paper.
the deepest feature map has been translated, we translate the next (shallower and less semantic feature map), conditioned on the translated deeper layer. In this manner, the translation of the shallower map preserves the general structure of the translated deeper one, but adds finer details, which are not encoded in the deeper feature maps. We repeat this procedure until the original image level is reached. Below we describe the training and the inference processes in more detail.
Pre-processing: We extract high-level semantic features from input images from both domains, A and B, by feed-forwarding the images through the pre-trained VGG-19 (Simonyan & Zisserman, 2014) network. Next, we sample five of the resulting deep feature maps, specifically conv_i_1 (i = 1, 2, 3, 4, 5), where each map has progressively coarser spatial resolution, but a larger number channels. We denote the i-th sampled feature map for image a ∈ A as ai. Since propagation through the pre-trained VGG-19 network may yield features in any range, while the range of the synthesized features is usually known, to ease the translation, we first normalize each channel, of every layer i, by calculating its mean and standard deviation across the domain and then we clamp the normalized feature values to the range of [−1, 1]. While the clamping is a potentially harmful irreversible operation, we did not observe any adverse effect on the results. We use Ai (Bi) to denote set of all normalized deep feature maps of level i, extracted from images in domain A (B).
Inference: We perform the translation in a coarse-to-fine fashion. Thus, the translator from domain A to B, actually consists of a sequence of translators { G5B , G 4 B , . . . , G 1 B } , where each translator is responsible for translating the i-th feature map layer ai, from Ai to Bi conditioned on the previously translated deeper layer b̃i+1 (except for the deepest layer translator G5B , which is unconditioned).
Finally, G0B uses feature inversion to convert b̃1 to obtain the translated image. The translators G i A from domain Bi to Ai are defined symmetrically. The entire inference pipeline is shown in Figure 3.
Feature inversion: In all the results we show, e.g., Figure 1, we visualize the output of the various translators by pre-training a deep feature inversion network (per domain), for each layer i = 1, . . . , 5, following Dosovitskiy & Brox (2016). The network aims to reconstruct the original image given the feature map of a specific layer, regularized by adversarial loss so that the reconstructed image would lie in the manifold of natural images. For more details we refer the reader to Dosovitskiy & Brox (2016). The specific settings used in our implementation are elaborated in Appendix A.1.
Deepest layer translation: We begin by translating the deepest feature maps, encoding the highestlevel semantic features, i.e.,A5 andB5, hence, our problem is reduced to translating high-dimensional tensors. Our solution builds upon the commonly used CycleGAN framework (Zhu et al., 2017). Specifically, we use the three losses proposed in Zhu et al. (2017). First, in order to generate deep features in the appropriate domain, we utilize an adversarial domain loss Ladv. We simultaneously train two translators G5A, G 5 B which try to fool domain-specific discriminators, D 5 A, D 5 B (for domains A5, B5, respectively). However, differently from Zhu et al. (2017) and other image translation methods (Huang et al., 2018; Mo et al., 2018), we have found LSGAN (Mao et al., 2017) not to be well-suited for our task, leading to mode collapse or convergence failures. Instead, we found WGAN-GP (Gulrajani et al., 2017) more effective, thus, the adversarial loss for translation from X to Y is defined as
Ladv (GY , DY , X, Y ) = E x∼PX [DY (GY (x))]− E y∼PY [DY (y)]+λgp E ŷ∼PŶ
[ (‖∇DY (ŷ)‖ − 1)2 ] , (1)
where GY : X → Y is the translator, DY is the target domain discriminator, λgp = 10 in all our experiments, and PŶ is defined by uniformly sampling along straight lines between ỹ ∼ G (PX) and y ∼ PY . For more details we refer the reader to Gulrajani et al. (2017). Second, for regularizing the translation to one-to-one mapping, we add the cycle constraint,
Lcyc(GX , GY , X, Y ) = E x∼PX ‖GX (GY (x))− x‖+ E y∼PY ‖GY (GX (y))− x‖, (2)
where ‖ · ‖ stands for the L1 norm. Finally, as in Zhu et al. (2017), we have found it helpful to use an identity loss, which guides the networks to preserve common high level features,
Lidty(GX , GY , X, Y ) = E x∼PX ‖GX (x)− x‖+ E y∼PY ‖GY (y)− y‖. (3)
The entire loss combines these components as follows L5 =Ladv ( G5B , D 5 B , A5, B5 ) + Ladv ( G5A, D 5 A, B5, A5 ) + λcycLcyc(G5A, G5B , A5, B5) + λidtyLidty(G5A, G5B , A5, B5), (4)
where λcyc and λidty were set to 100 in all our experiments.
Coarse to fine conditional translation: Consider two successive layers, ai ∈ Ai and ai+1 ∈ Ai+1, where the latter has already been translated, yielding b̃i+1 as the translation outcome (see Figure 3). We next perform the translation of the layer ai to yield b̃i, using the translator GiB , conditioned on b̃i+1. Note that GiB is effectively a function of all the previously translated layers.
The architecture of GiB is schematically shown in Figure 4. Since shallower layers encode less of the semantic content of the image, it is more difficult to learn how they should be deformed, and thus they are used to transfer “style”, while the “content” comes from the already translated deeper layer. Inspired by Huang et al. (2018), we add an adaptive instance normalization (AdaIN) (Huang & Belongie, 2017) component, whose parameters are learned from the current layer. Thus, several layers of GiB are normalized according to the AdaIN component. G i A, which is designed symmetrically, is learned simultaneously with GiB , as shown in Fig.4(a).
The loss for training these shallower translators is roughly the same as that used for training the deepest translation: it consists of adversarial domain loss, cycle constraint loss, and identity loss.

heading: MSCOCO
text: While we formalize the adversarial loss unconditionally, similarly to equation 1, the cyclic loss is now conditioned: ∥∥∥GiA (GiB (ai, b̃i+1) , ai+1)− ai∥∥∥+ ∥∥GiB (GiA (bi, ãi+1) , bi+1)− bi∥∥, and the same is true for the identity loss
∥∥GiA (ai, ai+1)− ai∥∥+ ∥∥GiB (bi, bi+1)− bi∥∥. We train the pairs of translators one layer at a time, starting from G5A and G 5 B . More details regarding the implementation and the training of the translators are included in Appendix A.1.

heading: 4 EXPERIMENTS
text: We evaluate our methods on several publicly available datasets: (1) Cat↔ dog faces (Lee et al., 2018), which contains 871 cats images and 1364 dogs images and does not require high shape deformation; (2) Kaggle Cat↔ dog (Elson et al., 2007) dataset with over 12, 500 images in each domain (the images may contain part of the object or several instances); (3) Challenging MSCOCO dataset (Lin et al., 2014), specifically, zebra ↔ elephant and zebra ↔ giraffe (the number of images is each category is reported in (overall there are 1917 zebras, 2547 giraffes and 2144 elephants). We note that no previous method has used MSCOCO, without supervision in the form of segmentation.
Level 3Level 2Level 1Original Level 4 Level 5
adv cyc& adv & adv idty cyc& adv idtyOriginal & cyc idty
Figure 6: Translation of the 5th (deepest) layer with different loss combinations. Using all three components yields the best result.
Our deepest translators, i.e., G5A, G 5 B , consist of encoder-decoder structure with several strided convolutional layers followed by symmetric transpose convolutional layers. We use group normalization (Wu & He, 2018) and ReLU activation function (except the last layer, which is tanh). The conditional generators, consist of learned AdaIN layer, achieved by several strided convolutional layers followed by fully connected layers. The content generator has also several convolutional layers and one single transpose convolutional layer which doubles the spatial resolution (Figure 4(right)). In practice we only train G5, G4, G3, and apply feature inversion directly on the output of the latter, with negligible degradation. For the exact layer’s specifics we refer the reader to Appendix A.1 and to our (soon to be published) code. We run each layer for 400 epochs with fixed learning rate of 0.0001 and Adam optimizer (Kingma & Ba, 2014). On a single RTX 2080 reaching the final image takes around 2.5 days, including the final inversion network training.
Several translation examples are presented in Figure 5. Our translation is able to achieve high shape deformation. Note that our translations are semantically consistent, in the sense that they preserve the pose of the object of interest, as well the number of instances is mostly preserved. Furthermore, partial occlusions of such objects, or their cropping by the image boundaries are correctly reproduced. See for example, the translations of the pairs of animals in columns 5–6. More results are provided in Appendix A.2.

heading: 4.1 ABLATION STUDY
text: We analyze two main elements of our method. Loss components First, we validate the use of CycleGAN loss components. As shown in Figure 6, we translate the 5th (deepest) layer with and without cycle, identity and adversarial losses. The best approach is achieved by using all of the losses, which balance each other.
VGG layers In addition, In Figure 7 we compare translation of different VGG-19 layers. Evidently, shallower layers introduce spatial constraints, thus, limiting the translation in the sense of shape’s changes. The shallowest layer can hardly change the shape of the input image, which may explain the failure of traditional image translation methods. In Table 1, we use the common FID score (Szegedy et al., 2015) to show that the cascaded translation achieves better translation compared to individual layer translation. Additional results are shown in Appendix A.4
Type of pre-trained network While our method is conceptually agnostic to the type of features extraction network, we still require the extracted features to represent high-level semantic. Therefore, VGG-19 layers, which are the most common features used for image generation tasks (Aberman et al.
(2018); Dosovitskiy & Brox (2016); Gatys et al. (2016); Liao et al. (2017)), was chosen. Nonetheless, we experiment with a fine-tuned version of VGG-19 and different network architecture as shown in Figure 9. We fine-tuned VGG to classify between zebras and giraffes and trained our translation networks using the resulting features. As can be seen, the translation results are inferior to the results achieved by the standard VGG-19. This maybe attributed to VGG-19 fixating on the unique differences between zebras and giraffes, unrelated to the translation, such as background. For more about the extracted features, we refer the reader to Appendix A.1.3. In addition, in Figure 9, we examine a different network, AlexNet, also pretrained on ImageNet. We have observed that the deepest image translation was not able to generate a valid shape of zebra or giraffe. AlexNet is composed of 5 convolutional layers, where the first one uses a stride of 4. Therefore, the resulting features have less spatial encoding, especially at the deeper layers, which may explain the difficulty to invert and translate these features.

heading: 4.2 COMPARISON TO OTHER METHODS
text: We compare our result with leading image translation methods, i.e. CycleGAN (Zhu et al., 2017), MUNIT (Huang et al., 2018), DRIT (Lee et al., 2018) and GANimorph (Gokaslan et al., 2018).
Quantitative comparison: In order to perform a quantitative comparison, we use the common FID score (Szegedy et al., 2015), as reported in Table 2. Our method achieved the best FID score on five out of the six cross-domain translations for which this score was measured.
Qualitative comparison: In Figure 8 we show several challenging translation examples. While traditional image translation methods struggle to preform translations with such drastic shape deformation, our method is able to do so thanks to its use of the pre-trained VGG-19 network.
The success of our method can also be explained and visualized by examining the translated deep features. We feed forward every image, original and translated, through the entire VGG network, extracting the last fully-connected layer (before the classification layer). We project this vector (of size 4096) to 2D, using t-SNE, as shown in Figure 9. It may be seen that the distribution of the translated vectors (in cyan) is closest to that of the target domain (in red) when using our method.
Limitations Our method achieves translations with significant shape deformation in many previously unattainable scenarios, yet, a few limitations remain. First, the background of the object is not preserved, as the background is encoded in the deep features along with the semantic parts. Also, in some cases the translated deep features may be missing small instances or parts of the object. This may be attributed to the fact that VGG-19 is generally not invertible and was trained to classify finite set of classes. In addition, since we translate deep features, small errors in the deep translation may be amplified to large errors in the image, while for image-to-image translation method that operate on the image directly, small translation errors would typically be more local. Please note that, similarly to CycleGAN and GANimorph, our translation is deterministic.

heading: 5 CONCLUSIONS
text: Translating between image domains that differ not only changes in appearance, but also exhibit significant geometric deformations, is a highly challenging task. We have presented a novel image-toimage translation scheme that operates directly on pre-trained deep features, where local activation
patterns provide a rich semantic encoding of large image regions. Thus, translating between such patterns is capable of generating significant, yet semantically consistent, shape deformations. In a sense, this solution may be thought of as transfer learning, since we make use of features that were trained for a classification task for an unpaired translation task. In the future, we would like to continue exploring the applications of powerful pre-trained deep features for other challenging tasks, possibly in different domains, such as videos, sketches or 3D shapes.

heading: A APPENDIX

heading: A.1 TRAINING DETAILS
text: A.1.1 Hyper parameters
In all our experiments, unless stated otherwise, we use Adam optimizer [16] with β1 = 0.5, β2 = 0.999. The learning rate was set to 0.0001 and the batch size to 10. During training, random crop and image mirroring is applied. Our training methodology follows WGAN-GP (Gulrajani et al., 2017), thus for one generator update we update the discriminator four times.

heading: A.1.2 NETWORK ARCHITECTURE
text: Feature Inversion Our implementation is similar to Dosovitskiy & Brox (2016). We train an individual feature inversion network for VGG layer, where each layer has different channels (512, 512, 256, 128, 64). All layers utilize Leaky ReLU nonlinearity (0.2) and employ no normalization. The last layer utilizes Tanh. All inversion networks, first apply three non-strided convolutional layers, with N input number of channels, equal to the number of channels each deep layer has. Next, several transpose convolutional layers are applied, each doubles the resolution of the image and decreases the channel resolution (by factor of 2) until the image resolution 224 is achieved (thus, different amount of ConvTranspose layers per layer). The final layer is a non-strided convolutional layer followed by Tanh layer. Together they project the features back to the original image dimensions and range (number of output channels is 3). For the discriminator we have used Patch GAN discriminator, with four strided convolutional layers, each utilizes batch normalization (except the first one) and Leaky ReLU. For the adversarial metric, only here, we have used LS-GAN. Here we also set the batch size to 25.
Deepest layer translation The input to the deepest translation network is conv_5_1, thus, the input size is 14× 14× 512 (recall the input image size is 224× 224). The identity and cycle losses are multiplied by λidty = λcyc = 100. The architecture is reported in Table.3. The networks is relatively small and achieve good results in a few hours on a single GPU (RTX2080). We use the WGAN-GP optimization method, updating the generator once for every four discriminator updates.
Coarse to fine conditional translation The coarse-to-fine generator, for generating level i, has two inputs: the current source VGG level and the previous translated VGG features (i + 1). An AdaIN component, acts on on the current deep features and normalizes several layers in the translator
itself. We report the AdaIN component structure for generating layer four in Table.4. The architecture can be extended easily to other layers. The core components of the translator, which takes as input the previous translated layer, are reported in Table.5.

heading: A.1.3 DIFFERENT NETWORKS
text: VGG-19 fine-tuned We fine-tuned VGG-19 by fixing all layers but the fully connected layers and conv_5_i (i = 1, 2, 3, 4). We replaced the final fully connected layer with new fully connected layer of size 4096 × 2 corresponding to the two domains. We trained the classifier for 50 epochs, with batch size of 25 and learning rate of 0.01.
AlexNet We extract each of the 5 convolution of AlexNet as different layers. Each layer was normalized, in a similar manner as was described for VGG-19.

heading: A.2 MORE COMPARISON RESULTS
text: In this section we show more results, not presented in the paper, for zebra↔giraffe, zebra↔elephant and cat↔dog translations.

heading: A.3 NON-SHAPE DEFORMATION TRANSLATION
text: Our method is also suited to none-shape deformation tasks, as in the case of dataset(1) (Lee et al., 2018).

heading: A.4 COARSE TO FINE TRANSLATION
text: We here present the translation of each layer. The translation of each shallower layer is conditioned on the translation result of the previous layer, and learns to add fine scale and appearance, such as texture. At every layer, in order to visualize the generated deep features, we use a network pre-trained for inverting the deep features of VGG-19, following the method of Dosovitskiy & Brox (2016).

heading: A.5 NEAREST NEIGHBOR COMPARISON
text: In this section we show side by side, source images, our translation and the three nearest neighbors in the target domain. We use the LPIPS metric, presented in Zhang et al. (2018). This metric is based on L2 distance of deep features extracted from pre-trained network. In our case we use the default settings proposed by Zhang et al. (i.e. alex net). As we show, the closest image in the target dataset vary in pose, scale and content (i.e. different parts of the objects).

Review summaries:
The paper proposes a new method for image-to-image translation. The problem of most existing methods is that they are good in translating style  (e.g from photo to Van Gogh) but do not allow for significant changes in shape (e.g. from zebra to giraffe). The authors address this by performing the translation in a cascaded fashion starting from a semantic (deep) level (fifth layer of VGG). The underlying idea is that translating at this more semantic level puts less spatial constraints on the final resulting images (making translations from zebra to giraffe possible). After the fifth layer is translated other layers are translated conditioned on the previous translation results.