Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: * Summarize what the paper claims to do/contribute.
This paper claims to extend existing image translation works, like CycleGAN, to domain pairs that are not similar in shape. It is proposed to do so by using a VGG network trained on classification (I assume on Imagenet), extracting features from the two domains and learn 5 CycleGANs to translate for each level of the feature hierarchy. At each level of the hierarchy the translation from the previous level is used to condition the translation for the current level. During inference, the final image translation is done by "feature inversion" (a technique proposed in Dosovitsikiy and Brox, 2016) from the final feature layer. The technique is show on example from a number of pairs of domains like Zebra-to-Elephant (and back), Giraffe-to-Zebra (and back), Dog-to-Cat (and back) and is compared with a number of baselines qualitatively and quantitatively with the FID score.  

* Clearly state your decision (accept or reject) with one or two key reasons for this choice.
Weak Reject.

Major reasons:
- The problem itself, as stated in the introduction, seems ill-posed to me. One of the struggles I had while looking through the results was to understand what the images should be looking like. ie What should a zebra translated to a giraffe look like? The motivation for such a problem is also not immediately clear either. 
- Most of the resulting images do not seem "translated" to me. As stated in the paper (end of p.2) "one aims to transform a specific type of object without changing the background." As one can see in eg Fig. 1 the resulting translations are completely different images with the foreground object of the new domain in roughly similar poses. The background in most cases does not persist. What I suspect is actually happening here is that the high-level semantics from the first image are used as some sort of noise to generate new images from the new domain. One question I had, for example: could we be getting similar results if we used the VGG bottleneck as the noise vector in an InfoGAN? Since the VGG network is pretrained and used in the same way in both domains, I imagine we would be seeing something very similar. (and it would be def. preferrable to tuning 10 GANs!)

* Provide supporting arguments for the reasons for the decision.
Some of the decisions made in the paper were unclear and not supported adequately. The questions (in rough order of importance) that made some of the contributions unclear to me:
- Why wasn't a final translator used for the final image, conditioned on the final \tilde{b}_1? 
- Is the VGG network pretrained on ImageNet? Why wasn't another task used that could be retaining more of the relevant features? eg on semantic segmentation
- Could this be used for networks pretrained on other datasets? Presumably ImageNet has information about the animals translated in this paper. Even better, could we somehow learn these features for the domain pairs automatically somehow?
- How meaningful is the FID score really in this case?
- How were the 10 GANs tuned?

* Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
- It is mentioned on p.4 that "clamping is potentially a harmful irreversible operation" but that harmful results were not observed. As I was reading that I was wondering how these results would actually look like. 
- On p. 6 it is mentioned that the number of images for 2 categories are reported in another paper. I think it'd take less space to actually report the number of images here.
- On p.7 it is mentioned that the number of instances is preserved, however it should be made clear that it's is perserved in some (or most if that is what was observed) of the examples.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new cascaded image-to-image translation method to address the I2I tasks where the domains have exhibit significantly different shapes. The proposed method train cycle GAN on different levels of feature extracted by pre-trained VGG and combine the futures with the AdaIN layer to keep the correct shape from the deep features. 

Pros:
1. The proposed method seems to work well on different shape I2I datasets without using semantic masks compared to previous works.
2. The idea of cascaded translators sounds simple and reasonable which can probably benefit other related tasks. The way of applying AdaIn to combine features of different levels is also a nice trick to keep the correct shape from deep features.
3. The paper writing is OK, but some explanation and organization should be improved as mention in cons.

Cons:
1. Some figures are hard to understand without looking at the text. For example, in Figure 1, the caption does not explain the figure well. What does each image, the order, and the different sizes mean?  As to Figure 3, the words “top left image”, “right purple arrows” are a bit confusing.
2. The “Coarse to fine conditional translation” section describes the conditional translation in the shallow layers. I suggest mentioning it in previous sections for easy understanding.
3. As to the t-SNE visualization in Figure 9, different methods seem to use different N-D to 2-D mapping functions. This may lead to an unfair comparison.

Suggestions:
1. The authors use the pre-trained classification network VGG for feature extraction and then train dedicated translators based on these features. I wonder if the authors also tried finetuning VGG on the two domains or training an auto-encoder on the two domains. The domain-specific knowledge may help to improve the results and alleviate the limitations presented in the paper, e.g. background of the object is not preserved, missing small instances or parts of the object due to invertible VGG-19.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new method for image-to-image translation. The problem of most existing methods is that they are good in translating style  (e.g from photo to Van Gogh) but do not allow for significant changes in shape (e.g. from zebra to giraffe). The authors address this by performing the translation in a cascaded fashion starting from a semantic (deep) level (fifth layer of VGG). The underlying idea is that translating at this more semantic level puts less spatial constraints on the final resulting images (making translations from zebra to giraffe possible). After the fifth layer is translated other layers are translated conditioned on the previous translation results. 

The method is compared to other translation methods including DRIT, MUNIT, GANimorph. Both visually and quantitatively as measured by FID. The results of the proposed method are superior. No comparison to TransGaGa is provided (but I could not find code for this method).

My recommendation is borderline accept. The proposed method is simple. The experimental results are limited but show both visually and quantitatively superior results. Especially FID scores are much better. I think the paper could be improved in explaining the conceptual novelty of the paper (especially with respect to GANimorph and TransGaGa).

1. I like the idea of applying the cycle consistency to the deeper layers rather than at the pixel level. Are there other methods which  do this ? It could be highlighted more as part of the contribution.
2. An ablation study should be added (in FID scores). I would like to see the necessity of the cascade (which is in the title) confirmed: results for only translating a single layer (3,4, or 5)  should be compared to translating 3,4,5 together as in proposed method. 
3. Would it be possible to not use pretrained feature from VGG-19 ? This might also be a limitation. In principle, I guess you could train everything end-to-end, or is this impractical because of the feature inversion. 
4.The authors could add some text on the lack of diversity for the translations in the limitation section. I understand there is no diversity and the translation is deterministic. 
 
In general, I think the paper clearly explains what it does, and it also shows cases that it performs better than state-of-the-art. The paper could be much improved in its analysis of the reasons for its better performance, analyzing key aspects of its design like cycle GAN on features, pretrained VGG features and the use of cascaded generation of the final image. 

review_title: Official Blind Review #1

Meta-review: The paper addresses image translation by extending prior models, e.g. CycleGAN, to domain pairs that have significantly different shape variations. The main technical idea is to apply the translation directly on the deep feature maps (instead of on the pixel level). 
While acknowledging that the proposed model is potentially useful, the reviewers raised several important concerns:
(1) ill-posed formulation of the problem and what is desirable, (2) using fine-tuned/pre-trained VGG features, (3) computational cost of the proposed approach, i.e. training a cascade of pairs of translators (one pair per layer). 
AC can confirm that all three reviewers have read the author responses. AC suggests, in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject