Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a domain adaptation approach by extending the CycleGAN with 1) task specific loss functions and 2) loss imposed over both pixels and features. Experiments on digit recognition and semantic segmentation verify the effectiveness of the proposed method.

Strengths:
+ It is a natural and intuitive application of CycleGAN to domain adaptation. 
+ Some of the implementation techniques may be useful for the future use of CycleGAN or GAN in other applications, e.g., the regularization over both pixels and features, etc.
+ The experimental results are superior over the past.
+ The translated images in Figure 6 are amazing. Could the authors show more examples and include some failure cases (if any)?

Weaknesses:
- The presentation of the paper could be improved. I do not think I can reproduce the experimental results after reading the paper more than twice. Many details are missing and some parts are confusing or even misleading.  As below, I highlight a few points and the authors are referred to the comments by Cedric Nugteren for more suggestions.

-- Equation (4) is incorrect.
-- In the introduction and approach sections, it reads like a big deal to adapt on both the pixel and feature levels. However, the experiments fail to show that these two levels of adaptation are complementary to each other. Either the introduction is a little misleading or the experiments are insufficient. 
-- What does the “image-space adaptation” mean?
-- There are three fairly sophisticated training stages in Section 4.2. However, the description of the three stages are extremely short and ambiguous. 
-- What are exactly the network architectures used in the experiments?

- The technical contribution seems like only marginal innovative. 
- The experiments adapting from MNIST to SVHN would be really interesting, given that the MNIST source domain is not as visually rich as the SVHN target. Have the authors conducted the corresponding experiments? How are the results? 

Summary:
The proposed method is a natural application of CycleGAN to domain adaptation. The technical contribution is only marginal. The results on semantic segmentation are encouraging and may motivate more research along this direction. It is unfortunate that the paper writing leaves many parts of the paper unclear. 

=========================================
Post rebuttal:

The rebuttal addresses my first set of questions. The revised paper describes more experiment details, corrects equation (4), and clarifies some points about the results. 

This paper applies the cycle consistent GAN to domain adaptation. I still think the technical contribution is only marginally innovative. Nonetheless, I do not weigh this point too much given that the experiments are very extensive. 

The rebuttal does not answer my last question. It would be interesting to see what happens to adapt from MNIST to SVHN, the latter of which contains more complicated background than the former. 

review_title: Novelty incremental, results encouraging, writing could be improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper essentially uses CycleGANs for Domain Adaptation. My biggest concern is that it doesn't adequately compare to similar papers that perform adaptation at the pixel level (eg. Shrivastava et al-'Learning from Simulated and Unsupervised Images through Adversarial Training' and Bousmalis et al - 'Unsupervised Pixel-level Domain Adaptation with GANs', two similar papers published in CVPR 2017 -the first one was even a best paper- and available on arXiv since December 2016-before CycleGANs). I believe the authors should have at least done an ablation study to see if the cycle-consistency loss truly makes a difference on top of these works-that would be the biggest selling point of this paper. The experimental section had many experiments, which is great. However I think for semantic segmentation it would be very interesting to see whether using the adapted synthetic GTA5 samples would improve the SOTA on Cityscapes. It wouldn't be unsupervised domain adaptation, but it would be very impactful. Finally I'm not sure the oracle (train on target) mIoU on Table 2 is SOTA, and I believe the proposed model's performance is really far from SOTA.

Pros:
* CycleGANs for domain adaptation! Great idea!
* I really like the work on semantic segmentation, I think this is a very important direction

Cons:
* I don't think Domain separation networks is a pixel-level transformation-that's a feature-level transformation, you probably mean to use Bousmalis et al. 2017. Also Shrivastava et al is missing from the image-level papers.
* the authors claim that Bousmalis et al, Liu & Tuzel and Shrivastava et al ahve only been shown to work for small image sizes. There's a recent work by Bousmalis et al. (Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping) that shows these methods working well (w/o cycle-consistency) for settings similar to semantic segmentation at a relatively high resolution. Also it was mentioned that these methods do not necessarily preserve content, when pixel-da explicitly accounts for that with a task loss (identical to the semantic loss used in this submission)
* The authors talk about the content similarity loss on the foreground in Bousmalis et al. 2017, but they could compare to this method w/o using the content similarity or using a different content similarity tailored to the semantic segmentation tasks, which would be trivial.
* Math seems wrong in (4) and (6). (4) should be probably have a minus instead of a plus. (6) has an argmin of a min, not sure what is being optimized here. In fact, I'm not sure if eg you use the gradients of f_T for training the generators?
* The authors mention that the pixel-da approach cross validates with some labeled data. Although I agree that is not an ideal validation, I'm not sure if it's equivalent or not the authors' validation setting, as they don't describe what that is.
* The authors present the semantic loss as novel, however this is the task loss proposed by the pixel-da paper.
* I didn't understand what pixel-only and feat-only meant in tables 2, 3, 4. I couldn't find an explanation in captions or in text


=====
Post rebuttal comments:
Thanks for adding content in response to my comments. The cycle ablation is still a sticky point for me, and I'm still left not sure if cycle-consistency really offers an improvement. Although I applaud your offering examples of failures for when there's no cycle-consistency, these are circumstantial and not quantitative.  The reader is still left wondering why and when is the cycle-consistency loss is appropriate. As this is the main novelty, I believe this should be in the forefront of the experimental evaluation. 
review_title: Great problem and idea, but without adequate experiments that show that cycle-consistency is the cause of the improvement

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper proposes  a natural extension of the CycleGAN approach. This is achieved by leveraging the feature and semantic losses to achieve a more realistic image reconstruction. The experiments show that including these additional losses is critical for improving the models performance.  The paper is very well written and technical details are well described and motivated. It would be good to identify the cases where the model fails and comment on those. For instance, what if the source data cannot be well reconstructed from adapted target data? What are the bounds of the domain discrepancy in this case? 
review_title: This paper extends the previous work on CycleGAN by coupling it with adversarial adaptation approaches. The extension includes a new feature and semantic loss in the overall objective of the CycleGAN. While this extension is straightforward, it is novel. The experimental validation is extensive and clearly shows the benefits of the proposed extension. 

Meta-review: I concur with two of the reviewers: the work is somewhat incremental in terms of technical novelty (it's effectively CycleGANs for domain adaptation with a couple of effective tricks) and the need/advantage of the cycle consistency loss is not demonstrated sufficiently. The only solid ablation evidence seems to the the SVHN-->MNIST experiment from post-submission; I would personally like to see this kind of empirical proof extended much further (the fact that Shrivastava et al.'s method doesn't work well on GTA-->Cityscapes is not itself proof that cycle consistency is needed). With more empirical evidence I can see this paper being a good candidate for a computer vision conference like CVPR or ICCV.
Decision: Reject