Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: * sec.2.2 is about label-preserving translation and many notations are introduced. However, it is not clear what label here refers to, and it does not shown in the notation so far at all. Only until the end of sec.2.2, the function F(.) is introduced and its revelation - Google Search as label function is discussed only at Fig.4 and sec.2.3.
* pp.5 first paragraph: when assuming D_X and D_Y being perfect, why L_GAN_forward = L_GAN_backward = 0? To trace back, in fact it is helpful to have at least a simple intro/def. to the functions D(.) and G(.) of Eq.(1). 
* Somehow there is a feeling that the notations in sec.2.1 and sec.2.2 are not well aligned. It is helpful to start providing the math notations as early as sec.2.1, so labels, pseudo labels, the algorithm illustrated in Fig.2 etc. can be consistently integrated with the rest notations. 
* F() is firstly shown in Fig.2 the beginning of pp.3, and is mentioned in the main text as late as of pp.5.
* Table 2: The CNN baseline gives an error rate of 7.80 while the proposed variants are 7.73 and 7.60 respectively. The difference of 0.07/0.20 are not so significant. Any explanation for that?
Minor issues:
* The uppercase X in the sentence before Eq.(2) should be calligraphic X
review_title:  This paper considers a generative approach in semi-supervised setting, which is basically a combination of cycleGAN and the Apple's S+U leaning.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Review, ICLR 2018, Simulated+Unsupervised Learning With Adaptive Data Generation and Bidirectional Mappings

Summary:

The paper presents several extensions to the method presented in SimGAN (Shirvastava et al. 2017). 
First, it adds a procedure to make the distribution of parameters of the simulation closer to the one in real world images. A predictor is trained on simulated images created with a manually initialized distribution. This predictor is used to estimate pseudo labels for the unlabeled real-world data. The distribution of the estimated pseudo labels is used produce a new set of simulated images. This process is iterated. 
Second, it adds the idea of cycle consistency (e.g., from CycleGAN) in order to counter mode collapse and label shift. 
Third, since cycle consistency does not fully solve the label shift problem, a feature consistency loss is added.
Finally, in contrast to ll related methods, the final system used for inference is not a predictor trained on a mix of real and “fake” images from the real-world target domain. Instead the predictor is trained purely on synthetic data and it is fed real world examples by using the back/real-to-sim-generator (trained in the conjunction with the forward mapping cycle) to map the real inputs to “fake” synthetic ones.

The paper is well written. The novelty is incremental in most parts, but the overall system can be seen as novel. 

In particular, I am not aware of any published work that uses of the (backwards) real-to-sim generator plus sim-only trained predictor for inference (although I personally know several people who had the same idea and have been working on it). I like this part because it perfectly makes sense not to let the generator hallucinate real-world effects on rather clean simulated data, but the other way around, remove all kinds of variations to produce a clean image from which the prediction should be easier.

The paper should include Bousmalis et al., “Unsupervised Pixel-Level Domain Adaptation With Generative Adversarial Networks”, CVPR 2017 in its discussion, since it is very closely related to Shirvastava et al. 2017.

With respect to the feature consistency loss the paper should also discuss related work defining losses over feature activations for very similar reasons, such as in image stylization (e.g. L. A. Gatys et al. “Image Style Transfer Using Convolutional Neural Networks” CVPR 2016, L. A. Gatys et al. “Controlling Perceptual Factors in Neural Style Transfer” CVPR 2017), or the recently presented “Photographic Image Synthesis with Cascaded Refinement Networks”, ICCV 2017.
Bousmalis et al., “Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping”, arXiv:1709.07857, even uses the same technique in the context of training GANs.

Adaptive Data Generation:
I do not fully see the point in matching the distribution of parameters of the real world samples with the simulated data. For the few, easily interpretable parameters in the given task it should be relatively easy to specify reasonable ranges. If the simulation in some edge cases produces samples that are beyond the range of what occurs in the real world, that is maybe not very efficient, but I would be surprised if it ultimately hurt the performance of the predictor. 
I do see the advantage when training the GAN though, since a good discriminator would learn to pick out those samples as generated. Again, though, I am not very sure whether that would hurt the performance of the overall system in practice.

Limiting the parameters to those values of the real world data also seems rather restricting. If the real world data does not cover certain ranges, not because those values are infeasible or infrequent, but just because it so happens that this range was not covered in the data acquisition, the simulation could be used to fill in those ranges. 
Additionally, the whole procedure of training on sim data and then pseudo-labeling the real data with it is based on the assumption that a predictor trained on simulated data only already works quite well on real data. It might be possible in the case of the task at hand, but for more challenging domain adaptation problem it might not be feasible. 
There is also no guarantee for the convergence of the cycle, which is also evident from the experiments (Table 1. After three iterations the angle error increases again. (The use of the Hellinger distance is unclear to me since it, as explained in the text, does not correspond with what is being optimized). In the experiments the cycle was stopped after two iterations. However, how would you know when to stop if you didn’t have ground truth labels for the real world data?

Comparisons:
The experiments should include a comparison to using the forward generator trained in this framework to train a predictor on “fake” real data and test it on real data (ie. a line “ours | RS | R | ?” in Table 2, and a more direct comparison to Shrivastava). This would be necessary to prove the benefit of using the back-generator + sim trained predictor.


Detailed comments:
* Figure 1 seems not to be referenced in the text.
* I don’t understand the choice for reduction of the sim parameters. Why was, e.g., the yaw and pitch parameters of the eyeball set equal to those of the camera? Also, I guess there is a typo in the last equality (pitch and yaw of the camera?).
* The Bibliography needs to be checked. Names of journals and conferences are inconsistent, long and short forms mixed, year several times, “Proceedings” multiple times, ...

review_title: Review: Some interesting ideas, some missing related work and comparisons

final_score: rating: 3: Clear rejection
review_content: General comment:

This paper proposes a GAN-based method which learns bidirectional mappings between the real-data and the simulated data. The proposed methods builds upon the CycleGAN and the Simulated+Unsupervised (S+U) learning frameworks. The authors show that the proposed method is able to fully leverage the flexibility of simulators by presenting an improved performance on the gaze estimation task.

Detailed comments:

1. The proposed method seems to be a direct combination of the CycleGAN and the S+U learning. Firstly, the CycleGAN propose a to learn a bidirectional GAN model between for image translation. Here the author apply it by "translating" the the simulated data to real-data. Moreover, the mapping from simulated data to the real-data is learned, the S+U learning framework propose to train a model on the simulated data.

Hence, this paper seems to directly apply S+U learning to CycleGAN. The properties of the proposed method comes immediately from CycleGAN and S+U learning. Without deeper insights of the proposed method, the novelty of this paper is not sufficient.

2. When discussing CycleGAN, the authors claim that CycleGAN is not good at preserving the labels. However, it is not clear what the meaning of preserving labels is. It would be nice if the authors clearly define this notion and rigorously discuss why CycleGAN is insufficient to reach such a goal and why combining with S+U learning would help.

3. This work seems closely related to S+U learning. It would be nice if the authors also summarize S+U learning in Section 2, in the similar way they summarize CycleGAN in Section 2.2.

4. In Section 2.2, the authors claim that the Cycle-consistency loss in CycleGAN is not sufficient for label preservation. To improve, they propose to use the feature consistency loss. However, the final loss function also contains this cycle-consistency loss. Moreover, in the experiments, the authors indeed use the cycle-consistency loss by setting \lambda_{cyc} = 10. But the feature consistency loss may not be used by setting \lambda_{feature} = 0 or 0.5. From table Two, it appears that whether using the feature-consistency loss does not have significant effect on the performance.

It would be nice to conduct more experiments to show the effect of adding the feature-consistent loss. Say, setting \lambda_{cyc} = 0 and try different values of \lambda_{feature}. Otherwise it is unclear whether the feature-consistent loss is necessary.

review_title: Interesting application of GAN type training. But not good enough.

Meta-review: Split opinions on paper: 6 (R1), 3 (R2), 6 (R3). Much of the debate centered on the novelty of the algorithm. R2 felt that the paper was a straight-forward combination of CycleGAN with S+U, while R3 felt it made a significant contribution. The AC has looked at the paper and the reviews and discussion. The topic is very interesting and topical. The experiments are ok, but would be helped a lot by including the real/synth car data currently in appendix B: seeing the method work on natural images is much more compelling. The approach still seems a bit incremental: yes, it's not a straight combination but the extra stuff isn't so profound. The AC is inclined to accept, just because this is an interesting problem.
Decision: Accept (Poster)