Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission proposes an interesting way to match synthetic data to real data in a GAN type architecture.
The main novelty are parametric modules that emulate different transformations and artefact that allow to match the natural appearance.

several points were raised during the discussion:

1. the proposed method is more model driven that previous GAN models. But does it pay off? how would a traditional GAN approach perform? The mentioned effects like blur, lighting and background could also potentially be modelled by upsamling network that directly predicts the image. I would assume that blur and lighting can be modelled by convolutions. transformations to some extend by convolutions - or spatial transformer networks.
The answers of the authors only partially addresses the point. The key proposal of the submission seems parameterised modules that can be trained to match the real data distribution. but it remains unclear why not a more generic parameterisation can also do the job. E.g. a neural network - as done in regular GANs. The benefit of introducing a stronger model is unclear. Using a render engine to generate the initial sample appearance if of limited novelty.


2. how does it compare to traditional data augmentation techniques, e.g. noise, dropout, transformations. you are linking to keras code - where data augmentation is readily available and could be tested (ImageDataGenerator)
The authors reply that plenty of such augmentation was used and more details are going to be provided in the appendix. it would have been appreciated if such information was directly included in the revision - so that the procedure could be directly checked. right now - this remains a point of uncertainty.

3. How do the different stages (\phis) effect performance? which are the most important ones?
The authors do evaluate the effect of hand tuning the transformation stages vs. learning them. it would be great to also include results of including/excluding stages completely - and also reporting how much the initial jittering of the data helps.

While there is an interesting idea of (limited) novelty to the paper, there are some concerns about evalations and comparisons as outlined above. In addition, only success on a single dataset/task is shown. Yet the task is interesting and seems challenging. Overall, this remains makes only a weak recommendation for acceptance.
review_title: RenderGAN: Generating Realistic Labeled Data

final_score: rating: null
review_content: - the proposed method is more model driven that previous GAN models. But does it pay off? how would a traditional GAN approach perform? The mentioned effects like blur, lighting and background could also potentially be modelled by upsamling network that directly predicts the image. I would assume that blur and lighting can be modelled by convolutions. transformations to some extend by convolutions - or spatial transformer networks.
- how does it compare to traditional data augmentation techniques, e.g. noise, dropout, transformations. you are linking to keras code - where data augmentation is readily available and could be tested (ImageDataGenerator)
- How do the different stages (\phis) effect performance? which are the most important ones?
review_title: pre-review questions

final_score: rating: null
review_content: Hello,

An interesting idea! A couple of questions:

1. Would it make sense to use a differentiable renderer (like OpenDR by Loper & Black) in your framework ?

2. Have you tried a baseline approach to synthetic data generation with simple parametric distributions of orientations and augmentations, hand-designed or estimated with simple statistics? Basically, do you really need a GAN to learn them in this relatively simple scenario, or can a human quickly make up a reasonable distribution of transformations which will perform as well ?

3. Do you think your work is related to Style and Structure Adversarial Networks https://arxiv.org/pdf/1603.05631.pdf , Spatial Transformers https://arxiv.org/abs/1506.02025 ?

4. Have you had a chance to run experiments on other datasets? 

Thanks.
review_title: pre-review questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the problem of decoding barcode-like markers depicted in an image.  The main insight is to train a CNN from generated data produced from a GAN.  The GAN is trained using unlabeled images, and leverages a "3D model" that undergoes learnt image transformations (e.g., blur, lighting, background).  The parameters for the image transformations are trained such that it confuses a GAN discriminator.  A CNN is trained using images generated from the GAN and compared with hand-crafted features and from training with real images.  The proposed method out-performs both baselines on decoding the barcode markers.

The proposed GAN architecture could potentially be interesting.  However, I won’t champion the paper as the evaluation could be improved.

A critical missing baseline is a comparison against a generic GAN.  Without this it’s hard to judge the benefit of the more structured GAN.  Also, it would be worth seeing the result when one combines generated and real images for the final task. 

A couple of references that are relevant to this work (for object detection using rendered views of 3D shapes):

[A] Xingchao Peng, Baochen Sun, Karim Ali, Kate Saenko, Learning Deep Object Detectors from 3D Models; ICCV, 2015.

[B] Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views. Francisco Massa, Bryan C. Russell, Mathieu Aubry. CVPR 2016.

The problem domain (decoding barcode markers on bees) is limited.  It would be great to see this applied to another problem domain, e.g., object detection from 3D models as shown in paper reference [A], where direct comparison against prior work could be performed.  

I found the writing to be somewhat vague throughout.  For instance, on first reading of the introduction it is not clear what exactly is the contribution of the paper.  

Minor comments:

Fig 3 - Are these really renders from a 3D model?  The images look like 2D images, perhaps spatially warped via a homography.  

Page 3: "chapter" => "section".

In Table 2, what is the loss used for the DCNN?

Fig 9 (a) - The last four images look like they have strange artifacts. Can you explain these?

review_title: The proposed model has potential merits, but the paper is missing a critical baseline in the evaluation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an approach to generating synthetic training data for deep networks, based on rendering 3D models and learning additional transformations with adversarial training. The approach is applied to generating barcode-like markers used for honeybee identification. The authors demonstrate that a classifier trained on synthetic data generated with the proposed approach outperforms both training on (limited) real data and training on data with hand-designed augmentations. 

The topic of the paper — using machine learning (in particular, adversarial training) for generating realistic synthetic training data — is very interesting and important. The proposed method looks reasonable, and the paper is written well. The downside is that experiments are limited to a fairly simple and not-widely-known domain of honeybee marker classification. While I am sure this is an important task by itself, in order to demonstrate general applicability of the method and to allow comparison with existing techniques, experiments on some standard and/or realistic datasets would be very helpful. Overall, I recommend acceptance, but encourage the authors to perform experiments on more datasets.

I appreciate that the authors added a baseline with manually designed transformations. This strengthens the paper.

As Reviewer3 points out, it would be interesting to analyze if restricting GAN to a fixed set of transformations is necessary here, and which transformations are most important. Perhaps this would provide some guidelines for designing sets of transformations for more complicated scenarios.

The authors should tone down their claims such as “Our method is an improvement over previous work  <...> Whereas previous work relied on real data for training using pre-trained models or mixing real and generated data, we were able to train a DCNN from scratch with generated data that performed well when tested on real data. “. This is not a fair comparison: the domain studied by authors in this work is much simpler than what was studied in these previous works, so this comparison is not appropriate.
review_title: Review

Meta-review: This paper was fairly well received by the reviewers in terms of the underling idea but the fact that a very specialized problem was the focus of the paper held back reviewers from giving stronger ratings. The question of what sorts of baselines would be reasonable was discussed extensively as the reviewers felt that other credible baselines should be included. The authors argue certain baselines are not appropriate but they were not able to clearly sway the reviewers to a more positive rating based on their response to this issue. We recommend a workshop invitation for this paper.
Decision: Invite to Workshop Track