Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Problem: There has been a plethora of work on image synthesis from a given layout of objects or label maps. However, it is not clear what has led to those results because there are no fixed backbone, optimization, training data, and evaluation protocol in each of them. This paper introduces a methodology to study three approaches (G2im, LostGAN, OC-GAN) that input a layout of objects to synthesize a new image.

What does the study include?: The study primarily assesses the ability of each model to (1) learn an effective mapping on the given data distribution; (2) generalize to unseen conditioning to seen object combinations; and (3) generalize to unseen conditioning to unseen object combinations. 

Conclusions of the study: (1) The current approaches are able to nicely learn an effective mapping on the given data distribution and are able to generalize to unseen conditioning of seen object configurations. However, they struggle for unseen conditioning of unseen object configurations. (2) The authors further found three essential components that help in getting good performance. These are: (a) instance-wise spatial conditional normalization layer that increases the robustness of the model to unseen conditioning; (b)  scene-graph perceptual similarly helps improve scene generation; (c) improving the quality of label map leads to better results. 


Setup of the study: 

1. Data: There are three parts: (a) $D_{s}$: seen data on which the model is trained on; (b) $D_{u}$: unseen data in the validation set that has similar object combinations like $D_{s}$; and (c) $D_{u^2}$: unseen data in the validation set that does not have similar object combination like $D_{s}$. The authors use COCO-stuff dataset in the evaluation.

2. Evaluation Protocol: Following metrics are used in the study: (a) Precision; (b) Recall; (c) Consistency; (d) FID; (e) LPIPS-based Diversity Score (DS); and (f) object classification accuracy or image-to-set prediction F1 score. 

Once you understand the setup, the analysis in Section-4 can be quickly understood. All the approaches follow a similar trend.


Major concerns with the study: 

1. In my understanding (developed using the three-axis of study), one of the underlying goals is to contrast between memorization and generalization. *This study is incomplete without an extensive comparison with simple nearest neighbors*. The current model fairs well on $D_{s}$ that is a good sign about memorization. However,  $D_{u}$ is not too different from  $D_{s}$ in this case, especially when considering the COCO-stuff dataset. The examples shown in Figure-3 and Figure-4 can be very easily seen in the training set. 

2. It is even more alarming to see that the approaches under study are not able to generalize to $D_{u^2}$. This is the scenario when we only change the object combinations. 

3. Proper setup of nearest neighbors in both the input space and output space needs consideration. Without this, I find it hard to accept any claims about the different approaches.

4. The evaluation and the related work should also include Tan et al., CVPR 2019 (Text2Scene: Generating Compositional Scenes from Textual Descriptions).

Minor Point: 

5. Section-4.5 (ground-truth masks): Though no example is provided in this paper, I looked through the generated masks in the original paper and it seems to be another issue that may be orthogonal to this study. I am not really sure how this point fits well to this study.
review_title: Examination needs extensive nearest neighbors in the evaluation

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### Summary ####
This paper studies the problem of scene conditional image generation with a focus on the evaluation of existing works towards unseen complex scene generation on the COCO-Stuff dataset. Specifically, it evaluates the model performances from three aspects, namely, image generation from seen conditionings, unseen fine-grained conditionings, and unseen coarse conditionings. For each evaluation, it computes the precision, recall, conditional consistency, F1-score, object accuracy, FID and diversity score for both object-wise and scene-wise measures. 

#### Comments ####
This paper studies an important problem in scene conditional image generation. Reviewer appreciates the experimental efforts and reasonably thorough evaluations of existing methods. However, reviewer feels this is an okay submission but not good enough.

W1: Most of the “findings” described in the evaluations are either known or somewhat expected. Reviewer fails to obtain in-depth understandings and insights through reading the paper. Based on the evaluations, the proposed ablations studies (see Table 1) also failed to advance this field further. Though the paper contains several tables with ablation studies or side-by-side comparisons, it is hard to learn more from just the numbers in the table. Reviewer would highly suggest to either provide in-depth analysis (e.g., at category level, or at bounding box level) if you choose to go this route (see my points in W2) or propose a novel method that generalizes much better than previous work.

W2: The current analysis of scene conditional generation is very preliminary and limited. Reviewer would like to see the analysis per category or per bounding box (currently the performance evaluation is mixed). In addition, it would be good to visualize the learned feature maps of the existing work to see whether the model has achieved high-level understanding of the task or not, similar to what has been done before [Ref1-4]. This way, the quality and impact of the paper can be greatly improved, as readers will gain much better understanding from reading the paper than the current form.

[Ref1] GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, Bau et al. In ICLR 2019. 

[Ref2] Seeing What a GAN Cannot Generate, Bau et al. In ICCV 2019.

[Ref3] Understanding Neural Networks Through Deep Visualization, Yosinski et al. In ICML 2015 Deep Learning Workshop.

[Ref4] Visualizing and Understanding Convolutional Networks, Zeiler and Fergus. In ECCV 2014.

review_title: [Official Review] 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper discusses the achievements and limitations of existing conditional generative models. As expected, while existing methods can generalize reasonably well to unseen layouts of seen object combinations, none of the methods can work well to unseen object combinations.

Strengths:
The paper provides extensive experiments to verify the performances of different methods under various conditions. The results should be valuable to audiences interested in future developments in the area.

Weaknesses:
There’s only one dataset that is being evaluated. Given that this paper is trying to test performances under various situations, it would be better to evaluate on more datasets to verify the results are not biased.

The conclusion is rather not surprising. I feel the paper would be stronger if it could provide more insight to the problems, especially on how to improve the generalization problem of existing methods. That is the interesting part and would really become a big contribution to the society. In the current form, it feels more like this paper only points out a problem that is already expected.

While the authors have identified some components that seem particularly useful in improving performances (Sec 4.5), I feel it is still not enough. In particular, most of the paragraphs are still simply spelling out the comparison results, without careful explanation of the insights. The authors should design more experiments to verify the hypotheses, explain the observed phenomena, and give details about the learned lessons. How these components are identified in the first place is worth mentioning too. Could there be other components that the authors are missing that could potentially be useful? Did the authors experiment with something else that turned out to be rather incremental? I think these are the questions that readers would want to know. 

review_title: Official Blind Review #4

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 

The paper provides a set of comparisons among different scene generation methods. It assesses ability of the models to fit the training set (seen conditionings), generalize to unseen conditionings of seen object combinations, and generalize to unseen conditionings composed of unseen object combinations. It finds that these models fit the training distribution with a moderate success, display decent generalization to unseen fine-grained conditionings, and have significant space for improvement when it comes to generating images from unseen coarse
conditionings. 

############################################################

Strengths:

The authors provide a comprehensive set of experiments to compare performance of different scene generation methods using several metrics. This can be helpful for researchers to assess strengths and weaknesses of each model and its components, and helps them to gain insights into which aspect of models need to be improved.  

###########################################################

Weaknesses:

1. While the comparisons among different scene generation methods are valuable, there are concerns about novelty of the paper especially since similar works have been published considering other computer vision tasks (e.g. [A, B]). The authors assess existing models in different settings and report their findings. 
2. There are some concerns regarding the overall setup for the experiments. Out of the three cases considered, the ability of a model to fit its training set (case 1) is not very interesting practically as we are more interested in the model’s generalization. Case 3, generalizing to unseen conditionings composed of unseen object combinations, is also not expected as the models are not particularly trained to be generalizable to unseen coarse conditionings. If one is seeking models with generalization ability to unseen coarse conditionings, he/she needs to incorporate a form of transfer/meta-learning and train the models differently. Generalization to novel categories is also an issue in object-level GANs.      
3. The paper claims that it is very hard to assess which models perform better due to “models being trained to fit different data splits, using different conditioning modalities and levels of supervision, and reporting inconsistent quantitative metrics (e.g. repeatedly computing previous methods’ results using different reference distributions, and/or using different image compression algorithms to store generated images), among other uncontrolled sources of variation”. However, I do not see a clear evidence supporting this. Each of the other papers (LostGAN, OC-GAN, etc.) provides a set of comparisons with other methods. The authors need to note specifically which setups are inconsistent among different papers. They report that LostGAN-v2 outperforms other models in most tasks. This is consistent with results reported in the LostGAN-v2 paper (although they evaluate their method on a smaller number of metrics).  

##############################################################

Reason for rating: 

While the paper provides a systematic evaluation of scene generation methods, there are some concerns regarding novelty and the proposed setup. I hope the authors clarify these in the rebuttal. 

##############################################################

Additional comments:

There are some minor grammatical errors in the paper, and it needs further proofreading. 

##############################################################

References:

[A] Are GANs Created Equal? A Large-Scale Study; Lucic et al.; NeurIPS 2018 

[B] A metric learning reality check, Musgrave et al., ECCV 2020


################################################################

After author response: The authors have addressed my comment about inconsistent evaluation setups among different papers. However, I sill think novelty of the paper is limited as it is a conditional counterpart of [A]. As mentioned by other reviewers, findings of the paper are quite incremental and are in line with LostGAN-v2 although the authors use a more consistent evaluation setup. 
Overall, I keep my current rating. 

review_title: The paper provides a systematic evaluation of scene generation methods, but there are some concerns regarding novelty and the proposed setup.

Meta-review: The paper received mixed reviews that overall lean negative. 

The main concern shared by reviewers is the novelty of the findings. Although the paper presents a systematic study that certainly has value, reviewers do not find sufficient insights from the analysis. The ACs agree with the reviewers that the paper is below the bar for acceptance. 
Decision: Reject