Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work proposes to represent individual visual relations in an image as unnormalized probabilistic densities following the scheme of Energy-Based Models, by which the modeling of each relation can be factorized and enables their composition to generate images. According to the reported experiments, this kind of relation description can help generate and edit images with multiple composed relations and understand semantically equivalent relational scene descriptions, which is also generalizable to unseen dataset during training. 
main_review: This paper employs the EBM to model composed relations, as a product of individual learned probability distributions for each relation description, thus offers more flexibility in generating images with multiple composed visual relations. Though the experiments were conducted upon synthetic and somewhat simple datasets, this approach shows certain potentials in understanding semantically equivalent relations, which is quite hard by previous methods and can be generalized to unseen scenarios.

Overall, this study is helpful to researchers in the fields of image generation and visual understanding. This paper is well written, clearly illustrated, and appropriately structured. 

In my opinion, the proposed method inherits the main merits from recent EBM-based image generation methods, that try to employ EBM to learn how to compose entities or other elements in the scene (Du et al. 2019, Du et al. 2020, etc.)  Thus this paper may be less significant if it looks like an extension to these prior arts (even the model architectures are similar to those from Du et al. 2021). I acknowledge that this paper has validates that the EBM models can compose visual relations in image generation, but a comprehensive discussion about the technical uniqueness of the proposed method would be encouraging.

I have a little bit of concern about the effectiveness of the proposed method since the testing scenarios were simple and a little bit far away from generating real scenes. 

Moreover, since the language parser converts a textual description into a series of relation descriptions, I am curious whether the language parser can disambiguate multiple entities from the same categories and attributes? And then whether the proposed method can generate multiple similar objects rather than one object, if the relation descriptions do not mark the identities of these objects.

The references require significant modification, about the format, missing items, duplicated entries, etc. 

If possible, adding algorithm snippets would help the readers to understand the training/testing pipeline. 
limitations_and_societal_impact: Yes, the authors have discussed the limitations and potential negative societal impact of this work in the section of conclusion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces an energy-based model to compose separate relations in a factorized manner.  A relational scene description is decomposed into individual relations, and modeled as the product of individual probability distributions across relations. The proposed method is shown to be able to generate images with good quality from the relational descriptions and understand the semantic differences between relational descriptions. Zero-shot evaluation also suggest that the proposed approach can generalize to unseen relational descriptions.
main_review: Overall, I think this paper conduct a comprehensive evaluation of the proposed method and demonstrate its power on several tasks: image generation, image editing, relational understanding. The model performance is strong, the proposed method outperforms StyleGAN2 for almost all cases both quantitatively and qualitatively. 

I would like to learn from the authors about the following questions. 

`Quantitative Comparison for Image Generation`

Is it a standard practice to perform binary classification on the generated images? How should we interpret the accuracy? Could a model somewhat trick this score? For example, in the qualitative analysis, "ours" seem to do better than the baselines, but the accuracy is lower.

In [1], the author proposed to use object detection metrics as the quantitative measure. And a graph-based relational similarity score is used to test the correct placement of objects, without requiring the model to draw the objects in the same exact locations. Would these more finegrained metric be better than a global match through a binary classifier between image and relational scene descriptions? 

Also, human evaluation on the generated images to compare different model performance might be a more convincing quantitative measure.

[1] Keep Drawing It: Iterative language-based image generation and editing: https://nips2018vigil.github.io/static/papers/accepted/13.pdf 

`Application to more realistic scenes`

Could the method be extended to more realistic scenes other than the artificial ones in CLVER and iGibson? 

I would imagine there are several challenges here. One is the lack of such detailed relation descriptions for real images, especially for complex scenes. In addition, the common practice to quantitatively evaluate image generation results for real scenes is FID or inception score.  Whether the proposed method can improve FID or inception score is unclear. Third, there will be more complex relation/attribute/object for real scenes, larger vocabulary size for the learned embedding. My guess is CLIP text encoder might perform better, as it has seen large scale image-text pairs that might cover a wide range of relation/attribute/object. I would be happy to hear authors' thoughts on this.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed image generation and image editing methods using the energy-based model for learning relational interaction. The proposed method can be summarized as : starting from N discrete relation and their features can be represented as a probability distribution function. This probability distribution is modeled using an energy-based model.  An energy-based model is an unnormalized probability distribution and trained with contrastive divergence. Monte Carlo simulation has been used iteratively to sample from the probability distribution. Afterward, a text encoder has been used to model relation triplet for every relation.

Authors have conducted experiments on two datasets Clever and iGibsion for multiple task categories.  In addition, a brief ablation experiment has been done for semantic equivalence and zero-shot generalization.
main_review: The paper is clearly written and easy to understand. The proposed energy-based model on image generation is relatively novel, but its application is somewhat limited due to the lack of global relation context.
limitations_and_societal_impact: There are a few limitations of the paper :
1. Experiment conducted on the synthetic dataset, this raises the doubt its performance on real relational dataset like visual genome.
2. How the interdependency of the relationship can be exploited using this framework or can be extended?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use EBM to model relations s in a factorized manner. The developed technique was applied to scene generation and experiments show that  a factorized decomposition allows more faithfully to both generate and edit scenes.
main_review: - The novelty of the paper is potentially limited. Basically [1] used EBM to compose objects, while this paper uses EMB to compose relations. However, there's no fundamental difficulty in the adaption. 
- Do the authors regard encoding the relation as a set (which I think is commonly used in knowledge graph) as the key innovation here? If not, I think it is necessary to add baselines such as (StyleGAN + set encoding). It is worth knowing how much improvement comes from changing encoding.
- Why is the baseline (StyleGAN + CLIP) better on iGibson compared with Clevr? It seems to me that iGibson has more complex scenes.
- The result looks pretty good, which accounts for my decision to suggest borderline accept.

[1] Du, Yilun, Shuang Li, and Igor Mordatch. "Compositional visual generation with energy based models." Advances in Neural Information Processing Systems 33 (2020): 6637-6647.

-------- post rebuttal update

I have checked the reviews and rebuttals of this submission. I lean towards acceptance. However, I still think the novelty of the submission is somewhat limited, so I keep my original rating as borderline accept.
limitations_and_societal_impact: Yes they were adequately discussed.

I don't think the readers can learn much from the method section - it was simply re-introducing the same EBM as on textbooks. On the other hand, it is not clear how these components are connected. I would suggest the authors to refurbish the writing a bit.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper demonstrates the potential usage of our models across tasks of compositional multi-object relational image generation, editing, and even generalization on unseen datasets. The proposed model represents each relation as an unnormalized density (an energy-based model), enabling it to compose separate relations in a factorized manner. The experimental results significantly outperform baseline approaches and even can generalize to a previously unseen relation description.

main_review: **Strengths:**
1) Overall, this paper is well written, and the technical details are easy to follow.
2) The main idea of a framework that factorizes and composes separate object relations is novel.
3) I found it very compelling that the approach can generalize to a previously unseen relation description.


**Weaknesses:**

1) **Motivation.** The authors mentioned in line 28, ‘’We posit that an issue with these models is that the language encoder is not compositional with respect to the underlying relations in an image, thus they fail to consider every single relation that describes a scene.’’ I would like to see evidence for that (citations or experiments that prove that this is the case). In general, I assume that there are language encoders that could contain enough compositionality, such as GPT3, and etc. 

2) **Related Work.** As the authors mentioned in the related work section, the proposed idea has similarities with scene graphs. In the scene graph-to-image task, we model a set of triplets (<subject, relation, object>) as a structure and generate from it an image/video. I would like to mention [1], which model set of triplets (with the notion of time) to generate videos given a structure. Additionally, [2] generates videos given a set of descriptions the same as the authors proposed. I believe they could be mentioned.

	[1] Compositional Video Synthesis with Action Graphs, ICML 2021.

	[2] Concept Grounding with Modular Action-Capsules in Semantic Video Prediction.

3) **Human eval.** In my opinion, Tables 1 and 2 are missing human evaluation. Since generation and semantic equivalence are challenging to examine with current metrics, I suggest performing human analysis.

4) **Relation Descriptions Vs. Scene Graphs (SGs).** I think it is interesting to understand the difference between relation descriptions-to-image and scene graphs-to-image tasks. Of course, both tasks aim to do the same, but it is still unclear which is a more appropriate direction for image generation, image editing, and generalization capabilities. Thus, I would expect to see a comparison of the proposed model and other existing SG-to-image methods.  


I raised some of my concerns above. Overall, I think it is an interesting paper that can be valuable to the community. I am open to the authors' feedback and other reviewers' opinions.


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper presents a quite novel framework to factorize and compose separate object relations, with the ability to generate and edit images with composed relations. The performance is significantly better than StyleGAN2 CLIP and encouraging. One of the limitations that reviewers are concerned about is that most of the experiments are conducted on synthetic data, and more analyses on real-world scenes are expected. All reviewers carefully read the authors' responses, have extensive discussions and agree on the novelty and explanation of this work. Considering the high quality of this paper and consistent positive comments by reviewers, The AC recommends accepting this paper. 


Decision: Accept (Spotlight)