Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method (Slot2Seq) to adapt the recent DALL-E (text-to-image) model to perform image-to-image composition. The aim is to simultaneously learn latent concepts from base images that can then apply to the generation process (as opposed to input text, which contains somewhat discretized "concepts" already). Experiments with 4 datasets (including a mirrored version of CLEVR) demonstrate that the Slot2Seq approach is effective for novel image generation from slots, reconstruction, and out-of-distribution generation.
main_review: STRENGTHS
- Identifies an area of opportunity between DALL-E (which has strong generation and imagination abilities, conditioned on explicit concepts) and Slot learning models (which can recover images based on slots but have limited imaginative ability) and addresses the gap with an image composition model
- Good exploration of the weaknesses of pixel-mixture decoders for image generation from slots, including its inability to leverage strong decoders and independence / lack of dependence between slots/concepts.
- The model demonstrates promise in out-of-distribution composition, and the experimental setup there is well founded.

WEAKNESSES
- Lacks detail on why CLEVR-Mirror was mirrored from CLEVR and how/why it was used in place of CLEVR.
- Would appreciate more clarity on architectural / modeling choices, such as the decision to use DVAE over VQ-VAE (which is used for ImageGPT).
- Stemming from above, this paper could also use a discussion of modeling ablations or comparisons to assess whether the slot2seq approach as a whole is responsible for the empirical gains in generation quality or if there is a strong dependence on architecture.
- It would be useful to discuss or experiment with imposing an order on the concept/slot prompt, as there may be an existing hierarchy when imagining an image (similar to the somewhat unspoken ordering exists for adjective ordering in English)
- Would like to see more than just comparison between mixture decoders and Slot2Seq - for example, [1] introduces energy-based models for composition, which seem to be applicable in this setting and would serve as a good point for contextualizing the overall impact of this work.


Additional References to include:
[1] Du, Yilun, Shuang Li, and Igor Mordatch. "Compositional visual generation with energy based models." Advances in Neural Information Processing Systems 33 (2020): 6637-6647.
summary_of_the_review: The paper introduces a simple way to perform image composition using slot attention and a GPT-based decoder in place of pixel-mixture decoders. The ablation study is promising, but I would like to see the inclusion of additional baselines and/or more ablation studies to properly contextualize this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a  model that uses visual prompts to generate images. Basically, visual prompts are interesting and inspiring. And compared with existing works, the proposed model shows better performance.
main_review: Strengths:

S1: The idea is inspiring and the visual prompts seem meaningful and interesting.
S2: The performance is fine based on the qualitative examples.

Weaknesses:

My major concern is that the experiments are simple. The datasets used in the paper are synthetic and easy to model. So I am wondering the performance on natural images, such as ImageNet, MSCOCO and Celeb1M, can the proposed model still learn meaningful prompts and what are the prompts for natural scenes? Another weakness is that the author does not conduct a human evaluation on the generated images, so it is difficult to judge how well the model performs.
summary_of_the_review: Basically, the idea in the paper is interesting, but the experiments should be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Illiterate DALL·E, a zero-shot image generation model inspired by slot attention and DALLE without text. The main idea is to leverage object-centric representations into image generation. The method demonstrates the zero-shot generation of novel images without text and better quality in generation than the models based on mixture decoders.

main_review: Strengths:
1. Overall, this paper is well written, and the technical details are easy to follow. 
2. The main idea of learning compositional slot-based representations with DALLE regardless of text is interesting.
3. The experiment results strongly support the benefits of the object-centric approach.

Weaknesses:

**Synthetic Data.** As shown in multiple works [1,2,3], the Slot Attention and similar methods are currently limited to toy data like moving 2D objects or very simple 3D scenes and generally fail at more realistic data with complex textures. The authors proposed to modify DALLE, which is a zero-shot image generation model that works well on natural images, to work on synthetic datasets. It is hard for me to see how this approach generalizes to real and more complex data.

[1] Multi-object representation learning with iterative variational inference, ICML 19.

[2] Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.

[3] Track, check, repeat: An EM approach to unsupervised tracking, arxiv 2021.


**Slots Initialization.** The authors mentioned the slot-decoding dilemma and the pixel-independence assumption that their model solves. But, as [1] and others suggested, using learnable query vectors instead of Gaussian-initialized slots helps the slots to learn on a unique embedding (similar to DETR). Thus, I am not sure the two mentioned problems above are related to the mixture decoder as the authors suggested but more to the slot encoding (and not the decoder). I would be happy to hear the author's thoughts about it.

[1] Self-supervised Video Object Segmentation by Motion Grouping, ICCV 2021


**Experiments.** The authors constantly compared to the Slot Attention model, but is it the correct comparison? Why not compare it to the DALLE, which does not use object-centric representations? Besides the fact that Slot Attention does not have an excellent image generator, the main point of this work is to show that adding compositional representations to DALLE improves the model (in contrast to adding a better decoder to the Slot Attention, unless I missed something).  

Additionally, few other object-centric approaches use object-centric representations for image generation and manipulation, which the authors are not comparing at all. For example, Scene-Graph-to-image methods [1,2,3] aim to add compositionality representations for image generation. It should be at least discussed why the authors approach is better than using a more structured approach for image generation.
	
[1] Learning Canonical Representations for Scene Graph to Image Generation, ECCV 2020.

[2] Specifying object attributes and relations in interactive scene generation, ICCV 19.

[3] Image Generation from Scene Graphs, CVPR 18.


**Novelty Clarification.** I am somehow confused about the novelty of the paper. Do the authors want to emphasize that adding a more robust decoder framework from DALLE improves the Slot Attention model, or adding the object-centric representations enhances DALLE generation capabilities? I am not sure what the authors want to highlight, but the latter is the more interesting research question to me.


**Relation to Prior Work.** As mentioned earlier, few object-centric approaches try adding compositional representations for image/video synthesis and manipulation, which the authors are not discussing. I believe they could have been mentioned in the Related Section. I am writing a few below.

[1] Learning Canonical Representations for Scene Graph to Image Generation, ECCV 2020.

[2] Compositional Video Synthesis with Action Graphs, ICML 2021.


**DALLE.** The authors mentioned DALLE as a model which already has compositionality capabilities (See “However, from the perspective of compositionality, this success is somewhat expected because the text prompt already brings the composable structure. That is, the text is already discretized into a sequence of concept modules.”) I can't entirely agree with the authors, DALLE lacks compositionality as the results suggested by the authors (See the example “a stack of 3 cubes. a red cube is on the top, sitting on a green cube. the green cube is in the middle, sitting on a blue cube. the blue cube is on the bottom” in https://openai.com/blog/dall-e/). It seems the Slot Attention has much better compositionality capabilities than DALLE.





summary_of_the_review: My main concerns are that I cannot see how the proposed approach can generalize to more realistic domains, and I am confused about the main paper's story. Overall, I like this paper and its contribution, but I think the authors should highlight and show that their new model improves DALLE and not the Slot Attention model since DALLE could potentially benefit object-centric information for zero-shot image generation in real-life datasets. Furthermore, since DALLE has shown vulnerability to compositionality, I would expect the proposed author's approach to resolving it, and thus I feel this paper misses an excellent opportunity to do it. 

I am open to the authors' feedback and other reviewers' opinions.


After Rebuttal
-----------------------------

After reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal. The rebuttal addresses most of my concerns. I am leaning towards acceptance of the paper. I vote for 6.
review_title: null

Meta-review: The paper addresses the problem of generating images by combining visual components. These components are learned during pretraining, forming a dictionary of visual concpets which plays the role of text in DALLE. The technique is based on DALLE and slot attention approach to generate VQ codes in a way that is consistent.

Reviewers had various concerns, including (1) that using synthetic images makes it easier to combine visual components' (2) that the novelty and relation to literature was not clear enough (3) missing ablations.  The authors provided a detailed rebuttal which addressed reviewer concerns in a convincing way. 

One remaining issue of the paper is the writing. The paper fails to clearly explain the workflow (what are input and output during pretraining, training and inference), and how compositionality is controlled (what can be used for conditioning). As a consequence, it requires substantial effort to understand the idea of the paper, and what real problems can be solved with the proposed approach . 

The paper can be accepted to ICLR, but it is expected that the writing would be improved. The abstract and introduction should make concrete statements about what the approach does, what problems it solves and how it can be used for the various tasks as disucussed in the experiments
Decision: Accept (Poster)