Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Image which is a text-to-image diffusion model. The major finding of this work is that using large language models pretrained only on text data as text encoders are effective for text-to-image generation and can benefit from the scaling power of language models. Dynamic thresholding and Efficient U-Net architecture are proposed to improve the training effectiveness and efficiency of the diffusion model. SOTA experimental results are achieved on COCO FID and the proposed DrawBench benchmark.
strengths_and_weaknesses: Strengths:

* The paper is well written and presented.
* The major finding of this work is that using large language models pretrained only on text instead of trained on paired image-text data only on text data is insightful.
* The experiment results in generating photorealistic with text alignment are amazingly impressive. 

Weaknesses:
* The technique contributions of this paper are limited. The proposed dynamic thresholding heuristic and U-Net architecture are somehow technically incremental. However, dynamic thresholding seems to be very effective for training the diffusion model. If some theoretical justification can be included, I will increase my rating.

* It is not clear if we should pursue the text language encoding models or paired image-text models if we have enough image and text data. In another word, if we have enough paired image-text data in the future, can the paired image-text encoding models outperform the text language encoding models.
questions: Please refer to the weaknesses for my questions.
limitations: The limitations are well-addressed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Summary:

The paper proposes a text to image diffusion model (Imagen) that generates high fidelity images that accurately match the prompts. The work uses powerful language models (T5-XXL) that have been trained on text corpora that eventually aids in improved language understanding of the final model. Additionally, the paper makes modifications to the existing diffusion models by having dynamic thresholding along with some architectural changes to the U-Net to make it more efficient. With this work, Imagen becomes the state-of-the-art on MS-COCO based on the FID scores surpassing the models that are trained on it. The authors further introduce a benchmark, DrawBench, to evaluate the quality and accuracy of the image synthesis for a battery of challenging prompts.


Strengths:
- The paper makes an important contribution by using the models trained on surplus text corpora (unpaired data) rather than learning from purely paired image-text data. Their experiments also confirm that scaling the pretrained text encoders improves the image generations.
- Imagen model is one of its kind to achieve unprecedented zero-shot results on MSCOCO on the FID metric – surpassing all the existing models that are directly trained on the dataset. 
- The paper invents a plethora of ‘tricks’ that facilitated image generations. It includes Efficient U-Net, having dynamic thresholding, text cross-attention layers in the super resolution model, and noise conditioning augmentation. It will be interesting to see if some of these tricks (or principles) can be used for improved generative modeling beyond the scope of the paper.
- The paper introduces DrawBench that aids in evaluating the quality of the model generations across various dimensions. It provides us with a way to systematically compare complementary text to image generative models along these dimensions. 
- It was nice to see that the paper uses Human evaluations to compare Imagen with existing models using fidelity and alignment scores. In my experience, CLIPScore might not be a very reliable metric to judge the image-text alignment every time. 

Suggestions and Weaknesses:

- [1] showed that scaling the data and the model are important factors in improving performance of zero shot models. The authors show that scaling the model size in terms of text encoder is indeed important for improved performance, however, it might be worthwhile to test how well scaling laws hold in the data dimension. Along the same lines, the Table 1 of the paper compares zero-shot models with different model size and training data. For a fairer comparison of how well the architectural choice is doing, it might be good to compare with an equivalent version of Imagen in terms of model parameters and/or training dataset size.
- Interestingly, the paper shows that Imagen achieves lower preference rate while generating people when compared to the set with no people. It is not clear if the model is finding it difficult to draw people because it has seen a lot of images without people as compared to the images with people in the training dataset or is it ‘inherently’ difficult to create photorealistic people with this architecture? How does Imagen rate when compared to the existing text to image models on the quality of images with and without people? 


Typos/Edits:
- Figure 4’s y-axis labels are inconsistent FID-10K vs FID@10K
- Line 338 - wrote ‘Imagen’ two times

Missing references:
- Technically, DALLE-mini also leverages the power of a BART encoder pretrained on text-only corpora. The authors should include it in their related work and discuss it a bit.

*References*

[1] Combined Scaling for Open-Vocabulary Image Classification: https://arxiv.org/pdf/2111.10050.pdf

[2] DALLE-mini: https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-Explained--Vmlldzo4NjIxODA#our-dall-e-model-architecture

strengths_and_weaknesses: Mentioned in the main review
questions: Mentioned in the main review
limitations: Mentioned in the main review
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper uses large transformer language models and diffusion models for text-conditioned image generation. The image generation results are impressive, achieving the state-of-the-art FID results on COCO. This paper uses a dynamic thresholding technique to sample images. This technique achieves better results based on the claims. This paper also builds a new benchmark, DrawBench, for text-to-image tasks.
strengths_and_weaknesses: (+) This paper combines transformer language models and diffusion models for image generation. Even though the techniques are not novel, the image generation results are impressive. 

(+) This paper observes that using larger transformer language models improves the image generation results significantly.

(+) This paper proposes a new benchmark for evaluating text-to-image tasks.

(-) The novelty is limited. The method is a simple combination of transformer language models and diffusion models.

(-) Using large models and massive data always seems helpful for better results and zero-shot generalization. This has become an obvious "fact" based on recent papers, such as DALLE [1], and CLIP[2]. However, purely having good results without solid methodology contribution is hard to be accepted.

[1] Learning Transferable Visual Models From Natural Language Supervision 
[2] Hierarchical Text-Conditional Image Generation with CLIP Latents





questions: The results are impressive, but many text-conditioned image generation works, such as DALLE-2, Parti (just come out), also have good performance. Most of these models use massive data and large models. So it seems like the improvements come from the usage of massive data and large models, which are usually not released. 

This paper is a combination of existing techniques, such as language models and diffusion models. 
Could the authors explain, except using the massive data and large models, what is the technical contribution of this paper and what is the novelty?

This paper would have much greater impact if it could distinguish itself from other recent/concurrent works. What makes it different from DALLE-2 and how do those differences influence the model's performance and use cases?

Will the authors release their code or data for supporting research in this area?

Is it ok to have such a large figure in the second page. It seems do not fit the NeurIPS template. 
limitations: This paper claimed their limitations and potential negative societal impact.

(-) This paper has solid results to show the effectiveness of the proposed method. However, the good results are most likely from using large data and large models. This paper simply combines transformer language models and diffusion models, which is not novel. The dynamic thresholding technique is more likely a trick for training models.

(-) This is a good paper with impressive results, but it seems the novelty is limited and not a good match for NeurIPS.
review_title: null

Meta-review: metareview: This paper proposes Imagen that uses large transformer language models and diffusion models for text-to-image generation. The major finding is that using large language models pretrained only on text data as text encoders are effective. Dynamic thresholding and Efficient U-Net architecture are proposed to improve the training effectiveness and efficiency of the diffusion model. 

It received scores of 578. All the reviewers agree that the image generation results are impressive, and the zero-shot results on COCO are strong. This paper also proposes a new benchmark for comprehensively evaluating text-to-image tasks. On the other hand, Reviewer 9tx5 pointed out that one major concern is that the novelty is quite limited.

Overall, the AC thinks that the paper presented impressive results and has great significance, therefore, the AC would like to recommend acceptance of the paper.
confidence: Certain
Decision: Accept