Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to utilize the retrieval image(s) for extending the model's ability for generating images. In this way, the model takes the text prompt as the input, by fusing with the noisy predicted by the retrieval images, it can generate the images that the dataset seldom contains. To balance the generation ability among text prompts and retrieved images, the paper also introduces a hand-crafted weighted scheme. Several experiments have been conducted to generate the efficiency of the method.
strength_and_weaknesses: Strengths:

+ The proposed idea of taking the power of retrieved images for generating better results is simple but effective. The effectiveness has also been demonstrated through the experiments.

+ I enjoy reading this paper, and it writes generally well, though with some typos (e.g., "on two dasets:").

+ This paper works on a recent hot topic, which is text-based image generation. The idea of customizing with more conditional images can push forward the development of this area. 

+ I appreciate the limitations 

Weaknesses:

+ Some parts of the model descriptions may not be clear. To my understanding, the Re-Imagen allows multiple retrieved images as inputs, but how exactly the model balances them or takes them as inputs? How is the model architecture designed? Does this mean the number of retrieved images should be the same as the training? What if the retrieved images share much diverse appearance, how the model handles such a case?

+ I feel the way of improving the classifier-free guidance is rather heuristic. Are there any insights that why the undesired imbalance happens?

+ Though the idea is effective, its underlying technical contribution is rather limited, as adding the retrieval images as another condition is straightforward for implementing the such retrieval-image-augment idea. 

+ As an image generation work, I would like to see much more visual results than the paper and the supplementary document provided. It is encouraged to release a project webpage containing more results for demonstrating the generality of the proposed method.

+ Will the model be released publicly? If not, how this work can contribute to the research community is encourag
clarity,_quality,_novelty_and_reproducibility: The paper writes clearly. Because of the simple idea, I believe it can be easily reimplemented. However, as far as I know, the behind diffusion model has not been released, which makes it hard for reproducibility and benefits the general research. The authors should justify how their works can contribute to general research without many computational resources properly.
summary_of_the_review: Generally speaking, the effective idea and good results make me lean toward acceptance. However, the rather hand-crafted designed combination way makes the method rather heuristic. The technical contribution is still limited. Thus, I can only put it a little bit over the boundary.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an approach to deal with the problem, that state-of-the-art models often have difficulty generating images of uncommon entities. The main contributions are 1) a novel retrieval-augmented text-to-image model Re-Imagen. 2)  interleaved classifier-free guidance. 3) a benchmark.
strength_and_weaknesses: **Strength**

1) The issue that this paper tried to solve is important and critical in nowadays text2image generation. The topic is meaningful.
2) This paper is well-written and easy to follow.
3) The qualitative results is compelling.

**Weaknesses**

1) Research topic definition changing.

The research topic is text-to-image generation, which is presumed to take text as input and image as output. However, though this work means to solve the problem of uncommon entities generation in text-to-image generation, it brings subtle but certainly important changing of the "research topic definition". An extra input of a set of knowledge base is needed for the inference of this model, which makes the problem to be like text (with knowledge base)-to-image generation, with more strong conditions. The changing of the definition mainly brings a problem that the usage bar for normal users is actually pulled up.

2) The comparisons on Table 1, Table 2 and Table 3 have some concern in fairness.

(a) For the comparion between Re-Imagen (=BM25; B=COCO; k=2) to Imagen on Table 1, (i) the # of Params are different, (ii) the training set is different (iii) the metric FID-30K and Zero-shot FID-30K is different, it seems to be arbitrary and misleading to say: “Re-Imagen (with the COCO database) can achieve a significant gain on FID-30K without fine- tuning: roughly a 2.0 absolute FID improvement over Imagen.”. 

(b) For the comparison on Table 2, the # of Params are different, it hard to deduce meaningful and solid conclusions. The factors should be disentangled to disclose the key insights and observations in the experiments.

(c) For the comparison on Table 3, the motivation why these three types of visual entities (dog breeds, landmarks, and foods) are picked may be clarified. A column of results of broader entities can be more convincing.



clarity,_quality,_novelty_and_reproducibility: 1. Will the code and dataset be public available? If yes, which parts of the code/data used will be released?
2. For the datasets that used in this work, what the licences are? Some concerns are about the validity of the large-scale datasets.
summary_of_the_review: This paper tried to solve an important problem, and the method is reasonable and works. The main concerns are in the fairness in the comparisons in experiments and the reimplementation of this method for both data and model.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a retrieval augmented diffusion model for the task of text to image synthesis. The idea is to condition the diffusion model on the external knowledge base in addition to the text embeddings that are used in text-to-image diffusion models. Conditioning on the retrieved nearest neighbor embeddings happens using the cross-attention layer. During inference, the model retrieves the image-text pairs corresponding to the input text prompt and then samples the result. Experiments show SOTA results on zero-shot FID on COCO, other benchmark datasets and human evaluation.
strength_and_weaknesses: Strengths:

The use of external knowledge base to augment the diffusion models is intuitive and a powerful idea. It is very hard to encode the entire world knowledge in a 2.5B parameter model, hence, these models are bound to fail in rare modalities. External knowledge base comes to rescue in this case.

The idea proposed by authors is simple, yet quite effective (as shown in  experimental studies). This model can also be finetuned from the original text-to-image model (Imagen in this case), which is something really nice. This way, a lot of compute can be saved.

The modification to the existing classifier free guidance to have optional conditioning on both nearest neighbor image features and text is also interesting.

The experimental results are strong. Both on zero-shot FID and human evaluation, the authors achieve SOTA.

Weakness:

I have a few comments, mainly with respect to some of the design choices made. I feel like the authors mainly mentioned these choices, and did not have a good discussion / justification.

On the choice of number of nearest neighbors during training: The authors just mention that they use k=2. But why just k=2? I can imagine that as k increases, the number of queries and keys in the cross attention would increase, hence the computational burden would go high. Is that the reason why k=2 was chosen? In a ideal case, performing an ablation on k would be good. But I can understand that ablations might be too expensive in these models. Hence, having a discussion on why this decision was made and what considerations were taking into account to end up in this decision would help the research community.
On the choice of number of nearest neighbors during inference: Should the same k be used during training and inference? While it might be true that performing an ablation on k during training might be infeasible, performing an ablation on k during inference might not be that hard. It would be great if authors had done that. K is an important parameter in retrieval, hence understanding the sensitivity of k might be beneficial.
Modified classifier-free guidance: The modification to the classifier free guidance makes sense to me. In the implementation, the authors randomly flip a coin and choose one of the two scores - text score and neighbor score in reverse diffusion. Why should this be done? Why not use a weighted combination of both scores. That is, why not use $\eta \hat{\eps}_{p} + (1-\eta)\hat{\eps}_{n}$ as the score? This might increase the NFE’s, but it might be worthwhile to study if this gives any gains. An ideal experiment would be to plot NFE-FID tradeoff for using this combined score vs using the proposed algorithm.
I would have liked it if authors provided more visualizations. One of the power of techniques like Imagen is the ability to image zero-shot content. For instance, “Corgi lives in a house made out of sushi”. But none of such visualizations is provided in this paper. I am wondering if this ability got worse due to the nearest neighbor finetuning because such samples are not present in retrieval dataset. COCO FID might not measure this as captions in COCO are not containing such novel texts. A discussion on this might be beneficial.  
A discussion on computational overhead: This is related to 1, but I am curious about what the additional computational overhead is because of the added attention tokens?
BM-25 seems to be a very simple retrieval algorithm. Will you get more gains by using CLIP text?

clarity,_quality,_novelty_and_reproducibility: Clarity: The ideas in the paper are reasonably well conveyed. However, as I pointed above, more discussions on design choices might have been nice.

Quality: This is a good quality work. Results are quite strong.

Novelty: The idea of using retrieval in diffusion models is not something new. There are many concurrent papers as authors point out. However, showing these impressive results at this scale is indeed a contribution, one that is valuable to the community.

Reproducibility: This is something I am a bit concerned about. Many of the ablations and design choice discussions are missing. The authors did provide other details, but adding some more detailed discussions would help with reproducibility.

summary_of_the_review: I think the paper is a good contribution to the community since it presents good results. However, as I pointed above, some of the ablations, discussions and visualizations are missing. Adding these would have made it a very strong submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This proposes a retrieval-augmented text-to-image model Re-Imagen for text and image guided image generation. The proposed method Re-Imagen interleaves classifier-free guidance during sampling to ensure both text alignment and entity fidelity and achieve good performances on the two datasets. This paper also introduces a more challenging benchmark EntityDrawBench for the text-to-image generation.
strength_and_weaknesses: Strength: 
1) This paper introduces an interesting viewpoint for text-to-image generation by involving the retrieval-based images during the generation procedure. 
2) The proposed methods achive good performances on the datasets and the newly-introduced benchmark may be helpful for the text-to-image task.

Weakness:
1) The comparison is not fair in the experiments. As for the most of the previous methods, the images are generated from the only text inputs. While, the proposed methods involve the images retrieved from texts by the datasets, which will make a big difference on the evaluation of the generated images, especially the FID scores. 
2) The novelty is limited. Although the retrieval-augmented methods achieve good performances and provide another viewpoint for image-guided text-to-image generation, the multiple conditions for diffusion models are not novel. The UNet architecture is also widely used in the diffusion models.
clarity,_quality,_novelty_and_reproducibility: Clarity: The clarity of this paper is good. The pipeline of the proposed methods is clear and the illustrations are pretty helpful for the readers although there are still lots of typos in the paper.

Quality:The completion of this paper is still good. However the comparisons in the experiments may be not fair. The experiment results especially the FID score cannot support the effectiveness  of the proposed methods.

Novelty: The novelty of this paper is limited. Although  this paper introduces a new challenging benchmark for text-to-image generaiton, the methods of  this paper are not very novel. The Interleaved Classifier-free Guidance seems an implementation trick for the multiple conditions for diffusion models.

Reproducibility: The clarify of the proposed methods are clear. It is possible to reproduce the performances of the proposed methods but it may be very challenging due to the lackness of the key codes and details.
summary_of_the_review: This paper introduces a novel retrieval-augmented text-to-image model and a more challenging benchmark for text-to-image generation. The retrieval-augmented methods may be helpful for the application for the text-to-image generation. However, the retrieved image involving in the image generation procedure makes it not fair to be compared with the previous methods which generates the images only from the texts. The experiements cannot support to prove the effectiveness of the proposed methods. Futhermore, the novelty of the proposed methods is limited. This paper is not good enough for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Paper Summary:
Authors present a modification over Imagen that provides additional context to the diffusion model in the form of images and text from a text-based search of a database. Approach yields SOTA on existing benchmarks, and authors propose a new benchmark that is stratified by frequency in the dataset.


Review Summary:

Pros:

- Simple but effective approach (T7pn, UC3H, oSTW)
- Well written (T7pn, xjWB)
- Popular topic (T7pn, xjWB)
- Well documented limitations (T7pn)
- Good results (xjWB, UC3H, oSTW)

Cons:
- Some descriptions of model architecture not clear (T7pn) -- Authors provided more details.
- Technical contribution / novelty is somewhat limited (T7pn, oSTW) -- Authors admit that while the idea is straightforward, the engineering required to get it to work well was not trivial.
- Needs more visual examples (T7pn) -- Authors added examples.
- Authors are changing the nature of the input by introducing a database, input is now text and database (xjWB, oSTW) -- Authors argue the inference speed is similar and the database is fixed and abstracted away from the user.
- Comparisons could be improved to be more direct/fair (xjWB) -- Authors added smaller model to the comparisons.
- Need more details on design choices (UC3H) -- Authors have responded with more information.

AC Recommendation: Accept. All reviewers except one lean accept because the paper is well written, is simple and effective, and posts good results and discussions. One lean reject reviewer felt the novelty was limited. However AC feels while technical complexity is low, this approach of creating a hybrid between a model and a database retrieval system has not been considered previously and it appears to be a novel perspective on this highly popular topic.

justification_for_why_not_higher_score: Some novelty concerns remain.
justification_for_why_not_lower_score: Results still hold value to community.
Decision: Accept: poster