Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to apply retrieval in text-to-image generation based on an encoder-decoder architecture. K-nearest neighbors of some image database is computed on top of the encoder output, and a transformer-based diffusion network is applied on these neighbors to produce the final output image.

The main idea of the paper is that by leveraging the retrieval process, smaller training dataset can be used, and even out-of-distribution image generation is possible by choosing appropriate database from which to draw the k-nearest neighbors.

In settings controlled for backbones author show significant improvements on common image generation metrics.
strength_and_weaknesses: Strengths
- [S1] The application of retrieval is interesting and tackling limited training data is a practical problem.
- [S2] The controlled settings show strong performance.

Weaknesses
- [W1] Benchmark seems weak. Not a lot of baselines, and main results are only zero-shot but it's not clear that this is the focus of the paper.
- [W2] Some drawbacks of retrieval-based methods are not well discussed. For example, the choice of k is highly dataset specific, how would someone select this in practice? The proposed method also do not scale well to larger dataset if every query involves a retrieval against the whole dataset.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. 
Quality is okay, it could use more experiments. 
The method is not novel but the application is pretty interesting.
summary_of_the_review: I recommend weak accept. The experimentation could be stronger, but the idea and the application is quite interesting. The problem this paper tackles is relevant.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for generating images from text with an addition of a retrieval component. Authors use pretrained text and image encoders and train diffusion-based generative models that use an image embedding and a set of image embeddings closest to the target one to reconstruct the image. During inference, instead of the image embedding, a text embedding is used (which comes from the frozen clip encoder, pretrained on image-text pairs). Authors demonstrate that the retrieval component is crucial for image quality both in online and in human studies and also extend the functionality for global image manipulation, which looks rather convincing.
strength_and_weaknesses: Strengths:
1) The paper presentation is overall good. Illustrations make sense, most of the ideas are clearly expressed.
2) The results look good, both FID and user studies. I was a bit surprised that having just embeddings as input (without the access to raw images) is enough to make a big difference. I guess the neighbors provide the hint of how far the query is from the real images, so it allows the model to prefer directions to the real embeddings.

Weaknesses:
1) A big red flag for me is the claim that no text pairs were used during the training. To reproduce the model from data one would have to pretrain the encoders on the paired data, so this claim is misleading. I highly recommend to put it into correct context to avoid reader confusion.
2) There are some details missing. E.g. I could not find the value of K during training, which should help interpret Fig. 9 (maybe the optimal number of neighbors is just the train one?)
3) Not clear whether the solution is going to be open-sourced.


Misc: Sometimes there is kNN in the text/figures, sometimes it is KNN, sometimes knn. Sometimes it is k, sometimes K, sometimes it is kNN. It also makes sense to explicitly state that Image quality and Text-alignment are the human evaluation metrics (e.g. it is mentioned for table 2, but not for table 1). 
clarity,_quality,_novelty_and_reproducibility: From novelty point of view, it seems that most of the building blocks of the solution were known before (the retrieval part, the generation part). Maybe the image manipulation method stands out; at least I never saw a similar idea. 

I also would recommend discussing paper on semi-parametric image generation which used raw pixels as input, like Qi, Xiaojuan, et al. "Semi-parametric image synthesis." CVPR2018 or Iskakov, Karim. "Semi-parametric image inpainting." arXiv preprint arXiv:1807.02855 (2018). 
summary_of_the_review: Overall, I think this is a decent paper. I my main criticism is about claim "training ... model without any text" which might very misleading. I believe it has to be fixed.

Update after the discussion - my comments were address during the rebuttal period, so I updating the rating (given the code will be open-sourced).
Also I suggest the authors to update Figure 3 - the kNN part there is hard to understand (both what it does and what is the output)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to improve the training performance and overall quality of the recently introduced text-to-image generative framework. By leveraging a kNN-based retrieval method, the proposed method 1) adds more flexibility to the training data, 2) greatly reduces the computational cost. Extensive experiments and ablation studies are conducted to prove the efficacy and the efficiency of the model. 
The description of the technical approach is confusing and vague. In Section 3, the authors firstly criticize the use of CLIP embedding since it alone cannot accurately bridge the gap between the text and image distributions. However, the authors still choose to use the CLIP pretrained model to do the job without any modifications. Without any proper justifications on how the proposed retrieval model could solve the problem, the practical motivation of the paper is lacking.
strength_and_weaknesses: Strengths:
1.	The paper is generally well-written and easy to follow.
2.	The paper introduces an easy approach to facilitate the training of text-free text-to-image generation.
3.	The experiments are comprehensive, it demonstrates the strength of the proposed method from different perspectives.
Weakness:
1.	The technical novelty might be limited, its model is conceptually similar to several previous works (e.g., CLIP, CogView, and LAFITE). The main contribution lies in the training strategy (kNN Retrieval) rather than in the techniques.
2.	The Retrieval Model (Section 3) is described very vaguely. The provided details are not sufficient to understand the module and re-implement it.
3.	It would be also helpful to release the code and pre-trained model for subsequent research.
clarity,_quality,_novelty_and_reproducibility: Given the limited motivation and technical novelty of the work, I don’t think the current version meets the standard of the ICLR. 
summary_of_the_review: This is a good paper with promising experimental results, with some aspects of the presentation that should be improved and clarified. Also, I have the following questions regarding to the paper.

Questions to Authors:
1.	Could the authors provide more justifications on how could adding the retrieval part bridge the gap between the text and image distributions?
2.	How does the encoders map text descriptions and image samples to a joint feature space? Do you merely use the pretrained models?
3.	The subsection of “Image generation network” also lacks sufficient details. How does the generation network generate new images with conditioned on other similar instances? The equations in the subsection don’t indicate any signals that this is a conditional generation framework. 
4.	How does the kNN search avoid the problem of “curse of dimensionality”?
5.	Without any details of the parameter setting and providing model details, the experimental merit is also hard to evaluate.
6.	The scalability of the model is highlighted as one of the primary contributions. However, the corresponding scalability analysis experiment is not found in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to train a conditional diffusion models by removing the necessary of annotation pairs. More detailed, they utilize the kNN retrieved image embedding as condition signal, and can further utilize caption embedding due to the almost-perfect joint distribution between text and images.  The framework also facilitates the generation of out-of-distribution sampling. One extra bonus is they can remove the necessity of mask annotation when conducting local semantic manipulations. 

Experimental results demonstrate effectiveness of the proposed method on zero-shot text-to-image generation, the metric is FID and user study around image quality and text-alignment.
strength_and_weaknesses: Pros:

-  The method framework is well motivated to sample from the out-of-distribution images.
-  Competitive results compared with several baselines.
-  Related works are quite thorough for new beginners.

Cons:

- Writing is a little rush, the preliminary knowledge of diffusion model is missing, which makes the method part is very unfriendly to novice of this area.  

- Why KNN can solve text-to-image models annotation problem in abstract? It seems that your method is condition on retrieved features instead of text/caption.
- Figure 3 is too rush, what is MMMM, 9143? I am totally missed about what you are trying to express in this figure. The data flow is full of confusion.
- Evaluation details are missing, how to generate condition in your sampling process? Where do those conditioned text/image come from?  Where is the reference set from? How many samples are used to calculate the FID in Table 1 and 2? 
- Training and network details are missing, how many parameters in your method, is the parameter number almost the same between your method and the proposed baselines. How many GPUs did you use? Learning rate? optimizer? batch size? epoch number? training time? weight decay? 


Minor comments:

- The full comparison is available in the supplement.
- The dataset intro should go to experiments part.
- Figure 6,9 is saved as vector image, pdf.
- The font size of Figure 6 is too small.


clarity,_quality,_novelty_and_reproducibility: Quality: 8/10

Clarity: 8/10

Novelty: 5/10

Reproducibility: 8/10
summary_of_the_review: The motivation is well consolidated, while too many details about training/inference/preliminary knowledge are missing. Also, the paper is not easy to follow. I suggest the authors to spend more time about the writing of this paper.

=======

post-review update:

The paper has been improved a lot based on my concerns about description, preliminary background, training details, etc. I think the quality of current version is enough for ICLR. I also share similar concern about the marginal novelty of the proposed methods with other reviewers.  Since advantages outweigh disadvantages, I upgraded my result.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an interesting approach for using a set of K nearest embeddings for text-to-image generation using diffusion models. The basic rationale is that CLIP image embedding may not be well aligned with CLIP text embeddings. So, for text-to-image generation, rather than specifying a conditioning input using text embedding of the input prompt, one can use K nearest image embeddings as input to the diffusion model. This paper bridges the gap between fully parametric and non-parametric models and shows that KNN diffusion models can achieve interesting results using small models. 

Pros:
- An interesting approach has been proposed for introducing non-parametric components into text-to-image diffusion models.
- The paper archives empirically strong results using small models

Cons:
- Most reviewers have found many missing details in the original submission (ranging from evaluation protocol to training and architecture details). Some of these details are added during the rebuttal period. 
- Writing and presentation could be improved.
justification_for_why_not_higher_score: Overall the reviewers agree that this paper presents an interesting approach for bridging between parametric and non-parameteric approaches for text-to-image diffusion models. The experimental results support the claims and present surprising strong results at times. However, given the concerns regarding the clarity of writing and missing experimental details, the reviewers recommended the poster acceptance. 

Final note: The reviewers in the AC-reviewer meeting brought up a potentially concurrent work [1] that also builds on top of retrieval ideas in diffusion models. However, given the concurrency of this work, the AC and reviewers decided not to penalize the current submission for novelty threats from the concurrent works. **However, they all agree that the related work [1] should be discussed in details in the related work section of this submission.** The authors are strongly encouraged to discuss this work and contrast their method against it in the camera-ready version.

[1] Semi-Parametric Neural Image Synthesis Blattman et al., https://arxiv.org/abs/2204.11824v3
justification_for_why_not_lower_score: Given the potential impact of KNN diffusion models, the reviewers and AC believe that the paper should be given a chance to be presented at ICLR.
Decision: Accept: poster