Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tries to use an additional database D to do generation. The idea is very simple and straightforward. During training for a training image x, they use it to find the k images from the D by using CLIP image encoder and K nearest neighbor. Then they feed those retrieved image CLIP embeddings into their generative model as extra information.

Because of the shared space between text and image, once their model is trained they can do text2image generation and class conditional generation even though the model is trained unconditionally. 

They studied what is the best value of k, and propose a top-m sampling strategy for a trade-off between diversity and fidelity.  


 
strengths_and_weaknesses: Strengths:
Using an additional database to help generation. (I have a mixed feeling about this point as using an external bank of images is not a new philosophy in image generation)
The paper writing is very clear and easy to follow. 
They achieve state-of-the-art unconditional generation results on ImageNet with less trainable parameters.

Weakness:

It lacks novelty. The model architecture follows LDM. Conditioning on CLIP embedding to enable the power of text2image generation is similar to DALLE2, and should be attributed to CLIP.  

My main concerns/doubts are listed in the "Questions" section. 
questions: Here are my major concerns for this paper regarding the setting and high-level: 

1, The main claim of this paper is having an additional database. But, they do not justify why they want to have a disjoint D? What if D is the same as the training data? They should have shown what are the benefits of using different images as D. They have something similar in L289, but in this case k=1. They should properly study a model with k=4 when D is the same as training images. 

2, They do not show enough ablation studies on the database. For example, how would different databases influence the model's performance and behavior? Although they studied the patch size used in OpenImage in the supp, those database features are all from the OpenImage, thus still similar in my mind.

3,  Similarly as above, they should also study the effect of the same database on the different training sets. They only conducted the training on ImageNet, what about the other dataset such as the single domain dataset (FFHQ, LSUN)? In these cases, how would the current database (OpenImage) influence the models? Does the database still provide useful information?  If not, why does someone need the database? 

4, A lot of studies were conducted on text-to-image generation, zero-shot, etc.  But actually, the reason why they can do this is mainly because of CLIP, not because of the extra dataset. Thus I do think they present the paper appropriately. If they want to highlight these properties, then they should make their story based more on CLIP. If they still want to make their main story be the database, then they need to thoroughly study the concerns I listed above. 

5, What is the true limitation of this paper? Although they mentioned limitations in the supp, but I do not think they take it seriously. What they mentioned are general problems for all diffusion models such as sampling speed. I hope to see they talk about the limitation unique to their approach. 

Here are my major questions regarding the existing experiments and statements:

6, In table 1, they miss the LDM baseline.

7, In table 1, they should also list the database size, instead of only parameters. Since storage is also a main concern in a real-world application. They mentioned it in the footnote on page 4, but they should explicitly list it in table 1 to show the extra storage cost.

8, When talking about DALLE2, they said DALLE2 prior training requires paired data and they do not (L287), which I do not think is accurate. Since the CLIP training also needs paired data. Actually, I do not fully understand the purpose of L279-L295, if the goal of this part is to show their approach does not need to train a prior, then they need to show why a feature database is better than a prior trained on the features from the database.   


Minor concerns:

1, in L157, I am not sure if LDM is the first one to propose cross attention. It seems GLIDE also uses cross attention to fuse image and text features.

2, for fig2, they can do a user study instead of only show 4 images.  


 

 
 

limitations: My major question for this paper was they did not have enough study and analysis for training data and extra database, which is the key claim of their paper. Since they addressed them well in the rebuttal, I am happy to raise my score. 

I still have the same concern about the way they present the paper. The main claim of this paper is using retrieval to help generation, BUT, the other very important component in their approach is CLIP which I do not think is replaceable. In other words, if I replace CLIP with normal Resnet trained on Imagenet, I do not think one can play around with text anymore. But I can still call my paper as "Semi-Parametric Neural Image Synthesis" or my model as "RDM". Thus I think their title and story is a bit over claim or not precise if one prefer.     

In summary, the contribution of this work outweighs my concern. But I highly suggest to put their ablation on D and X conducted in the rebuttal to the main paper since these ablations backup their main claim. Also consider adding CLIP into the story when present the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes retrieval-augmented generative model, which is a semi-parametric model conditioned on retrieved features.
strengths_and_weaknesses: Strengths:

The proposed method is straight-forward and effective, good performances are achieved with smaller model size;

The paper is well-written and easy to follow;

The use of pre-trained CLIP model enables image generation, text-to-image generation and class-conditioned generation; Good experiment results are obtained on image generation task;


Weaknesses:

Some related works [1, 2] which also perform generation conditioned on CLIP features should be discussed and compared. Specifically, [1] also retrieve image features based on CLIP multi-modal joint feature space, and train the model based on retrieved features. The only difference is that [1] trains a diffusion model in a latent space;

The text-to-image generation results on MS-COCO dataset is worse than related works;

Diversity of generation is influenced

[1]. KNN-Diffusion: Image Generation via Large-Scale Retrieval. Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, Yaniv Taigman.

[2]. LAFITE: Towards Language-Free Training for Text-to-Image Generation. Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, Tong Sun.
questions: In Table 1, ADM and RDM perform sampling with different hyper-parameters. Some more results are suggested for fair comparison (e.g. ADM with 100 steps or RDM with 250 steps).

Inception Score on MS-COCO dataset for text-to-image generation is suggested to be reported.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper attempts to solve the problem of image synthesis (unconditional, conditional, text-guided) in a semi-parametric way. It extends IC-GAN [1] by introducing an external retrieval dataset (used for kNN search), and a pretrained fixed CLIP encoder (used for encoding image and text). The proposed method could be used on various image synthesis frameworks (e.g. Diffusion-based, Autoregressive-based models).

A fundamentally similar concurrent work is kNNDiffusion [2] (also mentioned by the authors in Appendix C)

[1] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance-conditioned gan. *Advances in Neural Information Processing Systems*, 34, 2021.

[2] Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. *arXiv preprint arXiv:2204.02849*, 2022.
strengths_and_weaknesses: Strengths

- The proposed method could be trained with images only, while allowing tasks including conditional, unconditional, text-guided image synthesis. This is achieved by aligning the latent space to a pretrained fixed CLIP encoder.
- Good experiments of introducing the idea of retrieval based image synthesis in diffusion models and autoregressive models.

Weaknesses

- One major difference the authors have mentioned against IC-GAN is the external retrieval dataset (as IC-GAN use the same retrieval and training dataset). How does changing retrieval dataset to a different one affect the results? Does the diversity matters or is it the dataset size matters? If we use the same training and retrieval dataset in this work, does the result differ a lot?
- Domain transfer by replacing the retrieval dataset is also presented in IC-GAN, what is the fundamental difference here?
- It seems that the quantitative experiments of text-to-image task on COCO, and class conditional synthesis on ImageNet against SOTAs are missing.
- In Figure 2, using text representation only performs better than using kNN samples and the combination of these two. Does this mean NN is not important for text-to-image synthesis? Then, what is the fundamental reason to introduce NN retrieval here.
questions: Please see weaknesses above. Overall, I think this paper is a good extension of IC-GAN by introducing the retrieval-based methods to diffusion and autoregressive models, along with the combination with CLIP. However, some ablation studies, quantitative experiments, and more in-depth analysis with the proposed method are missing. If the authors could address these, I am willing to raise my score.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method of using an external database of images to conditional a smaller generative image model for neural synthesis. Since the model is conditioned on the external database, some amount of domain transfer can be granted through changing the exemplars in the database. This external database also tackles a scalability problem, as retrieving nearest neighbor exemplars from the database is more efficient than trying to parameterize larger and larger datasets into a generative model.

Post Comments: Given the rebuttal of the authors addressing my concerns and an improvement of the ethics statement in the paper, I feel it's appropriate to increase my overall rating. I feel many of the timings per exemplar are a bit misleading, given that these are taken over large batched and amortized.
strengths_and_weaknesses: Strengths:
Use of an external database and retrieving nearest neighbor exemplars. Additionally, a database of images may not even be needed if you are pre-computing CLIP embeddings. This allows for a very efficient 2048 bits of representation per image.

With increasingly complex datasets, the semi-parametric models increase in recall performance (which is not true of the fully parametric baselines). 

Weaknesses:
Domain transfer isn't explicitly shown. For example, a database of natural images used for training and cartoon images for exemplars to generate a "cartoon tiger" or "animated bear" would show a domain transfer from natural to animation images. The domain transfer examples shown are essentially the same visual domain as the training and the exemplars.

FID isn't state of art compared to classifier guided ADM, even with about 80% of the parameters.


questions: 1. Figure 1 is very misleading. There is a missing graph discontinuity on the X axis for model params (e.g. 0 is at 325 M params). Visually this is showing that the semi parametric models are around 1/8th the size when in fact they are 2/3rds to ~5/6ths the size. Additionally, this graph shows a flattening of FID as the database increases. Is this true of a model whose |theta| + |D| is the same size as the ADM w/classifier baseline?

2. L56, retrieving the exemplars at inference time is 0.95 ms. Since these images also need to be CLIP encoded, what is the time or compute spent to encode |D|.

3. One of the claims of this paper is domain transfer being possible with a change of the image database. Where any visually distinct domain transfers experimented with (i.e. style transfer).

4. What does the distribution of training data look like for parametric and semi-parametric approaches? Do semi-parametric models need less training time or less examples?
limitations: As in all generative work, there is the danger of generating images that can be harmful (used to incite fear, riots, harassment). The authors mention no limitations or negative societal impact of their work which I believe is an oversight. I don't think this research has any increased risk over other generative works, in fact, with careful curation of the database, this may actually *decrease* the potential to generate harmful content. The authors should more carefully consider the societal implications of their work.
review_title: null

Meta-review: metareview: This paper tackles the general image synthesis problem (unconditional, conditional, text-guided) using a semi-parametric manner. It first retrieves relevant samples from external dataset, and use them as additional conditions for image generation. It is verified with different image synthesis frameworks, e.g. Diffusion-based and Autoregressive-based models. The comprehensive experiments demonstrate the effectiveness of the proposed semi-parametric image generation methods, compared with baselines.

The paper received all positive review rating scores after some discussions, leading to an ``Accept'' decision overall.
confidence: Certain
Decision: Accept