kNN-Diffusion: Image Generation via Large-Scale Retrieval
Authors: Shelly Sheynin
Oron Ashual
Adam Polyak
Uriel Singer
Oran Gafni
Eliya Nachmani
Yaniv Taigman
Venue: ICLR 2023 poster (2023)

Abstract: Recent text-to-image models have achieved impressive results. However, since they require large-scale datasets of text-image pairs, it is impractical to train them on new domains where data is scarce or not labeled.
In this work, we propose using large-scale retrieval methods, in particular, efficient k-Nearest-Neighbors (kNN), which offers novel capabilities: (1) training a substantially small and efficient text-to-image diffusion model using only pre-trained multi-modal embeddings, but without an explicit text-image dataset, (2) generating out-of-distribution images by simply swapping the retrieval database at inference time, and (3) performing text-driven local semantic manipulations while preserving object identity. To demonstrate the robustness of our method, we apply our kNN approach on two state-of-the-art diffusion backbones, and show results on several different datasets. As evaluated by human studies and automatic metrics, our method achieves state-of-the-art results compared to existing approaches that train text-to-image generation models using images-only dataset.

Published as a conference paper at ICLR 2023
KNN-DIFFUSION: IMAGE GENERATION VIA LARGE- SCALE RETRIEVAL
Shelly Sheynin', Oron Ashual',
Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, Yaniv Taigman Equul Contribution
Meta AT
she lysheynin, oron) @meta. com ccm
Fooees high fiving'
'Hapoy magical wizard
skateboarding
'Octopus playings
Original
'With bow tie!
'Clown'
"Office chinchilla'
grummryyy cat eating ramenr
leddy bear in space
A superhero girl
FFuur eves'
'Anary'
'In love'
Figure (a) Samples of stickers generated from text inputs, (b) Semantic text-guided manipulations applied to the "Original" image without using edit masks.
ABSTRACT
Recent rext-to-image models have achieved impressive results. However, since they require large-scale datasets of text-image pairs, it is impractical to train them new domains where data is scarce using large-scale retrieval methods, in particular, efficient d-Nearest-Neighbor (kNN), which offers novel capabilities: (I) training a substantially small and ef- ficient text-to-image diffusion model using only pre-trained multi-modal embed- dings, but without an explicit text-image dataset, (2) generating out-of-distribution images by simply swapping the retrieval database at inference time, and (3) per- forming text-driven local semantic manipulations while preserving object identity. To demonstrate the robustness of our method, we apply our kNN approach two state-of ofe--ff-theee art diffusion backbones, and show results on several different datasets. As evaluated by human studies and automatic metrics, method achieves state- of-the-art results compared to existing approaches that train text-to-image gener- ation models using images-only dataset.
labeled. In this work, propose
INTRODUCTION
Large-scale generative models have been applied successfully to image generation tasks (Gafni al., 2022 Ramesh et al.) NichoNicchll
Nichol et al.) [2021; Saharia et al. 20222] Yu et al. (202), and have shown extending human creativity using editing and user control. However, these models face several significant challenges: (i) Large-scale paired data requirement. To achieve high-quality results, text-to-image models rely heavily on large-scale datasets of (text, im- age) pairs collected from the internet. Due to the requirement of paired data, these models cannot be applied to new or customized domains with only unannotated images. (ii) Computational cost and efficiency. Training these models highly complex distributions of natural images usually requires scaling the size the model, data, batch-size, and training time, which makes them challenging to train and less accessible the community. Recently, several works proposed text-to-image models
outstanding capabilities

---

Published as a conference paper at ICLR 2023
trained without an explicit paired text-image datasets. Liu al. (2021) performed a direct opti- mization to a pre-trained model based on a CLIP loss (Radford et al. (021)) Such approaches are
time- time-consuming, since they require optimization for each imput. Zhou et al.
202I proposed train-
ing with CLIP image embedding perturbed with Gaussian noise. However, to achieve high-quality
results, an additional model needs to be trained with an annotated text-image pairs dataset.
In this work, we introduce a novel generative model, ANN-Diffusion, which tackles these issues and progresses towards more accessible models for the research community and other users. Our model leverages a large-scale retrieval method, k-Nearest-Neighhort (kNN) search, in order to train the model without an explicit text-image dataset. Specifically, our diffusion model is conditioned on two inputs: (1) image embedding (at training time) or text embedding (at inference), extracted using pre-trained CLIP encoder, and (2) KNN embeddings, representing the l most similar images in the CLIP latent space. During training, we assume that no paired text is available, hence condition only on CLIP image embedding and on & additional image embeddings, selected using the retrieval model. At inference, only text inputs are given, so instead of image embeddings, we use the text embedding that shares a joint embedding space with the image embeddings. Here, the KNN image embeddings are retrieved using the text embeddings.
The additional KNN embeddings have three main benefits: (1) they extend the distribution of con- ditioning embeddings and ensure the distribution is similar in train and inference, thus helping to bridge the gap between the image and text embedding distributions (see Fig. 5 (2) they teach the model learn to generate images from target distribution by using samples from that distribution. This allows generalizing to different distributions at test time and generating cut-of-distribution sam- ples; (3) they hold information that does not need to be present the model, which allows it to be substantially smaller. We demonstrate the effectiveness of our KNN approach in Sec. T
To assess the performance of our method, we train our model on two large-scale datasets: the Public Multimodal Dataset (Singh et al. [202]) and an image-only stickers dataset collected from the Inter- net. We show state-of-the-an art zero-shot results on MS-COCO (Lin et al.] 2014), LN-COCO (Pont- Tuset et al. 2020) and CUB (Wah et al. 2011). To further demonstrate the advantage of retrieval methods text-to-imant image generation, we train two diffusion backbones using our kNN approach: continuous (Ramesh et al., 2022) and discrete (Gu et al., 2021]. In both cases we outperform the model trained without KNN. In comparison to alternative methods presented in Sec. we achieve state-of woff-the-att results in both human evaluations and FID score, with only 400 million parameters and 7 seconds inference time.
Lastly, we introduce new approach for local and semantic manipulations that is based on CLIP and KNN, without relying on user user-provided masks. Specifically, we fine-tune our model to perform local and complex modifications that satisfies given target text prompt. For example, given the teddy bear's image in Fig. and the target text "holds a heart", our method automatically locates the local region that should be modified and synthesizes a high-resolution manipulated image in which (1) the teddy bear's identity is accurately preserved and (2) the manipulation is aligned with the target text. We demonstrate our qualitative advantage by comparing our results with two ofte-of-theean models, Text2Live (Bar- Tal et al. 2022) and Textual Inversion (Gal et al. (022), that perform image manipulations without masks (Fig.
and 22).
We summarize the contributions of this paper as follows: (1) efficient model that utilizes a large-scale retrieval method for training text-to-image model with only pre-trained multi-modal embeddings, but without an explicit text-image dataset. (2) We demon- strate efficient -ntor--distribution generation, which is achieved by substituting retrieval databases. (3) We present new approach for local and semantic image manipulation, without utilizing masks. (4) We evaluate our method on two diffusion backbones, discrete and continuous, as well as on several datasets, and present state-of-the-e art results compared to baselines.
We propose KNN-Diffusion, novel and
RELATED WORK
Text-to-image models. Text-to-image generation is a well-studied task that focuses gener- ating images from text descriptions. While GANs (Xu et al.) 2018 Zhu et al.] [019 Zhang et al.)
2021 and Transformer- based methods (Ramesh et al. [022 Gafni al. 2022] Yu al. 2022 Ding
et al. 2021) have shown remarkable results, recently impressive results have been attained with dis-
2

---

Published as a conference paper at ICLR 2023
'A bell dag looks the camera while wearing huc"
"Blckk and white cat is sitting the of book shelf very close the TV
KChosts playing haskechalll
'House inside of volcano"
'Boses bloooiing on cloud!
sainbow colored apple
"mage of white ziraffe!
Nearest Naighbor
LARITE
FuseDream
Ours
Figure 2: Qualitative comparisons with baselines. Neighbor is the first kNN of the text in PMD dataset.
crete Gu et al.
2021 and continuous (Nichol et al. 2021 Saharia et al.
2022 Ramesh et al. 2022
Rombach et al.
2022) diffusion models. Most recent works trained diffusion models conditioned
on text embeddings extracted using pre-trained text encoder (Saharia et al. 2022 Yu et al. 2022) or image embedding extracted using CLIP (Ramesh et al., 2022p. While producing impressive re- sults, all previous works described above are supervised and trained with paired text-image datasets. Several works have proposed training text-to-image models without an explicit text-image dataset. FuseDream (Liu et al. 2021) proposed direct optimization to a pre-trained generative model based on CLIP loss. This method relies on a pre-trained GAN and requires a time-consuming optimization process for each image. LAFITE (Zhou et al.] (02T recently demonstrated text-to-image gener- ation results without requiring paired text-image datasets. Here, the CLIP embeddings are used interchangeably at train and test condition GAN-based model. The joint text-image embedding enables inference given a text input, whereas in training the model is fed with the visual embed- ding only. However, the gap between the text and image distributions the joint embeddings space leads to results with substantially lower quality, as we show in our experiments. To overcome this gap, LAFITE added noise to the image embeddings during training. Our remedy to this gap is to condition the model on the retrieval of an actual image embeddings, using text-image joint space.
Retrieval for generation. The Information Retrieval (IR) literature tackles the challenge of re- trieving a small amount of information from large database, given a user's query. A simple, yet efficient retrieval mechanism is to retrieve the K nearest neighbors (kNN) between the query and the entities in the database in some pre-calculated embedding space (Bijalwan et al. 2014). The database allows the model to leverage extensive world-knowledge for its specific task Borgeaud et al. (202I). Recently, language models were augmented with a memory component, allowing them to store representations of past inputs (Wu et al., 2022). The latter were then queried using a lookup operation, improving performance in various benchmarks and tasks. Retrieval models have been used for various tasks in learning problems, for example, language modeling (Borgeaud et al.) 2021), machine translation (Gu et 2018),, question answering (Lee et al. 2019) and image gen- cration (Tengg et al. 20200 Qi et al.) 2018). RetrieveGAN (Tseng et al. 2020) uses differentiable retrieval module for image generation from a scene description, ketrievalFuss (Siddiqui
2021) proposed a neural 3D scene reconstruction based on retrieval system. SIMS TQi et al. (2018) pro- posed generating an image using semantic layout and compatible image segments that are retrieved from image segments database, and (Iskakov) 2018) showed that the use of retrieval database in inpainting task significantly boosts visual quality. this work we utilize the kNN retrieval meeh- anism over the shared text-image embedding space, CLIP [Radford et al.) [222I). Using extensive ablation studies, we show the importance of the retrieval model both for training and inference, and demonstrate its large impact performance. KNN-Diffusion significantly outperforms prior work
3

---

Published as a conference paper at ICLR 2023
Train
Inference
Image encoder
KNN retrieval (Embedding space)
Text encoder
with pink hat"
Image embedding
Text embedding
KNN image embeddings
Decoder
Figure 3: The overall framework of our ENN-Diffusion model. In both training and inference, the decoder is conditioned on CLIP embedding, and kNN image embeddings. During training, we condition the model on image CLIP embedding,
its KNN image embeddings extracted using retrieval method. At inference time, given an input text, the KNN image embeddings are retrieved based on the CLIP text embedding that shares a joint embedding space with the image embedding.
with zero-shot FID of 12.5, including RDM (Blattmann al. 2022) FID of 22.1). concurrent work which similarly to our approach, proposes conditioning LDM (Rombach et al. 2022) on kNN.
Multi-modal feature learning. Learning joint and aligned feature space for several modalities challenging, as requires alignment between the modalities (paired datasets), whose distribu- tions may vary. Specifically, the joint feature space of vision-and-language has been long-standing problem. CLIP (Radford et al. 2021) successfully tackled this by leveraging contrastive learning over large dataset of text-image pairs. BLIP (Li al 2022). (Mu et al. 2021) and FLAVA (Singh et al. 2021). followed this idea and further improved the joint representation. The joint represen- tation shown to hold strong semantic alignment between the two modalities, enabling image generation (Liu et al., [022 Wang et al.) 2022). image manipulation (Patashnik et al.] 2021 Avra- hami et al. 2022b), and image captioning (Mokady et al.) (021). In this work leverage the joint representation in two ways: (i) enabling textless training with only visual data, while using text at
inference time, and (ii) creating an efficient embedding space for the use of the retrieval model.
METHOD
Our main goal is to facilitate language-guided generation of user-specified concepts while using images-only dataset during training. A possible way to achieve this goal is to use a shared text- image encoder that will map text-image pairs into the same latent space, thus allowing training with candidate for this encoder is CLIP,
an image embedding, and inferring from text embedding. which has been trained with a contrastive loss on large-scale dataset of text-image pairs. However, as we show quantitatively in Tab. I and qualitatively in Fig. 15 cannot accurately bridge the gap between the text and image distri bistributions.
16 5 CLIP embeddings alone order to reduce this gap, several methods have been proposed. The closest work to ours is LAFITE, which perturbs the CLIP image embedding with adaptive Gaussian noise. Under the assumption that there large paired text-image dataset, |Ramesh et al. (2022) have proposed prior that is used during inference, and is trained generate possible CL.PP image embeddings from a given text caption. In this regard, propose using large-scale and non-trainable image embedding index as an integral part of the diffusion process. Our method, ANN-Diffuusion assumes that only image data and a pre-trained multi-modal text-image encoder provided during training. As shown in Fig. 3] comprised of three main components: (1) multi-modal text-image encoder (CLIP); P) (2) A retrieval model A data structure containing image embeddings, which is indexed for fast kNN search; (3) An image generation network- A trainable diffusion-based image generation model, conditioned on the projected retrievals. For both training and inference, the image generation network conditioned on K additional image embeddings, chosen using the retrieval model to ensure similar distribution of the condition in training and inference. The following sections describe these components.
model is
Retrieval model. Our retrieval model has three non non-trainable modules: a pre-trained text en- coder fext (CLIP text encoder), pre-trained image encoder fimn (CIIP image encoder) and
4

---

Published as a conference paper at ICLR 2023
Ours
Text2LiVE
Inversion
ginal
Joker
Bower
Angel
Rainbow
Devil
Ghost
Ghost
Ghost
Or giral
Black shirt
Blue pants
Holds a heart
Sitting
Witha
Raising left hand
Raisinng left hand
Raising left hand
Original
With ahat
Sad
Surprised
Angry
Holds a heart
With tie
With Lie
With lie
Figure 4: Results for text-guided image manipulations without using masks. The original image is shown in the left column, our manipulated images shown in the center. The images of [Bar-Tal al. (2022); Gal et al. (2022) were generated using the authors' official code. The full comparison is available the supplement.
an index H. The encoders map text descriptions and image samples to a joint multi-modal d- dimensional feature space R4_ The index stores an efficient representation of the images database H (Semg (i) ii Σ} where denotes the dataset of images. During training, we the index to efficiently extract the & nearest neighbors in the feature space of the image embed- ding famg(1) (1) E _d knning (I, k) == arg minkew ((fimg(1) h) where is a distance function and arg min output the minimal & elements. The set {Simg (!), knning (I, k) } is used as the condition to the generative model. During inference, given a query text 4, an embedding St (1) is extracted. The generative model is conditioned
this embedding and its k nearest neighbors from the database knnear (f, k) arg minkew s(feaz (f), h). During training, we add embeddings of real images, by applying the retrieval method to the input image embedding. The extracted kNN should have large enough distribution to cover the potential text embedding. During inference, the kNN are retrieved using the text embedding (See Fig. [7]. In all our experiments we use the cosine similarity metric as the distance function s, k 10 for the number of nearest neighbors and d 512. The full implementation details can be found in Sec. 6.6 in the supplement.
Image generation network. In order to demonstrate the robustness of our method, apply KNN approach on two different diffusion backbones: Discrete (Gu et al. 202T) and Continuous
(Nichol al. 2021 Sohl-Diickttiin al. 2015 Ho al., 2020 Dhariwal & Nichol 2021). Al-
though very different in practice, these models share the same theoretical idea. Let Zo ~ ((zo) be
Markov chain that adds noise sample from our images distribution. A forward diffusion process
at each step ( (1) The reverse process, Po ( xu), is a denoising process that removes noise from an initialized noise state. At inference time, the model can generate an output, start- ing with noise and gradually removing it using pg. For additional background on diffusion models please refer to Sec. in the supplement.
In the discrete diffusion model, g( (/=n))/nn1)) == (x,n),,((Znn))) where v(rn) is one-hot vec- tor with entry 1 at In, and Q,, is a transition matrix, modeling the probability to move from state In-1 to Zn. using uniform probability over the vocabulary and pre-defined probability for ad- ditional special /MASK] token. We can compute the reverse transition distribution according to:
pe(xx-1/Zn y) := Ek. ((1 [Z,, (e æn, vo.y) y) where To is a discrete vector, tokenized by the VQGAN (Esser et al. (2021) encoder and y is the conditioning signal. For modeling pe we have followed (Gu et al. 2021) and used a conditional Transformer (Vaswani et al. 2017).
On ).o) and (e v) = ((Ha(Znn y). Ep(xn. v)). Here, the noise function is Gaussian noise. Fol- lowing (Ho et al. 2020 Nichol et al. (021) we trained a model eg to predict the added noise using standard mean-squares error loss: L : 2,o
In the continuous diffusion model, q(
:= (a Va,,n-1. 1,
[||e ea Ea(#n, n,, =||²] where ea
is a U-net model and y is the conditioning signal.
5

---

Published as a conference paper at ICLR 2023
Thst ambending
Mear tod and
Moan of bodt and NM Image anbeaddnng
Mean-el teal Inape enoodding
Vean text 20
ISME distioonion visualization
similrrty
cosine
0.05
0.08
Figure 5: iSNE visualization of 500 random text-image CLIP embeddings pairs taken from COCO validation. The leftmost figure demonstrates the gap hetween the text and image distributions. By gradually adding kNN the mean CL.IP embedding of the text, the gap decreases, demonstrating the importance of the kNN.
Table 1: Results for zero-shot Text-to-Image generation on the MS-COCO, CUB and LN- COCO test sets. Image- quality and Text-alignment report the percentage of majority human raters votes favor of our method when comparing between certain model and ours.
DALLE
Label
KSN
Paired
zero-shot FiD
30
Image-only
25
AupcDrsam
vi Deltiinn 20
15
Fors 353
MS-COCO
CUB Im. qual.
LN-COCO
10 Ours low
Txt qual. align.
Im.
Txt align.
Im. Txt
Pari
Model
FID4
FID
FIDI
qual. align.
irager
Number of Parameters (Billions)
LAFITE
26.9 72.1 65.3
89.7
74.0
59.6
42.8
68.4 61.9
FuseDream
21.2 64.0
79.3
50.2
79.1
60.9
37.5
71.1 59.0
Figure 6: FID on MS-COCO, including models trained image-only datasets and text-image datasets.
no-KNN
32.8 70.8
68.3
95.1
81.0
61.2
65.0
61.4 59.8
Ours
12.5
42.9
35.6
In both cases, we condition our model on y = (fimg (xa), knnssn (Zg. k)) where fimg (20) is the CLIP image embedding, knnjing k) is the k: nearest neighbors in the feature space of the image embedding. Following (Ramesh et al. 2022; Roobacc et al.. 2022) conditional injection, we con- dition our model on the image CLIP embedding, and the KNN clip embeddings by applying cross attention in the attention layers of the architecture. We sample both our models using Classifier Free Guidance (CFG) (Nichll et al. 102 Ho & Salimans [02II. Since CFG was originally proposed for continuous models, we propose a method for using it with discrete models as well. Full im- plementation details of the discrete and continuous models can be found in Sec. 6.7 and Sec. 6.8 respectively, in the supplement.
3.1 TEXT-ONLY IMAGE MANIPULATION
The majority of previous works the task of image manipulation either rely on user-provided
masks (Nichol et al.] 2021 Avrahami et al., 2022b ab. or limited to global editing (Crowson
cet al., 2022 Kim et al. (2022). Recently, several works (Bar-Tal et al., 2022 Hertz et al., 2022] Gal et al. 2022) have made progress with local manipulations without relying user edited masks. Nevertheless, most of the techniques suffer from several shortcomings: (1) They enable local texture
changes, yet cannot modify complex structures, (2) they struggle to preserve the identity of the object, for example, when manipulating humans, (3) they require optimization for each input.
We address these issues by extending KNN-Diffusion to perform local and semantic-aware image manipulations without any provided mask. Illustration of the approach is provided in Fig. 18 and Fig. [9 in the supplement. For this task, the model is trained predict the original image from a manipulated version. Specifically. we create a manipulated version of the image, which differs from the original image only in some local area. Given a random local area M in the image the manipulated image Imanip is constructed by replacing the area with the corresponding nearest neighbor: Imanip = I. (1 - M) + namm (I, 1) M, where nn.m, (I. 1) is the the nearest neighbor obtained after aligning it with I using the BCC alignment algorithm (Evangelidis & Psarakis 2008). The model then receives as input the manipulated image, together with the CLIP embedding of the original image only in the local area: famg (I M). This CLIP embedding represents the required modification that should be applied to the manipulated image in order to predict the original image. During inference, instead of using the CL.IP embedding of the local area, the desired modification is

---

Published as a conference paper at ICLR 2023
Table 2: Results on the stickers dataset. We report the percentage of human raters prefer our method over the baselines with respect to image quality and text aligmment. Discrete no-KNN refers to VO-diffusion. and Continuous no-KNN, to DALL-E2 decoder, both trained without an explicit text-image dataset.
Ours Discrete
Ours Continuous
Model
FID
Image quality
Text alignment
Image quality Text alignment
Cppp-ap
55.5
71.6
69.2
67.0
68.3
LAFITE
58.7
63.5
59.9
76.0
71.2
no-KNN
52.7
72.1
67.6
66.8
69.4
Ours
40.8
represented using the CLIP embedding of the text query. We modified the model to be capable of receiving as a condition both the manipulated image and the CLIP embedding of the local area.
4 EXPERIMENTS
First, we conduct qualitative and quantitative comparisons on MS-COCO, LN-COCO and CUB datasets. To further demonstrate the advantage of our method, we provide comparison on an image- only stickers dataset, where we apply our approach on two diffusion backbones. Next, we demon- strate image manipulation and out-of-distribution capabilities. Finally, to better assess the effect of each contribution, an ablation study is provided.
Datasets and Metrics. For photo-realistic experiments, our model was trained only on the im- ages (omitting the text) of modified version of the Public Multimodal Dataset (PMD) used by FLAVA (Singh et al. 2021). More information about the dataset is available in See. 6.44 of the sup- plement. To further demonstrate the capabilities of our method, we collected 400 million sticker images from the web, containing combinations of concepts such as objects, characters/avatars and text. The collected stickers do not have paired text, and are substantially different from photo- realistic data. Furthermore, since they have no paired text, they were not part of CLIP's training data, which makes the text-to-image generation task more challenging.
Evaluation metrics are based on objective and subjective metrics: (i) FID (Heusel et al. 2017 is an objective metric used to assess the quality of synthesized images, (ii) human eval evaluation we ask hu- man raters for their preference, comparing two methods based on image quality and text alignment. We used 600 image pairs; five raters rated each pair. The results are shown as a percentage of ma- jority votes in favor of our method over the baselines. We report the full human evaluation protocol in the supplement. We chose to omit Inception- -ccore, since it is shown by Barratt & Sharma (2018) to be misleading metric for models that were not trained on Imagenet.
4.1 QUALITATIVE AND QUANTITATIVE RESULTS
We begin by comparing our model, trained on the PMD dataset, with the previous works LAFITE and FuseDream, that trained on image-only datasets. To demonstrate the advantage of using a retrieval method in text-to-image generation, we trained a model variant, no-ANN. This baseline was trained solcly on image embeddings (omitting the kNN), while during inference, the images were generated using the text embedding. Tab. T displays zero-shot results on three different datasets: MS-COCO, CUB and LN-COCO. We follow the evaluation protocol of LAFITE, reporting our results 30,000 images from MS-COCO validation set without training, nor using it's training partition in the KNN index. Similarly, we follow LAFITE for CUB and LN-COCO evaluation. As can be seen, our model achieves the lowest FID score in all scenarios. In addition, human evaluations rate our method as better aligned to text and with the highest images quality. In Fig. 2 T5 and [T] we present a qualitative comparison between the methods. One can observe that while the simple retrieval baseline outputs non non-generated images with high-quality, the images generated by our method are more faithful to the input text. To further demonstrate the effectiveness of our method, we present in Fig. G| comparison of our model with the latest text-to-image models trained on paired text-image datasets: DALL-E, Cog View, VQ-Difffusio,, GLIDE, LDM, Make-A-Scena DALL-E2, Parti and Imagen. As can be seen, our model achieves comparable results to recent models trained with full text-image pairs (e.g LDM, GLIDE), despite being trained on an image- only dataset, with significantly lower computational costs. The results demonstrate that leveraging an external retrieval database allows to compensate for different trade-offs, in particular, reducing the number of parameters in the model. Additional samples are provided in Fig. 13 in the supplement.
7

---

Published as a conference paper at ICLR 2023
SA friendhrraodd walarorka
"The world tree lanor Dorgupts
gatewny
NNw York City Roodod bycliiate clange Chiike
Scapaer
"The prttww ny to the nocm deatthy Zreesaww Bebsirs(
Aesthetic
Unaesthetic
Image search engine
Stickers
Figure Comparison between various indexes used by the same model. Aesthetic. Images from the first quantile of an aesthetic classifier, (2) Unaesthetic. Images from the last quantile of an aesthetic classifier, (3) Image search engine. Images retrieved from Google Images, (4) The stickers index.
Text-to-sticker generation. As the sticker dataset does not have paired text, and is substantially different from photo-realistic data, it allows to illustrate the advantage of our model on image- only dataset. A selection of stickers generated by our model is presented in Fig. [ and Fig. [4] [2 To demonstrate the importance of using kNN image-only datasets, evaluate two diffusion backbones. To this end, we trained a continuous diffusion model (Ramesh et al.) 2022) and discrete diffusion model (Gu al., 202I), both conditioned the kNN image em- beddings. For each backbone, we compare our method with the following baselines: (1) no-kNN- this baseline was trained using both the continuous and the discrete methods conditioned only on image CLIP embedding, without using KNN. In the discrete case, we trained a VO-diffusion model, while the continuous case, we trained a re-implementation of DALL-E2's decoder (without prior). (2) DALL-E2+ DALL-E22Clippaap baseline here, we first captioned the entire sticker dataset using Clip- Cap (Mokady et al., [202T), then trained DALL-E2 decoder on the captioned dataset. (3) LAFITE we trained LAFITE language- free model on our stickers dataset using the authors' published code. We present the results Tab, 2 The FID calculated over a subset of 3, 000 stickers, generated from the ClipCap captioned dataset. As be seen, model achieves the lowest FID score. In addition, it outperforms all baselines in human evaluation comparison. using continuous and discrete backbones. In particular, compared with the same model trained without kNN, model achieves significantly higher favorability in both text alignment and image quality.
approach
4.2 APPLICATIONS
Text-only image manipulation. We demonstrate the manipulation capabilities of our model in Fig. 4 and 20 Furthermore, we qualitatively compare our model with Text2LIVE. (Bar-Tal et al.) 2022] and Textual Inversion (Gal al. 2022), using the authors' published code. Text2L.VE pro- edit layer that is composed over the original input, using generator trained for each training image. Textual Inversion utilized the pre-trained Latent Diffusion model to invert the input image into token embedding. The embedding is then used to compose novel textual queries for the generative model. Fig. 4 shows representative results, and the rest are included in Fig. 21 and 22 in the supplement. In contrast to model, baseline methods lack text correspondence or they do not preserve the identity of the object. Since Tex12LiVE is optimized to perform local changes, it has the difficulty changing the structure of the object (e.g. the "raising his hand" example in Fig. 4). Textual Inversion baseline changes the identity of the object because it struggles recon- structing the textual representation of the source image. Our model, on the other hand, can perform challenging manipulations that are aligned with the text, while preserving the object identity.
posed generating
8

---

Published as a conference paper at ICLR 2023
Poi
OL
2Z
33-
344
10
23
94
ARM
15
23
with MSH
430
4.21
2.55
6.00
3330
300
Figure 8: Mean aesthetics score of the generated images as fune- tion of the conditioned kNN mean aesthetics score.
Figure 9: MS-COCO test FID score various K's in: (1) Zero-Shot (2) Index includes AS-COCO train sub- set. No KNN trained with KNN, but did not employ KNN in inference.
Figure 10: MS-COCO test FID score for different model sizes. As can be seen, adding kNN to the model allows it to be smaller, while having better performance.
oitt---disttiittionn generation. Using the retrieval index part of the generation process en- ables using different databases during inference, without fine-tuning. This allows generatig images from distributions that were not part of the training set, enabling out-of-distribution generation. This novel capability is demonstrated with the same model trained on PMD, using three different retrieval databases: (i) A stickers database presented in Sec. 4 (ii) Aesthetic database: This database is con- structed by filtering images according to a classifier score. Let C' be a classifier that for each image i I outputs a score
C(i). This classifier enables filtering the kNN using L S s < H, where L and H are low and high thresholds, respectively. Here, we use an open source pre-trained aesthetics classifier A (Christoph Schuhmann 2022): y For each text input & T, we apply A on the kNN, and then divide the KNN into five equal quantiles based on A score. As can be seen in Fig. X using KNN with higher aesthetics score result in generated images with higher aesthetics mean score. (iii) Image search engine: Generative models are stationary in the sense that they are unable to learn new concepts after being trained, hence fine-tuming is required to represent new styles and concepts. online image search engine, which allows the model to adapt to new data without additional fine-tuning. A qualitative comparison of all three methods is shown in Fig 7
Here,
4.3 ABLATION STUDY
We conclude our experiments with an ablation study, quantify the contribution of our different components. We provide ablation study on index size and different kNN conditioning approaches in 665)of the supplement. Number of nearest neighbors. The results Fig. demonstrate the importance of applying the retrieval mechanism during training and inference. Here, we evaluate our model, trained on PMD dataset, with different numbers of kNN during inference. Furthermore, we examined the baseline no-kNN, in which during inference, the model is conditioned only on the text embedding (t), without using kNN. Best performance is achieved using 10 neighbors. Scalability analysis. To evaluate the effectiveness of our approach different model sizes, we trained three additional models with varying sizes for both settings with and without KNN. As can be seen in Fig. TO utilizing kNN consistently improves performance for all sizes. Furthermore, performance improvement can be achieved using much smaller models with KNN. For example, the 35A KNN model outperforms the 400M model without KNN.
CONCLUSION
"We shall always find, that every idea which we examine is copied from a similar impression" ". [Hume (1788). In this paper, we propose using a large-scale retrieval method in order to train a novel text- to-image model, with only pre-trained multi-modal embeddings, but without an explicit text-image dataset. Our extensive experiments demonstrate that using an external knowledge-bal base alleviates much of the model's burden of learning novel concepts, enabling the use of a relatively small model. In addition, it provides the model the capability of learning to adapt to new samples, which it only observes during test time. Lastly, we present a new technique utilizing the retrieval method for text- driven semantic manipulations without user-pror esoviddo masks. As evaluated by human studies and automatic metrics, our method is significantly preferable to the baselines in terms of image quality and text alignment.

---

Published as a conference paper at ICLR 2023
REFERENCES
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint urXiv:2206.02777 02779, 2022a.
Omri Avrahami, Dani Lischinski, and Ohad Fried, Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022b.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1r01v:6607.06450 2016.
Artem Babenko and Victor Lempitsky. The inverted multi-index. IEEE transactions on pattern analysis and machine intelligence, 37(6):1247-1260 2014.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. arXiv preprint arXiv:2204.02491, 2022.
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv:1801.0197 2018.
Vishwanath Bijalwan, Vinay Kumar, Pinki Kumari, and Jordan Pascual. Knn based machine learning approach for text and document mining. International Journal of Database Theory and Applica- tion, 7(1):61-70, 2014.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björ Ommer. Semi- parametric neural image synthesis. In Advances in Neural Information Processing Systems, 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George van den Driessche, Jean-Ban -antistt Lespiau, Bogdan Damoc, Aidan Clark, et al. Im- proving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.044. 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Romain Beaumont Christoph Schuhmann. Aesthetic predictor. httpss://iithu/ com/
LAION-AT COOON-AAaaestiin wwt.a///tttttttreeeelieeeoo predictor 2022.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204. 2000::2200008888,,, 2022.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text
data created by the people, for the people. In NewrIPS Datasets and Benchmarks, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint 1oXiv:2/05.05233. 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to- rext-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 12873-12883, 2021.
Georgios Evangelidis and Emmanouil 7. Psarakis. Parametric image alignment using enhanced correlation coefficient maximization. IEEE transactions on pattern analysis and machine intelli- gence, (0(10):18 858-1865, 2008.
10

---

Published as a conference paper at ICLR 2023
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.00 01611,, 2022.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2013.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided neural machine translation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Bain- ing Guo. Vector quantized diffusion model for text-to-image synthesis. ArXiv, abs/21 14.1222, 2021.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint 0rX1622008.06666, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 01220066.12200 11239, 2020.
David Hume. An enquiry concerning human understanding, 1748.
Karim Iskakov. Semi-parametric image inpainting. arXiv preprint arXiv:1807.02855. 2018.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization nearest neighbor search. IEEE transactions on pallern analysis and machine intelligence, 33(1):117-128 2010.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-5444 2019.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2426-2435, 2022.
Ivan Krasin, Tom Duerig, Neil Alldrin, Andreas Veit, Sami Abu-El-Haija, Serge Belongie, David Cai, Zheyun Feng, Vittorio Ferrari, and Victor Gomes. Openimages: public dataset for large- scale multi-label and multi-class image classification., 01 2016.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016.
URL
org/ abs/ 1602 07332
Kenton Lee, Ming-Wai Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv: wwXiv:/906.000 19960000000 2019.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified lasion-language understanding and generation. arXiv preprint arXiv:2201. 12086, 2022.
11

---

Published as a conference paper at ICLR 2023
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer viston, pp. 740-755. Springer, 2014.
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fuse- dream: Training-free text-to-image generation with improved clip+ gan space optimization. arXiv preprint arXiv:21 2021.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipeap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0973: 2021.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets
language-image pre-training. arXiv preprint arXiv:21 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishknn, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2112.10911 2021.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 1m2text: Describing images using 1 mil- lion captioned photographs. In J. Shawe- Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran As- http://pss...pppppp//////1//
Zemel, P. Bartlett, F. Pereira, and K. Q-
sociates, Inc., 2011, URL @dd9db5e033dam 7tteee//5aaaa11////e/aee-29-00
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 2085-2094, 2021.
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connect- ing vision and language with localized narratives. In European Conference on Computer Vision, pp. 647-664. Springer, 2020.
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi Semi-parametric image synthesis. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 8808- 8816, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. [tthtttss: http://:///.xw./////////// hbtp:/bb//110 /ttbs//1333 00020
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-eo-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 1487, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565, 2018.
12

---

Published as a conference paper at ICLR 2023
Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Re- trievalfuse: Neural 3d scene reconstruction with a database. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 125777 2021.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela, Flava: A foundational language and vision alignment model. arXiv preprint arXiv:21 12..044222. 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265 PMLR, 2015.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceed- ings of the 44th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, pp. 2443-2449, 2021.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. The new data and new challenges in multimedia research. CoRR, abs/1503.01817, 2015. URL http::t////axii.orrrr wwtp:///g/aab/aaoooeooeee 111./11113111111111 01817
Hung-Yu Yu Tseng, Hsin- Ying Lee, Lu Jiang, Ming-Hsuan Yang. and Weilong Yang. Retrievegan: Image synthesis via differentiable patch retrieval. In European Conference on Computer Vision, pp. 242-257. Springer, 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEF/CVF International Confer- ence on Computer Vision, pp. 1905-1914, 2021.
Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. Clip-gen: Language-free training of text-to-image generator with clip. arXiv preprint arXiv:2203.00338 2022.
Yuhuai Wu, Markus N Rabe, Delesley Hutchins, and Christian Szegedy. Memorizing transformers.
arXiv preprint Xiv:2203300811 2022.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1316-1324, 2018
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206.1078) 2022.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for eex-ttoimmae generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 833 842, 2021.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. LAFITE: towards language-free training for text-to-image generation. CoRR, abs/2111.Jt 11.1111.3779. 2021. URL htthtt::: //tp://aaavvvv//aa http//awwavvaaaaaaaaaa//eaa
orggabs/////ooooooorroggor/...er abs/ /21111111111 13792
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative ad- versarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 5802-5810, 2019.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
black cat with beanie on it's head
Novazell
bike on the stoet with lot of snow
hoto
This is living room cabin
wiree
luiio and fork toporee with
Aadd agpornsd sichee steed blaad
vodding
was
VQ
Ours
FuseDream
Cog View
Lañiee
NN
GT
Diffusion
Figure 11: Samples from COCO validation
14

---

Published as a conference paper at ICLR 2023
Chicken waiter serving
Virtual reality
Monkey eats hamburger
Scared fish an suit
Clown unicorn
Good Night
Goodnight sleep
Alpaca space
Gargoyle a party
Cauliflower crying
wearing VR headset
J
Hot headed
Painterly pokemon playing synthesizer
avatar
Music band made fruits
impressionist painting
confused
F
f
Panda ing guitar
doing ballet
rendering playing basketball
Singing
Muscle riding wave in Hawaii
Dog programming computer
Radha Krishna dancing in garden
Shark wearing birthday
Avocado playing ukulele and apple is watching
Cinematic Ilama with lighting
BUNS $10
Vhne monu
dog hotdog costume
Unicorn with rainbow horn, waving standing on grass
Celebrating frog
Alien using VR
words
Figure 12: A selection of stickers generated using the continuous ANN-Diffusion model.
15

---

Published as a conference paper at ICLR 2023
A brown shiny rose flower
A dog using typewniter
Aobbot tanning
Fuchsia iguanodon
A white finetruck
Race driver futu
Blue ants
Lucifer danccng with Jesus
Pineapple shaped refr gerator
surfer wearing three-piece men's suit
Rocking chair on water
Allen cartoon
Kitchen from the atlanta ntintis
pink watermelon
Image of grey tiger
Green robot vacuum
baby cooking spaghetti
Flying
Baby pictures gransparents
Figure 13: Additional samples generated from challenging text inputs using the photo-realistic model
16

---

Published as a conference paper at ICLR 2023
Thes tomato favorite of high-cless chefs
Bald headed mimes
Vomit candy
Alln carloon
Mf i
Squirrel wearing
parda carrying grocery bags
people
wearing T-shirt
1el S kas
bavloté
Singing eggplants
Penguin drives
Queen Esther
One legged striped rabbit
seser
Chihuahua pulling coach
mushroom
laughing purple parcupine
Dollar combing
Black mop head
Elaphant sitting on lon
Monkey eeting pickle
asting oatmeal
Figure 14: selection of stickers generated using the discrcte KNN-Diffusion model.
17

---

Published as a conference paper at ICLR 2023
6.1 BACKGROUND
Continuous diffusion process Diffusion models are latent variable models that aim to model distribution (2n) that approximates the data distribution ((ra). Specifically, they model forward process the space of Ty from data to noise. Given a sample from the data distribution To ~ g(2o), this process produces Markov chain of latent variables .1...,,TT by progressively adding Gaussion noise to the sample:
q (x-1) == (ma; / 1. 3,I)
(1)
where 3g is variance schedule. As presented previously by (Ho et al. 2020), the latent variable zt be expressed directly as linear combination of noise and zo:
Ft varzo + -ve- T e~ ~~,,
where at == -I (1- order to sample from the data distribution q(ro), we define the "reverse process" p(r;-1 (x,) which samples first from g(zr) and then samples reverse steps - (2,, until
Tip
Since the data distribution is unknown, we need to train model to approximate it. Note that when T large enough, the noise vector ær nearly follows an isotropic Gaussian distribution. This suggests learning model (o (xx) to predict mean μ사 and covariance matrix Eg:
To train this model, po( (1 t), Es (x1,,t))
replace ( by predicting the noise cs(2,, added zu using equation 2 and we get this objective function:
L == E,w,11,T), (00- ||| ep c0( 4, =
4))
where is an optional conditioning signal (such as text/image embedding or low resolution image).
Discrete diffusion process Let Fn. E fl..... hxw be the indices of the allocated codebook vectors extracted by pre-trained VQGAN (Esser et al., (021) encoder. The forward process of diffusion model g( (4n-1) 1) is a Markov chain that adds noise at each step. Moreover, the reverse process ( [n, 3o ). is denoising process that removes noise from initialized noise state.As presented by (Gu al. 202T). forward diffusion process is given by:
(5) where v(xn) is one-hot vector with entry 1 at Fa, and Q is the probability transition matrix from state Fn-1 to Fn.
q( xn-1) = v (zn QQ, (In-1)
The reverse process is given by the posterior distribution:
(1In-1/En mo) (u (In, Q,, Q,,)),,,K,,, ((x (2n-))) ((n=1))))1( v(zo))
(6)
of (xn v(00)
where Q =,, ......Q. Q
Inspired from mask language modeling (Devlin el al. 2018), they proposes corrupting the tokens by stochastically masking some of them. Specifically, additional special token [MASK] is pro- posed, so for each token there are (V+1) discrete states. The transition matrix is formulated as, By adding small amount of unifrom noise to the categorial distribution, the transition matrix can be formulated as,
Onn +
8n
Bn
3
3n
an +
3n
B
0
Q,
8n
Bn
an
3n
0
Yn
7n
7n
where On E [0, 1], 3, (1 an 7n /n//V and 7n the probability of token be replaced with [MASK] token. Each token has a probability of in to be replaced by the [MASK] token, V V to be resampled uniformly and on V %n) to be unchanged.
18

---

Published as a conference paper at ICLR 2023
6.2 ADDITIONAL SAMPLES
In Fig. 16 and 15 we present a visual comparison of our discrete model, trained on the stickers dataset with (1) the kNN extracted during inference, (2) the same model without using kNN in infer- ence. As can be seen, the images generated by our model are better aligned to the corresponding text compared the baselines. While the baselines fail with challenging prompts, our model produces high-quality images that align with the text, and composes multiple concepts correctly.
COCO Validation Set Comparison Fig. presents a qualitative comparison with Fuse- Dream (Lu et al. 2021), CogView (Ding et al. (021] and VQ-Diffusion (Gu et al. 2021) the COCO validation set. Note that both Cog View and VQ-Diffusion have been trained on an Image- Text paired dataset, whereas model was not trained on the COCO dataset, nor used it in the retrieval model.
Additional samples generated from challenging text inputs provided in Figs. 13
14 and Fig. 12
K- Nearest Neighbors
K K-Neettst Neighbors
Ours
no-kNN
Ours
no-kNN
birthday cake todily
drawing dog sitting
bear shaper
on table-
space
of bassmass Land-art style
Inpressionisn
style
Figure 15: Comparison of model, trained on PMD with kNN extracted in inference, (2) the model without using kNN in inference. While KNN lack information regarding text semantics, our model considers both text semantics and the KNN, thus proving the advantage of using both the text and the KNN embeddings.
Text
Ouze
Hearrsst ned gabobs
thout
Text
Cuze
Hearest neighbore thout
cinterela
kooking
leagy
slgg
Ang'y
A bouquet of suallowes van Cough
Fobu,
Puppy be steting
Nalke
Tecidy bear space:
Angry
Ucainiia
mountiin
shape
Figure 16: Qualitative comparison of stickers generated using the discrete kNN-Diffusion model, 10 Nearest Neighbors to the text in the CLIP embedding and a discrete model that does not kNN.
19

---

Published as a conference paper at ICLR 2023
6.3 HUMAN EVALUATION PROTOCOL
For all of our human evaluation experiments, we used Amazon Mechanical Turk. For each ex- periment, we used 600 samples, each scored by five different people. The preferred sample was determined according to majority opinion. For each baseline comparison, we asked two questions (in different experiments): "Which image of a higher quality?" and "Which image best matches the text?".
6.4 DATASETS
The modified PMD dataset is composed of the following set of publicly available text-image
datasets: SBU Captions (Ordonez et al. 20IT), Localized Narratives (Pont-Tuset et al. 2020), Con- ceptual Captions (Sharma et al. 2018). Visual Genome (Krishna et al., 2016). Wikipedia Image
Text (Srinivasan al. [002T,, Conceptual Captions 12M (Changpinyo et al. (021). Red Caps (De-
sai et al.) 2021). and a filtered version of YFCC100M (Thomee et al. 2015 In total, the dataset
contains 69 million text-image pairs.
6.5 ABLATION STUDY
Index size As one can expect, increasing the index size at inference time improves performance. To demonstrate this hypothesis, we evaluated our model with an index containing 10%, 30%, 50% and 70% images of PMD dataset, and obtained FID scores of 13.92, 13.85, 13. 72, and 13.65 respectively.
kNN conditioning We examined several different approaches to KNN input conditioning: (i) for- warding the kNN embeddings and the single image embedding through self-attention layer before feeding the contextualized + embeddings to the model, (ii) feeding the model with one embed- ding, computed using cross-attention between the image embedding and the kNN embeddings, and, (iii) feeding the model with the image embedding concatenated with a learned linear projection of the KNN embeddings. These variants received FID scores of 18.3, 22.4, 34.1 respectively.
6.6 RETRIEVAL MODEL
The retrieval model is implemented using FAISS (Johnson et al., 2019). FAISS is an efficient database, capable of storing billions of elements and finding their nearest neighbors in millisee- onds. In the pre-processing phase, for each image in the dataset, we store the image index and its corresponding CLIP image embedding. During training, given a training image, we extract its CLIP image embedding and search for its 10 (see Fig. 9 nearest neighbors in the dataset based the cosine similarity distance.
For efficient search during training and inference, use an inverted file index. As in Babenko & Lempitsky (2014), we define Voronoi cells the d- dimensional space (where d 512 is the CLIP embedding dimensional space), s.t each database vector falls in one of the cells. During search time, only the embeddings contained in the cell the query falls in and few neighboring ones are compared against the query vector. In addition, to fit the index of our large-scale datasets on 128GB RAM server, we compress the code size from 512 32/8 Jegou et al. (210)). In Algorithm [ we include pseudocode of the core of the implementation of the retrieval database.
non-exhaustive search: For this,
2048 Bytes to 256 Bytes using optimized product quantization (Ge al.) [013]]
6.7 DISCRETE KNN MODEL
We provide additional implementation details for the discrete diffusion model. Additional training details can be found in Tab.
Vector Quantization For token quantization, we use VQ VAE and adapt the publicly available VQGAN(Esser et al.) (2011) model, trained on the OpenImages(Krasinn et al.) (0166) dataset. The encoder downsamples images 32 x 32 tokens and uses codebook vocabulary with 2887 elements.
20

---

Published as a conference paper at ICLR 2023
Image Tokenization In discrete generative model model images sequence of dis- crete tokens. To this end, we utilize a vector-quantized variational auto-encoder (VQ-VAE) (Van) Den Oord al. 2017) as image tokenizer. VQ-VAE consists of three components: (i) an encoder, (ii) a learned codebook, and, (iii) decoder. Given an image, the encoder extracts a latent representa- tion. The codebook then maps each latent vector representation to its nearest vector in the codebook.
Finally, the decoder reconstructs the image from the codebook representation. VQ-VAE is trained with the objectives of reconstruction and codebook learning. VQ-GAN (Esser al. 202T) adds an adversarial loss term that tries to determine whether the generated image is fake or real. This added term was shown improve reconstruction quality.
Transformer We follow Gu al. (2021) and train a decoder-only Transformer. The decoder module contains 24 transformer blocks, each containing full attention, cross-attention for the con- catenated conditioner, and a feed-forward network. The timestamp is injected using Adaptive Layer Normalization
(Ba et al. 2016). The decoder contains 400 million parameters.
Classifier-free guidance We sample our diffusion models using classifier-free guid- ance (CFG) (Ho & Salimans 2021; Nichol et al. 2021 Ramesh et al. (022). CFG is performed by
extrapolating an unconditional sample the direction of conditional sample. To support uncondi- tional sampling, previous work had to fine-tune (Nichol et al. (021)) their models with 20% of the conditional features nullified. This enabled them to sample unconditional images from the model
using the null condition, y' = O', the null vector. We found that we can generate unconditional samples from our model using null conditioning without fine-tuning We hypothesize that by conditioning the model on a null vector, the cross-attention component is also nullified, resulting in no contribution to the diffusion process. During inference, each step of the diffusion process we generate two images: conditional image logits, po p v). conditioned on the desired multi-modal embedding y, and the unconditional image logits, pa ( (rn, y'), conditioned on null embedding. Then, the final image for diffusion step is sampled from
pe(xn-1lzn v) =pe(In-1l 1/)+
X(po(En-1 - po(2n-1lzn.5/
where is a scale coefficient. the highest FID scores the validation set. Note that the above extrapolation occurs directly on the logits output by pe. in contrast to GLIDE (Nichol et al. 2002)), which extrapolates the pixel values.
our experiments, we set = 8, which was found to yield
Training Objective For completeness we are adding the training objective of the discrete model. The network is trained to minimize the variational lower bound (VLB):
w Lo + L1 + ..... LN-1 1 + LN;
Co = log po po 11 (I), knnang k),
fn-1 DKL(g (In 12n, v0) |I po (1 100n-1/2n5 (ong(()) (1,
LN DKL N (zo) p(zx))
Where p(æx) is the prior distribution of timestep N = 100, Simg (T) is the CLIP image embedding, knnang (I, k) the & nearest neighbors in the feature space of the image embedding. The full details can be found in Gu et al. (2021)
6.8 CONTINUOUS KNN MODEL
We provide additional implementation details for the continuous diffusion model. Additional train- ing details can be found in Tab.
Decoder. We followed Nichol et al.) 2021 Ho et al. 2020; Ramesh et al.) 2022) and
re-implemented a diffusion U-net model. Specifically, we modify the architecture described in (Ramesh et al.) 2022) by allowing multiple CLIP embeddings Since we do not have a paired text-image dataset, we removed the text transformer, and thus the text embedding. In particular, we use 512 convolution channels, 3 residual blocks, 64 heads channels and attention resolution of 32, 16 and 8. Similarly to our discrete model, we trained two models (1)
the condition to the model.
21

---

Published as a conference paper at ICLR 2023
(tx Fing()
Jomg (
KNN
KNN
(a) Training
(b) Inference
During training, only the image I is given (red), whereas during inference only the text f is given (blue). In order to bridge the gap between the two distributions during training. we leverage the K nearest neighbors that should have large enough distribution (dashed cone) cover the potential text embedding (i.e. cos(b) < (a)) During inference, the opposite applied.
Figure 17:
no-kNN conditioned only on CLIP image embedding during training, (2) kNN conditioned on CLIP image embedding and its kNN. Finally, we enable classifier-free guidance by randomly set- ting the CLIP embeddings zero 10% of the time. As demonstrated in Tab. 2 find that humans prefer our model over no-kNN 66.8% of the time for image quality and 69.4% of the time for text alignment.
Huper-Resolutiin As the decoder generates images with 64 x 64 resolution, we up-sampled the images to 256 256 using the open-source super resolution of (Nichol et al., 2021). To further up- sample the images to 512 512 and 1024 x 1024 used the open-source super resolution provided by (Wang et al. 2021).
addding the training objective of our continuous model. Following Ho et al. (2020); Nichol et al. (2021) we are using mean-squared error loss to predict the noise:
Training Objectives For completeness
L: E.-(1 (...
co(xn. n, v11l
where is U net model and y (Jimg (zo), knning (vo, k)).
Discrete
Continuous
Number of nearest neighbors
10
10
Diffusion steps
100
1000
Noise schedule
cosine
Sampling steps
100
250
Model size
400M
1B
Sampling variance method
analytic 0.1
Dropout
Weight decay
4.5e-2
Batch size
512
1600
Iterations
150K
500K
Learning
4.05-4
1.4e-4
optimizer
AdamW
AdamW
Adam B2
0.96
0.9999
Adam
1.0c-8
1.0c-8
EMA decay
0.99
0.9999
warmup
5000
25000
#GPUs
128 A100
200 A100
Table Training details our models
22

---

Published as a conference paper at ICLR 2023
Algorithm Pseudo-code implementation for the construction of the retrieval model, training and sampling using conditioning KNN.
Retrieval model construction
def training (dataset: train image dataset):
//inverted index of 50k centreids, //with optimized product quantization to 256B idx_cfg "0P0255_/VV5000__22233XX index faiss. inl.sssinnde__fctrrryyoto idx_cfg, faiss.MRTRIA (NNKR_PRODUCT)
ivf faiss. tetrsertttttnnneeiiieeeee clustering index = faiss. index inwee_ccpuepppueeeeee_pe ctrpeeee_ppp___ee aa_ato_llllllllllallaa gpus (faiss. IndexF lat (d) wwf.clusteringgindee index = clustering indeex index
train dataset
for image random. sample (dataset, 1000000) train_dataset, aappndddaseet..ppeddd index.train (triinntrrinddttasstt for image in dataset:
append
(image))
11
12
index.ad (CLIP_image) (CLiP_image_enccdrr: (image))
return index
15
Training
def training (I:FAISS index, image, k:Number NN, t:timestamp 10, D-1])
CLIP hel.imiaaeeeecdeeeee (image)
KNN I. search (image_encoding k)
condition concatenate ([ ("image_encoding kNN])
image_T add_noise (image, t)
image_0 diffusion model (image_1 condition)
loss criterion (image0, image)
return loss
Sampling
def sampling (I: FAISS index, text, k Number of NN)
KNN I. search TLIP_text_enccder (text)
text_encoding
k)
conaition concatenate ([text ([text_encoding,, KNN])
image sample stappeee_niiseeeeeeeeeeeeeeeeeee for t in IT-1, I-2, 01:
noise (T)
image aiffusion_modee (image, t, condition) return image
23

---

Published as a conference paper at ICLR 2023
6.9 TEXT-ONLY IMAGE MANIPULATION
Our approach is illustrated in Fig. 18 Additional manipulation examples are provided in Figs. 20 The full comparison with the baselines provided in Fig. 21 and 22 We also provide in Fig. 19 several examples for the process of the manipulated images construction.
Manipulation
TRAIN
Original image
PM
CLIP
Condition
I' tired
(1)
(3)
(5)
Pm
tired
INIMMM)NNNM
Input image
Diffusson Model
Nearest
F'm tired
Pm tired
Neighbor NN
NN-M
(2)
(4)
Manipulation Prompt face of male child"
INFERENCE
CLIP
Condition
Original image
(5)
Input image
Diffusion Model
P' tired
An illustration of our manipulation approach. During training: Given training image (1), the model extracts its first nearest neighbor (2). Next, random local area in the training image is selected (3), and the manipulated image is constructed by replacing the area with the corresponding nearest neighbor (4). The model then receives as input the manipulated image and the clip embedding of the local area that needs to be restored (5). During inference: Given an input image and text query "A face of a male child", the model receives input the image (4) and the clip embedding of the modifying text (5).
Figure 18:
24

---

Published as a conference paper at ICLR 2023
Original
Random local area
1-NN
Aligned 1-NN
Manipulated image
RA
Figure 19: Illustration of the manipulated image construction process during training. Given original im- age, we select random local area, and extract the first nearest neighbor (I-NN). Using ECC alignment, align the nearest neighbor with the original image replace random local area with it's corresponding nearest neighbor local area. The model then receives as input the manipulated image, together with the CLIP embedding of the local area, and tries to predict the original image.
25

---

Published as a conference paper at ICLR 2023
Original
Raising hand
Raising hands
Blue pants
Black shirt
Holds heart
Princess
Sitting
Original
Smiling
Angry
Sad
Surprised
With tic
With hat
Holds a heart
Original
Brown
Blue racing
Flowers
Sketch
Pellowracing
Stars
Colorful
GO
Original
With
With
Dancing
With
Red lipttick
Angry
original
Smiling
Angry
Surprised
Figure 20: Additional manipulation examples, generated using our model.
26

---

Published as a conference paper at ICLR 2023
Original
Angry
Sad
Surprised
With a tie
With hat
Holds heart
Original
OG
Red lipstick
Angry
Sleeping
Surprised
In love
With Bow tie
Original
Joker
Ghost
Rainbow
Devil
Angel
Original
Raising left hand
Blue pants
Black shirt
Sitting
With a tie
Holds heart
Figure 21: comparison to TeX12LIVE Bar- Tal et al, 2022). For each input image, the bottom row corresponds images generated by model, and the top row corresponds images generated by the Tex12LIVE model.
27

---

Published as a conference paper at ICLR 2023
Original
Angry
Surprised
aith tie
With hat
Holds heart
LOVE
Original
Red lipstick
Angry
Sleeping
Surprised
love
With Bow
Original
Joker
Boxer
Ghost
Rainbow
Devil
Angel
Original
Raising left hand
Blue pants
Black shirt
Sitting
With tie
Holds heart
Figure comparison to Textual Inversion (Gal al. 2022). For each input image, bottom row corre- sponds to images generated by our model, and the top row corresponds to images generated by the Textual Inversion model.
28


Reviews:
[{"review_id": "rxLzgStTwB", "reviewer": "Reviewer_yR24", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to apply retrieval in text-to-image generation based on an encoder-decoder architecture. K-nearest neighbors of some image database is computed on top of the encoder output, and a transformer-based diffusion network is applied on these neighbors to produce the final output image.\n\nThe main idea of the paper is that by leveraging the retrieval process, smaller training dataset can be used, and even out-of-distribution image generation is possible by choosing appropriate database from which to draw the k-nearest neighbors.\n\nIn settings controlled for backbones author show significant improvements on common image generation metrics.", "summary_of_the_review": "I recommend weak accept. The experimentation could be stronger, but the idea and the application is quite interesting. The problem this paper tackles is relevant.", "strength_and_weaknesses": "Strengths\n- [S1] The application of retrieval is interesting and tackling limited training data is a practical problem.\n- [S2] The controlled settings show strong performance.\n\nWeaknesses\n- [W1] Benchmark seems weak. Not a lot of baselines, and main results are only zero-shot but it's not clear that this is the focus of the paper.\n- [W2] Some drawbacks of retrieval-based methods are not well discussed. For example, the choice of k is highly dataset specific, how would someone select this in practice? The proposed method also do not scale well to larger dataset if every query involves a retrieval against the whole dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PYJZmJCqnD", "reviewer": "Reviewer_5XrX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for generating images from text with an addition of a retrieval component. Authors use pretrained text and image encoders and train diffusion-based generative models that use an image embedding and a set of image embeddings closest to the target one to reconstruct the image. During inference, instead of the image embedding, a text embedding is used (which comes from the frozen clip encoder, pretrained on image-text pairs). Authors demonstrate that the retrieval component is crucial for image quality both in online and in human studies and also extend the functionality for global image manipulation, which looks rather convincing.", "summary_of_the_review": "Overall, I think this is a decent paper. I my main criticism is about claim \"training ... model without any text\" which might very misleading. I believe it has to be fixed.\n\nUpdate after the discussion - my comments were address during the rebuttal period, so I updating the rating (given the code will be open-sourced).\nAlso I suggest the authors to update Figure 3 - the kNN part there is hard to understand (both what it does and what is the output)", "strength_and_weaknesses": "Strengths:\n1) The paper presentation is overall good. Illustrations make sense, most of the ideas are clearly expressed.\n2) The results look good, both FID and user studies. I was a bit surprised that having just embeddings as input (without the access to raw images) is enough to make a big difference. I guess the neighbors provide the hint of how far the query is from the real images, so it allows the model to prefer directions to the real embeddings.\n\nWeaknesses:\n1) A big red flag for me is the claim that no text pairs were used during the training. To reproduce the model from data one would have to pretrain the encoders on the paired data, so this claim is misleading. I highly recommend to put it into correct context to avoid reader confusion.\n2) There are some details missing. E.g. I could not find the value of K during training, which should help interpret Fig. 9 (maybe the optimal number of neighbors is just the train one?)\n3) Not clear whether the solution is going to be open-sourced.\n\n\nMisc: Sometimes there is kNN in the text/figures, sometimes it is KNN, sometimes knn. Sometimes it is k, sometimes K, sometimes it is kNN. It also makes sense to explicitly state that Image quality and Text-alignment are the human evaluation metrics (e.g. it is mentioned for table 2, but not for table 1). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DTzuBDDOs_4", "reviewer": "Reviewer_myk8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to improve the training performance and overall quality of the recently introduced text-to-image generative framework. By leveraging a kNN-based retrieval method, the proposed method 1) adds more flexibility to the training data, 2) greatly reduces the computational cost. Extensive experiments and ablation studies are conducted to prove the efficacy and the efficiency of the model. \nThe description of the technical approach is confusing and vague. In Section 3, the authors firstly criticize the use of CLIP embedding since it alone cannot accurately bridge the gap between the text and image distributions. However, the authors still choose to use the CLIP pretrained model to do the job without any modifications. Without any proper justifications on how the proposed retrieval model could solve the problem, the practical motivation of the paper is lacking.", "summary_of_the_review": "This is a good paper with promising experimental results, with some aspects of the presentation that should be improved and clarified. Also, I have the following questions regarding to the paper.\n\nQuestions to Authors:\n1.\tCould the authors provide more justifications on how could adding the retrieval part bridge the gap between the text and image distributions?\n2.\tHow does the encoders map text descriptions and image samples to a joint feature space? Do you merely use the pretrained models?\n3.\tThe subsection of “Image generation network” also lacks sufficient details. How does the generation network generate new images with conditioned on other similar instances? The equations in the subsection don’t indicate any signals that this is a conditional generation framework. \n4.\tHow does the kNN search avoid the problem of “curse of dimensionality”?\n5.\tWithout any details of the parameter setting and providing model details, the experimental merit is also hard to evaluate.\n6.\tThe scalability of the model is highlighted as one of the primary contributions. However, the corresponding scalability analysis experiment is not found in the paper.\n", "strength_and_weaknesses": "Strengths:\n1.\tThe paper is generally well-written and easy to follow.\n2.\tThe paper introduces an easy approach to facilitate the training of text-free text-to-image generation.\n3.\tThe experiments are comprehensive, it demonstrates the strength of the proposed method from different perspectives.\nWeakness:\n1.\tThe technical novelty might be limited, its model is conceptually similar to several previous works (e.g., CLIP, CogView, and LAFITE). The main contribution lies in the training strategy (kNN Retrieval) rather than in the techniques.\n2.\tThe Retrieval Model (Section 3) is described very vaguely. The provided details are not sufficient to understand the module and re-implement it.\n3.\tIt would be also helpful to release the code and pre-trained model for subsequent research.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B5GtRoJPZCX", "reviewer": "Reviewer_o44L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to train a conditional diffusion models by removing the necessary of annotation pairs. More detailed, they utilize the kNN retrieved image embedding as condition signal, and can further utilize caption embedding due to the almost-perfect joint distribution between text and images.  The framework also facilitates the generation of out-of-distribution sampling. One extra bonus is they can remove the necessity of mask annotation when conducting local semantic manipulations. \n\nExperimental results demonstrate effectiveness of the proposed method on zero-shot text-to-image generation, the metric is FID and user study around image quality and text-alignment.", "summary_of_the_review": "The motivation is well consolidated, while too many details about training/inference/preliminary knowledge are missing. Also, the paper is not easy to follow. I suggest the authors to spend more time about the writing of this paper.\n\n=======\n\npost-review update:\n\nThe paper has been improved a lot based on my concerns about description, preliminary background, training details, etc. I think the quality of current version is enough for ICLR. I also share similar concern about the marginal novelty of the proposed methods with other reviewers.  Since advantages outweigh disadvantages, I upgraded my result.", "strength_and_weaknesses": "Pros:\n\n-  The method framework is well motivated to sample from the out-of-distribution images.\n-  Competitive results compared with several baselines.\n-  Related works are quite thorough for new beginners.\n\nCons:\n\n- Writing is a little rush, the preliminary knowledge of diffusion model is missing, which makes the method part is very unfriendly to novice of this area.  \n\n- Why KNN can solve text-to-image models annotation problem in abstract? It seems that your method is condition on retrieved features instead of text/caption.\n- Figure 3 is too rush, what is MMMM, 9143? I am totally missed about what you are trying to express in this figure. The data flow is full of confusion.\n- Evaluation details are missing, how to generate condition in your sampling process? Where do those conditioned text/image come from?  Where is the reference set from? How many samples are used to calculate the FID in Table 1 and 2? \n- Training and network details are missing, how many parameters in your method, is the parameter number almost the same between your method and the proposed baselines. How many GPUs did you use? Learning rate? optimizer? batch size? epoch number? training time? weight decay? \n\n\nMinor comments:\n\n- The full comparison is available in the supplement.\n- The dataset intro should go to experiments part.\n- Figure 6,9 is saved as vector image, pdf.\n- The font size of Figure 6 is too small.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster