Retrieval-Augmented Diffusion Models
Authors: Andreas Blattmann
Robin Rombach
Kaan Oktay
Jonas Müller
Björn Ommer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image Synthesis
Deep Generative Models
Retrieval-Augmented Models

Abstract: Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Much of this success is due to the scalability of these architectures and hence caused by a dramatic increase in model complexity and in the computational resources invested in training these models. Our work questions the underlying paradigm of compressing large training data into ever growing parametric representations. We rather present an orthogonal, semi-parametric approach. We complement comparably small diffusion or autoregressive models with a separate image database and a retrieval strategy. During training we retrieve a set of nearest neighbors from this external database for each training instance and condition the generative model on these informative samples. While the retrieval approach is providing the (local) content, the model is focusing on learning the composition of scenes based on this content. As demonstrated by our experiments, simply swapping the database for one with different contents transfers a trained model post-hoc to a novel domain. The evaluation shows competitive performance on tasks which the generative model has not been trained on, such as class-conditional synthesis, zero-shot stylization or text-to-image synthesis without requiring paired text-image data. With negligible memory and computational overhead for the external database and retrieval we can significantly reduce the parameter count of the generative model and still outperform the state-of-the-art.

Retrieval-Augmented Diffusion Models
Andreas Blattmann" Robin Rombach"
Kaan Oktay Jonas Müller Björn Ommer
LMU Munich, MCML & IWR, Heidelberg University, Germany
Abstract
Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Much of this success is due the scalability of these architectures and hence caused by dramatic increase in model complexity and in the computational resources invested in training these models. Our work questions the underlying paradigm of compressing large training data into ever growing parametric representations. We rather present orthogonal, remi-parametric approach. We complement comparably small diffusion or autoregressive models with a separate image database and retrieval strategy. During training retrieve set of nearest neighhors from this external database for each training instance and condition the generative model on these informative samples. While the retrieval approach is providing the (local) content, the model is focusing on learning the composition of scenes based on this content. As demonstrated by our experiments, simply swapping the database for one with different contents transfers a trained model post-hoc to novel domain. The evaluation shows competitive performance on tasks which the generative model has not been trained on, such as class condittonni synthesis, zero-shot stylization or text-to-image synthesis without requiring paired text-image data. With negligible memory and computational overhead for the external database and retrieval we can significantly reduce the parameter count of the generative model and still outperform the state-of-the-an
Introduction
Deep generative modeling has made tremendous leaps; espe- cially in language modeling as well as in generative synthesis of high-fidelity images and other data types. In particular for im-
25
ours [0]
ours [8] + |D]
ages, astounding results have recently been achieved [22 [5
[6
20
|o]
[D] 51K
[59] and three main factors can be identified the driving forces behind this progress: First, the success of the transformer (88
[3] 68K
ADM w/
15
B5K
fler
has caused architectural revolution in many vision tasks [19] for image synthesis especially through its combination with au- toregressive modeling [22
P = 169K
10
400
450
500
550
600
(88. Second, since their rediscovery, diffusion models have been applied to high-resolution image generation 766 78 3 and, within very short time, new
Model params [M]
Figure 1: Our semi-parametric model
outperforms the unconditional SOTA
model ADM [5]] ImageNet [ and reaches the class-conditional ADM (ADM w/ classifier). while re- ducing parameter count. [D]: Number of instances in database at inference; [0]: Number of trainable parameters.
standards in generative image modeling [15 34 63 [59. Third, these approaches scale well [58 59 37 811: particular when considering the model- and batch sizes involved for high-quality
models [ 56 58 59 there is evidence that this scalability of central importance for their performance.
However, the driving force underlying this training paradigm models with ever growing numbers of parameters [81 that require huge computational resources. Besides the enormous demands in energy consumption and training time, this paradigm renders future generative modeling more and more exclusive to privileged institutions, thus hindering the democratization research. Therefore, we here
"The first two authors contributed equally this work. wtt.sahub..commooo/ccceee
Code is available at http//sstttttceeemeeeeeee/eaaaaa l
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

puople ialomander the grass"
grbrnsakinned pandl.."
teddy bear riding matorcycle"
'Imagg of munkey vitii the for leopard.'
Text rept. only
Teat repc. and NNs
NNs only
Figure 2: As retrieve nearest neighbors in the shared text-image space provided by CLIP, we text prompts as queries for exempla-basee synthesis. We observe our RDM to readily generalize to unseen and fictional text prompts when building the set of retrieved neighbors by directly conditioning on the CLIP text encoding Φcurp(ees (Ges) (top row). When using óctur(Geu) together with its k nearest neighbors from the retrieval database (middle row) or the & nearest neighbors alone without the text representation, the model does show these generalization capabilities (bottom row).
present an orthogonal approach. Inspired by recent advances in retrieval auggeetee NLP 89, we question the prevalent approach of expensively compressing visual concepts shared between distinct training examples into large numbers of trainable parameters and equip a comparably small generative model with large image database. During training. our resulting pari-erraettric generative models access this database via a nearest neighbor lookup and, thus, need not learn to generate data 'from scratch'. Instead, they learn compose new scenes based retrieved visual instances. This property not only increases generative performance with reduced parameter count (see Fig. [ and lowers compute requirements during training. Our proposed approach also enables the models during inference to generalize to new knowledge in form of alternative image databases without requiring further training, what can be interpreted form of post-hoc model modification [4. We show this by replacing the retrieval database with the WikiArt 666 dataset after training, thus applying the model zero-shot stylization.
Furthermore, our approach formulated indepently of the underlying generative model, allowing us present both eetrieval-augmentee diffusion (RDM) and autoregressive (RARM) models. By searching in and conditioning on the latent space of CLIP [57] and using scaNN (28 for the NN-
search, the retrieval causes negligible overheads in training/inference time (0.95 ms to retrieve 20 nearest neighbors from a database of 20M examples) and storage space (2GB per 1M examples). We show that semi-parametric models yield high fidelity and diverse samples: RDM surpasses recent state-of-the-art diffusion models in terms of FID and diversity while requiring less trainable parameters. Furthermore, the shared image-text feature space of CLIP allows for various conditional applications such as text-to-image or class-conditional synthesis, despite being trained on images only (as demonstrated in Fig. (2). Finally, we present additional truncation strategies control the synthesis process which can be combined with model specific sampling techniques such as classifier-free guidance for diffusion models 32 or top-k sampling (3 for autoregressive models.
Related Work
Generative Models for Image Synthesis. Generating high quality novel images has long been
challenge for deep learning community due to their high dimensional nature. Generative adversarial
networks (GANs) 25 excel synthesizing such high resolution images with outstanding quality [5]
39 40 70 while optimizing their training objective requires some sort of tricks 27 [54] 53] and
their samples suffer from the lack of diversity 80 I 55 50). On the contrary, likelihood-baat methods have better training properties and they are easier to optimize thanks to their ability to
capture the full data distribution. While failing to achieve the image fidelity of GANs, variational
autoencoders (VAEs) 43 61 and flow-based methods [16] I7 facilitate high resolution image
generation with fast sampling speed [4 45). Autoregressive models (ARMs) [10, 85| 87 68
succeed in density estimation like the other likelihood- methods, albeit at the expense of computational efficiency. Starting with the seminal works of Sohl-Dickstein et al. (76 and Ho et

---

Training via conditioning on neighbors
Inference with fexxible conditionings
Sumi-PPrannetric Model P
D train
p8
D
D
2
D
neighbor ratrioval
Likelhhood- based conditional ganerative model
retrioned neighbora
Traineble
retrieval becedt on tott
D
fivedd encodfer cuay
retrooved neighbors
funed
Pe
(t.p. CLUP
Non- Trainable
P9
o. diffusion model autorogresstre model
Figure 3: A semi- semi--prameen parametric generative model consists of a trainable conditional generative model (decoding
head) pa (xl-). ). an external database D containing visual examples and sampling strategy & to obtain a subset Mb S D, which serves as conditioning for ps. During training, & retrieves the nearest neighbors of each target example from D, such that Pe only needs to learn to compose consistent scenes based on M cf. Sec 3.2
During inference, can exchange D and Ex. thus resulting in flexible sampling capabilities such post-hoc conditioning on class labels (E)) or text prompts (Ei) of. Sec. 3.3 and zero-shot stylization, of. Sec. [4.3]
al. [33. diffusion-based -hased generative models have improved generative modeling of artificial visual 35 9 65). Their good performance, however, comes the expense of high training costs and slow sampling. To circumvent the drawbacks of ARMs and diffusion models, several two-stage models are proposed to scale them to higher resolutions by training them on the compressed image features [86, 60 [22] 93 63 [1). However, they still require large models and significant compute resources, especially for unconditional image generation [5] on complex datasets like ImageNet [13] or complex conditional tasks such as text-to-image generation [56] 588 [26 63)
systems [15 44 90
To address these issues, given limited compute resources, we propose trade trainable parameters for external memory which empowers smaller models to achieve high fidelity image generation.
Retrieval-Augmenneer Generative Models. Using external memory to augment traditional models has recently drawn attention in natural language processing (NLP) 41 42 52 29). For example, RETRO 4 proposes a retrieval-enhanced transformer for language modeling which performs on par with state-of-the-at models [6] using significantly less parameters and compute resources. These serreval--uugenters models with external memory turn purely parametric deep learning models into semi- camimmrraettric ones. Early attempts [51 74 83 91 in retrieval -aggeeetd visual models do not use an external memory and exploit the training data itself for retrieval. In image synthesis, IC-GAN [8] utilizes the neighborhood of training images to train a GAN and generates samples by conditioning on single instances from the training data. However, using training data itself for retrieval potentially limits the generalization capacity, and thus, we favor an external memory in this work.
3 Image Synthesis with Retrieval-Angmented Generative Models
Our work considers data points as explicit part of the model. In contrast to common neural genera- tive approaches for image synthesis [ 40) 70 60
22 TO 9. this approach is not only parameterized by the learnable weights of neural network, but also (fixed) set of data representations and non-learnable retrieval function, which, given a query from the training data, retrieves suitable data representations from the external dataset. Following prior work in natural language modeling [4), we implement this retrieval pipeline as a nearest neighbor lookup.
Sec. and Sec. 3.2 formalize this approach for training
-aggeeeed diffusion and autore- gressive models for image synthesis, while Sec. (33 introduces sampling mechanisms that become available once such a model is trained. Fig. 3 provides an overview over approach.
3.1 Sarrieval-Enhancee Generative Models of Images
Unlike common, fully parametric neural generative approaches for images, define semi- parametric generative image model Po.D,E. (x:) by introducing trainable parameters and non-
trainable model components D, Ex. where D = {y 1 is a fixed database of images y/1 E R x3 that is disjoint from our train data X. Further, & denotes a (non-trainable) sam- pling strategy to obtain a subset of 'D based on query i.e. Ex:22 D + M(h,, where M C D and [M(() = k. Thus, only 0 is actually learned during training.
3

---

Importantly, (x, D) has to be chosen such that it provides the model with beneficial visual repre- sentations from D for modeling x and the entire capacity 8 can be leveraged to compose consistent scenes based on these patterns. For instance, considering query images & E RR XH,x x3, valid strategy (x; D) a function that for each æ: returns the set of its nearest neighbors, measured by given distance function d(x,,)) -).
Next, we propose to provide this retrieved information to the model via conditioning, i.e. we specity a general semi-parametric generative model as
De.D,E. (x) = pe( Ex(x, D)) = po(x M
(1)
In principle, one could directly use image samples y E _M to learn 0. However, since images
contain many ambiguities and their high dimensionality involves considerable computational and
storage cos we use fixed, pre-trained image encoder & to project all examples from M onto a
low-dimensional manifold. Hence, Eq. [I reads
Po.D.Ca (I) po (a (d(v) y (x(x,
where pe (x(--) is a conditional generative model with trainable parameters which we refer to as decoding head. With this, the above procedure can be applied to any type of generative decoding head and is not dependent on its concrete training procedure.
3.2 Instances of Semi-P Semi-Paramee Parametric Generative Image Models
During training we are given a train dataset X' = {x:] M of images whose distribution p(z) we want to approximate with po,D. (x)). Our train-time sampling strategy & uses query example I ~ p(x) to retrieve its & nearest neighbors y E 'D by implementing d(x:, y) as the cosine similarity in the image feature space of CLIP (57) Given sufficiently large database DD this strategy ensures that the set of neighbors & (x, D) shares sufficient information with x and, thus, provides useful visual information for the generative task. We choose CLIP to implement εx, because embeds images in a low dimensional space (dim = 512) and maps semantically similar samples to the same neighborhood, yielding an efficient search space. Fig. 4 visualizes examples of nearest neighbors retrieved via a B/3- vision transformer [19] backbone.
x
Figure 4: k = 15 nearest neighbors from D for given query æ when parameterizing d(æ, :) with CLIP [57]
Note that this approach can, in principle, turn any generative model into a semi-parametric model in the sense of Eq. (2). In this work we focus on models where the decoding head is either implemented as a diffusion
autoregressive model, motivated by the success of these models in image
synthesis 33 15
56, 58 22)
To obtain the image representations via ø, different encoding models are conceivable in principle. Again, the latent space of CLIP offers some advantages since it (i) very compact, which (ii) also reduces memory requirements. Moreover, the contrastive pretraining objective (iil) provides shared space of image and text representations, which beneficial for text-image synthesis, as we show in Sec. (42 Unless otherwise specified, á = ΦCLI is set in the following. We investigate alternative parameterizations of @ Sec. F.22
Note that with this choice, the additional database 'D can also be interpreted as a fixed embedding layer of dimensionality [D] x512 from which the nearest neighbors are retrieved.
3.2.1 Detrieval-Augmenter Diffusion Models
In order to reduce computational complexity and memory requirements during training. we follow 63 and build on latent diffusion models (LDMs) which learn the data distribution in the latent space
= F(()) of a pretrained autoencoder. We dub this retrieval-augmente latent diffusion model
RDM and train it with the usual reweighted likelihood objective 33), yielding the objective [76 33
min L = E.pc))
c0( 4, (ocur(v) E Ca 1)))1
2 NNot that D is essentially part of the model weights
For a database of IM images and using 32-bit precision, this equals approximately 2.048 GB

---

where the expectation is approximated by the empirical mean over training examples. In the above equation. Eo denotes the UNet-based [64 denoising autoencoder as used in [ 63 and (1, ..... 7) denotes the time step 76 33 To feed the set of nearest neighbor encodings
(CLI((y) into co, we use the cross-attention conditioning mechanism proposed in [63]
3.2.2 Retrieval-Augmmented Autoregressive Models
Our approach is applicable to several types of likelihood- based methods. We show this by augmenting diffusion models (Sec. [3..1T) as well as autoregressive models with the retrieved representations. To implement the latter, we follow 22 and train autoregressive transformer models to model the distribution of the discrete image tokens En = Exx) of a VQGAN [22] 86). Specifically, as for RDM, we train retrieval- -aggeeeee autoregressive models (RARMs) conditioned on the CLIP embeddings ΦCLI(()) of the neighbors y, so that the objective reads
min L - Bp(x).=,- -F(x)
Σ log p(2)) 2((()) (dcIIp()) (x, D11]
()))
(4)
where we choose a row-major ordering for the autoregressive factorization of the latent Ag. condition the model on the set of neighbor embeddings ΦCLP (CLIP(Ex(: D)) via cross-attention
We
[8
3.3 Inference for Retrieval-Augmenntd Generative Models
Conditional Synthesis without Conditional Training Being able to change the (non-learned) D and &a at test time offers additional flexibility compared to standard generative approaches: Depending on the application, it is possible to extent/restrrict 'D for particular exemplars; or to skip the retrieval via Ea altogether and provide set of representations {ØCLAP(U [y+)]/_ directly. This allows to use additional conditional information such as a text prompt or a class label, which has not been available during training, to achieve more fine-grained control during synthesis.
For text-to-image generation, for example, our model can be conditioned in several ways: Given a text prompt Cext and using the text-to-image retrieval ability of CLIP, we can retrieve k neighbors from D and use these as an implicit text-based conditioning. However, since we condition on CLIP representations ΦCLIP, we can also condition directly on the text embeddings obtained via CLIP's language backbone (since CLIP's text-image embedding space shared). Accordingly, it is also possible to combine these approaches and use text and image representations simultaneously. We show and compare the results of using these sampling techniques in Fig. 2
Given a class label c, we define a text such as 'An image of a ((c).' based on its textual description Me) or apply the embedding strategy for text prompts and sample pool E(()), k < / for each class. By randomly selecting & adjacent examples from this pool for a given query c, we obtain an inference-time class-conditional model and analyze these post-hoc conditioning methods in Sec. 4.2
For unconditional generative modeling, we randomly sample a pseudo-query ă E D to obtain the set <xat ci; D) of its i nearest neighbors. Given this set, Eq. (2) can be used to draw samples, since (x(l))) itself is a generative model. However, when generating all samples from po.D.E. (2) only from one particular set (Ø)) we expect Po.D.E. (x)) to be unimodal and sharply peaked around a. When intending to model a complex multimodal distribution p(x)) of natural images, this choice would obviously lead weak results. Therefore, construct proposal distribution based on D where
{x E X
T E (xx D)H
for ã D.
k: k
This definition counts the instances in the database D which are useful for modeling the training dataset X. Note that (ō) only depends on X' and D, what allows us to precompute it. Given (T) we can obtain set
p = fa x pa(w (d(v) y E Ex(2, D)H) ã~ (
(6)
of samples from the model. We can thus draw from the unconditional modeled density Pe.D.sa &u. (x) by drawing x' ~ Uniform(P..
By choosing only fraction m E (0,1] of most likely examples ă PD))), we can artificially truncate this distribution and trade sample quality for diversity. See Sec. [D.]] for detailed description of this mechanism which we call top-m sampling and Sec. 4.5 for an empirical demonstration.
5

---

RDM
RARM ImageNet
ImageNet [B]
FFHQ 38
Mb (+)
Samples
NNs in train set
Figure 5: Samples from our unconditional models together with the sets of MM (æ) of retrieved neighbors for
the pseudo query ă, cf. Sec. Sec.B.3 and nearest neighbors from the train set, measured in CLIP [S7] feature space. For ImageNet samples generated with m 0.01, guidance with 2.0 and 100 DDIM steps for RDM and m 0.05, guidance scale s 3.0 top-k 2048 for RARM On FFHQ we use 8 1.0, m 0.1.
Experiments
This section presents experiments for both retrieval-augmenned diffusion and autoregressive models. To obtain nearest neighbors we apply the ScaNN search algorithm
[28] in the feature space of a pretrained CLIP-ViT-B/32 (57). Using this setting, retrieving 20 nearest neighbors from the database described above takes ~ 0.95 ms. For more details on our retrieval implementation, see Sec. F. For
quantitative performance measures
FID [1I. CLIP-FID [48] Inception Score (IS) 67 and
Precision-Readt [47], and, for the diffusion models, generate samples with the DDIM sampler with 100 steps and 7 1. For hyperparameters, implementation and evaluation details cf. Sec.
4.1 Semi- Parimeearetttic Image Generation
Drawing pseudo-queries from the proposal distribution proposed in Sec. 3.3 and Eq. (6) enables semi-pate camimparametric unconditional image generation. However, before the actual application, we compare different choices of the database Denin used during training and determine an appropriate choice for the value & of the retrieved neighbors during training.
LDM
RDM-COCO
RDM-WA
0.6
100
Precision
0.6
Recall
75
0.5
50
0.4
0.4
0
25
50
75
25
50
75
25
50
75
Epoch
Epoch
Epoch
Figure 6: Comparing performance metries of RDMs with different train databases Dinin with those of an LDM baseline the dogs-subset of ImageNet (13): find that having database of diverse visual instance from visual domains similar to the train dataset X' (as RDM -COCO) improves performance upon param-tai baseline. Increasing the size of the database further boosts performance, Icading to significant improvements of RDMs the baseline despite having less trainable paramcters.
Finding a train-time database Draain Key to a successful application of remi-parametric models is choosing an appropriate train database Panin as has to provide the generative backbone pe with useful information. We hypothesize that a large database with diverse visual instances is most useful for the model, since the probability of finding nearby neighbors in D for every train example is highest for this choice. To verify this claim, we compare the visual quality and sample diversity of three RDMs trained on the dogs-subset of ImageNet [3] with i) WikiArt (66) (RDM- WWA), ii) MS-COCO ( (RDM-COCO) and iii) 20M examples obtained by cropping images (see App. F.I from OpenImages 446) (RDM-OI) as train database Denin with that of an LDM baseline with 1.3x more parameters. 6 shows that i) database Dimin whose examples are from different domain than those of the train set X' leads to degraded sample quality, whereas ii) a small database from the same domain as X' improves performance compared to the LDM baseline. Finally, ii) increasing

---

the size of mini further boosts performance in quality and diversity metrics and leads to significant improvements of RDMs compared LDMs.
Method
FIDI
CLIP-FID|
Precisiont Recallf
Method
FID
CLIP-FIDJ CCIP-score?
ISt
reln
ROM-IN
5.91
5.32
3.92
4.44
0.74
0,51
LAFITE 994
26.94
26.02
RDM-OI
12.28
11.31
4.59
0.69
0.55
RDM-IN
27.28
18.12
0.29
24.17
RDM-IN/OI
17.23
16.82
8.86
9.75
0.52
0.60
RDM- 01
22.08
13.16
0.30
24.31
RDM-OU/N
10.81 12.01
3.84
4.41
0.81
0.39
Table I: Generalization to new databases. Left: We train RDMs on ImageNet with OpenImages (RDM-OL Ol) and the train dataset itself (RDM-IN). By exchanging the train and inference databases between the two models we that RDM-OI which trained with database disjoint from the train set generalizes better new inference databases. Right: Quantitative comparison against LAFITE
on zero-shot texl-to-image synthesis.
For the above experiment we used Panin n = D. This is in contrast to prior work [ which conditions generative model on the train dataset itself, i.c., Parin = X. Our choice is motivated by the aim to obtain a model as general as possible which can be used for more than one task during inference, introduced in Sec. 3.3 To show the benefits of using Dusin n X & we use ImageNet [33 as train set X' and compare RDM-OI with an RDM conditioned on X' itself (RDM- IN). We evaluate their performance
the ImageNet train- and validation-sets in Tab. D which shows RDM-OI to closely reach the performance of RDM-IN in CLIP-FID (48) and achieve more diverse results. When interchanging the test-time database between the two models, i.e., conditioning RDM-OI on examples from ImageNet (RDM-OI/N) and vice versa (RDM-IN/OI) we observe strong performance degradation of the latter model, whereas the former improves in most metrics and outperforms RDM-IN in CLIP-FID, thus showing the enhanced generalization capabilities when choosing Prain n X' = O. To provide further evidence of this property we additionally evaluate the models on zero-shot text-conditional on the COCO dataset [7] in Tab. [ Again, we observe better image quality (FID) as well as image-text alignment score) of RDM AMMOOI which furthermore outperforms LAFITE [94] in FID, despite being trained on only a third of the train examples.
How many noighbors to retrieve during training? As the number kirain of retrieved nearest neighbors during training has strong influence on the prop- erties of the resulting model after training, we first identify hyperparameters obtain model with opti- mal synthesis properties, Hence, we parameterize pe with a diffusion model and train five models for different Krain E {1, 21 4, 8, 16) on ImageNet [3]. All models use identical generative backbones and computational resources (details in Sec. F.2.Ip Fig. shows resulting performance metrics assessed on 1000 samples. For FID and IS we do not observe significant trends. Considering precision and recall, however, we see that increasing kimai trades consistency for diversity. Large kirain causes recall, i.e. sample diversity, to deteriorate again.
0.50
44.0
Recall
Procis
KNNN
K:NNN
Figure 7: Effect of Auaiin
We attribute this regularizing influence of non-redundant, additional information beyond the single nearest neighbor. which is fed to the respective model during training, when Kirain > 1. For knain E {2, 4, 8} this additional information is beneficial and the corresponding models appropriately mediate between quality and diversity. Thus, we use k = 4 for our main RDM Furthermore, the numbers of neighbors has significant effect on the generalization capabilities of our model for conditional synthesis. e.g. text-to-image synthesis as in Fig. 2 We provide an in-depth evaluation of this effect in Sec. 4.2 and conduct similar study for RARM Sec. E.4
Qualitative results. Fig. shows samples of RDM /RARM trained on ImageNet as well as RDM samples on FFHQ [38 for different sets M (7) of retrieved neighbors given pseudo- query v ~ (#). We also plot the nearest neighbors from the train set to show that this set is disjoint from the database D and that our model renders new, unseen samples.
Quantitative results. Tab. 2 compares our model with the recent state-of-the-ar diffusion model ADM [5] and the semi-parametric GAN- model IC-GAN [ (which requires access to the training set examples during inference) in unconditional image synthesis on ImageNet [3] 256 x 256.
To boost performance, we use the sampling strategies proposed in Sec. 3.3 (which is also further detailed in See. D.I. With classifier-free guidance (c.f.g.), our model attains better scores than IC-GAN and ADM while being on par with ADM-G [15] The latter requires an additional classifier and the labels of training instances during inference. Without any additional information about training data, e.g., image labels, RDM achieves the best overall performance.
7

---

Method
FID
IS
Precision
Recallf
Nparams
traón
vol
train
vol
trgin
vol
IC-GAN
18. 17.17
15.60*
59.00"
0.77
0.73
0.21
0.23
191M
conditioned on train set, add. aug.
ADM 15
26.21 32.50*
39.70
0.61
0.63
554M
250 steps
ADM-G 15
33.03
32.92
0.56
0.65
618M
250 steps, c.g., $=1.0
ADM-G
15
12.00
95.41
0.76
0.44
618M
250 steps, c.g., s=10.0
RDM-OI (ours)
24.50
21.28
45.29
0.60
0.54
0.65
0.66
400M
100 steps, = =0.1
RDM-OI (ours)
19.08
16.89
62.78
0.57
0.62
0.56 0.57
400M
100 steps, m = 0..1
RDM-OI (ours)
13.22
12.29
70.64
0.72
0.65
0.56
0.5T
400M
100 steps, c.I... 8 = 1.75, m 0.1
RDM-OI (ours)
13.60
13.11
87.58
0.79
0.73
0.51
0.50
400M
100 steps, c.f.g., 8- -1.,, m = 0.02
RDM-01 01 (ours) 12.21
11.31
77.93
0.75
0.69
0.55 0.55
400M
100 steps, c.I.g. -15, m 0.05
RDM-IN (ours) 5.91
5.32
158.76
0.74
0.74
0.51 0.53
400M
100 steps, c.f.g., 1.5,
0.05
Table 2: Comparison of RDM with recent the methods for unconditional image generation on
ImageNet [33. While cff.g. denotes classifier-free guidance with a scale parameter as proposed in (32), c.g. refers to classifier guidance [TS], what requires classifier pretrained on the noisy representations of diffusion models to he available. ": numbers taken from R
For m 0.1, our retrieval aaggenneeee diffusion model surpasses unconditional ADM for FID, IS, precision and, without guidance, for recall. For 8 = 1.75, we observe bisected FID scores compared to our unguided model and even reach the guided model ADM-G, which, unlike RDM, requires classifier that is pre-trained on noisy data representations. The optimal parameters for FID are m 0.05, 8 = 1.5, as in the bottom row of Tab. - Using these parameters for RDM-IN results in a model which even achieves similar FID scores than state of the class-conditions inference. Overall, this shows the strong performance of RDM and the flexibility of top-m sampling and c.f.g., which we further analyze in Sec.
Method P-GAN
CLIP--FI CI.IP-Prec CLIP-Rec
69
4.87
Style-GANz
39
2.90
LDM
63
2.12
0.48
LDM (equal Nanaaa
2.65
0.87
0.44
RDM Of
1.92
0.93
0.35
Table 3: Quantiative results on FFH [38]
RDM- OI samples generated with 0.1
and without classificr-free guidance.
models on ImageNet [63 15
70 without requiring any labels during training or
4.5 Moreover we train exact replicate of our ImageNet
RDM-OI on the FFHQ 38) and summarize the results in Tab. 3 Since FID 31 has been shown to be "insensitive to the facial region" [8] we again use Clse-based metrics. Even for this simple dataset, our retrieval-base based strategy proves beneficial, outperforming strong GAN and diffusion baselines, albeit at the cost of lower diversity (recall).
4.2 Conditional Synthesis without Conditional Training
60.0
Text-to-Image Synthesis In Fig. 2 we show the zero-shot text- to-image synthesis capabilities of our ImageNet model for user defined text prompts. When building the set M (Cwaa by di- rectly using i) the CLIP encodings (CLIP(Ces)) of the actual textual description itself (top row), we interestingly see that our model generalizes to generating fictional descriptions and transfers attri- butions across object classes. However, when using ii) (CCI((((s)) together with its k nearest neighbors from the database DD as done in (2). the model does not generalize to these difficult conditional inputs (mid row). When iii) only using the k CLIP image representations of the nearest neighbors, the results are even worse (bottom row). We evaluate the text-to-image capabilities of RDMs on 30000 examples from the COCO validation set and compare with LAFITE (94). The latter is also based on CLIP space, but unlike our method, the image features are translated to text features by utilizing supervised model in order to address the mismatch between CLIP text and image features. Tab, I summarizes the results and shows that our RDM-O OI obtains better image quality measured by the FID score.
53.6
50.5
coconi womorioo
6.29
80000
40.5
0.3
6.5
010
0.25
Figure 8: We observe that the num- ber of neighbors ktrain retrieved during training significantly impacts the generalization abilities of RDM See Sec. 4.2
Similar to Sec. 4.1 we investigate the influence of kirin on the text-to-image generalization capability of RDM. To this end we evaluate the zero-shot transferability of the ImageNet models presented in the last section to text-conditional image generation and, using strategy i) from the last paragraph, evaluate their performance on 2000 captions from the validation set of COCO 7 Fig. 8 compares the resulting FID and CLIP scores on COCO for the different choices of kirain- As a reference for the train performance, we furthermore plot the ImageNet FID. Similar to Fig- 7 we find that small Ktain lead to weak generalization properties, since the corresponding models cannot handle misalignments between the text representation received during inference and image representations it
trained on. Increasing kirain results in sets M (a) (x) which cover a larger feature space volume, what regularizes the corresponding models to be more robust against such misalignments. Consequently,
the generalization abilities increase with knain and reach an optimum at kirain = 8. Further increasing
Arain results in decreased information provided via the retrieved neighbors (cf. Fig. 4 and causes deteriorating generalization capabilities.
8

---

'A brown bear." 'A yellow bird.'
We note the similarity of this approach to [59,, which, by directly conditioning on the CLIP image representations of the data, es- sentially learns to invert the abstract image embedding. In our
framework, this corresponds to fx(x) = (CLIP()) (i.e., no external database is provided). In order to fix the misalignment between text embeddings and image embeddings, [59] learns a conditional diffusiin model for the generative mapping between these represen- tations, requiring paired data. We argue that our retrieval aarieeel-augmeented approach provides orthogonal approach to this task without requir- ing paired data. To demonstrate this, we train an "inversion model" as described above, i.e., use & (z) (C)I(()) with the same number of trainable parameters and computational budget as for the study
Bow
Text-to-image general- ization needs generative prior retrieval. See Sec. 4.2
Figure 9:
Fig. [ When directly using text embeddings for inference, the model renders samples which generally
resemble the prompt, but the visual quality is low (CLIP score 0.26 + 0.05, FII) ~ 87). Modeling the
prior with a conditional normalizing flow [8] 62] improves the visual quality and achieves similar
results in terms of text-consistency (CLIP score 0.26 + 0.3, FID ~ 45), albeit requiring paired data.
See Fig. for qualitative visualization and Appendix F.2.1 for implementation and training details.
'Tench'
VValure"
"Grey Fox
'Tiger'
Teeddy Bear'
'Moped'
'Harvester'
'Espresso'
Figure 10: RDM can be used for condiiconiiional generation on ImageNet despite being trained without class labels. To achieve this during inferenee, we compute pool of nearby visual instances from the database 'D for each class label based on its textual deseription, and combine it with its nearest neighbors conditioning.
Synthesis Similarly we can apply our model zero-shot class-conditional image synthesis as proposed in Sec. 3.3 Fig. 10 shows samples from our model for classes from ImageNet. More samples for all experiments can be found in Sec. G
4.3 Zero-Shot Text-Guided Stylization by Exchanging the Database
In our semi- paniiearraetricc model, the retrieval database D is an explicit part of the syn- thesis model. This allows novel applica- tions, such replacing this database after training to modify the model and thus its output. In this section we replace Dewin of the ImageNet-RDDM, built from Open- Images with an alternate database Dayle which contains all 138k images of the WikiArt dataset [66]. As in Sec. 4.2 we retrieve neighbors from Dayie via a text prompt and use the text-retrieval strategy tii). Results are shown in Fig. 11 (top row). Our model, though only trained on Ima- geNet, generalizes to this new database and is capable of generating artwork-like images which depict the content defined by the text prompts. To further emphasize the effects of this post-hoc exchange of D. show samples obtained with the same procedure but using Duain (bottom row).
'A basket full of fruits."
"A playing piano.
stap.
sable
Dayie
Drainn
Figure 11: Zero-shot text- stylization with our ImageNet-RDMM Best viewed when zoomed in.
4.4 Increasing Dataset Complexity
To investigate their versatility for complex generative tasks, we compare cerimparamettic models to their mully-parametro counterparts when systematically increasing the complexity of the training data p(x). For both RDM and RARM, we train three identical models and corresponding fully parametric baselines (for details cf. Sec. F.2p on the dogs-, mammals- and animals- of ImageNet [13]. cf. Tab. 7 until convergence. Fig. 12 visualizes the results. Even for lower-complexity datasets such as IN-Dogs, our semi-parametric models improve over the baselines except for recall, where RARM performance slightly worse than a standard AR model. For more complex datasets, the performance gains become more significant. Interestingly, the recall scores of our models improve with increasing complexity, while those of the baselines strongly degrade. We attribute this to
9

---

RPAM
KM
0.39
870
0.39
823
840
0.52
835
SePerdd
1efaires 140gg
BiCagee
HoFameb
Datwwel
Detawet
Figure 12: Assessing our approach when increasing dataset complexity as in Sec. 4.4 We observe that
performance gaps between semí- and polly-parametrie models increase for more complex datasets. the explicit access of underrepresented ones via the (T), of Eq. (6. whereas a standard generative model might focus only the modes containing the most often occurring classes (dogs in the case of ImageNet).
models to nearby visual instances for all classes including
4.5
Quality-Diversity Trade-Offs
Top-m sampling. In this section, we evaluate the effects of the top-m sampling strategy introduced in
We train a RDM the ImageNet T3] dataset and assess the usual generative performance metrics based on 50k generated samples and the entire training set (5) Results are shown in Fig. 3 For precision and recall scores, we observe truncation behavior similar to other inference- time sampling techniques (S 1S 32 23]: For small values of m, we obtain coherent samples, which all come from a single or a small number of modes, as indicated by large precision scores. Increasing m, on the other hand, boosts diversity at the expense of consistency. For FID) and IS, we find sweet spot for m = 0.01, which yields optima for both these metrics. Visual examples for different values of m are shown in the Fig. 16. Sec.
Sec. 3.3
E.5 also contains similar experiments for RARM
0.
Recail
D03
20
501
141
05
Precision
0.7
15
50
(i) Quality-diversity trade-offs when applying top-m sampling.
Assessing the effects of classifier free guidance.
Figure 13: Analysis of the quality-diversity trade-offs when applying top-m sampling and classifier- free guidance.
Classifier-free guidance. Since RDM is a conditional diffusion model (conditioned on the neighbor encodings ((y)),
apply classifier-free diffusion guidance [32] also for unconditional modeling. Interestingly, we find that we can apply this technique without adding an additional 0-label to account for a purely unconditional setting while training 6p. as originally proposed in 32 and instead use a vector of zeros to generate an unconditional prediction with cg. Additionally, this technique can be combined with top-m sampling obtain further control during sampling. In Fig. 13b we show the effects of this combination for the ImageNet-model as described in the previous paragraph, with m {0.01, 0.1) and classifier scale (1.0, 1.25, 1.5, 1.75, 2.0, 3.0), from left right for each line. Moreover we qualitatively show the effects of guidance in Fig. 18 demonstrating the versatility of these sampling strategies during inference.
5 Conclusion
This paper questions the prevalent paradigm of current generative image synthesis: rather than compressing large training data in ever-growing generative models, we have proposed to efficiently store an image database and condition a comparably small generative model directly on meaningful samples from the database. To identify informative samples for the synthesis tasks at hand we follow an efficient retrieval-basee approach. In the experiments our approach has outperformed the state of the art various synthesis tasks despite demanding significantly less memory and compute. Moreover, it allows (i) conditional synthesis for tasks for which has not been explicitly trained, and (ii) post-hoc transfer of a model to new domains by simply replacing the retrieval database. Combined with CLIP's joint feature space, our model achieves strong results on text-image synthesis, despite being trained only on images. In particular, our retrieval-based approach eliminates the need train an explicit generative prior model the latent CLIP space by directly covering the neighborhood of given data point. While we assume that our approach still benefits from scaling, it shows a path to more efficiently trained generative models of images.
A Acknowledgements
This work has been funded by the Deutsche Forschungsgemeinschaat (DFG, German Research Foundation) within project 42170392 and the German Federal Ministry for Economic Affairs and Energy within the project KI- Kl-Absicherung- Safe AI for automated driving.
10

---

References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference muchine learning, pages 214-223. PMLR, 2017.
[2] Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. arXiv preprint arXiv:2204. 02849, 2022.
[3] Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Björm Ommer. ipoke: Poking a still image for controlled stochastic vidco synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14707-14717, October 2021.
[4] Scbastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- Bapttste Lespiau. Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprint
2021.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rX.v:1809.I 10096 2018.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:180 13:1877-1901, 2020.
[7] Holger Caesar. Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. pages 1209-1210 1218, 2018, doi: 10.1 100.119//VPP... 2012200110001 8.00132. URL WtRLhhtp::// httt::///penaccess . thecvf com/
content_cvpr. 2018/html////// /Caesarr CDCD- Stuff_ Thing _a___andd_______ CVPR_ 2018, 1appreppppppp___- .ttll
[8] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance-conditioned gan. Advances in Neural Information Processing Systems. 34, 2021.
[9] Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-rcsolution training for high-resolution image synthesis. arXiv preprint arXiv:2204.00
2022.
[10] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun. David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMI.R, 2020.
[11] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. ArXiv, abs/1604.061 (664006664,,, 174,, 2016.
[12] Katherine Crowson. Tweet on Classifier-free guidance for autoregressive models. htttps://tiitee
com htnm///vversaaaeeiingga////////e http:///wwinggttttteuaeeaaee /ttpe///4788935876666669333
2022.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenct: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Ieee, 2009.
[14] Emily Denton. Ethical considerations of generative ai. Al for Content Creation Workshop, CVPR, 2021. URL (RRp//ttss//drrl http://rrrve.oo.ll....... cow homm/file/dll
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[16] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation.
arXiv preprint arXiv: /410.85 2014.
[17] Laurent Dinh, Jascha arXiv: 1605.088000000 2016.
and Samy Bengio. Density estimation using real nvp. arXiv preprint
[18] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017 Conference Track Proceedings. OpenReview.net, 2017. URL httphttt/pp/eeree///// net fott/forum?idd
HkpbnH91x
[19] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint urXiv:2010. 11929, 2020.
[20] Patrick Esser, Robin Rombach, and Björ Ommer. A note on data biases in generative models. arXiv preprint wwXi:20022.05166 2020.
[21] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advonces in Neural Information Processing Systems, 34, 2021.
[22] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873 128812283,,22 2021.
11

---

[23] Angela Fan, Mike Lewis. and Yann N. Dauphin. Hierarchical neural story generation. CoRR. abs/1805.04833, 2018. URL Jttttp////arzi http:://arziv.oog/bbba////// arg/abs/1805. 04833
[24] Mary Anne Franks and Ari Ezra Waldman. Sex, lies, and videotape: Deep fakes and free speech delusions. Md. I.. Rev. 78:892, 2018.
[25] lan Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014,
[26] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint arXiv:2111, 14822, 2021.
[27] Ishaan Guirajani. Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[28] Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. pages
PMLR, 13-18 18 Jul 2020. URL https
//proceedings mlr. prwss/ v119//11111//1 www19///g/0..... html html
[29] Kelvin Guu, Kenton Lce, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retricval augmented language model pre-training. In International Conference on Machine Learning, pages 3929-3938. PMLR, 2020.
[30] Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus), 2016. URLAttps://arxxv.. abs/ 08415
[31] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Adv. Neural Inform. Process. Syst., pages 6626-6637, 2017.
[32] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
[33] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[34] Jonathan Ho. Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022.
[35] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.0 234404348 2022.
[36] Niharika Jain, Alberto Olmo, Sailik Sengupta, Lydia Manikonda, and Subbarao Kambhampati. Imperfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat selfie lenses. arXiv preprint arXi::2001.099
2020.
[37] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CORR, 154/2001.08361,
[38] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 44100, 2019.
[39] Tero Karras. Samuli Laine. Mïka Aittlla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pages 81 8110-8119. 2020.
[40] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
[41] Urvashi Khandelwal, Omer Levy. Dan Jurafsky. Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172 2019.
[42] Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zeulemoyer, and Mike Lewis. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710 2020.
[43] Diederík Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[44] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.006:: 2021.
[45] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
12

---

[46] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings. Ivan Krasin, Jordi Pont-Tuset. Shahab Kamali, Stefan Popov, Matteo Malloci. Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR. abs/181 11118111009922 2018. URL http:///://arx/vv.oo owg/abss/.rrr////////a//a/b 1811. 00982
[47] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. CoRR, abs/1904,06991, 2019. URL http://aa/a org/ arg//bb/1904..... 1904. 06991
[48] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. CoRR, 2bs/2203.00026, 2022.
[49] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain gencralization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[50] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. Advances in neural information processing sysiems, 31, 2018.
[51] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. arXiv preprint arXiv:2202.1 11333, 2022.
[52] Yuxian Meng, Shi Zong. Xiaoya Li, Xiaofei Sun, Tianwei Zhang, Fei Wu, and Jiwei Li. Gnn-Im: Language modeling based global contexts via gnn. arXiv preprint arXiv:2110.08 2021.
[53] Lars Mescheder. Sebastian Nowozin, and Andreas Geiger. The numerics of gans. Advances in neural information processing systems, 30, 2017.
[54] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 2481-3490. PMLR, 2018.
[55] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dick Dick-Dissttin. Unrolled generative adversarial networks. arXiv preprint arXiv: /61 1, 02.02/163, 2016.
[56] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. /0741, 2021.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Muchine Learning, pages 8748-8763, PMLR, 2021.
[58] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[59] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 06/2,, 2022.
[60] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals, Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[61] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[62] Robin Rombach, Patrick Esser, and Björn Ommer. Network-too-network translation with conditional invertible neural networks. In NeurlPS, 2020.
[63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffiusion models. arXiv preprint arXiv:2112. 10752, 2021.
[64]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCA/ (3), volume 9351 of Lecture Notex in Computer Science, pages 234-241. Springer, 2015.
[65] Chitwan Saharia. Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:200.006666, 2021.
[66] Babak Salch and Ahmed M. Elgammal. Large-scale classification of fine-art paintings: Learning the right metric the right feature. CoRR, abs/1505. 00855, 2015. http:/tt: wwwpexiivvoor/aaaa org//as////reeeeeeee/eeee http//abs/1050... 10850008555500005
[67] Tim Salimans, 1. Goodřellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016
13

---

[68] Tim Salimans. Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 170.05771.. 2017.
[69] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. CoRR, 0bs/2111.01001 2021. URL httt.tttts/:/aiitt wwwp/r/g///////eee/o.//eee ats//11b 00000
[70] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Sealing stylegan to large diverse datasets. arXiv preprint arx.v:220..00273, 2022.
[71]
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of elip-filtered 400 million image-text pairs, 2021.
[72] Christoph Schuhmann, Romain Bcaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[73] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
[74] Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Retrievalfuse: Neural 3d scene reconstruction with a database. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12568-122 12577, 2021.
[75] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2C: diftussinnnonn models for few-shot conditional generation. CoRR, abs/116.006819, 2021. /tt.l/ttttt//arxi.or////aaaa a106
06819
[76] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thernodynamics. In International Conference on Machine Learning, pages 2256-2265, PMLR, 2015.
[77] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arX.v:2010.02.52,, 2020.
[78] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[79] Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. CoRR, abs/201 1.1455,, 2020.
[80] Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational lcarning, Advances in neural information processing systems, 30, 2017.
[81] Rich Sutton. The bitter lesson, 2019. URL BitterLesson. ]tm]
[82] Antonio Torralba and Alexci Efros. Unbiased look dataset bias. In CVPR 2011, pages 152211128. IEFE, 2011.
[83] Hung- Yu Tseng, Hsin- Ying Lee, Lu Jiang, Ming-IIsuan Yang. and Weilong Yang. Retrievegan: Image synthesis via differentiable patch retrieval. In European Conference Computer Vision, pages 242-257. Springer, 2020.
[84] Arash Vahdat and Jan Kautz. Nvae: A deep hicrarchical variational autoencoder. Advances in Neural Information Processing Systems. 33: 13619967-1977, 2020.
[85] Aaron Van den Oord, Nal Kalehbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[86] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[87] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747-1756. PMLR, 2016.
[88] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems. 30, 2017.
[89] Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers, CoRR, 0lx/2203.08913 2022. doi: 00448550arxxi.2203..8999 /RL.hhtts::///o.orrgg http:////ooooo//.../ 48550//rx1 2203.08913 08913
[90] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, arXiv preprint 12.00221 2021.
14

---

[91] Rui Xu. Minghao Guo, Jiaqi Wang, Xiaoxiao Li. Bolei Zhou, and Chen Change Loy. Texture memory- augmented deep patch-based image inpainting. IEEE Transactions Image Processing, 30:91 12-9124, 2021.
[92] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video genera- tion. arXiv preprint S.Ki:2203300488 2022.
[93] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. arXiv preprint arXiv:2 / 10.04627, 2021.
[94] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. arXiv preprint arXiv:2111.1379. 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] The code will be released, the data is publicly available and the additional instructions are provided in the supplemental material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes] The code and pretrained models will be released.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "wkHCZW12AWu", "reviewer": "Reviewer_fYVT", "summary": "This paper attempts to solve the problem of image synthesis (unconditional, conditional, text-guided) in a semi-parametric way. It extends IC-GAN [1] by introducing an external retrieval dataset (used for kNN search), and a pretrained fixed CLIP encoder (used for encoding image and text). The proposed method could be used on various image synthesis frameworks (e.g. Diffusion-based, Autoregressive-based models).\n\nA fundamentally similar concurrent work is kNNDiffusion [2] (also mentioned by the authors in Appendix C)\n\n[1] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance-conditioned gan. *Advances in Neural Information Processing Systems*, 34, 2021.\n\n[2] Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. *arXiv preprint arXiv:2204.02849*, 2022.", "questions": "Please see weaknesses above. Overall, I think this paper is a good extension of IC-GAN by introducing the retrieval-based methods to diffusion and autoregressive models, along with the combination with CLIP. However, some ablation studies, quantitative experiments, and more in-depth analysis with the proposed method are missing. If the authors could address these, I am willing to raise my score.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The proposed method could be trained with images only, while allowing tasks including conditional, unconditional, text-guided image synthesis. This is achieved by aligning the latent space to a pretrained fixed CLIP encoder.\n- Good experiments of introducing the idea of retrieval based image synthesis in diffusion models and autoregressive models.\n\nWeaknesses\n\n- One major difference the authors have mentioned against IC-GAN is the external retrieval dataset (as IC-GAN use the same retrieval and training dataset). How does changing retrieval dataset to a different one affect the results? Does the diversity matters or is it the dataset size matters? If we use the same training and retrieval dataset in this work, does the result differ a lot?\n- Domain transfer by replacing the retrieval dataset is also presented in IC-GAN, what is the fundamental difference here?\n- It seems that the quantitative experiments of text-to-image task on COCO, and class conditional synthesis on ImageNet against SOTAs are missing.\n- In Figure 2, using text representation only performs better than using kNN samples and the combination of these two. Does this mean NN is not important for text-to-image synthesis? Then, what is the fundamental reason to introduce NN retrieval here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tD0hRvAra13", "reviewer": "Reviewer_XT8s", "summary": "This paper proposes a method of using an external database of images to conditional a smaller generative image model for neural synthesis. Since the model is conditioned on the external database, some amount of domain transfer can be granted through changing the exemplars in the database. This external database also tackles a scalability problem, as retrieving nearest neighbor exemplars from the database is more efficient than trying to parameterize larger and larger datasets into a generative model.\n\nPost Comments: Given the rebuttal of the authors addressing my concerns and an improvement of the ethics statement in the paper, I feel it's appropriate to increase my overall rating. I feel many of the timings per exemplar are a bit misleading, given that these are taken over large batched and amortized.", "questions": "1. Figure 1 is very misleading. There is a missing graph discontinuity on the X axis for model params (e.g. 0 is at 325 M params). Visually this is showing that the semi parametric models are around 1/8th the size when in fact they are 2/3rds to ~5/6ths the size. Additionally, this graph shows a flattening of FID as the database increases. Is this true of a model whose |theta| + |D| is the same size as the ADM w/classifier baseline?\n\n2. L56, retrieving the exemplars at inference time is 0.95 ms. Since these images also need to be CLIP encoded, what is the time or compute spent to encode |D|.\n\n3. One of the claims of this paper is domain transfer being possible with a change of the image database. Where any visually distinct domain transfers experimented with (i.e. style transfer).\n\n4. What does the distribution of training data look like for parametric and semi-parametric approaches? Do semi-parametric models need less training time or less examples?", "limitations": "As in all generative work, there is the danger of generating images that can be harmful (used to incite fear, riots, harassment). The authors mention no limitations or negative societal impact of their work which I believe is an oversight. I don't think this research has any increased risk over other generative works, in fact, with careful curation of the database, this may actually *decrease* the potential to generate harmful content. The authors should more carefully consider the societal implications of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nUse of an external database and retrieving nearest neighbor exemplars. Additionally, a database of images may not even be needed if you are pre-computing CLIP embeddings. This allows for a very efficient 2048 bits of representation per image.\n\nWith increasingly complex datasets, the semi-parametric models increase in recall performance (which is not true of the fully parametric baselines). \n\nWeaknesses:\nDomain transfer isn't explicitly shown. For example, a database of natural images used for training and cartoon images for exemplars to generate a \"cartoon tiger\" or \"animated bear\" would show a domain transfer from natural to animation images. The domain transfer examples shown are essentially the same visual domain as the training and the exemplars.\n\nFID isn't state of art compared to classifier guided ADM, even with about 80% of the parameters.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "k4X14VFGaHZ", "reviewer": "Reviewer_aPkQ", "summary": "The paper proposes retrieval-augmented generative model, which is a semi-parametric model conditioned on retrieved features.", "questions": "In Table 1, ADM and RDM perform sampling with different hyper-parameters. Some more results are suggested for fair comparison (e.g. ADM with 100 steps or RDM with 250 steps).\n\nInception Score on MS-COCO dataset for text-to-image generation is suggested to be reported.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method is straight-forward and effective, good performances are achieved with smaller model size;\n\nThe paper is well-written and easy to follow;\n\nThe use of pre-trained CLIP model enables image generation, text-to-image generation and class-conditioned generation; Good experiment results are obtained on image generation task;\n\n\nWeaknesses:\n\nSome related works [1, 2] which also perform generation conditioned on CLIP features should be discussed and compared. Specifically, [1] also retrieve image features based on CLIP multi-modal joint feature space, and train the model based on retrieved features. The only difference is that [1] trains a diffusion model in a latent space;\n\nThe text-to-image generation results on MS-COCO dataset is worse than related works;\n\nDiversity of generation is influenced\n\n[1]. KNN-Diffusion: Image Generation via Large-Scale Retrieval. Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, Yaniv Taigman.\n\n[2]. LAFITE: Towards Language-Free Training for Text-to-Image Generation. Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, Tong Sun.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "IpJRHqnckp", "reviewer": "Reviewer_B1f9", "summary": "This paper tries to use an additional database D to do generation. The idea is very simple and straightforward. During training for a training image x, they use it to find the k images from the D by using CLIP image encoder and K nearest neighbor. Then they feed those retrieved image CLIP embeddings into their generative model as extra information.\n\nBecause of the shared space between text and image, once their model is trained they can do text2image generation and class conditional generation even though the model is trained unconditionally. \n\nThey studied what is the best value of k, and propose a top-m sampling strategy for a trade-off between diversity and fidelity.  \n\n\n ", "questions": "Here are my major concerns for this paper regarding the setting and high-level: \n\n1, The main claim of this paper is having an additional database. But, they do not justify why they want to have a disjoint D? What if D is the same as the training data? They should have shown what are the benefits of using different images as D. They have something similar in L289, but in this case k=1. They should properly study a model with k=4 when D is the same as training images. \n\n2, They do not show enough ablation studies on the database. For example, how would different databases influence the model's performance and behavior? Although they studied the patch size used in OpenImage in the supp, those database features are all from the OpenImage, thus still similar in my mind.\n\n3,  Similarly as above, they should also study the effect of the same database on the different training sets. They only conducted the training on ImageNet, what about the other dataset such as the single domain dataset (FFHQ, LSUN)? In these cases, how would the current database (OpenImage) influence the models? Does the database still provide useful information?  If not, why does someone need the database? \n\n4, A lot of studies were conducted on text-to-image generation, zero-shot, etc.  But actually, the reason why they can do this is mainly because of CLIP, not because of the extra dataset. Thus I do think they present the paper appropriately. If they want to highlight these properties, then they should make their story based more on CLIP. If they still want to make their main story be the database, then they need to thoroughly study the concerns I listed above. \n\n5, What is the true limitation of this paper? Although they mentioned limitations in the supp, but I do not think they take it seriously. What they mentioned are general problems for all diffusion models such as sampling speed. I hope to see they talk about the limitation unique to their approach. \n\nHere are my major questions regarding the existing experiments and statements:\n\n6, In table 1, they miss the LDM baseline.\n\n7, In table 1, they should also list the database size, instead of only parameters. Since storage is also a main concern in a real-world application. They mentioned it in the footnote on page 4, but they should explicitly list it in table 1 to show the extra storage cost.\n\n8, When talking about DALLE2, they said DALLE2 prior training requires paired data and they do not (L287), which I do not think is accurate. Since the CLIP training also needs paired data. Actually, I do not fully understand the purpose of L279-L295, if the goal of this part is to show their approach does not need to train a prior, then they need to show why a feature database is better than a prior trained on the features from the database.   \n\n\nMinor concerns:\n\n1, in L157, I am not sure if LDM is the first one to propose cross attention. It seems GLIDE also uses cross attention to fuse image and text features.\n\n2, for fig2, they can do a user study instead of only show 4 images.  \n\n\n \n\n \n \n", "limitations": "My major question for this paper was they did not have enough study and analysis for training data and extra database, which is the key claim of their paper. Since they addressed them well in the rebuttal, I am happy to raise my score. \n\nI still have the same concern about the way they present the paper. The main claim of this paper is using retrieval to help generation, BUT, the other very important component in their approach is CLIP which I do not think is replaceable. In other words, if I replace CLIP with normal Resnet trained on Imagenet, I do not think one can play around with text anymore. But I can still call my paper as \"Semi-Parametric Neural Image Synthesis\" or my model as \"RDM\". Thus I think their title and story is a bit over claim or not precise if one prefer.     \n\nIn summary, the contribution of this work outweighs my concern. But I highly suggest to put their ablation on D and X conducted in the rebuttal to the main paper since these ablations backup their main claim. Also consider adding CLIP into the story when present the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nUsing an additional database to help generation. (I have a mixed feeling about this point as using an external bank of images is not a new philosophy in image generation)\nThe paper writing is very clear and easy to follow. \nThey achieve state-of-the-art unconditional generation results on ImageNet with less trainable parameters.\n\nWeakness:\n\nIt lacks novelty. The model architecture follows LDM. Conditioning on CLIP embedding to enable the power of text2image generation is similar to DALLE2, and should be attributed to CLIP.  \n\nMy main concerns/doubts are listed in the \"Questions\" section. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept