Re-Imagen: Retrieval-Augmented Text-to-Image Generator
Authors: Wenhu Chen
Hexiang Hu
Chitwan Saharia
William W. Cohen
Venue: ICLR 2023 poster (2023)
Keywords: Diffusion Model
Information Retrieval
Knowledge Grounding
Image Generation

Abstract: Research on text-to-image generation has witnessed significant progress in generating diverse and photo-realistic images, driven by diffusion and auto-regressive models trained on large-scale image-text data. Though state-of-the-art models can generate high-quality images of common entities, they often have difficulty generating images of uncommon entities, such as `Chortai (dog)' or `Picarones (food)'. To tackle this issue, we present the Retrieval-Augmented Text-to-Image Generator (Re-Imagen), a generative model that uses retrieved information to produce high-fidelity and faithful images, even for rare or unseen entities. Given a text prompt, Re-Imagen accesses an external multi-modal knowledge base to retrieve relevant (image, text) pairs, and uses them as references to generate the image. With this retrieval step, Re-Imagen is augmented with the knowledge of high-level semantics and low-level visual details of the mentioned entities, and thus improves its accuracy in generating the entities' visual appearances. We train Re-Imagen on a constructed dataset containing (image,text,retrieval) triples to teach the model to ground on both text prompt and retrieval. Furthermore, we develop a new sampling strategy to interleave the classifier-free guidance for text and retrieval condition to balance the text and retrieval alignment. Re-Imagen achieves new SoTA FID results on two image generation benchmarks, such as COCO (\ie, FID = 5.25) and WikiImage (\ie, FID = 5.82) without fine-tuning. To further evaluate the capabilities of the model, we introduce EntityDrawBench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple visual domains. Human evaluation on EntityDrawBench shows that Re-Imagen performs on par with the best prior models in photo-realism, but with significantly better real-world faithfulness, especially on less frequent entities. 

Published as a conference paper at ICLR 2023
RE-IMAGEN: RETRIEVAL-AUGMEETED TEXT-TO-IMAGE GENERATOR
Wenhu Chen, Hexiang Hu, Chitwan Saharia, William W. Cohen Google Research
(wenhuchen, hexi ang, sahariac, wcohen) com
ABSTRACT
Research on text-to-image generation has witnessed significant progress in gener- ating diverse and photo-realistic images, driven by diffusion and rugresseeee models trained on large-scale image-text data. Though state-of-the-art models can generate high-quality images of common entities, they often have difficulty gener- ating images of uncommon entities, such as 'Chortai (dog)' or 'Picarones (food)'. -ext- Gener-
To tackle this issue, we present the Retrieval- Augmented ator (Re-Imagen), a generative model that uses retrieved information to produce high-fidelity and faithful images, even for rare or unseen entities. Given a text prompt, Re-Imagen accesses external multi-modal knowledge base to retrieve relevant (image, text) pairs and uses them as references to generate the image. With this retrieval step, Re-Imagen is augmented with the knowledge of high- level semantics and low-level visual details of the mentioned entities, and thus improves its accuracy in generating the entities' visual appearances. We train Re- Imagen on a constructed dataset containing (image, text, retrieval) triples to teach the model to ground on both text prompt and retrieval. Furthermore, we develop a new sampling strategy to interleave the classifier-fre guidance for text and re- trieval conditions to balance the text and retrieval alignment. Re-Imagen achieves significant gain on FID score over COCO and Wikilmage. To further evaluate the capabilities of the model, we introduce Entity Bench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple object categories including dogs, foods, landmarks, birds, and characters. Human evaluation EntityDrawBeer shows that Re- Re-Imagen significantly improve the fidelity of generated images, especially on less frequent entities.
INTRODUCTION
Recent research efforts on conditional generative modeling, such as Imagen (Saharia et al. 2022). DALL-E 2 (Ramesh et al. 2022). and Parti YYu et al. (022). have advanced text-to-image generation to an unprecedented level. producing accurate, diverse, and even create images from text prompts. These models leverage paired image-text data Web scale (with hundreds of millions of training examples), and powerful backbone generative models, ie., autoregressive models (Van Den Oord
al. 2017 Ramesh al. 2021 Yu et al. 2022), diffusion models (Ho al.) [0200] Dhariwal & Nichol 2021). etc, and generate highly realistic images. Studying these models' generation results, we discovered their outputs are surprisingly sensitive to the frequency of the entities (or objects) in the text prompts. In particular, when generating text prompts about frequent entities, these models often generate realistic images, with faithful grounding the entities" visual appearance. However, when generating from text prompts with less frequent entities, those models either hallucinate non- existent entities, or output related frequent entities (see Figure 1), failing to establish a connection between the generated image and the visual appearance of the mentioned entity. This key limita- tion can greatly harm the trustworthiness of text-to-image models in real-world applications and raise ethical concerns. In our studies, we found these models suffer from significant quality degradation in generating visual objects associated with under-represented groups.
In this paper, propose Retrieval- auureete- Text-to- Image Generator (Re- (Re-emag,n), which alle- viates such limitations by searching for entity information a multi-modas knowledge base, rather than attempting memorize the appearance of rare entities. Specifically, we define our multi-modal

---

Published as a conference paper at ICLR 2023
Imagen
Durs
imagen
Ours
Entry Rnce
Entry Reference
Barbedo Terceira is standing by a river.
Piccrones served with wind.
Entry Reference
Referance
Uningqushho served with wine.
Firds Are fyynn around Morument to the Fallin Shipyard Vickaaa of 1970.
Figure 1:
Comparison of images generated by Imagen and Re-Imagen on less frequent entities. We observe that Imagen hallucinates the entities while Re-Imagen maintains better faithfulness.
knowledge base encodes the visual appearances and descriptions of entities with collection of refer- ence <image, text> pairs'. To use this resource, Re-Imagen first uses the input text prompt to retrieve the most relevant <image, text> pairs from the external multi-modal knowledge base, then uses the retrieved knowledge as model additional inputs to synthesize the target images. Consequently, the retrieved references provide knowledge regarding the semantic attributes and the concrete visual appearance of mentioned entities to guide Re-Imagen to paint the entities in the target images.
The backbone of Re- Re-Imagen is cascaded diffusion model (Ho al. 2022), which contains three independent generation stages (implemented as U-Nets (Ronneberger et al.] 2015)) to gradually produce high-resolution (i.e., 1024 x 1024) images. particular, we train Re-Imagen constructed from the image-text dataset used by Imagen (Saharia et al. 2022), where each data in- stance associated with the top-k nearest neighbors within the dataset, based on text-only BM25 score. The retrieved top-k <image, text> pairs will be used as a reference for the model attend to. During inference, we design interleaved guidance schedule that switches between text guidance and retrieval guidance, which ensures both text alignment and entity alignment. We show some examples generated by Re-Imngen, and compare them against Imagen in [Figure I] We can qualita- tively observe that our images are more faithful to the appearance of the reference entity.
dataset
To further quantitatively evaluate Re-Imagen, we present zero-shot text-to-image generation results on two challenging datasets: COCO (Lin et al. 2014) and Wikilmages (Chang al. 2022 Re- Imagen uses an external non-overlapping image-text database as the knowledge base for retrieval and then grounds on the retrieval to synthesize the target image. We show that Re- Rmagen achieves the state-of-the. art performance for text-to-image generation on COCO and Wikilmages, measured in FID score (Heusel al.. 2017), among non-fine-tuned models. For the non-entity-centréc dataset COCO, the performance gain coming from biasing the model to generate images with similar styles as the retrieved in-domain images. For the entity-centric dataset Wikilmages, the perfor- mance gain comes from grounding the generation on retrieved images containing similar entities. We further evaluate Re-Imagen on more challenging benchmark Entity Benw to test the model's 's ability to generate variety of infrequent entities (dogs, landmarks, foods, birds, animated characters) in different scenes. We compare Re-Imaeen with Imagen (Saharia et al., (2022). DALL- E 2 (Ramesh et al. 2022) and StableDiffusion (Rombach et al.] (022) in terms of faithfulness and photorealism with human raters. We demonstrate that Re-Imagen can generate faithful and realistic images 80% over input prompts, beating the existing best models by at least 30% Entity- DrawBench. Analysis shows that the improvements are mostly coming from low-frequency visual entities.
To summarize, our key contributions are: (1) a novel retrieval -uggeette text-to-imanl model Re- Imagen, which improves FID scores two datasets; (2) interleaved classifier- free guidance during sampling to ensure both text alignment and entity fidelity; and (3) We introduce EntityDraw and show that Re- Rm-Inn can significantly improve faithfulness on less-frequent entities.
'The original Wikilmages database contains (entity image, entity description) pairs. It was crawled from Wikimedia Commons for visual question answering, and we repurpose it here for text-to-image generation.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Text-to-Image Difflusion Models There has been a wide-spread success (Ashual et al. 2022
Ramesh et al. 2022: Saharia al. 2022 Nichol al. 2021) modeling text-to-image generation with diflusion models, which has outperformed GANs (Goodiellow et al. 2014) and auto-regressive Transformers (Ramesh et 202T) photorealism and diversity (under similar model size), with- out training instability and mode collapsing issues. Among them, some recent large text-to-imalt models such as Imagen (Saharia et al. 2022], GLIDE (Nichol et al. 2021], J. and DALL-E2 (Ramesh) et al. 2022) have demonstrated excellent generation from complex prompt inputs. These models achieve highly fine-grained control over the generated images with text inputs. However, they do not perform explicit grounding over external visual knowledge and are restricted to memorizing the visual appearance of every possible visual entity in their parameters. This makes it difficult for them to generalize to model from memorizing, as models are encouraged to retrieve semantic neighbors from the knowl- edge base and use retrievals context paint the image. Re-Imagen improves the grounding of the diffusion models to real-world knowledge and is therefore capable of faithful image synthesis.
unseen entities. In contrast, Re-Imagen is designed to free the diffusion
Concurrent Work There are several concurrent works (Li et al. 2022 Blattmann et al., 2022 Ashual et al. 2022), that also leverage retrieval to improve diffusion models. RDM (Blattmann et al. (202)) is trained similarly to Re-Imagen, using examples and near neighbors, but the neigh- bors in RDM are selected using image features, and at inference time retrievals are replaced with user-chosen exemplars. RDM was shown to effectively transfer artistic style from exemplars to generated images. In contrast, our proposed Re-Imagen conditions on both text and multi-modal neighbors to generate the image includes retrieval at inference time and is demonstrated to improve performance on rare images (as well as more generally). KNN-Diffusion (Ashual et al., 2022) is more closely related work to us, it also uses retrieval the quality of generated images. How- ever, LNN-Diifuuiinn uses discrete image representations, while Re- Imagen uses the raw pixels, and Re-Imagen's 's retrieved neighbors can be <image, text> pairs, while KNN-Diffusion's are only im- ages. Quantitatively, Re-Imagen outperforms KNN-Diffusion on the COCO dataset significantly.
Others Due to the space limit, we provide an additional literature review in Appendix A
3 MODEL
In this section, we start with background knowledge, in the form of a brief overview of the cascaded diffusion models used by Imagen. Next, we describe the conerete technical details of how we incorporate retrieval for Re- Rmagen. Finally, we discuss interleaved guidance sampling.
3.1 PRELIMINARIES
Diffusion Models Diffusion models Sohh--Dickstenn et al. 2015) are latent variable models, param- eterized by 8, in the form of po (xx))
po (Forr dæ where .1, .y are "noised" latent versions of the input image To ~ g( Note that the dimensionality of both latents and the im- age is the same throughout the entire process, with 2p.r E Rd and d equals the product of <height, width, # of channels>. The process that computes the posterior distribution q (Zo) is also called the forward (or diffusion) process, and is implemented as a predefined Markov chain that gradually adds Gaussian noise to the data according to a schedule Sy;
q( 2D)
II [[( (=1
== N(zx; /1 3,01-1,
(1)
Diffusion models are trained to learn the image distribution by reversing the diffusion Markov chain. Theoretically, this reduces to learning to denoise o; (())))))) into æo, with a time re-weighted square error loss-sce see Ho et al. (2020) for the complete proof:
Ex.... || c) xulli
(2)
Here, the noised image is denoted as æ; := Var2o /1 ā,e, By is the ground-truth image, is
the condition, (O, I) is the noise term, a₁ 1- 3, and o₁ I as- To simplify notation,
3

---

Published as a conference paper at ICLR 2023
X
Edtt Cx.).
Cy' Two Chortai are running the field.
UNet
UNet
Query
Attention
Attention
Retrieval
Chortai breed dog
Database
An illustration of the text-to-image generation pipeline in the 64x diffusion model. Specifically, Re-Imagen learns UNet to iteratively predict c( Cn. Cp. t) that denoises the image. (cn; set of retrieved image-text pairs from the database; Cyi input text prompt; current time-step)
Figure 2:
we will allow the condition include multiple conditioning signals, such as text prompts Cp. low-resolution image input C₂ (which is used in super- resouutes, or retrieved neighboring images Cn (which are used Re-Imagen). Imagen (Saharia et al., 2022) uses a U-Net (Ronneberger al. 2015 to implement eg (2,,, c, 1). The U-Net represents the reversed noise generator as follows:
(2 c) == (æa v1 (ac c, c,t))//aa
During the training, we randomly sample minimize the difference between (2 and 2o according to Equation 2 At the inference time, the diffusion model uses DDPM (Ho et al., 2020) to sample recursively as follows:
2((0,) 1J) and image Bi from the dataset D, and
t c) 3 1 - ā₁
Var(1 ū;-1) 2t + v(1 āz-1)3t 1 āt
Lt-1
V1 21-d8 -,
(4)
The model sets æт Gaussian noise with 7 denoting the total number of diffusion steps, and then keeps sampling in reverse until step T = 0, i.e. ØT - -T-1 - ..., to reach final image 2o
For better generation efficiency, cascaded diffusion models (Ho et al. 2022 Ramesh et al. 2022 Sa- haria et al. 2022) use three separate diffusion models to generate high-resolution images gradually, going from low resolution high resolution. The three models 64x model, 256x super-resolution model, and 1024x super-resolution model gradually increase the model resolution to 1024 x 1024.
Classifier-free Guidance Ho & Salimans (2021) first proposed classifier-free guidance trade off diversity and sample quality. This sampling strategy has been widely used due to its simplicity. In particular, Imagen (Saharia al. 2022) adopts an adjusted e-prediction follows:
= w 60 (2,,, C, 1) - (w - 1) en (x :))
(5)
where is the guidance weight. The unconditional c-prediction c0 t) is calculated by dropping the condition, iee the text prompt.
3.2 GENERATING IMAGE WITH MULTI-MODAL. KNOWLEDGE
Similar to Imagen (Shariaa al. 2022). Re-Imagen cascaded diffusion model, consisting of 64x, 256x, and 1024x diffusion models. However, Re-Ina augments the diffusion model with the new capability of leveraging multimodal 'knowledge' from the external database, thus freeing the model from memorizing the appearance of rare entities. For brevity (and concreteness) we present below a high-level overview of the 64x model: the others are similar.
Main Idea As shown in Figure 2 during the denoising process, Re-Imagen conditions its generation result not only on the text prompt Cp (and also with c, for super-resolution, but on the neighbors On that were retrieved from the external knowledge base. Here, the text prompt C E ig is repre- sented using a T5 embedding (Raffel et al. 2020), with being the text length and d being the em- bedding dimension. Meanwhile, the top-k neighbors e, := |<image, text>1, ... <image, text> retrieved from external knowledge base B, using the input prompt p as the query and retrieval similarity function (p, B). We experimented with two different choices for the similarity function: maximum inner product scores for BM25 (Rohertson et al. 2009) and CLIP (Radford al. 2021)
4

---

Published as a conference paper at ICLR 2023
Id Sp
fdltp Sp 9p
cgltp Sp Cp
Attention
Dstack
Ustack
Gi Two Chortai are running the field.
Cn
f)
Chortai is a breed of dog
Figure 3: The detailed architecture of our model. The retrieved neighbors are first encoded using the DStack encoder and then used to augment the intermediate representation of the denoising image (via attenntttnn The augmented representation is fed to the UStack to predict the noise.
Model Architecture We show the architecture of our model in Figure 3 where we decompose the UNet into the downsampling encoder (DStack) and the upsampling decoder (UStack). Specifically, the DStack takes an image, a text, and a time step as the input, and generates a feature map, which denoted as (a,,, Cp: RF Fxxxx with F denoting the feature map width and d denoting the hidden dimension. We share the same DStack encoder when we encode the retrieved <image, text> pairs (with set zero) which produce set of feature maps fe(c,, 0) E K xxxxxFxx We then use a multi-head attention module (Vaswani et al. 2017) to extract the most relevant information to pro- duce new feature map fg (Zee Cp. Cn. t) Aftm(fo (2x, Cps f), fo(cn, 0)). The upsampling stack decoder then predicts the noise term Co (Zt, Cp. Ca, t) and uses it to compute 2h with Equation 3) which is either used for regression during training or DDPM sampling.
Model Training In order to train Re-Imagen, we construct new dataset KNN-Image based on the 50M Text-daastt used in Imagen. There are two motivations for selecting this dataset. (1) the dataset contains many similar photos regarding specific entities, which is extremely help- ful for obtaining similar neighbors, and (2) the dataset is highly sanitized with fewer unethical or harmful images. For each instance the 50M ImageText-dataset,, we search over the same dataset with text-to-text BM25 similarity to find the top-2 neighbors C, (excluding the query instance). We experimented with both CLIP and BM25 similarity scores, and retrieval was implemented with the KNN-ImageTText by minimizing the loss
ScaNN (Guo et al., 2020). We train Re-Imagen function of Equation 2] During training, we also randomly drop the text and neighbor conditions independently with 10% chance. Such random dropping will help the model learn the marginalized noise term c (z,, Cp, t) and cg(x,, Ca, t), which will be used for the classifier-free guidance.
Interleaved Classifier-free Guidance Different from existing diffusion models, our model needs to deal with more than one condition, i.e., text prompts c2 and retrieved neighbors C,, which allows new options for incorporating guidance. In particular, Re-Imagen could use classifier-free guidance by subtracting the unconditioned 6-predictions, or either of the two partially conditioned e-predictions. Empirically, we observed that subtracting unconditioned e-predictions (the standard classifier- free guidance of Figure 3.1) often leads to an undesired imbalance, where the outputs are either domi- nated by the text condition the neighbor condition. Hence, we designed an interleaved guidance schedule that balances the two conditions. Formally, we define the two adjusted c- c-predictions as:
é Wp Co ca Cp. Ca, t) (wp 1) co( Ca. t) én = Wn. 60 Cp, Ca, t) - (w, - 1) c0 tp,
(6)
where ép and én are the text-enhanced and neighbor-enhancee c-predictions, respectfully. Here, Wp is the text guidance weight and w, is the neighbor guidance weight. We then interleave the two guidance predictions by a certain predefined ratio 7). Specifically, at each guidance step, we sample [0. ] random number R, and R < TJ, use ép. and otherwise é,, We can adjust v to balance the faithfulness w.r.t text description or the retrieved image-text pairs.
4 EXPERIMENTS
Re- Re-Ime consists of three submodels: 2.5B 64x64 text-to-image model. 750M 256x256 super-rese ruper-resolution model and a 400M 1024x 1024 super-resolution model. We also have a Re- Re-imgeen- small with 1.4B 64x64 text-to-image model to understand the impact of model size.
5

---

Published as a conference paper at ICLR 2023
COCO
Wikilmages
Model
Params
FID
ZS FID
FID ZS FID
GLIDE Nichol
2021
12.24
DALL-E 2 RRamesh ct al
2022
<-5B
10.39
Stable- (Rombach et a
2022
1B
12.63
7.50
Imagen (Saharia et al. 2022
3B
7.27
6.44
Make-A A Scene (Gafni ct al 2022
4B
7.55
11.84
Parti Yu al. 2022
20B
3.22
7.23
Retrieval-Augmentte Model
KNN-Diffusion (Ashual et al.
2022
16.66
Memory-Driven Text-to-Image Image
(Li ct al.
2022
19.47
Re- Bee-Imaeen (B25;, B=IND; k=2)
3,6B
5,25
5,88
Re-Imagen (y=BM255 B=0OD; k=2)
3.6B
6.88
5.80
Re- moagennn--lllll (7=BM25; B=IND; k=2)
2.4B
5.73
6.32
Re- Re-Imagen-small (y=BM25; B=OOD; 2=2)
2.4B
7.32
6.04
Table I: MS-COCO results for text-to-image generation. We use a guidance weight of 1.25 for the 64x diffusion model and for our 256x super-resolution model. (f is fine-tuned)
We finetune these models on the constructed KNN- Image'Text dataset. We evaluate the model under two settings: (1) automatic evaluation on COCO and Wikilmages dataset, to measure the model's general performance generate photorealistic images, and (2) human evaluation on the newly troduced EntityDrawBBnchh to measure the model's capability to generate long-tail entities.
Training and Evaluation details The fine-tuning was run for 200K steps on 64 TPU-ve -44 chips and completed within two days. We use Adafactor for the 64x model and Adam for the 256x super- resolution model with a learning rate of le-4. We set the number of neighbors k=2 and set R=BM25 during training. For the image-text database B, we consider three different variants: (1) the in- domain training set, which contains non-overlapping small-scale in-domain image-text pairs from COCO Wikilmages, (2) the out-of-domain LAION dataset (Schuhmann al. (021)) containing 400M <image, text> crawled pairs. Since indexing Image'Tex< and LAION with CLIP encodings is expensive, only considered the BM25 retriever for these databases.
4.1 EVALUATION ON COCO AND WIKIIMAGES
In these two experiments, we used the standard non-interleaved classifier- frreeffferffeeeeeeee guidance Figure 3.1 with 70=0000 steps for both the 64x diffusion model and 256x super-resolution model. The guid- ance weight w for the 64x model is swept over [1.0, 1.25. 1.5, 1.75, 2.0], while the 256 256 super- resolution models' guidance weight is swept over [1.0. 5.0, 8.0, 10.0]. We select the guidance with the best FID score, which is reported in Table We also demonstrate examples in Figure 4
COCO Results COCO the most widely-used benchmark for text-to-image generation models. Although COCO does not contain many entities, it does contain unusual combinations of com- mon entities, so it is plausible that retrieval augmentation could also help with some challenging text prompts. We adopt FID (Heusel et al. 2017) score to measure image quality. Following the previous literature, we randomly sample 30K prompts from the validation set as input to the model. The generated images are compared with the reference images from the full validation set (42K). We list the results in two columns: FID-30K denotes that the model with access to the in-domain COCO train set, while Zero-shot FID-30K does not have access to any COCO data.
Re-Imagen can achieve significant gain FID by retrieving from external databases: roughly 2.0 absolute FID improvement over Imagen. performance even better than fine-tuned Make- A-Scene (Gafni al. (022).. We found that Re-Imagen retrieving from OOD database achieves less gain than IND database, but still obtains a 0.4 FID improvement over Imagen. When comparing with other retrieval-augmentee
models, Re- Re-gnn is shown to outperform KNN-Diffusion and Memory- Driven T2I models by a significant margin of 11 FID score. We also note that Re- Re-Imagen-smal is also competent in the FID, which outperforms normal-sized Imagen with fewer parameters.
As COCO does not contain infrequent entities, retrievals from the in-domain database mainly pro- vide useful 'style knowledge" for the model ground on. Re-Imagen can better adapt to COCO

---

Published as a conference paper at ICLR 2023
w/ retrioval
w/o rotrioval
OOCD
CoCO Retrieval
Style
slope
stending
A hard of cattle standing front church with stoople.
Whllmagee
LAION
Retrivaal
Enily Groanding
San Giorgio Maggiore church
Island of San Giorgio Maggiore
landscape San Giorgio Maggiore.
Figure 4: The retrieved top-2 neighbors of COCO and Wikilmages and model generation.
distribution, thus achieving better FID score. As be seen in the upper part of from Figure 4 Re-Imagen with retrieval generates images of the same style as COCO, while without retrieval, the output still high quality, but the style less similar to COCO.
Wikilmages Results Wikilmages is constructed based on the multimodal corpus provided in Web- QA (Chang et al. 2022), which consists of <image, text> pairs crawled from Wikimedia Commons We filtered the original corpus to remove noisy data (see Appendix BB. which Ieads to a total of 320K examples. We randomly sample 22K as our validation set to perform zero-shot evaluation, we further sample 20K prompts from the dataset as the input. Similar to the previous experiment, we also adopt the guidance weight schedule before and evaluate 256x256 images.
From Table 1 we found that using the OOD database (LAION) actually achieves better performance than using the IND database. Unlike COCO, Wikilmages contains mostly entity-focused images, thus the importance of finding relevant entities in the database is more important than distilling the styles from the training set-and since the scale of LAION-400M 100x larger than an in-domain database, the chance of retrieving related entities is much higher, which leads to better performance. One example depicted in the lower part of Figure 4 where the LAION retrieval finds 'Island of San Giorgio Maggiore', which helps the model generate the classical Renaissance-styl church.
4.2 ENTITY FOCUSED EVALUATION ON ENTITYDRAW
Dataset Construction We introduce BrawthDraw to evaluate the model's capability to gener- ate diverse sets of entities different visual scenes. Specifically, we pick various types of visual entities (dog breeds, landmarks, foods, birds, and animated characters) from Wikipedia Commons, Google Landmarks and Fandom to construct our prompts. total, we collect 250 entity-centric prompts for evaluation. These prompts are mostly very unique and we cannot find them the Internet. let alone the model training data. The dataset construction details are in Appendix C To evaluate the model's capability to ground on broader types of entities, 20 objects like "sunglasses, backpack, vase, teapot, etc' and write creative prompts for them. We compare generation results with the results from DreamBooth (Ruiz et al. (022) in inpApendix H.
also randomly select
We use the constructed prompt as the input and its corresponding image-text pairs as the 'retrieval' for Re-Imagen, generate four x024x 1024 images. For the other models, feed the prompts directly also to generate four images. We pick the best image of random samples to rate its Pho- torealism and Faithfulness by human raters. For photorealism, we rate if the image is moderately realistic without noticeable artifacts. For the faithfulness measure, we rate I if the image is faithful both the entity appearance and the text description.
Results We use the proposed interleaved classifier free guidance subsec tion 3.2p for the 64x diffusion model, which runs for 256 diffusion steps under a strong guidance weight of w=30 for both text and neighbor conditions. For the 256x and 1024x resolution models, we use constant guidance weight of and 3.0, respectively, with 128 and 32 diffusion steps.
hhttps:/ccmmte www.ee/wwwwiiiiii......
Paage

---

Published as a conference paper at ICLR 2023
Faithfulness Birds
Photorealism All
Model
Dogs
Foods
Landmarks
Characters
Broader
Imagen
0.28
0.26
0.27
0.84
0.10
0.54
0.98
DAILLEE
0.60
0.47
0.36
0.82
0.08
0.58
0.98
Stable-Diffusionn
0.16
0.24
0.24
0.68
0.08
0.46
0.92
Re- Re-lmagen (K=2)
0.80
0.80
0.82
0.92
0.54
0.80
0.98
Table 2: Human evaluation results for different models on different types of entities.
The inference speed is 30-40 sees for 4 images on 4 TPU-v4 chips. We demonstrate our human evaluation results for faithfulness and photorealism in Table 2
We can observe that Re-Imagen can in general achieve much higher faithfulness than the existing models while maintaining similar photorealism scores. When comparing with our backbone Imagen, we see the faithfulness score improves by around 40%, which indicates that our model paying aftention to the retrieved knowledge and assimilating it into the generation process.
Faithfulness on Frequent Entities
Faithfulness on Infrequent Entities
12
Imagen
Re-Imagen
Scores
4.8
0.10
0.00
0.70
0.14
3.72
0.49
0.44
5.42
0.8
0.16
0,168.22
6.0
Dogs
Foods
Landmarks
Birds
Characters
Dogs
Foods
Landmarks
Birds
Characters
Figure 5: The human evaluation scores for both frequent and infrequent entities.
We further partition the entities into 'frequent" and 'infrequent" categories based their frequency (top 50% as 'frequen'). We plot the faithfulness score for 'frequent' and 'infrequent' separately that Re- Re-Imaeen is less sensitive to the frequency of the input entities than
in Figure 5] We the other models with only minor performance drops. This study reflects the effectiveness of text-to- image generation models on long-tail entities. More generation examples are shown in Appendix F
4.3 ANALYSIS
Comparison to Other Models We demonstrate some examples from different models in Figure 6 As can be seen, the images generated from Re-Imagen strike good balance between text alignment and entity fidelity. Unlike image editing perform in-place modification, Re-Imagen the neighbor entities both geometrically and semantically according to the text guidance. As concrete example, Re-Imagen generates the Braque Saint-Germain (2nd row Figure 6) on the grass, in a different viewpoint from to the reference image.
transform
Impact of Number of Retrievals The number of retrievals K is an important factor for Re-Imagen. We vary the number of K for all three datasets to understand their impact on the model performance. From Figure 7| we found that on COCO and Wikilmages, increasing K from to many changes in the FID score. However, on UntityDraw/Bech increasing K will dramatically improve the faithfulness of generated image. indicates the importance of having multiple images to help Imagne ground on the visual entity. We provide visual examples in Appendix D
does not lead
Text and Entity Faithfulness Trade-off's our experiments, we found that there is a trade-off between faithefulness to the text prompt and faithfulness to the retrieved entity images. Based on Equation by adjusting 77, the proportion to generate images that explore this tradeoff: decreasing 7) will increase
&p and én in the sampling schedule, we can
control Re- Re-Imagen the entity's 's entity faithfulness but decrease the text alignment. We found that having 7) around 0.5 is usually 'sweet spot" that balances both conditions.
8

---

Published as a conference paper at ICLR 2023
Ours
Imagen
DALL-E2
Stable-Diffusion
Entiy Reference
placed grass.
Jibarito
A Braque Saint-Germain taking shower.
Braque Saint-Germaie
A flock of birds fly around Visoki Dečani church. Figure None-cherry picked examples from Entity DrawiyyDraaBende for different models.
Viscki Dečani
17
K=
Faithfu mines
5.80
6:02
0:02
6.20
0.51
COCO
Wclmaggs
Dogs
Foods
Landmarks
Binds
Characters
Figure 7: Ablation Study of retrieval number K on different datasets.
Cretan Hound is running on the moon.
Reference
# = 0.1
1) 0.4
f 0.50
1 == 0.50
f 1.0
Figure 8: Ablation study of interleaved guidance ratio 1) to show the trade-off.
5 CONCLUSIONS
We present Re-Imagen, a retrieval-augmennt diffuuion model, and demonstrate its effectiveness in generating realistic and faithful images. We exhibit such advantages not only through automatic FID measures on standard benchmarks (i.e., COCO and Wikilmage) but also through human evaluation of the newly introduced EntityDrawBenchh We further demonstrate that our model particularly effective in generating image from text that mentions rare entities.
Re-Imagen still suffers from well-known issues in text-to-image ext-to--mmae generation, which we review below in infsection 5) In addition, Re- Re-Imagen also has some unique limitations due to the retrieval-augne -ugmented modeling. First, because Re-Imagen sensitive to retrieved image-text pairs it is conditioned on when the retrieved image is of low quality, there will be negative influence the generated image. Second, Re-Imma Re-ann sometimes still fails to generate high-quality images with highly compositional prompts, where multiple entities involved. Thirdly, the super-resolution model is still not com- petent at capturing low-level details retrieved entities leading to visual distortion. In future work, plan further investigate the above limitations and address them.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Strong text-to-image generation models, i.e., Imagen (Saharia et al. 2022) and Parti (uu et al. 2022), raise ethical challenges along dimensions such as the social bias. Re-Imagen is exposed to the same challenges, as we employed Web-scale datasets that are similar to the prior models.
The retrieval- -aggeeetd modeling techniques of Re-Imagen have substantially improved the con- trollability and attribution of the generated image. Like many basic research topics, this additional control could be used for beneficial or harmful purposes. One obvious danger is that Emagnn (or similar models) could be used for malicious purposes like spreading misinformation, e.g. by pro- ducing realistic images of specific people in misleading visual contexts. On the other side, additional control has many potential benefits. One general benefit is that Re-Imagen can reduce hallucination and increase the faithfulness of the generated image to the user's intent. Another benefit is that useful for minorities and other users in
the ability work with tail entities makes the model smaller communities: for example, Re-Imagen is more effective at generating images of landmarks famous in smaller communities or cultures and generating images of indigenous foods and cultural artifacts. We argue that this model can help decrease the frequency-caused bias in current neural network-based Al systems.
Considering such potential threats to the public, we will be cautious about code and API release. In future work, we will explore framework for responsible use that balances the value of external auditing of research with the risks of unrestricted open access, allowing this work to be used in safe and beneficial way.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432-4441, 2019.
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inver- sion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18511-18521, 2022.
Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. arXiv preprint arXiv:2204. 0222:22204.02849 2022.
Andreas Blattmann, Robin Rombach, Kaan Oktay, and Björn Ommer. Retrieval-augmenned diffu- sion models. arXiv preprint 2.Xi::2244.11844 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Im- proving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04406 2021.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16495-16504, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Oran Gafni, Adam Polyak, Oron Ashual Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint wrXiv:2203.13131, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
10

---

Published as a conference paper at ICLR 2023
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chem, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference Machine Learning, pp. 3887-3896. PMLR, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-traiming. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pp. 3929 3938. PMLR, 13-18 Jul 2020. URL
press/ v11p/
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prome image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23: 47-1, 2022.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generaliza- tion through memorization: Nearest neighbor language models. In International Conference on Learming Representations, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. detrieval-augmentee genera- tion for knowledge- 9459-9474, 2020.
nowwlege-intensive intensive nlp tasks. Advances in Neural Information Processing Systems, 33:
Bowen Li, Philip HS Torr, and Thomas Lukasiewicz. Memory-driven text-to-image generation. arXiv preprint 2tt.::2008.00222.. 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6959-6969, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(140): 2020.

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061155, 2022.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends(R) in Information Retrieval, 3(4):333-389, 2009.
Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.057444 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ferrence Computer Vision and Pallern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint ar arXiv:2208. /2242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
Christoph Schuhmann, Robert Kaczmarczyk, Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NeurIPS Workshop Datacentric AI, number FZJ- 2022-00923. Jülich Supercomputing Center, 2021.
Yawar Siddiqui, Justus Thies, Fangehang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Re- trievalfuse: Neural 3d scene reconstruction with database. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12568-12577, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In International Conference on Machine Learn- ing. pp. 2256-2265 PMLR, 2015.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fdelity gan inversion for image attribute editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11379-11388. 2022.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2575-25854 2020.
Rui Xu, Minghao Guo, Jiaqi Wang, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Texture memory-augmented deep patch- based image inpainting. JEEE Transactions on Image Processing, 30:91 12-9124, 2021.
12

---

Published as a conference paper at ICLR 2023
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision, pp. 592-608. Springer, 2020.
Jan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexi A Efros. Generative visual manipu- lation on the natural image manifold. In European conference on computer vision, pp. 597-613. Springer, 2016.
13

---

Published as a conference paper at ICLR 2023
A EXTENDED LITERATURE REVIEW
As aforementioned in the main text, in this section, we provide an additional review of related works, on (I) aetreeell-uggmented Generative Models; and (2) Text-guided Image Editing.
Getrieval-Augmentee Generative Models Knowledge grounding has also drawn significant at- tention in the natural language processing (NLP) community. Different semi-parametric models like KNN-LM (Kaandelwal et al.] 2019). RAG (Lewis et al. 2020), REALM (Guu et al.. 2020), RETRO (Borgeaud et al. 2021) have been proposed to leverage external textual knowledge into the transformer language models. These models have demonstrated great advantages in increasing the language model's s faithfulness and reducing the computation/meemory cost. Such attempts have also been made in visual tasks like image recognition (Long al. (022), 2-D scene reconstruction (Sid- diqui et al. 2021), and image inpainting (Xu et al. 2021). Our proposed method follows the same theme incorporate visual knowledge into pre-trained text-to-image generation model to help the model generalize to long-tail entities or even unseen entities without scaling up the parameters.
Text-Guided Image Editing The work of text-guided image editing aims at preserving the ob- ject's appearance while changing certain contexts in the image. Previously, GANs (Goodfellow) al. 2014) have been used to achieve significant performance on image editing (Zhu et al. 2016 Abdal et al. 2019; Zhuu et al. 2020 Roich et al. 2021 Tov et al., [021] Wang et al., 2022 Alalui ctt al. 2022). The problem is also known as inversion as normally requires finding the initial noise vector added in the generation process. More recently, Prompt-to- tooooo--ppttttt (Hertz al. 2022) pro-
pose to use pre-trained text-image models for image editing. Image editing focused on performing in-place modifications to the input image, either changing the global styles editing local region specifically without modifying the object's appearance. However, we treat the retrieved image as 'knowledge" and ground it synthesize new images. Thus, we are not restricted in-place modifications and are able to perform more sophisticated transformations over the objects.
B WIKIIMAGES DATASET
The Wikilmages dataset is taken from WebQA (Chang et al. 2022). Images were crawled from Wikimedia Commons via the Bing Visual Search API. Since lots of Wikimedia's topics are not vi- sually interesting, the authors seeded with natural scenes and gradually refine the search pool to obtain more interesting images. The images are mostly containing entities from Wikipedia or Wiki- Data. However, the original dataset still contains heavy noises. Therefore, we apply further filtering to obtain the more plausible ones for image generation. Specifically, we remove all the image-text pairs with text lengths larger than 15 tokens and all the text with date or wiki-id information.
14

---

Published as a conference paper at ICLR 2023
C ENTITYDRAWBENCH
For dog breeds and birds, we sample 50 from Wikipedia Common as our candidates. For land- marks, we sample 50 from Google Landmarks (Weyand et al. 2020 our candidate. For foods, we sample 50 from Wikipedia as our candidates. For film characters, we collected 50 images from Starwars from Fandom We use appropriately paired source images as the retrieved 'knowledge" For each entity category, we write 5 prompt templates with an entity name placeholder, which de- scribes the entity in different scenes. Each entity will sample a template and replace the placeholder with the entity's name to generate a prompt, which is used as input to the text-to-image generation model.
Templates
Entity
Wikimed'a
Knowledoe
Food: [ENTITY] is served with red wine.
Promot
Landmreric A group of tourista is front of (ENTTT)]
Dog: ENTITY is fighting with another dog.
Figure 9: The construction process of EntityDrawBench. We first list entity names and then find their source images from Wikimedia, and finally generate prompts related these entities.
We list all the prompt templates Figure 10
Type
Template
Template
Template
Template
Template
Dog
[DOG akeeping on groundl
[DCG] rurning by
[DOC] cunching a frabex.
[DOG] saking showees
[OO]] lighting anothe dog.
(FO00) ploood grass.
[FOOO]
[FOOO] popcorn the
dog beside [FOOD
JFCC decorated
Landmark
[LANIDMARRK)
front
day BLANDMARK)
during the
J.ANDNAAPR
Bird
AB PIO) is docking on plas.
A [BFIC] la drinking witler.
A [BRAC] la Repping its wings
A (BIRO) diving from the diy.
[BIR]] swimming in the rVer.
Character
StarWara JENTTM
stancing the
sranding gantien.
(mNTTh)) whcchen
Figure 10: The BrawBeeD prompt templates for all the different entity categories.
of dog breeds
15

---

Published as a conference paper at ICLR 2023
D IMPACT OF RETRIEVAL NUMBER K
We change the retrieval number K from to to see its impact on the model output. We show some examples Figure 11 to demonstrate the advantage of having multiple retrievals to help the model better capture the visual appearance of the given entities.
Retrieval
Retrieval = 2
Reference
A Female Burgalese Pointer is running by the river.
Male Briquet Griffon Vendéen is catching a frisbee.
Caucasian Shepherd is running by the river.
Hallacas (a traditional dish) is decorated with flowers.
Figure 11: Generation Examples for setting K to and
16

---

Published as a conference paper at ICLR 2023
E CLASSIFIER GUIDANCE-FREE SAMPLING STRATEGY
We demonstrate different types of sampling strategy to leverage two conditions: standard joint con- dition guidance sampling, weighted guidance sampling, and our proposed interleaved guidance sam- pling.
The standard joint condition guidance only considers the joint diffusion score e( Cny Gp) to meet both conditions. In contrast, weighted guidance sampling uses the weighted sum of text-enhanced epsilon ?p and neighbor-enhaneed epsilon én. Our interleaved classifier guidance switches between Ep and é,, with ratio of 1) 7). We plot their conceptual difference in Figure 12 Essentially, and cp not have dependency in weighted sampling, however, they are dependent in interleaved sampling. In an extreme case where and Cp are contradictory to each other, the model will get stuck in a local region. In contrast, Interleaved sampling can alleviate this issue.
è = ncp(() + (1 - (x,
=
eg(xi)
c,(gg-1)
x
(x(x)
(p(x-11)
ex(x,,
xG-2
&
1/-
(x)x
1,(1)-1)
Figure 12: Weighted Guidance Sampling vs. Interleaved Guidance Sampling.
We compare 20 dog images generated from these three sampling strategies in BrawtyDDrw/Bench. We vary the number of diffusion step observe their human evaluation score curve and show case some generated outputs in Figure 13] As can be seen, the joint decoding is either dominated by the retrieval image by the text prompt. Weighted and Interleave can help balance the conditions generate better images. We also found that with less sampling steps K=200, "weighted" sampling actually achieves better results than "interleaved" sampling. However, as the sampling steps increase, our proposed "interleaved" sampling achieves better human evaluation score.
17

---

Published as a conference paper at ICLR 2023
EntityDrawBench Human Score vs. NFE
1.00
Interleaved
Weighted
Joint
0.80
Falthfulress
0.60
0.40
0.20
0.00
100
200
300
Interleave ...,((p, Sp..................... 86F .....
Average
Joint
n 8(x, cp) + (1 - 1) E(x,. c
wē(K, Cp. 9n) (w 1)8(x,)
Croatian sheepdog is living a house made of Sushi
Caucassian Shepherd is living in house made of Sushi
A burgalese pointer is living in house made of Sushi
Figure 13: Different classifier-fres guidance sampling strategy (Interleave is ours).
18

---

Published as a conference paper at ICLR 2023
F GENERATION EXAMPLES
We provide more generation examples in Figure 14 and Figure 13
Re-lmagen
Imagen
Stable-Dillusion
Entiy
Chliquiles
Chilaquiles with popcorns
Tomato bredle
Tomato bredie served with wine
Tri-colour Armant
Tri-colour Amant taking
Bergamasco shepherd
A Bergamasco shepherd dog catching friebee.
flock birds around
Daćani church.
Visoki Dačani
Figure 14: Extra None-cherry picked examples from EntityDraw Bench for different models.
19

---

Published as a conference paper at ICLR 2023
Re-Imagen
Imagen
DALLE-2
Entity Refenrence
a Rufous-tailed flycatcher is flapping its wings.
Rufous-tailed flycatcher standing on top a basketball.
Sultan Amir Ahmad Bathhouse on the moon.
Shan
dog is sitting front of Palau Güell.
Figure 15: Extra None-cherry picked examples from EntityDrawBBnnc for different models.
20

---

Published as a conference paper at ICLR 2023
G IMAGINARY EXAMPLES
We provide generation results for imaginary scene in Figure 16
Reference
Corgi
Corgi is living in house
made of Sushi
A Corgi is riding bicycle
Croatian sheepdog
A Croatian sheepdog is living in house made Sushl
A Croatian sheepdog is riding a bicycle.
Caucassian Shepherd
Caucassian Shepherd living in house made Sushi
Caucassian Shepherd is riding a bicycle.
Figure 16: Imaginary Scenes generated by Re-Imagen.
21

---

Published as a conference paper at ICLR 2023
H COMPARISON WITH DREAMBOOTH
We also add comparison to DreamBooth (Ruiz et al. 2022). We adopt almost the same input images from DreamBooth and display our generation results Figure 17
Figure 18 and Figure 19
Retrieved Images
Re-Imagen
A bvigari sunglasses at Mt. Fuj
bvigari sunglasses asses on top snow.
bvigari sur sunglasses
bvigari sunglasses jungle.
bvlgari sunglasess war by bear.
Input images
DreamBooth
[ sunglasses in
[[V sunglasses bear
[V] sunglasses Fuji
[V] sunglasses top
A [V] sunglasses with Eiffel the background
Figure 17: Imaginary Scenes generated by Re-Imagen
22

---

Published as a conference paper at ICLR 2023
Retrieved Images
BB
Re-Imagen
Kenzo Kids Backpack Grand Canyon
Wet Kenzo Kids Backpack Water
Kenzo Kide Backpack
Kanzo Kids Backpack at Boston
Konzo Kids Backpack with night sky
Input images
DreamBooth
backpack Canyon
backpack night sky
A [V] backpack in the Versailles
A wet [V] backpock in water
[V] backpock
Figure 18: Imaginary Scenes generated by Re- Re-Imagen.
23

---

Published as a conference paper at ICLR 2023
Retrieved Images
Re-Imagen
Sweejar Royal teapot floating in the sea
Sweejar Royal teapot pouring tea.
Sweedjr Royal teapot
Transparent Sweejar Royal leapot
bear is pouring from Sweejar Royal teapot.
Inpui images
DreamBooth
teapet Booting
tesper fleating milk
A bear pouring from A transpurent [V] uapet
[V] teapot pouring tea
a teapot
with milk inside
Figure 19: Imaginary Scenes generated by Re-Imagen.
24

---

Published as a conference paper at ICLR 2023
I FAILURE EXAMPLES
We found that Re-Imagen can also fail in lot of cases. We demonstrate a few examples in Figure 20. As can be seen, the model sometimes has few failure modes: (1) the text input prior like 'Zoom' will be interpreted as 'Zoom-in' picture by the model. (2) the model cannot ground the retrieval text on the retrieval image, for example, the model believes that only the 'beef tenderloin inside the bowl' is "Escudella" rather than the whole stew, therefore generating "beef tenderloin on the grass'. (3) the model can sometimes mess up conditions, example, the reference 'Australian Pinscher' and the 'rabbit' in the prompt gets mixed into single object.
strong
Ratrienat
Generattons
grans
chasing
Figure 20: Failure examples from Entity for dogs, landmarks, and foods.
25


Reviews:
[{"review_id": "hWKMcI0sitW", "reviewer": "Reviewer_UC3H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a retrieval augmented diffusion model for the task of text to image synthesis. The idea is to condition the diffusion model on the external knowledge base in addition to the text embeddings that are used in text-to-image diffusion models. Conditioning on the retrieved nearest neighbor embeddings happens using the cross-attention layer. During inference, the model retrieves the image-text pairs corresponding to the input text prompt and then samples the result. Experiments show SOTA results on zero-shot FID on COCO, other benchmark datasets and human evaluation.", "summary_of_the_review": "I think the paper is a good contribution to the community since it presents good results. However, as I pointed above, some of the ablations, discussions and visualizations are missing. Adding these would have made it a very strong submission.", "strength_and_weaknesses": "Strengths:\n\nThe use of external knowledge base to augment the diffusion models is intuitive and a powerful idea. It is very hard to encode the entire world knowledge in a 2.5B parameter model, hence, these models are bound to fail in rare modalities. External knowledge base comes to rescue in this case.\n\nThe idea proposed by authors is simple, yet quite effective (as shown in  experimental studies). This model can also be finetuned from the original text-to-image model (Imagen in this case), which is something really nice. This way, a lot of compute can be saved.\n\nThe modification to the existing classifier free guidance to have optional conditioning on both nearest neighbor image features and text is also interesting.\n\nThe experimental results are strong. Both on zero-shot FID and human evaluation, the authors achieve SOTA.\n\nWeakness:\n\nI have a few comments, mainly with respect to some of the design choices made. I feel like the authors mainly mentioned these choices, and did not have a good discussion / justification.\n\nOn the choice of number of nearest neighbors during training: The authors just mention that they use k=2. But why just k=2? I can imagine that as k increases, the number of queries and keys in the cross attention would increase, hence the computational burden would go high. Is that the reason why k=2 was chosen? In a ideal case, performing an ablation on k would be good. But I can understand that ablations might be too expensive in these models. Hence, having a discussion on why this decision was made and what considerations were taking into account to end up in this decision would help the research community.\nOn the choice of number of nearest neighbors during inference: Should the same k be used during training and inference? While it might be true that performing an ablation on k during training might be infeasible, performing an ablation on k during inference might not be that hard. It would be great if authors had done that. K is an important parameter in retrieval, hence understanding the sensitivity of k might be beneficial.\nModified classifier-free guidance: The modification to the classifier free guidance makes sense to me. In the implementation, the authors randomly flip a coin and choose one of the two scores - text score and neighbor score in reverse diffusion. Why should this be done? Why not use a weighted combination of both scores. That is, why not use $\\eta \\hat{\\eps}_{p} + (1-\\eta)\\hat{\\eps}_{n}$ as the score? This might increase the NFE’s, but it might be worthwhile to study if this gives any gains. An ideal experiment would be to plot NFE-FID tradeoff for using this combined score vs using the proposed algorithm.\nI would have liked it if authors provided more visualizations. One of the power of techniques like Imagen is the ability to image zero-shot content. For instance, “Corgi lives in a house made out of sushi”. But none of such visualizations is provided in this paper. I am wondering if this ability got worse due to the nearest neighbor finetuning because such samples are not present in retrieval dataset. COCO FID might not measure this as captions in COCO are not containing such novel texts. A discussion on this might be beneficial.  \nA discussion on computational overhead: This is related to 1, but I am curious about what the additional computational overhead is because of the added attention tokens?\nBM-25 seems to be a very simple retrieval algorithm. Will you get more gains by using CLIP text?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KKvYwDBc-Nj", "reviewer": "Reviewer_oSTW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This proposes a retrieval-augmented text-to-image model Re-Imagen for text and image guided image generation. The proposed method Re-Imagen interleaves classifier-free guidance during sampling to ensure both text alignment and entity fidelity and achieve good performances on the two datasets. This paper also introduces a more challenging benchmark EntityDrawBench for the text-to-image generation.", "summary_of_the_review": "This paper introduces a novel retrieval-augmented text-to-image model and a more challenging benchmark for text-to-image generation. The retrieval-augmented methods may be helpful for the application for the text-to-image generation. However, the retrieved image involving in the image generation procedure makes it not fair to be compared with the previous methods which generates the images only from the texts. The experiements cannot support to prove the effectiveness of the proposed methods. Futhermore, the novelty of the proposed methods is limited. This paper is not good enough for publication.", "strength_and_weaknesses": "Strength: \n1) This paper introduces an interesting viewpoint for text-to-image generation by involving the retrieval-based images during the generation procedure. \n2) The proposed methods achive good performances on the datasets and the newly-introduced benchmark may be helpful for the text-to-image task.\n\nWeakness:\n1) The comparison is not fair in the experiments. As for the most of the previous methods, the images are generated from the only text inputs. While, the proposed methods involve the images retrieved from texts by the datasets, which will make a big difference on the evaluation of the generated images, especially the FID scores. \n2) The novelty is limited. Although the retrieval-augmented methods achieve good performances and provide another viewpoint for image-guided text-to-image generation, the multiple conditions for diffusion models are not novel. The UNet architecture is also widely used in the diffusion models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AqP78s70jNj", "reviewer": "Reviewer_T7pn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to utilize the retrieval image(s) for extending the model's ability for generating images. In this way, the model takes the text prompt as the input, by fusing with the noisy predicted by the retrieval images, it can generate the images that the dataset seldom contains. To balance the generation ability among text prompts and retrieved images, the paper also introduces a hand-crafted weighted scheme. Several experiments have been conducted to generate the efficiency of the method.", "summary_of_the_review": "Generally speaking, the effective idea and good results make me lean toward acceptance. However, the rather hand-crafted designed combination way makes the method rather heuristic. The technical contribution is still limited. Thus, I can only put it a little bit over the boundary.", "strength_and_weaknesses": "Strengths:\n\n+ The proposed idea of taking the power of retrieved images for generating better results is simple but effective. The effectiveness has also been demonstrated through the experiments.\n\n+ I enjoy reading this paper, and it writes generally well, though with some typos (e.g., \"on two dasets:\").\n\n+ This paper works on a recent hot topic, which is text-based image generation. The idea of customizing with more conditional images can push forward the development of this area. \n\n+ I appreciate the limitations \n\nWeaknesses:\n\n+ Some parts of the model descriptions may not be clear. To my understanding, the Re-Imagen allows multiple retrieved images as inputs, but how exactly the model balances them or takes them as inputs? How is the model architecture designed? Does this mean the number of retrieved images should be the same as the training? What if the retrieved images share much diverse appearance, how the model handles such a case?\n\n+ I feel the way of improving the classifier-free guidance is rather heuristic. Are there any insights that why the undesired imbalance happens?\n\n+ Though the idea is effective, its underlying technical contribution is rather limited, as adding the retrieval images as another condition is straightforward for implementing the such retrieval-image-augment idea. \n\n+ As an image generation work, I would like to see much more visual results than the paper and the supplementary document provided. It is encouraged to release a project webpage containing more results for demonstrating the generality of the proposed method.\n\n+ Will the model be released publicly? If not, how this work can contribute to the research community is encourag", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9m0I9aZhW9E", "reviewer": "Reviewer_xjWB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an approach to deal with the problem, that state-of-the-art models often have difficulty generating images of uncommon entities. The main contributions are 1) a novel retrieval-augmented text-to-image model Re-Imagen. 2)  interleaved classifier-free guidance. 3) a benchmark.", "summary_of_the_review": "This paper tried to solve an important problem, and the method is reasonable and works. The main concerns are in the fairness in the comparisons in experiments and the reimplementation of this method for both data and model.\n\n", "strength_and_weaknesses": "**Strength**\n\n1) The issue that this paper tried to solve is important and critical in nowadays text2image generation. The topic is meaningful.\n2) This paper is well-written and easy to follow.\n3) The qualitative results is compelling.\n\n**Weaknesses**\n\n1) Research topic definition changing.\n\nThe research topic is text-to-image generation, which is presumed to take text as input and image as output. However, though this work means to solve the problem of uncommon entities generation in text-to-image generation, it brings subtle but certainly important changing of the \"research topic definition\". An extra input of a set of knowledge base is needed for the inference of this model, which makes the problem to be like text (with knowledge base)-to-image generation, with more strong conditions. The changing of the definition mainly brings a problem that the usage bar for normal users is actually pulled up.\n\n2) The comparisons on Table 1, Table 2 and Table 3 have some concern in fairness.\n\n(a) For the comparion between Re-Imagen (\u0000=BM25; B=COCO; k=2) to Imagen on Table 1, (i) the # of Params are different, (ii) the training set is different (iii) the metric FID-30K and Zero-shot FID-30K is different, it seems to be arbitrary and misleading to say: “Re-Imagen (with the COCO database) can achieve a significant gain on FID-30K without fine- tuning: roughly a 2.0 absolute FID improvement over Imagen.”. \n\n(b) For the comparison on Table 2, the # of Params are different, it hard to deduce meaningful and solid conclusions. The factors should be disentangled to disclose the key insights and observations in the experiments.\n\n(c) For the comparison on Table 3, the motivation why these three types of visual entities (dog breeds, landmarks, and foods) are picked may be clarified. A column of results of broader entities can be more convincing.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster