RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Authors: Yuanhuiyi Lyu
Xu Zheng
Lutao Jiang
Yibo Yan
Xin Zou
Huiyu Zhou
Linfeng Zhang
Xuming Hu
Venue: ICML 2025 poster (2025)
Keywords: Self-reflective Contrastive Learning
Real-object-based RAG
Primary area: applications->computer_vision

Abstract: Recent text-to-image generative models, e.g., Stable Diffusion V3 and Flux, have achieved notable progress. However, these models are strongly restricted to their limited knowledge, a.k.a., their own fixed parameters, that are trained with closed datasets. This leads to significant hallucinations or distortions when facing fine-grained and unseen novel real-world objects, e.g., the appearance of the Tesla Cybertruck. To this end, we present **the first** real-object-based retrieval-augmented generation framework (**RealRAG**), which augments fine-grained and unseen novel object generation by learning and retrieving real-world images to overcome the knowledge gaps of generative models. Specifically, to integrate missing memory for unseen novel object generation, we train a reflective retriever by **self-reflective contrastive learning**, which injects the generator's knowledge into the sef-reflective negatives, ensuring that the retrieved augmented images compensate for the model's missing knowledge. Furthermore, the real-object-based framework integrates fine-grained visual knowledge for the generative models, tackling the distortion problem and improving the realism for fine-grained object generation. Our Real-RAG is superior in its modular application to **all types** of state-of-the-art text-to-image generative models and also delivers **remarkable** performance boosts with all of them, such as a **gain of *16.18\%* FID score** with the auto-regressive model on the Stanford Car benchmark.

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Yuanhuiyi Lyu' Xu Zheng'
Lutao Jiang'
Yibo Yan
Xin Zou'
Huiyu Zhou 2
Linfeng Zhang
Xuming Hu 124
Abstract
Prompt
(a)
Cahartnuck speeiing wrong Mie Grear Hari"
Text-to-image Generative Model
Recent text-to-image generative models, e-g-, Sta- ble Diffusson V3 and Flux, have achieved notable progress. However, these models are strongly re- stricted to their limited knowledge, a.k.a., their own fixed parameters, that are trained with closed datasets. This leads to significant hallucinations or distortions when facing fine- ane-grained and un- novel real-world objects, e-g., the appearance of the Tesla Cybertruck. To this end, present the first real-object-baset retrieval-augmented generation framework (ReaIRAG), which aug- ments fine- fine-graing greigraindd and unseen novel object gener- ation by learning and retrieving real-world images to overcome the knowledge gaps of generative models. Specifically, to integrate missing mem- ory for unseen novel object generation, train reflective retriever by self-reflective contrastive learning, which injects the generator's 's knowl- edge into the sef-reflective negatives, ensuring that the retrieved augmented images compensate for the model's missing knowledge. Furthermore, the real-object-basse framework integrates fine- grained visual knowledge for the generative mod- els, tackling the distortion problem and improv- ing the realism for fine- fine-grained object generation. Our Real-RAG is superior in its modular applica- tion to all types of state-of-the-ee generative models and also delivers remarkable performance boosts with all of them, such as a gain of /6.18% FID score with the auto-regressive model the Stanford Car benchmark.
Image
Koeeiin Similriy
(b)
Prompt:
Text-o-lmage
"A Cytertnck
Generative Model
spendin aling
the Great Horl
Reflective
Ref.
Retriever
Image
Retriove
(c)
Prompt:
Text-o--mage
Cybertnck
Generative Model
Hall"
Figure I. (a) The pipeline of text-to -aa-tt---igeee generative models. (b) The framework existing retrieval -ugmennted methods. (c) The framework of our proposed RealRAG.
1. Introduction
Recent text-to-image generators have achieved notable progress in image synthesis from the given textual prompts. There are three mainstream types of generative models, including the U-Net-based diffusion model (Rombach et al., 2022a; Podell et al., 2023), the DiT-based diffusion model (Xiao et al., 2024; Sun et al., 2024), and the auto- regressive model (Esser et al., 2024; BlackForest, 2024). Typically, these models store all their visual memory (e.g.. the appearance of Big Ben) implicitly in the parameters of the underlying neural network, requiring a lot of pa- rameters(e.g. 10B). Furthermore, similar to the hallucina- tion problem Large Language Models (LLMs) (OpenAI, 2023; Touvron et al., 2023), the large-scale text-to-image generative models also show the same problem. Some gen- erated images include ghosting, distortions, and unnatu- ral elements when generating specific real-world objects. Therefore, these problems motivate the development of text- to-image generation models, which can integrate external visual knowledge (e.g., images from the web) to augment generative realism and accuracy for fine- mme-grained and unseen novel object generation.
text-to-image
'The Hong Kong University of Science and Technology (Guangzhou) GGanggx Zhuang Autonomous Region Big Data Research Institute 'Shanghai Jiao Tong University "The Hong Kong University of Science and Technology. Correspondence to: Xuming Hu <xuminghu @ hkust- .iasstgzz.du.n....
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RealRAG: Retricval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Retrieval-augmented generation (RAG) has shown promise in natural language processing (NLP) (Gao et al., 2023). To enhance the specific knowledge and minimize the hal- lucination of LLMs. a retrieval model first retrieves the most relevant documents for the input prompts (e.g., the questions), and then LLMs generate predictions powered by the recalled documents. However, text-to-image genera- tion, these RAG methods, which rely primarily on similarity matching, remain challenging. Specifically, Fig. (b) and (c), the candidate image (1), with the high- est similarity score of the text prompt, fails to improve the image generation. In contrast, candidate image (2), despite having a lower similarity score, complements the missing knowledge and augments the image generation. In this work, we propose a reflective retriever trained via self-reflective contrastive learning, aimed at retrieving images with miss- ing knowledge instead of the most relevant one.
regressive models (OmniGen (Xiao et al., 2024), Emu (Sun et al., 2024)). Note that, our RG is the first work to build a unified RAG framework for all types of text-to- image generative models. Our RealRAG consistently de- livers significant performance improvements with all these models, such as +5.77% on the Stanford Cars benchmark with Flux (BlackForest, 2024) and +20.48% on the Ox- ford Flowers benchmark with Emu (Sun et al., 2024). Ad- ditionally, evaluate the capability of unseen novel ob- ject generation, we collect recent novel objects from news web pages and construct human evaluations. Project page: https: /ttp:///c-/o //c////ccoo//// ly. gi sithub.... io/ /e///ealRRRa RRRA-AaeG////
shown in
2. Related Work
2.1. Text-to-Image Generation
The U-Net-based Stable Diffusion models (Rombach et al., 2022a; Podell et al., 2023) first perform universal image gen- eration from text prompts, which typically trained on large scale text-image paired dataset, a.k.a., LAION 5B (Schuh- mann et al., 2022). After the proposal of the Diffusion Trans- former (DiT) (Peebles & Xie, 2023), some research, such as Stable Diffusion V3 (Esser et al., 2024) and Flux (Black- Forest, 2024; Liu et al., 2025), utilize DiT as the backbone to develop DiT-based Diffusion Models for text-to-image generation. Recently, with the success of auto-regressive (AR) modeling in natural language processing (OpenAI, 2023; Touvron et al., 2023), some works have explored how to combine auto-regressive models with diffusion models to improve the understanding capability and further build a unified multi-modae model for both understanding and generation (Chen al., 2024; Xie et al., 2024; Zhou et al., 2024). These AR-based models, such as OmniGen (Xiao et al., 2024) and Emu (Sun et al., 2024), also show no- table performance on the text-to-image task. While these methods have achieved strong performance in text-to-image generation, they store all the knowledge in their pre-trained parameters, which leads to hallucinations and distortions when generating realistic objects. To address this limitation, we propose the real-object- based RAG framework to inte- grate missing knowledge and improve the ability to generate realistic images.
In this paper, we present the first real-object-based retrieval- augmented generation framework (RealRAG) which lever- ages real-world images to compensate for the missing knowl- edge inherent in generative models and improve realistic image generation. The core insight of RealRAG is to en- hance realism and reduce hallucination in text-to-image generative models powered by the reflective retriever, which is trained via self- reflective contrastive learning. In ad- dition, our proposed RealRAG demonstrates remarkable flexibility and can be applied across diverse text-to-imant generative models, yielding significant performance gains, e.g., a +1% gain with Emu (Sun et al., 2024) on the Stanford Cars benchmark (Krause al., 2013).
Specifically, we first generate images from the given text prompts using the specific eoxttt-image generative models. Given these generated images, we then sample the reflective negatives in the image database by selecting images with the highest cosine similarity of the generated images. In this way, these sampled negatives are similar to the generated images that store the generative models' visual memory. Second, we train the reflective retriever with self-reflective contrastive learning, which utilizes text prompts as posi- tives and trains with the sampled reflective negatives. In this way, the well-trained retriever can recall the prompt- relevant images also with missing knowledge for the genera- tive models. For example, as shown in Fig. I, our RealRAG retrieves the image ("Cybertruck "J, ], instead of the im- age with highest similarity (["A truck is speeding al log the Great Wall "]), which integrates miss- ing knowledge for generative models.
2.2. Retrieval-augmented Generation
Retrieval-augmentted generation has shown promise with NLP (Lewis et al., 2020; Guu et al., 2020). To incorporate external knowledge into a LLM (Gao et al., 2023; Jiang et al., 2023b), these methods retrieve documents relevant to inputs from an external database, subsequently, the LLM utilizes the recalled documents as references to generate accurate results. The external knowledge used is typically text database (Hashimoto et al., 2018; Khandelwal et al., 2019; Shi et al., 2023; Lyu et al., 2024b). However, the text
We apply our RealRAG to all types of state-of-the-ar text-to-image generative models, including U-Net-based diffusion models (SD V2.1 (Rombach et al., 2022a), SD XL (Podell et al., 2023)), DiT-based diffusion models (SD v3 (Esser et al., 2024), Flux (BlackForest, 2024)), and auto-
2

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Prompt: Cybertruck is speeding along the Great Wall"
Real-world Objects
Retrieve
Generator
weal-world Objects
Prompt:
Reflective Retriever
"A Cybertruck is speeding along the Great Wall"
Text Encoder
Query
Generator
Reflective Negative
Vision Encoder
Missing Memory Image
Positive
Augmented Result
Input images
Self-reflective Contrastive Learning
(a)
(b)
Figure 2. An overview of our RealRAG. (a) The pipeline of the real-object- based RAG, We propose the first rcal-object-baas retrieval- augmented generation framework, which leverages real-world images compensate for the knowledge gap inherent in generative models and augment realistic image generation, (b) The framework of self-reflective contrastive learning, which injects the generator's knowledge into the self-reflective negatives ensuring that the retrieved images compensate for the model's missing knowledge.
database is not direct and controllable for realistic image generation (Blattmann et al., 2022; Zheng et al., 2025). In this paper, conduct iision-based real-object- based database, which is collected by realistic images from public real-world datasets, including ImageNet (Deng et al., 2009), Stanford Cars (Krause et al., 2013), 3), Stanford Dogs (Dataset, 2011), 1), and Oxford Flowers (Nilsback & Zisserman, 2008). In this way, we augment the realism of the generative images with the real-object- database.
Lei et al., 2024), thereby enhancing cross-modal retrieval capabilities. These methods often incorporate techniques (e.g,, hard negative mining (Kalantidis et al., 2020; Robin- son et al., 2020) and balanced learning (Zhu et al., 2022a; Liu et al., 2022)) to improve the quality of the learned embedding space. Despite its success in query-document matching, images that best match a text prompt may not be the most valuable references for eet-too-mmae generative models (Zhang et al., 2021). Consequently, we propose self-reflective contrastive learning approach, which retrieves images containing the missing knowledge of the generative models rather than selecting the most relevant images.
2.3. Contrastive Learning for Retrieval
Contrastive learning has emerged as a powerful method for retrieval tasks, leveraging the principle of learning represen- tations by contrasting positive and negative samples (Khosla et al., 2020; Ke-Khac et al.. 2020). The approach aims to map semantically similar data points closer in the embed- ding space while simultaneously pushing dissimilar data points apart. Methods such as SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) have popularized this frame- work in vision tasks, whereas in the domain of multi-modal retrieval, models including CLIP (Radford et al., 2021) have demonstrated their effectivvness by aligning textual and visual representations. Recent research has extended con- trastive learning to various modalities, such as audio (Rad- ford et al., 2021; Sun et al., 2023; Likhosherstov et al., 2021; Guzhov et al., 2022; Mahmud & Marculescu, 2023; Girdhar al., 2023), video (Huang et al., 2023; Fang et al., 2021: Luo et al., 2022; Xue al., 2022; Zhu et al., 2023), point cloud (Zhang et al., 2022; Zhu et al., 2022b; Huang al., 2022; Guo et al., 2023), and tactile data (Yang et al., 2024;
3. Methodology
Problem Setting: Given a textual prompt T drawn from a text space TT the text-to-image generation task aims pro- duce corresponding image E I that accurately reflects the semantics of T. Formally, we model this task as learning conditional distribution p(I T)), which describes how likely an image is given the text T. In practice, we often parameterize this distribution with a neural generator Go (with parameters 0), yielding:
Po(1 T) P(I T),
given a new text prompt T, the learned generator Ga gener- ate an image: i PT T):
Alternatively, one might produce a deterministic output by taking, for example, the mode of Po(T (T). In either case,
3

---

RealRAG: Retricval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
the goal is ensure that i visually manifests the semantics conveyed by T.
Reffective Negatives
Generation Space
Generation Space
Overview: An overview of RealRAG is shown in Fig. 2. Specifically, as shown in Fig. 2 (b), we first train the reflective retriever via self-reflective contrastive learning (Sec. 3.1). Powered by self-reflective contrastive learning, the reflective retriever retrieves images with the generator's s missing knowledge. Subsequently, as shown in Fig. 2 (a), utilize the recalled image as the reference for the alistic image generation, which effectively addresses the text-to-image generation's hallucinations (Sec. 3.2). We now describe these technical components in detail.
Self-reflective Contrastive Learning
4 l2
Promptt C.arrrrkkk is speeitg olong the Grest Hail"
Prompie Cyherrrcc speeding along the Great Wisl!
Figure 3. The comparison of generation space. We show the gen- eration space of the normal RAG (left) and our RealRAG (right).
where F; is the vision encoder and F3 is the text encoder (As shown in Fig. 2 (b)).
3.1. Self-reflective Contrastive Learning
We propose self- reff-reflectiv contrastive learning to retrieve useful images for the generator. Our key insight is to train a retriever that retrieves images staying off the generation space of the generator, yel closing to the representation of text prompts. To this end, as shown in Fig. 2 (b), we first generate images from the given text prompts and then utilize the generated images as queries to retrieve the most relevant images in the real-object- based database. These most rele- vant images are utilized as reflective negatives. Concretely, we denote the generator as G. given a text prompt T, we generate image Ieen and extract its feature embedding Zgen by: Lgen - G(T), Zyen- Fg(ggen);
We perform our self-reflective contrastive learning by using normal in-batch negative and the reflective negative the same time. We calculate the similarity between the text prompt and negatives:
N
D Doer
(erp(Zz Dans/T)), Diet exp(Zy Zneg); Σ Jipps
(7)
where N is the training batch size, Daeg is the similarity between the text prompt and the in-batch negative, Dieg is the similarity between the text prompt and the reflective negative.
Lastly, the overall loss of self-reflective contrastive learning is:
(3)
expl ZZ
7ob5 /T)
L= --
(8)
where F; is the vision encoder for extracting visual embed- ding from input images. We then select the self-reflective negative by ranking the images from the database I with the cosine-similarity. We choose the highest one:
CoplZZy L00g /T) + Dney + Dref
based
where T is a temperature hyperparameter.
Powered by self-reflective contrastive learning, the reflective retriever integrates the missing knowledge of the generator, augmenting the realism and accuracy of the generated im- ages and effectively addressing the hallucination problems in the text-to-image generation.
Ineg = arg max sim any EI
(F (F((T))), Zgen)
(4)
The reflective negative, with the highest cosine-similarity score, includes the original knowledge of the generator and empowers the retriever to capture the real-object-based im- ages with missing knowledge of the generator. For training the retriever via self-reflective contrastive learning, first extract object feature embeddings (Zoby. 22rp.... .... Zodg } for
3.2. Real-object-basee Retrieval-augmentte Generation
Expanding upon our trained reflective retriever, we subse- quently retrieve the augmented real-object-basee images Iref based on the given text prompt T for the generator:
the real-object-baal bbaee- images (Tobj: -...... [m))) E I:
Iref = Retriever(T, (Tanj ... 1),
(9)
(Zoby Zaoj [oin} = F. obji 12 Pabj:... .... 1bb H), (5)
where Retriever is the trained reflective retriever, and in the scale of real-object-bane database.
we select the embedding Zoby of ground truth image, which is matched with the input text prompt T, from the object feature embeddings (Zibg: Z2:g).... ..., Z J.
We then generate image Trea based on the text prompt T and the augmented image Iref
Subsequently, we encode text prompt T as the query em- bedding Zg and the reflective negative Ineg as the negative feature embedding Znegi
lres = G(T, Ires):
(10)
where G is the generator. Our real-object- based RAG im- proves the knowledge of the generator and significantly
Zg = F.)),, Zung = Fi(Tne));
(6)

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
extends the generation space for the frozen generator. As shown in Fig. 3, powered by Eq. 8, the reference image is distributed in the distribution space outside the generator's s generation space and close to the text prompt embedding. In this way, the generation space of our real-object-based RAG can effectively expand toward the text prompt embedding.
to validate the realism of the generated images (Sec. 4.2). Furthermore, validate the ability of RealRAG to generate unseen novel objects, we also test our model on recently introduced novel objects (Sec.4.3).
Evaluation Metrics. We employ FID, CLIP-T, and CLIP- to compare the visual quality and realism of generated images from different methods. Specifically, FID measures how closely the distribution of generated images matches that of real images by comparing their feature statistics in the latent space of Inception V3, CLIP-T uses CLIP model to assess how well the generated image matches with its text prompt, essentially quantifying text-image correspondence. CLIP-I, on the other hand, focuses on measuring image- image similarity through CLIP, often comparing a generated image to reference or target image, thereby evaluating visual fidelity consistency across images.
3.3. Implementation
Our RealRAG can be flexibly implemented with different existing generators, including U-Net-based diffusion mod- els, DiT-based diffusion models, and auto-regressive models. It is the first work to build unified RAG framework for all types of text-lo-image generative models.
real-object-based Database. We collect our real-object- based database from wide-use real-world datasets, including ImageNet (Deng et al., 2009), Stanford Cars (Krause et al., 2013), Stanford Dogs (Dataset, 2011), and Oxford Flow- ers (Nilsback & Zisserman, 2008). We use the training set of these datasets to conduct our database.
4.2. Fine-grained Object Generation
4.2.1. EXPERIMENT SETUP
We use existing text-to-image generative mod- els as the generator to implement our RealRAG. Concretely, we implement our RealRAG with the following models: U-Net-based diffusion models (SD V2.1 (Rombach et al., 2022a), SD XL (Podell et al., 2023)), DiT-based diffusion models (SD V3 (Esser et al., 2024), Flux (Black 2024)), and auto-regressive models (OmniGen (Xiao et al., 2024), Emu (Sun et al., 2024)). Specifically, for autore- gressive models, they can directly generate images from the image condition. For diffusion models, we utilize Control- Net, which introduces a branch to stable diffusion, enabling the inclusion of additional imputs, to input image-based con- ditions. For example. First, we retrieve and sort the closest images. Next, we input the selected images into the Con- trolNet branch to control specific elements during the image synthesis process.
Generator.
To evaluate the realism of the generated images, we use the text prompt ["A photo of a [CLASS NAME] "] for image generation. We generate ten images for each class to get more reliable results by multiple sampling. We use fine-grained real-world datasets to evaluate the realism of the generated images by calculating the FID score, CLIP-T score, and CLIP-I score between the ground truth images and the generated images.
4.2.2. GENERATIVE RESULT
Quantitative Results. In Tab. 1, we apply our method with all the types of SoTA text-to-image generators. Our Real- RAG demonstrates significant performance gains with all these models on the three benchmarks, including an average gain of 6.19% on the Stanford Cars, a 3.62% gain on the Stanford Dogs, and an 8.94% gain on the Oxford Flowers. As shown in Tab. 1, our RealRAG achieves the most sig- nificant improvement (a 9.90% gain average) with the auto auto-regressive model, which shows the potential of our Re- aIRAG to enhance the development of the large-scale auto- regressive model. To further evaluate the generative quality and realism of fine-grained objects, we utilize a pre-trained classification model (OpenCLIP (Cherti et al., 2022)) to cal- culate the classification accuracy for the generated images. As shown in Tab. 2, our RealRAG achieves considerable im- provements of the classification performance, e.g., a 3.89% gain for the auto-regressive model.
Training Details. We train our reflective retriever based on the pre-trained CLIP model (Radford et al., 2021). We add a simple MLP layer at the end of the vision encoder of the CLIP model, to map the visual embeddings outside the generation space of the generator and close to the input prompt embedding. We utilize the frozen text encoder from the CLIP model as our text encoder.
4. Experiment
4.1. Datasets and Implementation Details
Datasets and Benchmarks. We evaluate our proposed RealRAG on three fine-grained real-world image datasets, including Stanford Cars (Krause et al., 2013), Stanford Dogs (Dataset, 2011), and Oxford Flowers (Nilsback & Zisserman, 2008), We use the test sets of these datasets
Qualitative Results We show the qualitative results in Fig. 4. The visual results show the ability of our Re to overcome hallucinations and significantly improve the realism and quality of fine- realistic image gener- ation. Specifically, for the autoregressive model, In the
5

---

RealRAG:
Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Table Evaluation of fine-grained object generation. We report the quantitative results the Stanford Cars (Krause et al., 2013), Stanford Dogs (Dataset, 2011), and Oxford Flowers (Nilsback & Zisserman, 2008) benchmarks.
Type
Method
Stanford Cars
Stanford Dogs
Oxford Plowers
CLIP-I CLIP-T T FID
CLIP-I 1 CLIP-T
FID
CLIP-I1
CLIP-T f FID
OmniGien (Xiao et al., 2024)
$5.69
11.25
77.35
72.47
17.92
67.94
82.78
20.72
66.22
AR Model
OmniGen W. Ours
57.11
14.48
74.98
78.74
18.30
63.10
84.65
20.98
54,42
Emu (Sun et al., 2024)
61.41
13.49
86.73
80.51
19.52
48.75
81.88
21.17
84.33
Emu W. Ours
63.07
15.30
70.55
80.39
19.60
45.04
85.52
22.42
63.85
A
+1.54
+2.52
-9.28
+3.08
+0.23
-4.28
+2.76
+0.75
-16.14
SD 1R.ombhbh et al., 2022a)
61.48
14.16
64.99
68.80
17.69
49.60
82.97
20.76
69.17
U-Net-based
SD V2.1 W. Ours
64.50
15.84
58.98
79.06
18.57
43.93
88.43
21.77
60.76
Diffusion Model
SDXL (Podell et al., 2023)
59.53
13.03
61.98
77.10
18.05
49.28
81.08
19.59
55.20
SDXL W. Ours
64.71
15.16
60.95
77.79
18.31
45.31
85.63
20.92
47,41
+4.10
+1.91
-3.52
+5.48
+0.57
-4,82
+5.01
+1.17
-8.
SD V3 (Esser et ul., 2024)
59.43
12.61
59.94
80.28
18.56
53.37
83.23
20.19
63.46
DiT-based
SD V3 W. Ours
60.01
14.80
54.60
18.62
53.28
82.68
20.73
60.89
Diffusion Model
Fux (BlackForest, 2024)
60.50
13.70
58.47
80.16
18.36
45.71
82.18
20.19
55.93
Flux W. Ours
62.81
14.46
52.28
83.53
18.64
42.25
86.18
21.46
53.32
+1.45
+1.48
-5.77
+2.07
+0.17
-1.78
+1.73
+0.91
-2.59
Model
Obj.
Original
RealRAG
Obj.
Original
RealRAG
Obj.
Original
RealRAG
Prompt: Hatchback
Prompt:
Prompt: Colt's foot
AR Model
Prompt: Martin Virage Convertible 2012
Prompt: Brabancon griffon
Prompt: Silverbush
U-Net- based
Diffusion
Model
Prompt: Audi 100 Wagon 1994
Prompt: Bluetick
Prompt: Ball moss
DIT-based
Diffusion
Model
Figure 4. The visual results of mne-grained object generation. We visually compare the images generated by the original generators and our RealRAG. We also add real- world images for reference.
prompt ["Audi TT Hatchback 2011"], the Real- RAG model produces highly realistic rendering the vehicle, capturing intricate details like the side mirrors and headlights, which are closer the real object compared to the original AR model with significant hallucination in the car's side. For the U-Net-based diffusion model, In prompt [ "'Silverbush" RealRAG ensures the correct flower structure and vibrant color tones, whereas the orig- inal model struggles with maintaining fine- man--gr details like petal shape. Lastly, compared with the original gen- erator, ReaIRAG delivers a compelling representation of
the breed's signature coat patterns and body posture from the ''Bluticc prompt. The results demonstrate that RealRAG generates high realism and correct objects and show the flexible application of the SoTA generators.
4.3. Unseen Novel Object Generation
4.3.1. EXPERIMENT SETUP
To evaluate the ability to generate unseen novel objects, collect several recently introduced objects (e.g., the Cy- bertruck) to form the input prompts. Specifically, we use
6

---

RealRAG: Retricval-augmentted Realistic Image Generation via Self-reflective Contrastive Learning
Real Objects
Text-to-lmage Generation
Real Objects
Text-to-lmage Generation
Cybertruck
(a) Prompt: A cybertruck is apeeding along the Great Wall.
Boeing 747 (1cf)
(b) Prompt: Boeing 747 (1cf) forced to land in the desert.
Flux
Ours
Flux
Ours
Buzj
(c) Prompt: The Burj Khalifa
Apple vision
(d) Prompt: doq wearing apple vision pro.
rorneed greenery below.
pro
Flux
Ours
Flux
Ours
Volocopter 2x
(e) Prompt: An electric air taxi (volocopter 2x) parked on the rocf.
DJI Avata
Prompt: DJI Avata 2 flying over the rainforeat.
Flux
Ours
Flux
Ours
Boston Dynamics "Spot"
(g) Prompt: Boston Dynamics
(h) Prompt:
Venus Flytrap in the
Venus Flytrap
"Spot" on mountains.
Flux
Ours
Flux
Ours
Figure 5. The visual results of unseen novel object generation.
LLM (e.g., ChatGPT) to generate prompts based on these novel objects. For the image database, we use the images from the Internet (Google Images), which include the novel objects. To ensure faimeess in comparison, we conduct both qualitative evaluations and human evaluations.
although the Flux can generate sense of ["a robot walks on mounta ins" ns" the shape and type of the [ (BBoston Dynamics Spot "] is inaccurate. Our Re- aIRAG can generate the correct and realistic objects, which shows the significant improvement of our RealRAG for unseen noval object generation.
4.3.2. GENERATIVE RESULT
Human Evaluation. We conducted a human evaluation to assess the accuracy of the generated images. Data ac- quisition primarily revolved around participants' subjective assessments of the generative accuracy of the unseen novel objects. We involved 26 participants in evaluation (More details are included in the Appendix.). We use 7- point Likert scale to evaluate the accuracy of the generated images in the human's 's view. As shown in Fig. 6, we ask participants to give score from I (low accuracy) to 7 (high accuracy) for the 4 of images. The results show that
Qualitative Results. In Fig. 5, we present the visual com- parison the SoTA model Flux (BlackForest,, 2024) and Our RealRAG. The visual results demonstrate the significant improvement in the unseen object generation of Real- RAG. As the case ["Cybertruck' shows, compared with the original result generated by Flux, our RealRAG generates the realistic shape for the Cybertruck and also syn- thesis the sense [ ("speeding along the Great Wall" "]; As the case [" ["Boston Dynamics S,

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Table The classification results of fine-grained object generation. We use the pre-trained classification model (OpenCLIP (Cherti al., 2022)) to test classification accuracy of the generated images from the three datasets. A higher accuracy indicates the generated images are more similar to real images. We report the average accuracy score (the full results of the three datasets are shown in Tab, 5 of the Appendix).
Table 3. Evaluation of fine-grained object generation. We report the quantitative results on the Stanford Cars (Krause et al., 2013).
Generator
Method
CLIP-I1
CLIP-T
FID
Zero-RAG
61.84
14.06
80.75
Emu (Sun et al., 2024)
Normal RAG
62.20
14.31
76.85
ReaIRAG
63.07
15,30
70.55
Zero-RAG
62.59
14,87
61.67
SD V2.1 (Romhach et al., 2022a)
Normal-RAC RAG
63.05
15.03
60.77
Type
Method
Average Acc.
RealRAG
64.50
15.84
58.98
OmniGen (Xiao et al., 2024)
35.26
Zero-RAG
61.20
13.96
54.25
Autoregressive
OmniGen W. Ours
38.96
Flux (BlackForest. 2024)
Normal-RAR
61.72
14.52
54.04
Model
Emu (Sun et al., 2024)
35.81
ReaIRAG
62.81
14,46
52.28
Emu W. Ours
39.90
+3.89
FID Score
Accuracy Score
SD V2. (Rombaach al., 2022a)
37.45
20
U-Net-based
SD V W. Ours
40.23
36
30
Diffusion Model
SDXL (Podell et al., 2023)
36.32
37
SDXL W. Ours
40.49
A
+3.48
SD V3 (Esser et al., 2024)
36.09
DiT-based
SD V3 W. Ours
39.03
Normal RAG
RealRAG
Diffusion Model
Fux (BlackForest, 2024)
37.29
Flux Ours
40.37
Figure 7. The results present the performance comparison Nor- mal RAG and our RealRAG with the checkpoint from the second, fourth, sixth, eighth, and tenth epoch.
+2.99
Flux
Ours RealRAG
7
fine-grained object generation setting. As shown in Ta- ble we compare the generative performance of three variants: Zero-shot RAG, Normal RAG, and RealRAG. Specifically, Zero-shot RAG is baseline approach that em- plays pre-trained CLIP model to retrieve relevant images based on cosine similarity, whereas Normal RAG trains the retriever via contrastive learning to retrieve the most relevant images. We evaluate each approach using three different generators-EEme (Sun et al., 2024) (AR model), SDXL (Podell al., 2023) (U-Net-based diffusion model), and Flux (BlackForest, 2024) (DiT-based diffusion model), The results highlight the effectiveness of our proposed self- reflective contrastive learning. Compared with the Normal RAG framework, RealRAG achieves substantial improve- ments both generation realism and accuracy. addition, we employ classification model to measure the classifica- tion accuracy of the generated images (see Table 5 in the Ap- pendix), which further demonstrates the robust performance of RealRAG for fine-gradd fine-grained, realistic image generation.
6-
5-
4-
3-
2-
1-
Set
Set 2
Set 3
Sef 4
Figure 6. The 7-point Likert scale of the human evaluation. The participant is asked to score the generated images from (low accuracy) to 7 (high accuracy), according to given text prompts.
all participants consistently rated the quality of RealRAG- generated images with mean score exceeding indicating significant performance gains compared to Flux. RealRAG exhibited a relatively small variance across samples, high- lighting its proficiency unseen novel object generation. Overall, the combination of higher mean scores and lower variance demonstrates that RealRAG not only better at generating higher-realinee images but also more reliable, participants consistently rated its outputs highly.
To further investigate the effectiveness of the reflective neg- ative, we evaluate the performance the checkpoints from the second, fourth, sixth, eighth, and tenth epochs. As shown Fig. 7, while our RealRAG does not improve as rapidly retrieval frameworks relying solely on similarity in the early stages of training, it ultimately surpasses the generative bottleneck in later training stages, powered by the reflective negative.
5. Ablation Study
To investigate and analyze the effectiveness of our pro- posed RealRAG, we first perform an ablation study on the
Lastly, the results of the t-SNE visualization in Fig. re- veal the differences between the representation spaces con-
8

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Ground Truth
Normal RAG
RealRAG
Impact Statement
This paper advances the field of machine learning, with a particular focus on text-to-image generation. It addresses the prevalent challenge of hallucinations and distortions in generated images, which arise from the limited knowl- edge stored in the fixed parameters of generative models- an issue frequently encountered in real-world applications. While there are potential societal implications, we feel none need to be specifically highlighted here.
Figure The t-SNE visualization the generated images.
Acknowledgments
This work was supported by Open Project Program of Guangxi Key Laboratory of Digital Infrastructure (Grant Number: GXDIOP2024015); Guangdong Provincial De- partment of Education Project (Grant No.2024KQNNCX0288;
structed by normal RAG and our RealRAG. The visual results demonstrate that the generative space of RealRAG expands more in the direction of ground-truth images.
Scientific Research Projects for the Higher-cducationall In- stitutions (Grant No. 20963, Education Bureau of Guangzhou Municipality; CuuangZhou--IKKUST(((Z)) Joint Funding Program (Grant No.2025500313957), Education Bureau of Guangzhou Municipality.
6. Discussion
Research Purpose: Different from RDMs (Rombach et al., 2022b; Blattmann et al., 2022; Sheynin et al., 2022: Chen et al., 2022) that use retrieval-augmented tech- niques to train or fine-tune a diffusion model and achieve out-of-distribution (OOD) image generation by switching databases, our RealRAG aims to use retrieval-aggmenteed techniques train or fine-tune a diffusion model and achieve -itoo--dittribution (OOD) image generation by switching databases.
References
Bai, H. Lyu, Y., Jiang, L., S., Lu, H., Lin, X., and
Wang. L. Componerf: Text-guided multi-object composi- tional nerf with editable 3d scene layout. arXiv preprint arXiv:2303.1388 2023.
Unseen novel objects: These refer to objects that appear after the generative models and retrieval models are trained. The generative model cannot generate these objects, and the retrieval model can't It easily retrieve relevant references by similarity. This is much more challenging issue.
BlackForest. Black forest labs; frontier ai lab, 2024. URL https: htapss///bbccccforstlbbs.s ai/.
Blattmann, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B. Retrieval-augmente diffusion models. Ad- vances in Neural Information Processing Systems, 35: 15309-15324, 2022.
Hallucination When Generating Fine-Grained Objects: Existing SoTA 12i models are pre-trained on large-scale scale text-image paired datasets. As a result, they tend to produce hallucinations, such as inaccuracy or unrealistic features, when generating fine-grained objects. This is problem inherent to large generative models.
Chen, B., Monso, D. M., Du, Y.. Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffusion forcing: Next-token pre- diction meets full-sequence diffusion. arXiv preprint arXiv:240.001392, 2024.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
7. Conclusion
In this paper, we proposed RealRAG, the first real-object- based retrieval-augmented generation framework. Our RealRAG augmented one-grained and unseen novel object generation by learning and retrieving real-world images to overcome the knowledge gaps of generative models. Our RealRAG achieved remarkable performance boosts and is compatible with all types of generative models. Addi- tionally, we examined the potential of the RAG framework for text-to-in to-imageeeeaaaea generation. For future work, we will con- tinue to explore more efficient RAG frameworks for the -ot-to--mmge generation task and other generation tasks.
Chen, W., Hu, H., Saharia, C., and Cohen, W. W. Re- imagen: Retrievallaugmmened text-to-image generator. arXiv preprint arXiv:2209.1 14491, 2022.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.07 143, 2022.

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Dataset, E. Novel datasets for fine-grained image catego- rization. In First Workshop Fine Grained Visual Cate- gorization, CVPR. Citeseer. Citeseer. Citeseer, volume 5,
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
pp. 2. Citeseer, 2011.
Deng, J., Dong, W., Socher, R., Li, L..J,, Li, and Fei-Fei, L.. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., and Tan, H. Lrm: Large recon- struction model for single image to 3d. arXiv preprint arXiv:231 1.13310000000 2023.
Esser, P., Kulal, S., Blattmann, A.. Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. Forty-first International Conference on Machine Learning. 2024.
Huang, J., Li, Y., Feng, J., Wu, X., Sun, X., and Ji, R. Clover: Towards unified video- Langual alignment and fusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14856- 14866, 2023.
Fang, H., Xiong, P., Xu, L., and Chen, Y. Clip2video: Mas- tering video-text retrieval via image clip. arXiv preprint wrXiv22106.11097 2021.
Huang, T., Dong, B., Yang, Y., Huang, X., Lau, R. W., Ouyang, W., and Zuo, W. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. arXiv preprint 2rXiv:2210.01000 2022.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint MxXiv:2312.10997 2023.
Jiang, L., Ji, R., and Zhang, L. Sdf-3dgan: A 3d object gen- erative method based on implicit signed distance function. arXiv preprint
2.0.::2333006..... 2023a.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Jiang. L., Li. H., and Wang. L. general framework to boost 3d gs initialization for text-to-3d generation by lexical richness. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 6803-6812, 2024a.
Jiang, L.. Zheng, X., Lyu, Y., Zhou, J., and Wang. L. Brightdreamer: Generic 3d gaussian generative framework for fast text-to-3d synthesis. arXiv preprint arXiv:2403. 1273, 2024b.
Gu, Z., Yan, R., Lu, J., Li, P., Dou, Z., Si, C., Dong, Z.., Liu, Q. Lin, C., Liu, Z. et al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. arXiv preprint arXiv:2501 03387707777... 2025.
Jiang. L., Lin, J., Chen, K., Ge, W., Yang, X., Jiang, Y., Lyu, Y., Zheng, X., and Chen, Y. Dimer: Disentangled mesh reconstruction model. arXiv preprint arXiv:2504. 150.7 2025.
Guo, Z., Zhang, R., Zhu, X., Tang. Y., Ma. X., Han, J., Chen, K., Gao, P., Li, X., Li. H., et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint 23X9v:2309.000 00615, 2023.
Jiang, Z., Xu, F.F. Gao, L., Sun, Z., Liu, Q., Dwivedi- Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. arXiv preprint 26X1v:2305.06993, 2023b.
Guu, K., Lee, K. Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-tra pre-training In International conference machine learning, pp. 3929- 3938. PMIR, 2020.
Kalantidis, Y., Sariyildiz, M. B., Pion, N., Weinzaepfel, P., and Larlus, D. Hard negative mixing for contrastive learning. Advances in neural information processing systems, 18:21798-21800 2020.
Guzhov, A., Raue, F., Hees, J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Khandelwal, U., Levy, O., Jurafsky, D.,, Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.0012.. 2019.
Hashimoto, T. B., Guu, K., Oren, Y.. and Liang, P. S. A retrieve-and-ede framework for predicting structured out- puts. Advances in Neural Information Processing Sys- tems, 31, 2018.
Khosla, P., Teterwak, P., Wang, C., Sama, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised
10

---

RealRAG: Retricval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
contrastive learning. Advances in neural information processing systems, 33: 33:18661-18673 2020.
Lyu, Y., Zheng, X., Zhou, J., and Wang. L. Unibind: Llm- augmented unified and balanced representation space bind them all. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 26752-26762, 2024b.
Krause, J., Stark, M., Deng, and Fei-Fei, L. 3d object rep- resentations for fine- tnee-griied categorization. In Proceed- ings of the IEEE international conference computer 2013.
vision workshops, pp.
Mahmud, T. and Marculescu, D. Ave-clip: Audioclip-based multi-window temporal transformer for audio visual event localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5158- 5167, 2023.
Le- P. H., Healy, G., and Smeaton, A. F. Contrastive representation learning: A framework and review. Ieee Access, 8: 19999977-193934, 2020.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Lei, W., Ge, Y., Yi, K., Zhang, J., Gao, D., Sun, D., Ge, Y., Shan, Y., and Shou, M. Z. Vit-lens: Towards omni- modal representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2024.
pp.
OpenAL. Gpt-4 technical report, 2023.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t. Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 1149459-94744 2020.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2023.
Podell, D., English, Z., Lacey. K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffiusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.0/99 2023.
Li, C.. Zhang. C., Cho, J., Waghwase, A., Lee, L.-H., Rameau, F., Yang, Y., Bae, S.-H., and Hong, C. S. Gen- crative ai meets 3d: A survey on text-to-3d in aige era. arXiv preprint ar arXiv:2355.06333 2023.
Poole, B., Jain, A., Barron, T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 000/2299.49990 14988, 2022.
Li, X., Zhang, Q., Kang, D., Cheng, W., Gao, Y., Zhang, J., Liang, 7.,, Liao, J., Cao, Y.P.,, and Shan, Y. Ad- vances in 3d generation: A survey, arXiv preprint wxXiv:2401.17800 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Likhosherstov, V., Arnab, A., Choromanski, K., Lucic, M., Tay, Y., Weller, A., and Dehghani, M. Polyvit: Co- training vision transformers on images, videos and audio. arXiv preprint arXiv:2111.1299) 2021.
Robinson, J., Chuang, C.-Y,, Sra, S., and Jegelka, S. Con- trastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
Liu, J., Zou, C., Lyu, Y., Chen, J., and Zhang, From reusing to forecasting: Accelerating diffusion models with taylorseers. arXiv preprint 27X1:22503.06923, 2025.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684- 10684-10695, June 2022a.
L Liu, 7., Xiong, C., Lv, Y., Liu, Z., and Yu, G. Universal lasiun-language dense retrieval: Learning a unified repre- sentation space for multi-modal retrieval. arXiv preprinl arXiv:2209.00179, 2022.
Rombach, R., Blattmann, A., and Ommer, B. Text-guided synthesis of artistic images with retrieval- dif- fusion models. arXiv preprint arXiv:2207. 13038, 2022b.
Luo, H., Ji, I.., Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293-304, 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294 2022.
Lyu, Y. Zheng, X., and Wang. L. Image anything: Towards crasoning-coheren and training-free free multi-modal image generation, arXiv preprint arXiv:2401. 1..01.76644 17664, 2024a.

---

RealRAG: Retricval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y. Knn-diffusion: Im- age generation via large-scale retrieval. arXiv preprint arXiv:2204.02849 2022.
Xue, H., Sun, Y., Liu, B., Fu, J., Song, R., Li, H., and Luo, J. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint WwXiv:22090644333 2022.
Shi, W., Min, S., Yasunaga, M., Sco, M., James, R., Lewis, M., Zettlemoyer, L.., and Yih, W.-t. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301.12662, 2023.
Yang. F., Feng. C., Chen, Z., Park, H., Wang, D., Dou, Y., Zeng, Z.., Chen, X., Gangopadhyay, R., Owens, A., et al. Binding touch to everything: Learning unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26340-26353 2024.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q- Yang. H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., and Yang, Y. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 333-842, 2021.
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J.. Huang, T., and Wang, X. Generative mul- timodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2024.
Zhang, R., Guo, 7., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, Y., Gao, P., and Li, H. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. Zheng, X., Weng, Z., Lyu, Y., Jiang, L., Xue, H., Ren, B., Paudel, D., Sebe, N., Van Gool, L.,, and Hu, X. Retrieval augmented generation and understanding in vision: A sur- vey and new outlook. arXiv preprint arXiv:2503. 18016, 2025.
Pattern Recognition, pp.
Sun, W., Zhang, J., Wang. J., Liu. Z., Zhong, Y., Feng. T., Guo, Y., Zhang, Y., and Barnes, N. Learning audio-visual source localization via false negative aware contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6420- 6429, 2023.
2022.
Tang. J., Chen, Z., Chen, X., Wang. T., Zeng, G., and Liu, Z. Lgm: Large multi-view gaussian model for high- resolution 3d content creation. In European Conference on Computer Vision, pp. 1-18. Springer, 2024.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXii:2408. 10039 2024.
Tang, Z., Yang, Z., Zhu, C., Zeng, M., and Bansal, M. Any- to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36:16083- 16099, 2023.
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang. Y., Jiang, W., Zhang, J., Li, Z., et al. Language- bind: Extending video- languag pretraining to n-modality by language based semantic alignment. arXiv preprint arXiv:2310.01852. 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023.
Zhu, J., Wang. Z., Chen, J., Chen, Y.-P. P., and Jiang, Y.- G. Balanced contrastive learning for long-taiil mailtd visual recognition. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6908- 6917, 2022a.
Wang, Q., Luo, Y., Shi, X., Jia, X., Lu, H., Xue, T., Wang, X., Wan, P., Zhang, D., and Gai, K. Cinemaster: A 3d-aware and controllable framework for cinematic text- to-video generation. arXiv preprint arXiv:2502.08639, 2025.
Zhu, X., Zhang, R., He, B., Zeng, Z., Zhang, S., and Gao, P. Pointelip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211. 2022b.
Xiao, S., Wang, Y., Zhou, J.. Yuan, H., Xing, X., Yan, R., Wang, S., Huang, T., and Liu, Z. Omnigen: Unified image generation. arXiv preprint 24X1v:2409.11340 2024.
Xie, J., Mao, W. Bai, Z., Zhang, D. Wang. W., Lin, K. Q. Gu, Y., Chen, Z. Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
12

---

RealRAG: Retricval-augmennted Realistic Image Generation via Self-reflective Contrastive Learning
A. Appendix
A.I. More Experimental Results
We show the full classification results of fine-grained object generation in Tab. 4. The results demonstrate that RealRAG effectively enhances the generative quality and realism. Notably, the augment of our RealRAG is not limited to the original generative ability of the generators. For example, although the SoTA model, Flux (BlackForest, 2024) achieves the best performance the Stanford Car, e.g.35.78 Accuracy score, RealRAG also effectively enhances the generative quality and realism (a 4.20% gain).
Table 4. The full classification results of fine-graines object generation.
Type
Method
Stanford Cars
Stanford Dogs
Oxford Flowers
Average Acc.
OmniGen (Xiao al., 2024)
32.80
40.93
32.04
35.26
Autoregressive Model
OmniGen W. Ours
37.45
44.71
34.72
38.96
Emu (Sun al., 2024)
33.94
41.32
32.18
35.81
Emu W. Ours
40.24
44.33
35.13
39.90
A
+5.48
+3.40
+2.82
+3.89
SD V2.1(Rombach et al., 2022a)
35.21
42.35
34.79
37.45
U-Net-based
SD V2.1 W. Ours
39.43
46.10
35.15
40.23
Diffusion Model
SDXL (Podell al., 2023)
35.20
41.06
32.69
36.32
SDXL W. Ours
40.55
44.76
36.17
40.49
A
+4.79
+3.73
+1.92
+3.48
SD V3 (Esser al., 2024)
33.90
41.26
33.12
36.09
DiT-based
SD V3 W. Ours
38.72
44.24
34.05
39.03
Diffusion Model
Fux (BlackForest, 2024)
35.78
42.11
33.98
37.29
Flux W. Ours
39.98
45.84
35.29
40.37
A
+4.51
+3.36
+1.12
+2.99
A.2. More Results in Ablation Study
We show the ablation study of fine-graines classification in Tab. 5. Specifically, Zero-shot RAG is the straightforward pipeline of the RAG, which simply uses the pre-trained CLIP model retrieve relevant images by cosine-similarity: Normal RAG trains the retriever via contrastive learning retrieve the most relevant images. The results show the effectiveness of our proposed self-reflective contrastive learning. Compared with the normal RAG framework, our RealRAG achieves significant improvements in generation realism and accuracy.
Table 5. The ablation study of fine-grained classification the three datasets.
Generator
Method
Stanford Cars
Stanford Dogs
Oxford Flowers
Average Acc. 1
Zero-RAG
37.18
42.88
34.27
38. 18.11
Emu (Sun et al., 2024)
Normal-RAs RAG
38.75
43.09
35.10
38.95
RealRAG
40.24
44.33
35.13
39,90
Zero-RAG
38.64
44.07
33.70
38.80
SD V2.1 (Rombach et al., 2022a)
Normal-RAG
39.07
44.92
35.21
39.73
RealRAG
39.43
46.10
35.15
40.23
Zero-RAG
36.27
44.31
34.59
38.39
Flux (BrackForest, 2024)
Normal- RAGG
37.84
45.01
34.96
39.27
RealRAG
39.98
45.84
35.29
40.37
A.3. Details and More Results Human Evaluation
We conducted a human evaluation to assess the realism and accuracy of the generated images. Data acquisition primarily revolved around participants' subjective assessments of the realism in fine- one-grained image generation and accuracy in unseen novel image generation. The evaluation consists of two tasks: (1) Text-to-image matching task and (2) Real/False task. The
13

---

RealRAG: Retrieval-augmentel Realistic Image Generation via Self-reflective Contrastive Learning
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
CASE CASE 2 CASE 3 CASE CASE 5 C CASE
CASE
REAL FAKE
Figure 9. The results of the Real/fake task. Case 1,2,3,4 are generated by our RcalRAG and case 5,6,7,8 are generated by the original generators.
human evaluation procedures were approved by the ethical committee.
Participants. The evaluation involved 26 participants, of whom 69.23% aged 18-24 and 30.77% aged 25-34. The gender distribution was 53.85% male and 46. 15% female. Also, 57.69% of the participants had previous experience with generative models, and 42.31% had no experience using AI generative models in the last six months.
Tasks and Measurement. We designed two tasks to evaluate the effectiveness of RealRAG for its ability to generate fine-grained and unseen novel objects. The two tasks include: (1) Text-to-image matching task and (2) Real/False task.
Text-to-image matching task. In this task, we collected four sets of images, and for each set, there were two images, one was generated by the original Flux model (BlackForest, 2024), and the other was generated by our RealRAG, We then asked the participants to rate how well the text prompt and the generative image matched via 7-point Likert scale. Images presented in random order, and the specific cases shown in Fig. 10.
Real/Fake task. In this task, we selected eight cases, which, four them are generated by original generators and others are generated by our ReaIRAG. Furthermore, we asked the participants whether the images were real or fake images (generated by AI). We show the task details in Fig. 11.
Results. We show the evaluation results of task (1) Fig.6 of the main paper. For the task (2), we present the human- evaluation results in 9ig.9. The results show that the images generated by RealRAG have got higher scores from the participants. To summarize, more than 70% of the participants found our images be realistic, which demonstrates the strong performance of our RealRAG for realistic image generation.
B. More Visualization
B.I. Qualitative Feature Visualization
We show more t-SNE visualization results in Fig. 12, which shows that the generative space of RealRAG expands more in the direction of ground-truth images.
B.2. Generative Results
We show more generative results for fine- ene-grained generative results Fig. 13. The results show the generative ability of our AG in realistic image generation.
14

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
text: A cybertruck is speeding along the Great Wall.
text: Boeing 747 (1ef) forced to land in the desert.
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you the matching of the images to the text? On scale of I (low) (high).
How would you rate the matching of the images to the text? On scale of (low) to (high).
Set
text: The Burj Khalifa Tower is surrounded by greenery below.
text: dog wearing apple vision pro.
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you rate the matching of the images to the text? On scale of (low) to (high).
How would you rate the matching of the images the text? On scale of) (low) to 7 (high).
Set
text: electric air taxi (volocopter 2x) parked on the roof.
text:
Avata flying over the rainforest.
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you the matching of the images to the text? On scale of 1 (low) (high).
How would you rate the matching of the images to the text? On scale of (low) to (high).
Set
text:
Boston Dynamics "Spot" walks on mountains.
text: Venus Flytrap in the desert.
Please review the left image and text carefully.
Please review the left image and carefully.
How would you rate the matching of the images to the text? On scale of I (low) to 7 (high).
How would you rate the matching the images the text? On scale of (low) to (high).
Set 4
Figure 10. The cases in Text-to-image matching task of the human evaluation.
15

---

RealRAG: Retrieval-augmeniee Realistic Image Generation via Self-reflective Contrastive Learning
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you rate the realism of the images? a scale (low) (high).
How would you rate the realism of the images? On a scale of (low) (high).
Case 1
Case 2
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you rate the realism of the images? On a scale of (low) to (high).
How would you rate the realism of the images? On a scale of 1 (low) to 7 (high).
Case 3
Case 4
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you rate the realism of the images? On a scale of 1 (low) to 7 (high).
How would you rate the realism of the images? On a scale of 1 (low) to 7 (high).
Case 5
Case 6
Please review the left image and text carefully.
Please review the left image and text carefully.
How would you rate the realism of the images? On a scale of 1 (low) to 7 (high).
How would you rate the realism of the images? On scale of 1 (low) to 7 (high).
Case 7
Case 8
Figure 11. The cases in Real/Fake task of the human evaluation. Case 1,2,3,4 are generated by our RealRAG and case 5,6,7,8 are generated by the original generators.
16

---

RealRAG: Retrieval-augmente Realistic Image Generation via Self-reflective Contrastive Learning
Stanford Car
Stanford Dog
Stanford Flower
AR Model
U-Net-bused
Diffusion
Model
DiT-bused
Diffusion Model
Ground Truth
Normal RAG
ReaIRAG
Figure 12. More t-SNE visualization results of generated images.
17

---

RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Real Obj.
Original
RealRAG
Real Obj.
Original
RealRAG
Real Obj.
Original
RealRAG
Prompt: Bugutti Veyron 16.4 Convertible 2009
Prompt: griffon
Prompt: Ostenspermmm
Prompt: Ford E-Series Wagon Van 2012
Prompt: Ibizan hound
Prompt: Pelargonium
Prompt: Chevrolet Cobalt SS 2010
Prompt: Saluki
Prompt: Rose
Prompt: Audi AS Coupe 2012
Prompt: Kelpie
Prompt: Red ginger
Prompt: Chevrolet Express Cargo Van 2007
Prompt: Mexican hairless
Prompt: Great masterwort
Prompt: Chevrolet Monte Carlo Coupe 2007
Prompt: Great Dane
Prompt: Mallow
Figure 13. More visual results of fine- fine-grained object generation.
18

---

RealRAG: Retricyal-augmenter Realistic Image Generation via Self-reflective Contrastive Learning
C. Application Value
For existing commercial text-to-image generative models, training them is cost-prohibitive. Therefore, a pipeline needed integrate real-time updated data from the internet into the generative model without any retraining, enabling the generation of unseen novel objects. On the other hand, in specific application scenarios such advertising creation and multi-modal generation (Tang al., 2023; Lyu et al., 2024a), users need generated images that meet their design requirements, while also ensuring that products (fine-grained objects) within the image remain realistic. This requires generative models to have the ability to generate both open-domain and fine- line-grained objects. Therefore, RealRAG focuses on reducing hallucinations in large (2i generators through RAG technology, enabling open-domain generators to generate specific fine-grained objects.
Beyond text-to-image (T2I) generation itself, several downstream tasks depend on and extend same technology, like 3D generation (Li et al., 2024; Jiang et al., 2023a; Li et al., 2023) and video generation (Singer et al., 2022; Gu al., 2025; Wang al., 2025). For example, (i) text-to-3D synthesis (Poole et al., 2022; Jiang et al., 2024a;b; Bai et al.. 2023) generally requires strong T2I backbone, while (ii) image-to-30 3D reconstruction (Jiang al., 2025; Hong et al., 2023; Tang et al., 2024) often begins with an image created by T21 model.
19


Reviews:
[{"review_id": "6zrHVsSsda", "reviewer": "Reviewer_vQw4", "summary": "The paper introduces RealRAG, a retrieval-augmented generation framework designed to enhance text-to-image models by addressing their inherent knowledge limitations. The main idea is to retrieve and integrate real-world images to supplement the generator's missing knowledge. The key innovation of RealRAG lies in its reflective retriever, trained using self-reflective contrastive learning. This method ensures that retrieved images effectively compensate for gaps in the model's knowledge. Experimental results comparing on fine-grained object generation and unseen novel object generation demonstrate its effectiveness.\n\n##update after rebuttal\nThe authors have addressed most of my concerns satisfactorily, I am therefore updating my recommendation to a weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank *Reviewer vQw4*  for the constructive comments on our work. We promise to revise the paper based on the comments. We will cite and discuss all the papers in the \"Essential References Not Discussed\" in the final version**\n\n# About the \"first work\" claim and the novelty.\nThe reference mentioned by the reviewer in the **Essential References Not Discussed** section, referred to here as RDMs, and our RealRAG both utilize retrieval-augmented techniques in text-to-image generation. However, the research problems and purposes we aim to solve are entirely different.\n* **Research Purpose of RDMs**: They use retrieval-augmented techniques to ***train or fine-tune*** a diffusion model and achieve ***out-of-distribution (OOD) image generation*** by switching databases.\n* **Research Purpose of RealRAG**: RealRAG uses retrieval-augmented techniques to ***mitigate hallucinations*** when generating fine-grained objects and to enhance the ability to generate ***unseen novel objects***, ***without any training*** of generative models.\n\nThere are **three key points** to pay attention to:\n1. **Out-of-Distribution Objects ≠ Unseen Novel Objects**:\n   - **Out-of-distribution objects**: In the RDMs work mentioned by the reviewer, out-of-distribution is defined as data from outside the domain of the training dataset, such as **\"Angry Bird.\"** In **Knn-diffusion**[3]. These objects existed during the generative model's training but were not part of the training set. As a result, the foundational generative model cannot generate them. RDMs use retrieval-augmented techniques, employing the CLIP model's similarity calculation ability to provide relevant references to the generative model, thereby solving the OOD problem.\n   - **Unseen novel objects**: These refer to objects that appear after the generative models and retrieval models are trained. The generative model cannot generate these objects, **and the retrieval model can't easily retrieve relevant references by similarity**. This is a much more challenging issue.\n\n   For OOD object generation, existing state-of-the-art text-to-image generators can generate images from almost all domains, as shown in the [link](https://realrag.github.io/AngryBird/), and the original FLUX model can generate **\"Angry Bird\"** very well, outperforming the retrieval-augmented KNN-diffusion. However, for unseen novel objects, such as the **\"cybertruck\"** (shown in **Fig. 5** of the main paper), FLUX still can't generate an accurate cybertruck. Therefore, the focus of this paper is on ***solving the problem of generating unseen novel objects.***\n2. **Hallucination Issue When Generating Fine-Grained Objects**:\n\n   Existing SoTA t2i models are pre-trained on large-scale text-image paired datasets. As a result, they tend to produce hallucinations, such as inaccuracy or unrealistic features, when generating fine-grained objects. This is a problem inherent to large generative models.\n3. **Application Value**:\n\n   For existing commercial text-to-image generative models, training them is cost-prohibitive. Therefore, a pipeline is needed to integrate real-time updated data from the internet into the generative model **without any retraining**, enabling the generation of **unseen novel objects**. On the other hand, in specific application scenarios such as advertising or poster creation, users need generated images that meet their design requirements, while also ensuring that **products (fine-grained objects) within the image remain realistic**. This requires generative models to have the ability to generate **both open-domain and fine-grained objects**. Therefore, RealRAG focuses on reducing hallucinations in large t2i generators through RAG technology, enabling open-domain generators to generate specific fine-grained objects.\n# Evaluation on ImageNet.\nThanks for the reviewer's suggestion. Please check the results in the rebuttal for ***Reviewer 5Kbn***.\n# More Baselines\nPlease check the results in the response for ***Reviewer HZx9***.\nWe also show more visual comparison here [link](https://realrag.github.io/RealRAG_Rebuttal/).\n# About the Database\nWe are sorry for the misunderstanding. For the fine-grained object generation setting, we use the Real-object-based Database. For the unseen novel object generation setting, as shown on line 042, we use the images from the Internet (Google Images), which include the novel objects.\n# More User Study\nThanks for your suggestions, we **try our best to add more cases and invite more participants** for the user study. Finally, we extant the scale of the user study to **50 participants and 20 cases**. The results can be found at the [link](https://realrag.github.io/UserStudy/).\n# About **the trade-off between performance and the inference time**.\nWe present the comparison results in the response for ***Reviewer HZx9***. The table shows that the average percentage increase in reasoning time is much lower than the percentage increase in performance.", "claims_and_evidence": "1. The paper claims to present the first real-object-based retrieval-augmented generation (RAG) framework. However, the term real-object-based appears to be primarily reflected in the fact that the retrieval dataset is constructed using real objects. This concept, however, is not novel, as prior works have also employed real-object-based datasets in a similar retrieval-augmented manner. For instance, KNN-Diffusion [1] utilizes the MS-COCO dataset, while RDM [2] leverages the ImageNet dataset to enhance generative processes. Given these precedents, this claim requires further clarification.\n\n2. Furthermore, the paper claims to be the first to establish a unified RAG framework applicable to all major categories of text-to-image generative models, including diffusion models and autoregressive models. However, a similar direction has already been explored in RDM, which proposes a RAG system capable of being integrated with multiple likelihood-based generative methods, also including both diffusion models and autoregressive models. In light of this, the assertion of novelty should be carefully contextualized with respect to prior works.\n\n[1]Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, \nYaniv Taigman. \"Knn-diffusion: Image generation via large-scale retrieval.\"\nInternational Conference on Learning Representations, 2023.\n\n[2] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, Björn Ommer. \n\"Retrieval-augmented diffusion models.\" Advances in Neural Information Processing \nSystems, 2022.", "methods_and_evaluation_criteria": "The proposed method utilizes several benchmark datasets, including ImageNet, Stanford Cars, Stanford Dogs, and Oxford Flowers, as the retrieval database. However, the evaluation of the proposed RealRAG framework is conducted exclusively on three fine-grained real-world image datasets: Stanford Cars, Stanford Dogs, and Oxford Flowers. \n\nThis raises two key concerns. \n\n1. Given that ImageNet is included in the retrieval database, it is unclear why the evaluation does not extend to general image generation using ImageNet. Such an evaluation could provide insights into the method’s performance on broader image retrieval and generation tasks. \n\n2. The inclusion of ImageNet as part of the retrieval database introduces an important trade-off: while a larger retrieval database may enhance retrieval diversity and coverage, it also increases retrieval time complexity. Therefore, it is crucial to analyze the computational impact of incorporating ImageNet into the retrieval database and determine whether the associated increase in retrieval time justifies the potential performance improvements.", "theoretical_claims": "This paper does not present any formal proofs or theoretical claims.", "experimental_designs_or_analyses": "1. The experimental comparison should be conducted against other retrieval-augmented image generation methods, as specified in the \"Essential References Not Discussed\" section. A comprehensive comparison with these methods will provide a clearer evaluation of the proposed approach's effectiveness and highlight its advantages and limitations in relation to existing techniques.\n\n2. The retrieval database used in the unseen novel object generation scenario requires further clarification. Specifically, it is important to confirm whether the database still consists of ImageNet, Stanford Cars, Stanford Dogs, and Oxford Flowers. Additionally, if these datasets are indeed used, it would be better to illustrate which images are retrieved in this unseen novel object generation case. Would the retrieved images provide useful informations.\n\n3. The number of image sets used for human evaluation appears to be relatively small, with only four sets being considered. This limited sample size may not be sufficient for a robust quantitative analysis, potentially affecting the statistical significance of the results. \n\n4. The type of embedding used to represent the constructed representation spaces in both the standard retrieval-augmented generation (RAG) approach and RealRAG in t-SNE visualization should be explicitly stated.", "relation_to_broader_scientific_literature": "The idea of exploring the generator's missing knowledge for retrieval-augmented generation task might be related to the broader scientific literature.", "essential_references_not_discussed": "This paper focuses on the field of Retrieval-Augmented Image Generation; however, the discussion of related work lacks a comprehensive review of prior retrieval-augmented generative models. \n\n[1]Robin Rombach, Andreas Blattmann, Björn Ommer. \"Text-guided synthesis of artistic \nimages with retrieval-augmented diffusion models.\" arXiv preprint, 2022.\n\n[2] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, Björn Ommer. \n\"Retrieval-augmented diffusion models.\" Advances in Neural Information Processing \nSystems, 2022.\n\n[3]Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, \nYaniv Taigman. \"Knn-diffusion: Image generation via large-scale retrieval.\"\nInternational Conference on Learning Representations, 2023.\n\n[4] Wenhu Chen, Hexiang Hu, Chitwan Saharia, William W. Cohen. \"Re-imagen: \nRetrieval-augmented text-to-image generator.\" arXiv preprint arXiv:2209.14491 (2022)\n\n[5] Huaying Yuan, Ziliang Zhao, Shuting Wang, Shitao Xiao, Minheng Ni,\nZheng Liu, and Zhicheng Dou. Finerag: Fine-grained retrieval-augmented\ntext-to-image generation. In Proceedings of the 31st International Confer-\nence on Computational Linguistics, pages 11196–11205, 2025.", "other_strengths_and_weaknesses": "The strengths and weaknesses have been presented above.", "other_comments_or_suggestions": "1. The illustrations in Figure 1, specifically images (1) and (2), appear to be misarranged. Their correspondence with the description provided in the Introduction section is inconsistent, leading to potential confusion for readers. \n\n2. In Section 2.1, the reference to the large-scale text-image paired dataset incorrectly cites \"LIANG 5B\" instead of \"LAION 5B.\" The correct dataset name should be \"LAION 5B,\" a widely recognized and publicly available resource. \n\n3. In the Experiments section, specifically on line 271 of page 5, there is an empty set of parentheses following the mention of the CLIP model, indicating a missing reference. \n\n4. In the qualitative results section, on line 308 of page 6, the discussion regarding the comparison with the AR model does not correspond to the correct text prompt.", "questions_for_authors": "1. It would be important for authors to explicitly clarify how their approach differs from and improves upon these prior works. If they acknowledge that these claims require revision, they should provide a more precise positioning of their contributions and add a more comprehensive discussion.\n\n2. Why was ImageNet excluded from the evaluation? Including it could provide a broader perspective on the method’s performance.\n\n3. The retrieval database includes ImageNet, which increases retrieval diversity but also adds computational cost. Could the authors provide a quantitative analysis of the retrieval time impact? \n\n4. The experimental comparison should include other retrieval-augmented image generation methods such as those listed in the \"Essential References Not Discussed\" section.\n\n5. In the unseen novel object generation scenario, does the retrieval database still include ImageNet, Stanford Cars, Stanford Dogs, and Oxford Flowers? If so, could the authors provide examples of retrieved images and explain whether they provide meaningful information for generating unseen objects? This would help clarify whether the retrieval process effectively supports novel object generation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CVLlK9fNTh", "reviewer": "Reviewer_5Kbn", "summary": "The paper introduces a novel retrieval-augmented generation (RAG) framework aimed at improving text-to-image generative models. Traditional generative models suffer from hallucinations and distortions when generating fine-grained or novel real-world objects due to their fixed training datasets. RealRAG overcomes this limitation by integrating external real-world images through a self-reflective contrastive learning approach. This technique ensures that the retrieved images supplement missing knowledge rather than merely matching text prompts based on similarity. RealRAG is adaptable to various generative architectures, including diffusion and auto-regressive models, yielding significant performance improvements. The method demonstrates superior realism in generating both fine-grained and unseen objects, outperforming existing retrieval-based models while maintaining modular compatibility across different generative approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank *Reviewer 5Kbn* for the constructive comments on our work. We are very grateful to the reviewer for recognising the novelty of our idea and the richness and rationality of our experiments.**\n\n# About the Database\n\n***We are sorry for the misunderstanding***. For the fine-grained object generation setting, we use the Real-object-based Database; For the unseen novel object generation setting, as shown on line 042, we use the images from the Internet (**Google Images**), which include the **novel objects**.\n\nWe show the results of using Internet data in the **Fig. 5 in the main paper**.\n\n# Evaluation on ImageNet\n\nThanks for the reviewer's suggestion. We add the experiments with the ImageNet. \n\n| Model        |          | ImageNet Val |          |\n| ------------ | :------: | :----------: | :------: |\n|              |  CLIP-I  |    CLIP-T    |   FID    |\n| Emu          |  53.55   |    14.18     |  25.56   |\n| Emu W. Ours  |  55.48   |    16.21     |  23.51   |\n| **Gain**     | **1.93** |   **2.03**   | **2.05** |\n| SDXL         |  53.74   |    14.53     |  17.57   |\n| SDXL W. Ours |  57.35   |    17.20     |  12.81   |\n| **Gain**     | **3.61** |   **2.67**   | **4.76** |\n| FLUX         |  54.41   |    14.77     |  16.84   |\n| FLUX W. Ours |  56.73   |    15.82     |  14.56   |\n| **Gain**     | **2.31** |   **1.05**   | **2.28** |\n\nOur RealRAG demonstrates significant performance gains for the SoTA t2i generative models on the ImageNet dataset. Furthermore, our insight is to **reduce hallucinations of the fine-grained object generation** (eg, various types of cars) and **the classes in ImageNet are coarse-grained** (eg, cars and planes). Therefore, the selected datasets (eg, Stanford Car) are more suitable for the evaluation.\n\n# More clarification about Image-conditioned generation\n\n*So sorry for the lack of a relevant background introduction.* Several methods have been proposed for inputting images into diffusion models. These include: 1) **Embedding-based approaches**, where image embeddings from CLIP are concatenated with timestamp embeddings in the UNet of diffusion models [Ref A, B]; 2) **Input Layer Concatenation-based approaches**, in which the latent representation of the input image is concatenated to the UNet's input, changing its input dimension from 4 to 8, with the additional input layer initialized to zero [Ref C]; 3) **ControlNet-based approaches**, where ControlNet [Ref D] introduces a branch to stable diffusion, enabling the inclusion of additional inputs; and 4) **Noise-based approaches**, which involve image editing methods [Ref E] that add noise to the input image, followed by denoising.\n\nWe use the version of our **Reflective Retriever + FLUX** as an example. First, we retrieve and sort the closest images. Next, we input the selected images into the FLUX’s ControlNet branch to control specific elements during the image synthesis process. The FLUX version is **black-forest-labs/FLUX.1-Canny-dev**, which allows for inputting a referenced image to guide the generation process. For further details, please refer to the code provided in the supplementary materials.\n\nIn general, equipped with our Reflective Retriever, we can significantly enhance the prompt-image consistency and feasibility.\n\n[Ref A] https://huggingface.co/lambdalabs/sd-image-variations-diffusers.\n\n[Ref B] https://huggingface.co/stabilityai/stable-diffusion-2-1-unclip.\n\n[Ref C] Zero-1-to-3: Zero-shot one image to 3d object.\n\n[Ref D] Adding Conditional Control to Text-to-Image Diffusion Models.\n\n[Ref E] Sdedit: Guided image synthesis and editing with stochastic differential equations.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical proof in this paper", "experimental_designs_or_analyses": "Yes, the experiment design is reasonable and relatively sufficient.", "relation_to_broader_scientific_literature": "es, related to real-world applications for enhancing generative models' ability to generate new concepts.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea for using contrastive learning to retrieve images as supplementary knowledge for pre-trained generative models is novel.\n2. The writing is easy to follow, and the figures help readers to understand the paper.\n3. Experimental results show that the proposed method significantly improves the generation ability on various datasets with fine-grained classes.\n\nWeaknesses:\n1. Although the idea is novel, I am wondering why not just search relative images from the internet rather than retrieve them from the image pool. It should also work well.\n2. The number of classes and dataset size is relatively small in the experiments part. Larger datasets with more complicated and challenging classes need to be evaluated.\n3. The image as a reference part is not clear to me. Existing generative models (including SD based and AR based) only support text condition as input, how would you use image as additional reference? By cross-attention or initialize the latent noise with the reference images? This is a very important part, and you need to explain it clearly in the main paper for readers to understand.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKIcPn3j2q", "reviewer": "Reviewer_HZx9", "summary": "This paper introduces RealRAG, a retrieval-augmented generation (RAG) framework that enhances text-to-image models by retrieving real-world images to improve realism, accuracy, and faithfulness to fine-grained and unseen objects. Unlike conventional text-to-image models that suffer from hallucinations due to their fixed knowledge within model parameters, RealRAG retrieves real-object images and incorporates them into the generation process. The core innovation is the Reflective Retriever, trained via self-reflective contrastive learning, which aims to integrate missing memory rather than just the most similar images. The framework is designed to be modular and compatible with various generative models, including diffusion models (U-Net-based, DiT-based) and autoregressive models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank *Reviewer HZx9*  for the constructive comments and insightful suggestions.**\n\n# Evaluation on More General Datasets\n\nThanks for the reviewer's suggestion. We add the experiments with the ImageNet dataset, please check the results in the response for ***Reviewer 5Kbn***. The results demonstrate that our RealRAG showcases strong performance on the general datasets.\n\n# Compared with Other RAG-based Methods\n\nThe reference mentioned by the reviewer in the **Essential References Not Discussed** section, referred to here as RDMs, and our RealRAG both utilize retrieval-augmented techniques in text-to-image generation. However, the research problems and objectives we aim to solve are entirely different.\n\n* **Research Purpose of RDMs**: They use retrieval-augmented techniques to ***train or fine-tune*** a diffusion model and achieve ***out-of-distribution (OOD) image generation by switching databases***.\n\n* **Research Purpose of RealRAG**: RealRAG uses retrieval-augmented techniques to ***mitigate hallucinations*** when generating fine-grained objects and to enhance the ability to generate unseen novel objects, ***without requiring the training of generative models.***\n\n**Out-of-Distribution Objects ≠ Unseen Novel Objects**:\n  * **Out-of-distribution objects**: In the RDMs work mentioned by the reviewer, out-of-distribution is defined as data from outside the domain of the training dataset, such as \"**Angry Bird**\"(Knn-diffusion[1]) These objects existed during the generative model's training but were not part of the training set. As a result, the foundational generative model cannot generate them. RDMs use retrieval-augmented techniques, employing the CLIP model's similarity calculation ability to provide relevant references to the generative model, thereby solving the OOD problem.\n  * **Unseen novel objects**: These refer to objects that appear after the generative models and retrieval models are trained. The generative model cannot generate these objects, ***and the retrieval model can't easily retrieve relevant references via only similarity-based retrieval***. This is a much more challenging issue.\n\nFor **OOD object generation**, existing state-of-the-art text-to-image generators can generate images from almost all domains, as shown in the [link](https://realrag.github.io/AngryBird/), and the original FLUX model can generate \"**Angry Bird**\" very well, outperforming the retrieval-augmented KNN-diffusion[1]. However, for **unseen novel objects**, such as the \"**cybertruck**\" (shown in **Fig. 5** of the main paper), FLUX still cannot generate an accurate cybertruck. Therefore, the focus of this paper is on ***solving the problem of generating unseen novel objects.***\n\nFinally, although RDM is not part of the same type of research as our RealRAG, we still **provide a comparison of their performance in the following table.**\n\n| Models          | CLIP-I | CLIP-T |  FID  |\n| --------------- | :----: | :----: | :---: |\n| RDM[2]          | 59.82  | 12.37  | 69.20 |\n| CLIP-similarity | 61.72  | 14.52  | 54.04 |\n| RealRAG         | **62.81**  | **14.46**  | **52.28** |\n\nOur RealRAG shows significant performance gain, and we also show more visual comparison between the baseline and our RealRAG here [link](https://realrag.github.io/RealRAG_Rebuttal/).\n\n# Computational Overhead Analysis\n\nWe present the retrieval time cost and the performance gain in the table below. The table shows that the average percentage increase in reasoning time is much lower than the percentage increase in performance. Our approach achieves significant performance gains over a limited increase in inference time.\n\n| Model | Original Time Cost | RAG Time Cost | Delay (%) | AVG FID | AVG RAG FID | Gain (%) | Gain / Delay |\n| :---- | :----------------: | :-----------: | :-------: | :-----: | :---------: | :------: | :----------: |\n| Emu   |        8.96        |     9.32      |   **3.57**    |  73.27  |    59.81    |  **18.37**   |     ***+5.15***     |\n| SDXL  |        5.94        |     6.30      |   **6.06**    |  55.48  |    51.22    |   **7.68**   |     ***+1.27***     |\n| FLUX  |       13.42        |     13.78     |   **2.68**    |  53.37  |    49.28    |   **7.66**   |     ***+2.86***     |\n\n# Failure Cases\n\nThis work, like original FLUX and Stable Diffusion, has not been designed to target multi-object generation, therefore, RealRAG has some limitations in multi-object generation. Here we show some failure cases of multi-object generation [link](https://realrag.github.io/FailureCase/). ***It is our future work to implement multi-object generation***.", "claims_and_evidence": "The claim of \"the first real-object-based retrieval-augmented generation framework\" is arguable. There are some previous works in real image-based RAG.", "methods_and_evaluation_criteria": "The experiments are limited to datasets that are mainly focused on one category, e.g., Stanford Cars, Stanford Dogs, and Oxford Flowers. The ability of the proposed model on general cases needs to be further studied.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Some previous work used image retrieval to improve image generation. For example, [1], [2], [3], and some more recent ones.\n\n[1] knn-diffusion: Image generation via large-scale retrieval. ICLR 2022.\n\n[2] Retrieval-augmented diffusion models. NeurIPS 2022.\n\n[3] Retrieval-augmented text-to-image generator. ICLR 2022.", "essential_references_not_discussed": "As mentioned above, [1] and [3] could be mentioned and discussed. Though [2] is mentioned, \"the text database is not direct\nand controllable for realistic image generation\" seems not proper for describing [2]. A better description is desired.", "other_strengths_and_weaknesses": "1. The paper is well-written and easy to follow.\n\n2. The qualitative results clearly demonstrate the advantages of the proposed method.\n\n3. The proposed Self-reflective Contrastive Learning is interesting and effective.", "other_comments_or_suggestions": "What does the \"!\" mean in \"SD V2.1\"? (Table 1 and 2).", "questions_for_authors": "1. Could the proposed method be evaluated on more general datasets, e.g., Imagenet?\n\n2. Could the proposed method be compared with other RAG-based methods?\n\n3. Could a detailed computational overhead analysis be added?\n\n4. Could more failure case analysis be discussed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: RealRAG introduces a novel framework for t2i generation to improve the fixed-knoledge generation models. It introduces self-reflective contrastive learning to train a reflective retriever. This retriever then identifies and retrieves “reflective negatives” from an external database to compensate for missing visual details not included in the generated image. Experiments show that RealRAG significantly improves synthesized images' realism and fine-grained accuracy across various architectures.

The paper received 3 weak accepts, with all reviewers recognizing its novelty and technical contributions and expressing satisfaction with the rebuttal. Based on these positive assessments, the area chairs recommended acceptance.
Decision: Accept (poster)