AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
Authors: Jingyuan Qi
Zhiyang Xu
Qifan Wang
Lifu Huang
Venue: NeurIPS 2025 poster ()
Keywords: retrieval augmented image generation
autoregressive image generation
discrete image tokens
Primary area: deep_learning
TL;DR: propose a new retrieval augmentation method for image generation

Abstract: We introduce  Autoregressive Retrieval Augmentation (AR-RAG), a novel paradigm that enhances image generation by autoregressively incorporating k-nearest neighbor retrievals at the patch level.
Unlike prior methods that perform a single, static retrieval before generation and condition the entire generation on fixed reference images,  AR-RAG performs context-aware retrievals at each generation step, using prior-generated patches as queries to retrieve and incorporate the most relevant patch-level visual references, 
enabling the model to respond to evolving generation needs while avoiding limitations (e.g., over-copying, stylistic bias, etc.) prevalent in existing methods. To realize AR-RAG, we propose two parallel frameworks: (1) Distribution-Augmentation in Decoding (DAiD), a training-free plug-and-use decoding strategy that directly merges the distribution of model-predicted patches with the distribution of retrieved patches, and (2) Feature-Augmentation in Decoding (FAiD), a parameter-efficient fine-tuning method that progressively smooths the features of retrieved patches via multi-scale convolution operations and leverages them to augment the image generation process. We validate the effectiveness of AR-RAG on widely adopted benchmarks, including Midjourney-30K, GenEval and DPG-Bench, demonstrating significant performance gains over state-of-the-art image generation models.

AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
Jingyuan Qi'
Zhiyang Xu"
Qifan Wang'
Lifu Huang
Virginia Tech
Meta
UC Davis
jingyq1evt. edu
Jfumuang@ucdavis..dd
(a) Vanilla Image Generation
(b) Image-Based Retrieval Augmentation
Prompt
Prompt
Generated Image
Retrieved Images
Generated Image
Prompt
(c) Patch-based Autoregressive Retrieval Augmentation (Ours)
Generation
Next Image Patch
Retrieval
Augmentation
Query
Key
Value
Query
Key
Value
Query
Key
Value
Generated Image
Figure 1: Comparison between Autoregressive Retrieval Augmentation (AR-RAG) for image generation in (c) and existing image generation paradigms in (a) (b). In AR-RAG, image patches in red boxes denote retrieval queries and keys, image patches blue boxes are retrieved values, and gray boxes with the question mark are next image patches be predicted. (Caption: white playing basketball on the court.)
Abstract
We introduce Autoregressive Retrieval Augmentation (AR-RAG), novel paradigm that enhances image generation by autoregressively incorporating k- nearest neighbor retrievals at the patch level. Unlike prior methods that perform a single, static retrieval before generation and condition the entire generation on fixed reference images, AR-RAC performs context-aware retrievals each gen- eration step, using prior-generated patches as queries to retrieve and incorporate the most relevant patch-level visual references, enabling the model to respond evolving generation needs while avoiding limitations (e.g.. over-copying, stylis- tic bias, etc.) prevalent in existing methods. To realize AR-RAG, we propose in Decoding (DAiD), training-fre -enning-fre alu--and-se decoding strategy that directly merges the distribu- tion of model-predicte patches with the distribution of retrieved patches, and (2) Peature-Augmeenattion in Decoding (FAiD), parameter- efffiiier-efficient fine-tuning method that progressively smooths the features of retrieved patches via multi-scale convolution operations and leverages them to augment the image generation pro- cess. We validate the effectiveness of A RR-RAG on widely adopted benchmarks, including Midjourney -3K, GenEval, RareBench,
two parallel frameworks: (1)
and DPG-Benedl
Jingyuan Qi and Zhiyang contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

demonstrating significant performance gains over state-of-the- art image generation models
1 Introduction
Recent advancements in image generation have demonstrated remarkable capabilities in producing
28 6 However, despite these improvements, the generated images often exhibit local distortions and inconsistencies, particularly in visual objects that possess complex structures [I], frequently interact with other objects and the surrounding scene [23] 27]. or are underrepresented in the training data [8. A promising approach to mitigating these challenges is aurieval-augmentel generation (RAG), which enhances the generation process by incorporating real-world images as additional references (8 3. While RAG has been extensively explored in the language domain [24 13, its application to image and multimodal generation remains largely underdeveloped. A few existing studies 3 g
photorealistic images based on user prompts [3] 30
39
10
45 47
48
50 [1
bridge this gap by performing a single-step retrieval based on the input prompt prior to generation, conditioning the entire image generation process on fixed visual cues (Figure (b)). However, as demonstrated in our pilot study (Section (Sectionn.22).. such static, coarse-grained retrieval approaches B [1] frequently introduce irrelevant or weakly aligned visual contents that persist throughout generation. Since the retrieved images are selected once, before decoding begins, and remain unchanged, these methods cannot respond to the evolving generation needs, resulting in over-copying of irrelevant details, stylistic bias, and the hallucination of unrelated visual elements. For example, as shown in Figure Tb), a basketball player present in the retrieved references, despite being irrelevant to the input prompt, unintentionally appears in the generated image.
In this paper, we propose Autoregressive Retrieval Augmentation (AR-RAG), a novel retrieval- augmented paradigm for image generation that dynamically and autoregressively incorporates patch- level thearest-neighiee (k-NN) retrievals throughout the generation process (Figure (c)). In contrast to prior methods that rely on static, coarse- -rrsined retrievals of entire reference images, typically using captions retrieval queries and keys, ARRAAG performs fine- mreegraided, step-wise retrieval at the image patch level. Specifically, as generation unfolds, AR-RAC leverages the already- generated surrounding patches localized queries to retrieve contextually similar patches from a pre-constructed patch-level database. This database is built by encoding real-world images into latent patch features, where each entry contains a patch embedding as value and the embeddings of its h-hop spatial neighbors as a key. During the generation of the next target patch (gray boxes in Figure [c). AR-RAG retrieves the top-K most relevant patches (blue boxes) by measuring similarity between the surrounding generated context patches (red boxes) and database keys (also red boxes). These retrieved patches are then integrated into the model to inform and enhance the prediction of the next patch, enabling the model to dynamically adjust to local generation needs. By conditioning on the evolving generation context as retrieval queries. AR-RAG ensures that retrieved visual references remain relevant throughout the generation process, encouraging local semantic coherence. Moreover, the patch-level retrieval allows for precise integration of visual elements without overcommitting to entire reference images, avoiding the limitations of over-copying or irrelevant conditioning observed in static retrieval.
To realize the AR-RAG framework, we introduce two parallel implementations: (1) Distribution- Augmentation in Decoding (DAiD), a training-free. plug-and-play decoding strategy that merges the model's 's predicted patch distribution with that of the retrieved patches. Specifically, the top-K retrieved patches are assigned probabilities inversely proportional to their normalized &g distances computed from the query and key patch embeddings. These probabilities are then lincarly combined with the model's native output distribution guide the next patch prediction, enabling retrieval-an -wwre generation without any additional training. (2) Feature-Angmenntttion in Decoding (FAiD), a parameter-efficient fine-tuning approach that integrates retrieved patches into the generation process through learned smoothing and blending mechanisms. Specifically, when generating the next image token, FAiD operates in two stages: (1) refining the retrieved patch features by adjusting them to better fit the local context of the already generated surrounding patches, based on parameterized convolutional operations of varying kernel sizes; and (2) blending the refined features of retrieved patches with the model's predicted feature representation for the next patch, based on compatibility
'Code and model checkpoints can be found at https:: wwwp:tthhhuccooLLLU--MMLL RAG

---

scores computed for each retrieved patch to quantify their alignment with the current generation context. To enable iterative refinement, we insert multiple FAiD modules selected transformer layers, where the output of each FAiD module, i.e., the context-aware retrieved features blended at that layer, forwarded as input to the next FAil) module in deeper layers. This progressive retrieval refinement mechanism allows the model to incrementally enhance its predictions as patch- level representations evolve through the network. We evaluate AR-RAG on five widely adopted benchmarks, including Midjourney-305 DPG-Bench [T8. Experimental results demonstrate that both DAiD and FAiD significantly improve the coherence and naturalness of generated images while introducing only marginal computational overhead.
P Geneval [14), RareBench [29]. T2I-Bench [19], and
The contributions of our work can be summarized as follows:
We propose AR-RAG, the first patch-level autoregressive retrieval augmentation framework which dynamically retrieves and integrates fine-grained visual content to enhance image generation, while avoiding limitations (e.g., over-copying, stylistic bias, etc.) prevalent in existing image-level retrieval augmentation methods.
We introduce Distribution-Augmnnaattiin Augmentation in Decoding (DAiD), training-free, plug-and-play decoding strategy that directly integrates the distribution of retrieved patches into that predicted by the image generation models, enabling easy integration into existing architectures.
We introduce Peature-Augmentationn in Decoding (FAiD), a parameter-efficient fine-tuning frame- work that progressively refines and blends retrieval signals via lightweight convolutional modules, enhancing spatial coherence and visual quality across layers.
Extensive experiments and analysis show that AR-RAG significantly improves performance of state-of-the-art image generation model across diverse metries. In particular, Janus-Pro with FAiD achieves 6.67 FID on Midjourney-38 and 0.78 overall score on GenEval, establishing a new state of the art among autoregressive image generation models of comparable scale.
2 Preliminary
Autoregressive Image Generation Models We implement both DAiD and FAiD based on Janus- Pro [9], autoregressive (AR) unified generation model, due its strong performance. Janus-Pro is initialized from a transformer-basee pre-trained large-language model [2], and employs quantized autoencoder 39 encode images into discrete image tokens. During multimodal pretraining, the model learns to predict a sequence of discrete image tokens [V1, UN!... conditioned on an input text prompt [t,, f2 ... The training objective formally defined as:
arg
(1 tMs U1 U1--)))))1)
(1)
where is the training corpus. This the same training objective used in our FAiD method in We argue that DAiD and FAiD can be extended to any image generation model that autoregressively predicts probability distributions of discrete image tokens such as LlamaGen (3
Section 3.3
Show-o 46 and VAR [40]
Quantized Autoencoder The quantized autoencoder used in Janus- Paous-Proodee consists of an encoder dene- decoder fice, and codebook Z. The encoder, a convolutional neural network, downsamples and compresses raw pixel inputs into compact patch representations. During the quantization process, each patch representation is mapped to an index in the codebook by identifying its nearest neighbor vector in the codebook. In the decoding stage, these patch indices are mapped back to their corresponding vector representations via the codebook, and the decoder, another convolutional neural network, reconstructs the image from these compact representations. In our implementation, we leverage this autoencoder to build the coupled database for Janus-pro which is detailed in Section 3.1

---

AR Model
Distrbution-Augmennt Decoding (DAiD)
Dook
Patch-based Retrieval
Generated Patches
Generated Next Image Patches
Doeege
h-hop surrounding patches
Darsiwal
Figure The decoding process in Distribution- Auguumnttnnnn in Decoding (DAiD).
3 AR-RAG: Patch-based Autoregressive Retrieval Augmentation
3.1 Patch-based Retrieval Database Construction
We build a patch-based retrieval database based on several large-scale, real-world image datasets, including CC12M [5] and JourneyDB [3 Specifically, for each image T, we encode it into N
patches using the quantized autoencoder (39], OFne from Janus-Pro: V Oene (I) RRVXV/NN where d is the hidden dimension, and Vi corresponds to the latent representation of the patch at position (). We utilize each patch vector Vay the value of database entry and the representation of its h-hop surrounding patches as the key. Here, the h--opp surrounding patch representation is formed by concatenating the vectors of adjacent patches centering around (i,j) top-to-bottom. left-to-right order. For example, for a patch at position (i,j), j), the 1-hop surrounding representation spans 8 surrounding patches [V (V(+-))--)
V (G+1))++) V(G-- V V(o(++) V ((+11)/111 V (++1))) V ((+1))+1)) where denotes the concatenation operation of image patch features. If a patch is located at the edge of the image and lacks certain surrounding patches, we
((-DD))
substitute each missing surrounding patch with zero vector
Distribution- -ugimtttionnn in Decoding (DAiD)
Given text prompt T, Janus-Pro autoregressively predicts sequence of image tokens [vi, 02 UN.UU where per-token probability is defined Equation [] As shown Figure 2 DAiD augments this process by incorporating probability distributions from retrieved image patches. Specifically, when Janus-Pro predicts the next image token V.j. we first utilize the codebook Z to convert Uy 's h- hop already generated surrounding patches into patch representations. If surrounding image tokens are available at a given position (c.g., when placeholder. Once compute the representation of 0: 's h-hop surrounding patches, as the retrieval query and retrieve the top-K most similar patch representations from the database constructed in Section 3.1 using distance. We denote the representations of the top-K retrieved patches as [V1, 22, ....... K and their corresponding l2 distances as [81, 82, ..., sk.. These retrieved representations are then mapped back to discrete token indices using the codebook: êx = Z(vx).
or j = 0), we use the zero vector as a
leverage it
To augment the generation process with the retrieved image tokens [01, ..,, úx], we create retrieval-based distribution Dretrieal E IRI over the entire codebook Z, where [Z] is the codebook size. Tokens not included in the top-K retrieved set are assigned probability of O. For tokens within the top-K, K, we compute their probabilities using softmax over their l2 distance to the query, scaled by retrieval temperature hyperparameter
D Dettrieval [x] = [ppp(i (p())) i = GG for some m {1,2, ..., K}
[0 [
(2)
otherwise
exp(-8k/T)
p(O))
Σ exp(-xxx77
This creates sparse distribution where only the top-K K retrieved tokens have non-zero probabilities. Finally, we merge this retrieval distribution with the model's predicted distribution Dmodd using

---

Generated Patches
Database
Feature Augmentation Decoding (FAID)
h
b
h
h
h
h2
b,
h
h
hp'
h
h"
h
Patch-based Image Retriever
Convolational Lager
Janus-Pro
Embeeddings
RMS Norm
RMS Norm
Self-Atth
SPB
+C*
Mi P Layer
RMS Norm
8; =.3
8, =0.2
FRD
Dacoder Layer
Figure 3: Overall architecture of Feature-Anngmentttion in Decoding (FAiD).
weighted average:
Dmerge = (1 A) mondel + Drettieval=
(4)
where X E [0, 1] is the retrieval weight hyperparameter controlling the influence of retrieved patches on the final distribution. The next token is then sampled from this merged distribution: U2j ~ Dmerge-
3.3 Feature-Augmennttionn in Decoding (FAiD)
While DAiD offers a training free approach to directly augment the probability distribution of predicted patches using retrieved ones, it suffers from noise propagation and limited flexibility in fully leveraging the fine-grained visual information the retrieved patches. We thus further propose FAiD, feature-based autoregressive augmentation strategy to enhance the image generation process. As illustrated in Figure 3 when predicting the next token Uú during image generation, we employ the same retrieval process described in Section 3.2 to obtain the top-K most relevant patches and their representations [V1, V2,... v from our database. To effectively incorporate them into the autoregressive generation process, FAiD consists of two steps: (1) refining retrieved patches to ensure coherence with the surrounding context of vty in the generated image, and (2) adaptively blending the representation of refined patches with the hidden state of the predicted next patch based learned compatibility scores. To enable progressive refinement of retrieved information as representations evolve through the network, insert FAID module for every L./ decoder layers of the generation model, where I. denotes the total number of decoder layers and is hyperparameter.
Multi-Scale Feature Smoothing The key of effective patch integration lies in ensuring spatial coherence between retrieved patches and the surrounding image context. To achieve this, propose f
multi-scale feature smoothing (Algorithm in Appendix A). where multi-scale convolutions are applied retrieved patches within the generation context, that the retrieved visual features smoothed to preserve structural and sty listicc consistency with the surrounding context of the predicted token. Specifically, each step when predicting the next image token tij. first construct 2D
spatial representation H' E TRVN /VNNx/NX of the current partially-generated image by arranging their hidden states [h/ b/g h/,-1- by] from the current decoder layer We use 0 vectors as placeholders for positions that have not yet been generated. Then, we transform the retrieved patch representations [V, ..., & *] into the generation model's hidden space by mapping each patch Ve to a discrete token index via the codebook Z and embedding it through the pretrained image embedding layer Embing:
h h hx] Embing (IZ(i1), .... Z(vx))
(5)
5

---

For each retrieved patch h3, we create copy of H' where position (i.j) (the location of Usy) is replaced with hg. We then apply convolution operations at multiple scales (2 x 2 through Q Q) to capture contextual patterns at different resolutions. To maintain computational efficiency, we only perform convolution operations when the kernel covers position (i, rather than processing the entire image. Each convolution kernel Conygxg produces refined representation lig for the retrieved patch at scale g. The final refined representation for each retrieved patch is computed as a weighted sum of these multi-scale features:
Q
h&g
softmax(11), hg
(6)
where 52 = [W2,,,,,, wQ] are learnable parameters that determine the importance of each scale.
Feature Augmentation After feature smoothing, some of the retrieved patch features may still not be able to fit into the surrounding neighbors and hence we need to lower their impact in the final repre- sentation. Thus, we compute a compatibility score for each of the refined patches. This is achieved by projecting each refined retrieved patch representation through a linear transformation parameterized by weight matrix W R lx yielding the score Sk h2 W" The final representation for the next image token Viy after layer is computed as:
(1-1) Myy +
AM
spha
1=1
Here, My is the residual, AM, the updated representation from the transformer layer 1, and K Ek= sghi is the contribution of the retrieved image patches.
4 Experiment Setup
Patch-based Retrieval Database To construct our patch-level retrieval database, randomly sample 5.7 million images from CC12M [5), 3.3 million from JourneyDB (38], and 4.6 million from DataComp [12). while ensuring that any samples included in the testing set excluded to prevent data leakage. Each image is encoded into a sequence of patch-level representations and image tokens using the same image tokenizer employed in the Janus-Pro model. For efficient similarity search, we implement our retriever using the FAISS library
22
Training Setup We adopt Janus-Pro-I1 9 and Show-o 446 as our backbone models and fine-tune them on a dataset of 50,000 image-caption pairs sampled from CC12M [5] and Midjourney-vō We empirically determine the optimal hyperparameters for DAiD and FAiD, and the complete hepraarrameter optimization experiment results be found in Appendix regarding the training dataset construction and implementation can be found in Appendix [B3]
C.2 Further details
Baselines To evaluate the effectiveness of our proposed methods, we adopt several state-of-the-art image generation approaches as baselines. including non-retrieval models such as LlamaGen [39]. LDM [4], Stable Diffusion (SDv1.5 and SDv3) [3] 10]. Pit-rtlalphaa (7], DALL-E 2 (32). Show- Janus-Pro 19, and FLUX. dev and image-based retrieval augmentation methods, including RDM 3 RA-CM3 48 and ImageRAG[ES (.5) Since pretrained models of RA-CM3 are not publicly available, we try our best to replicate their method based on Janus-Pro to ensure a fair comparison.
46
More details of training and implementation RA-CM3 can be found in Appendix [
Evaluation Benchmarks and Metrics To comprehensively evaluate our proposed methods,
employ five benchmarks: (1) GenEval [4], which assesses models' ability to generate images
with specific attributes and relationships described in text prompts; (2) DPG-Bench [18]. which
evaluates performance on detailed prompts with complex requirements; (3) RareBench 22) which is
designed to evaluate the ability of diffusion models to generate rare compositions of concepts; (4)
T2I-Bench [19), which focuses on fine-graince attribute binding across four key dimensions: color, shape, texture, and numeracy; and (5) Midjourney-333 [42). where we employ three complementary metrics: FID [7] for measuring statistical similarity between generated and real image distributions,
CMMD 21 for assessing alignment with human perception using CLIP embeddings, and FWD 41
1lava

---

Method
Params
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Overall f
Non Betrieval-Angment
PixArt-o
0.6B
0,98
0.50
0.44
0.80
0.08
0.07
0.48
LlamaGen
0.8B
0.71
0.34
0.21
0.58
0.07
0.04
0.32
SDv1.5
0.9B
0.97
0.38
0.35
0.76
0.06
0.43
SDv2.1
0.9B
0,98
0.51
0.44
0.85
0.17
0.50
Janus- Pro
1.0B
0.98
0.77
0.52
0.84
0.61
0.55
0.71
Show-o
1.3B
0.98
0.66
0.84
0.50
LDM
1.48
0.92
0.29
0.23
0.7
0.02
0,05
0.37
SD3 (d=24)
2.0B
0,98
0.74
0.63
0.67
0.34
0.36
0.62
2.6B
0.98
0.39
0.85
DALL-E2
6.5B
0.94
0.66
0.49
0.77
0.10
0.19
0.52
DALL-E
0.96
0.87
0.47
0.83
0.43
0.45
0.67
Transfusion
7.3B
0.63
Chameleon
34B
0.39
FLUX.I-dev
12B
0.67
-unmentel Model
1.48
0.28
0.71
0.36
ImageRAG
3.5B
0.93
0,06
0.03
0.37
0.01
0.03
0.24
Jamus-Pru
-RA-CM
L.BB
0.98
0.41
0.84
0.42
0.49
0.65 (-0.06)
DAiD (ours)
1.0B
0.98
0.82
0,54
0.87
0.63
0.49
0.72 (40.01)
FAiD (ours)
1.2B
1.00
0.50
0.86
0.73
0.78 (+0.07)
Table 1: Evaluation of text-to-image generation ability on GenEval benchmark. Note our methods
based on Janus-Pro highlighted in gray.
for evaluating spatial and frequency coherence through wavelet packet coefficients. For all three metrics, lower scores indicate higher quality generated images. Detailed descriptions of these benchmarks and metrics can be found in Appendix B.4
Results and Discussion
5.1 Text-to-Image Generation Results
Tables 1 2 and present performance comparisons across multiple benchmarks, where our AR- RAG methods consistently outperform existing approaches. Notably, previous perrival--uugenteee approaches such as RDM and ImageRAG perform worse than their non-retrieval counterparts (LDM and SDXL, respectively) on both GenEval and DPG-Bench. We provide detailed analysis for existing image-level retrieval methods and highlight the unique advantages of our AR-RAG AG frameworks the following discussion and Section 5.2 Appendix C.. provides benchmark analysis demonstrate the effectiveness of patch-level retrieval in our AR-RAG methods.
On GenEval, our methods show significant improvements in categories such "Two Obj." and "Position," " which demand accurate multi-object generation and spatial arrangement. These gains are largely due to the local and dynamic nature of our autoregressive patch-level retrieval. Consider the prompt "a green couch and an orange umbrella". a combination that rarely co-occurs in real- world images. Static full-image retrieval methods may retrieve references containing only one of the objects.
Method
Params
Global
Entity
Attribute
Relation
Other
Overall
Non
Gerrieval-Angmenner Model
PixArt-a
0.6B
74.97
97.32
78.60
82.57
76.96
71.11
SDv1.5
0.9B
74.63
74.23
75.39
73.49
67.81
63.18
Janus-Pro
1.0B
81.76
84.53
84.34
92.22
75.20
77.26
Lumina-Next
2.0B
82.82
88.65
86.44
80.53
81.82
74.63
SDXL
3.5B
83.27
82.43
80.91
86.76
80.41
74.65
Retrieval-Augee Aurreeeal-uggeentedd Model
RDM
1.4B
40.46
60.20
69.16
24.68
26.51
ImageRAG
3.5B
61.35
32.77
$3.87
60.38
18.42
19.82
Jamus-Pre Pro
RA-CM3
1B
81.76
81.03
83,32
90.60
70.80
73.76 (-3,50)
+DAiD (ours)
84.76
91.49
76.40
77.88 (40.62)
+FAiD (ours)
1.2B
82.67
85.80
85.38
92.30
76.80
79.36 (+2.10)
Table Evaluation of text-to-inne eoxt-to-image generation ability on DPG-Bench. Note our methods are based on Janus-Prr Pro highlighted in gray.

---

Taking these references as a global visual prior throughout the generation can lead the model to overfit irrelevant layouts or dominant visual structures in the retrieved examples. On DPG-Bench. which features dense and highly detailed prompts, the performance gap between our method and prior surrival--uugenter approaches becomes even more substantial. Similar as GenEval, existing image-level retrieval augmentation methods struggle to retrieve meaningful references when the number of distinct entities and attributes in prompt increases. In contrast, our autoregressive augmentation framework overcomes this limitation by dynamically retrieving patch-level visual features based on the evolving image context rather than the original prompt, enabling more targeted and effective augmentation.
On Midjourney-30Ks our proposed methods con- sistently outperform both Vanus-Pro and Show-o baselines across all three evaluation metrics. No-
Model
Params
CMMD
FID.
FWD.
RDM
1.4B
0.71
19.17
34.95
ImageRAG
3.5B
0.32
19.39
62.65
tably, despite operating locally at the patch level, our approach leads to a significant reduction in FID
Show-o
1.3B
0.09
11.47
2.57
DAiD (ours)
1,3B
0.08
9.28
2.49
scores, indicating improved global visual quality and closer alignment with the distribution of real
FAiD (ours)
1,5
0.06
7,93
1,73
Janus-Pro
1.0B
0.12
14.33
28.41
images. This suggests that context-aware, auto- regressive retrieval and refinement can propagate to enhance holistic image fidelity. Furthermore, the improvements in CMMD and FWD metrics con- firm our method's effectiveness in reducing visual distortions and enhancing coherence. These results also demonstrate that A AR-RAG delivers robust and architecture-agnoste improvements, validating
-RA-CM3 CM3
1.0B
0.13
12.40
20.57
DAID (ours)
1.0B
9.15
28.00
+ FAiD (ours)
1.2B
0.07
6.67
9.40
Table 3: Evaluation of text-to-image generation ability on the Midjourney-300 benchmark.
its broad applicability across different image generation backbones.
Tables AbblesAande and present our results on RareBench [29] and T2I-Bench [T9].
Method
Params
Color
Shape
Texture
Numeracy
On RareBench, our AR-RAG meth-
RDM
1.4B
0.666
0.555
0,538
0.382
ImageRAG
3.5B
0.670
0.543
0.528
0.344
ods demonstrate substantial improve-
ments over baselines, particularly in multi-object composition categories such "Concat," "Relation," and "Complex." where FAiD achieves 67%, 54%, and 65.5% respectively, significantly outperforming previous aetrieall-uugment methods. These gains highlight our method's effectiveness in generating rare or novel combinations by dynamically retrieving relevant patch-level features rather than relying solely on model memorization. On T2I-Bench, our methods outperform the Janus baseline across most categories. with particularly notable improvements in "Color" (0.703 vs. 0.652) and "Numeracy" (0.537 vs. 0.251). While the diffusion- based RDM achieves competitive performance on certain sub-categories such as "Texture," this reflects the inherent advantage of continuous diffusion models over discrete token-based autoregressive approaches, as the quantization process inevitably incurs information loss when converting between discrete image tokens and continuous image features. Nonetheless, our autoregressive approach demonstrates strong capability in attribute binding and counting tasks, validating the effectiveness of patch-level retrieval augmentation within the discrete token generation paradigm.
Janus-Pro
1.0B
0.652
0.455
0.449
0,251
DAID (ours)
1.0B
0.630
0.459
0.464
0.245
FAiD (ours)
1.2B
0.703
0.494
0.483
0.537
Table 4: Evaluation of text-to-image compositional reasoning on T2I-Bench.
Complex (single obj)
Concat (multi obj)
Relation (multi obj)
Complex (multi obj)
Method
Params
Property
Shape
Texture
Action
RDM
1.48
44.0
64.5
56.2
SDXL
3.5B
60.0
56.9
71.3
47.5
58.1
39.4
35.0
47.5
ImageRAG
3.5B
25.0
86,5
85,5
49.0
77.0
$7.5
36.0
55.5
Janus-Pro
1.0B
53.5
59.0
76.5
53.5
78.0
42.5
48.0
50.5
DAiD (ours)
1.0B
61.5
54.0
78.5
46.5
50.5
54.5
FAID (ours)
1.2
74.0
62.0
78.5
58.5
79,0
67.0
54.0
65.5
Table 5: Evaluation of text-to-image compositional reasoning on RareBench.
8

---

5.2 Qualitative Analysis
Figure [ illustrates these quan- titative improvements with rep- resentative examples from DPG- Bench (left three columns) and GenEval (right two columns). These examples demonstrate how autoregressive retrieval aug-
solitary careel slowly aubbles beside plush, ruund red couch.
The morning light filter cast soct high-top sneakers.
A realistic poctrait ofuylor swift with red scart.
photo bench.
sheepp
Janus-Pro
mentation improves the vanilla image generation models. The vanilla model struggles with
DAID
object interactions (e.g., col-
3, where shoes merge with coffee machine in the background), complex structures (e.g., columns 2 and 5, where camels and sheep have anatom- ically incorrect numbers of or- gans), and implausible configu- rations (e.g., column 4, where a chair exhibits an impossible design). Both DAiD and FAiD substantially reduce such local distortions, with FAiD yielding the highest visual quality. These results confirm that autoregressive retrieval effectively maintains object consistency and structural integrity throughout the generation process, particularly for complex objects and multi-object scenes.
FAID
Figure 4: Qualitative results of DAiD, FAiD and baselines.
(4) A photo an apple.
(b) photo of white dog and a blue potted plant.
Retrieved Image
ImageRAG
AR-RAG
Retrieved Image
ImageRAG
AR-RAG
photo green yellow
(d) photo of green orange
Retrieved Image
ImageRWG
AR-RAG
Retrieved Image
ImageRWGG
AR-RAG
Figure 5: Images generated by ImageRAG (35 and our AR-RAG. ImageRAG excessively copies retrieved images and does not follow user prompis.
Figure [guure3|preee presents a comparative analysis of conventional image-level and our autoregressive patch- level retrieval augmentation methods. By comprehensively examining images produced by Im- ageRAG alongside their corresponding retrieved reference images, we identify two critical chal- lenges inherent in image-level retrieval augmentation approaches. First, these methods tend to overcopy irrelevant visual elements from retrieved reference images into the generation outputs. As illustrated in Figure 5 (a), when generating image of apple, image-level retrieval ap- proaches retrieve a reference image showing an apple on tree branch and subsequently incorporate both the apple and the surrounding branches, despite the prompt making no mention of them. Similarly, for the prompt "a green cup and yellow bowl" Figure (b), the image-level retrieval augmentation ap- proach retrieves a green Starbucks cup and reproduces the
Single GPU (L40)
Model
Total (s)
Average (s)
pattern on the cup in the generated image, despite this el- ement not being part of the original instruction. This over- copying behavior directly compromises the instruction- following capability of generative models. Figure (c) demonstrates that when prompted to generate "A photo of white dog and blue potted plant," image-level retrieval
ImageRAG
879.64
8.80
Janus-Pro
457.74
4.58
DAID
459.34
4.59 (+0.22%)
FAiD
623.01
6.23 (+36.039) 03%
Table Inference time for generating 100 images on single L40 card.

---

methods produce an image containing only the white dog, omitting the blue potted plant entirely. Similarly, for "a photo of a green couch and an orange umbrella" in Figure the generated image fails to include the umbrella. This degradation in instruction following occurs because image-level retrieval biases the generation process toward the compositional structure of retrieved reference images, which may not align with the multi-object relationships specified in the prompt. In contrast, by autoregressively retrieving and integrating visual information at the fine-grained patch level rather than the image level, AR-RA RAG enables selective incorporation of relevant visual elements while maintaining independence from irrelevant contextual features present in the reference images.
5.3 Inference Time Cost
T the inference time comparisons across different models when generating 100 images using both a single L40 GPU. The DAiD method introduces only minimal increase in inference time compared to the base Janus-Pro-11 model, with average overhead of just 0.22%, demonstrating that DAiD maintains high computational efficiency. FAiD shows a more noticeable overhead of 36. 36.03% on single GPU due its autoregressive retrieval and feature blending operations. However, this increase remains reasonable given the substantial performance gains in generation quality, Overall, both DAiD and FAiD do not significantly compromise the inference efficiency of Janus-Pro, making them practical for real-world applications.
6 Related Work
Retrieval-augmentee generation (RAG) has emerged as powerful paradigm that enhances generative models by incorporating external knowledge during decoding [24
13
16
49 48, [5]
25
26 44).
Originally developed for natural language processing, RAG enables models to retrieve relevant documents to supplement parametric knowledge during response generation 44, and has been widely
adopted in many downstream tasks, such as knowledge-intensive tasks [24]. document fusion (20).
model pretraining [16), dialogue generation 37 and so on.
Beyond the text domain, prior research has explored enhancing image generation by incorporating external visual references. Early approaches 8 3 condition the diffusion process on retrieved images, typically encoded via CL.P or VAE encoders, to guide generation toward higher visual fidelity. KNN-Diffusion (36] extends this idea by leveraging k-nearest neighbor images improve zero-shot generalization to novel domains. Building on this retrieval-augmentt
framework, more recent methods [51 35 introduce adaptive retrieval pipelines that iteratively refine retrieved images based on feedback from multimodal large language models (MLLMs) analyzing the generated outputs. These methods enable context-aware and prompt-sensitive guidance during generation. Another line of work [48] encodes multimodal retrievals into discrete visual and text tokens, and uses them directly as contextual input to augment the generation process of multimodal large language model. All of these works differe from our method by that our method works on patch-level, enabling more fine grain retrievals and can dynamically adjust retrievals based on evolving generation states.
Conclusion
In this work, we propose Autoregressive Retrieval Augmentation (AR-RAG), novel retrieval paradigm that enhances image synthesis by leveraging k-nearest neighbor retrievals at the patch levef. Unlike traditional image-level retrieval approaches, AR-RAG enables fine-grained visual element integration while maintaining compositional flexibility. We introduce two parallel frameworks: (I) Distribution-Augmnnaaii in Decoding (DAiD), a training-free ffre approach that integrates retrieved patch distributions directly into generation, and (2) reature-Augmentatiion in Decoding (FAiD), which employs parameter- efficieett fine-tuning with multi-scale feature smoothing and compatibility- based feature augmentation. Extensive experiments across GenEval, DPG-Bench, and Midjourney330k demonstrate that AR-RAG significantly outperforms both conventional and strieval-augmentt baselines, particularly in handling complex prompts with multiple objects and specific spatial rela- tionships. Our methods substantially reduce local distortions in generated images, improving object consistency and structural integrity.
Acknowledgment
This research supported by the award No. #2238940 from the Faculty Early Career Development Program (CAREER) of the National Science Foundation (NSF). The views and conclusions contained
10

---

herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of the U.S. Govemment. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
[1] Trevor Ashby, Adithya Kulkarni, Jingyuan Qi, Minqian Liu, Eunah Cho, Vaibhav Kumar, and Lifu Huang. Towards effective long conversation generation with dynamic topic tracking and recommendation. In Proceedings of the 17th International Natural Language Generation Conference, pages 540-556, 2024.
[2] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint w...iv22411.029..5 2024.
[3] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi- parametric neural image synthesis. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[4] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jcan-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206-22444 2240. PMLR, 17-23 Jul 2022.
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. CoRR, abs/2102.08981, 2021.
[6] Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal
training and dataset, 2025.
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. CoRR, abs/2310.00426, 2023.
[8] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W. Cohen. Re-imagen: Retrieval- augmented text-to-image generator. In The Eleventh International Conference Learning Representations, 2023.
[9] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. CoRR, abs/2501. 17811, 2025.
[10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21 27, 2024. OpenReview.net, 2024.
[11] Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. CoRR, 1bs/2208.13753, 2022.
11

---

[12] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymiss, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh. Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Veneu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[13] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai. Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. language models: A survey. CoRR, abs/2312.10999 2023.
generation for large
[14] Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. CoRR, abs/2310.115111 2023.
[15] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for vision-and-language. arXiv preprini 12Xiv:2/22088644 2021.
[16] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages
PMLR, 2020.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6629 6640, Red Hook, NY, USA, 2017. Curran Associates Inc.
[18] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment, 2024.
[19] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-comphench: A compre- hensive benchmark for open-world compositional text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[20] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 874-880, Online, April 2021. Association for Computational Linguistics.
[21] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In CVPR, pages 9307-9315, 2024.
[22] Jeff Johnson. Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
[23] Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: native skeleton -gudee diffusion model for human image generation. CoRR, abs/2304.04269, 2023.
[24] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenttdd generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
12

---

[25] Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 01-22800-22444 2023.
[26] Weizhe Lin, Jingbiao Mci, Jinghong Chen, and Bill Byme. Prefimr: Scaling up fine-grained late- inte-interaction multi-modal retrievers. arXiv preprint arXi::2402.08327, 2024.
[27] Wenquan Lu, Yufei Xu, Jing Zhang, Chaoyue Wang, and Dacheng Tao. Handrefiner: Refining malformed hands generated images by diffusion-baase conditional inpainting. In Jianfei Cai, Mohan S. Kankanhalli, Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu, editors, Proceedings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne, VIC, Australia, November 2024, pages 7085-7093. ACM, 2024.
28 October 2024
[28] Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Hou, and Saining Xie. Transfer between modalities with metaqueries. CoRR, abs/2504.06256, 2025.
[29] Dongmin Park, Sebin Kim, Tachong Moon, Minkyu Kim, Kangwook Lee, and Jaewoong Cho. Rare-to-frequent Unlocking compositional generation power of diffusion models on rare concepts with LLM guidance. CoRR, abs/2410.22376, 2024.
[30] Dustin Podell. Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL.: improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Openkeview.net, 2024.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pages 8748-87666 8763, 2021.
[32] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXivv2204.06125 2022.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CoRR, abs/21 12.12122.. 2021.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10699 June 2022.
[35] Rotem Shalev-Arkuhhnn. Rinon Gal. Amit H. Bermano, and Ohad Fried. Imagerag: Dynamic image retrieval for reference-guided image generation, 2025.
[36] Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. KNN-diffusion: Image generation via large-scale retrieval. In The Eleventh International Conference Learning Representations, 2023.
[37] Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval aug- mentation reduces hallucination in conversation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[38] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
13

---

[39] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. CoRR. abs/2406.06525, 2024.
[40] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 84839-84865. Curran Associates, Inc., 2024.
[41] Lokesh Veeramacheneni, Moritz Wolter, Hilde Kuehne, and Juergen Gall. Fréchet wavelet distance: A Nomain-agnostio metric for image generation. In The Thirteenth International Conference Learning Representations, 2025.
[42] Vivym. Midjourney prompts dataset. https://huggingter httto://hhggigggaacc..o///tttttt midjourney-prompe 2023. Accessed: 20-4-04-11.
[43] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang. Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang. Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need. CoRR, abs/2409. 18869, 2024.
[44] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang. Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint urXiv:2311.17133 2023.
[45] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: efficient high-resolution image synthesis with linear diffusion transformers. CoRR, abs/2410. 10629, 2024.
[46] Jinheng Xie, Weijia Mao. Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. CoRR, abs/2408. 12528, 2024.
[47] Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Modality-specialized synergizers for interleaved vision-language generalists. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
[48] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovee, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. Retrieval- augmeeted multimodal language modeling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 39755-39769. PMLR, 2023.
[49] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmenned language models robust to irrelevant context. CoRR, abs/2310.015588 2023.
[50] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. CoRR, Abs/2309.02591, 2023.
[51] Huaying Yuan, Ziliang Zhao, Shuting Wang, Shitao Xiao, Minheng Ni, Zheng Liu, and Zhicheng Dou. FineRAG: Fine-grained eerreval-auummetee text-to-image generation. In Owen Ram- bow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 3 31st International Conference on Computational Lin- guistics, pages 11196-11205, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.
14

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We claim in the Abstract and Introduction that introduce Autoregressive Re- trieval Augmentation, a novel paradigm that enhances image generation by autoregressively incorporating k-nearest neighbor retrievals at the patch level.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Not theory paper
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification:
Will release after acceptance
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwiiicgguie//eeeeeeeeeic CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to the cost of each experiment, we do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics sattps://nnt :ttta:::///euriss.oo httpp//s...ccbboceccccccccteeeea
Answer:
[Yes]
Justification:
We respect NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the papers for models, code and datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppawpptseeee www.reeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
19

---

Answer:
[No]
Justification:
Code and model will be released after acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowd sourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
20

---

Justification:
LLM is not the core method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httees////earrppcccc////f/rrrerp for what should or should not be described.
21

---

A Multi-Scale Feature Smoothing Algorithm
Algorithm Ajillustrates the multi-scale fea- ture smoothing, which is the core computa- tional procedure for refining retrieved patch representations within their generation text. This algorithm ensures that retrieved visual elements are spatially and stylisti- cally coherent with the surrounding image content through systematic multi-scale con-, volution operations.
Algorithm 1: Multi-Scale Feature Smoothing
Input: Image Representations H' E RVNxVVXD Retrieved Patch Representations
[hi, h2...., hg]. Next Patch Index (i,5)
Output: Updated hidden states [b,, ha......................... hx foreach h, [hi..... .., .ix]
2
for g q= 2 to Q do
Initialize tensor: M E R
The algorithm processes each retrieved patch representation h, independently, ap- plying convolution operations at multiple* scales ranging from 2 x 2 Q kernlss." For each scale q; the algorithm initializes temporary feature tensor M E Rx
Initialize tensor: h,
R.;
for m q down to 1 do
for n = q down to do
H H
m i m. n:
j , q n|:
M Conv axnv axa (H
and an accumulation vector h2 RD. The nested loops over indices m and system-9 atically extract local patch features from different spatial windows around the target!o position (i. Each extraction operation Hie Hi m i 9 m.j n
h, += Conv yxy (M):
h h
Q-T
j + q n| captures a local neighborhood
of size q q centered varying offsets from the target position.
The extracted local features undergo two-stage convolution processing. The first convolution operation Conv qxg transforms the local patch features into an intermediate representation stored in Mm effectively capturing contextual relationships within each local window. The second convolution operation Conv 9xg processes the accumulated intermediate features to produce scale-specific refined representations. This two-stage design enables the algorithm first capture local contextual patterns and then integrate them into a coherent scale-specific feature representation.
2
After processing all scales for given retrieved patch, the algorithm computes the final refined representation by averaging the scale-specific features. The normalization factor (Q - 1) accounts for the number of scales processed, ensuring consistent feature magnitudes across different retrieved patches. This averaging operation effectively combines multi-scale contextual information into single refined representation that preserves both fine-grained details from smaller kernel sizes and broader contextual patterns from larger kernel sizes. The resulting refined patch representations maintain spatial coherence with the surrounding generation context while preserving the essential visual characteristics of the retrieved content.
B Experiment Setup
B.1 RA-CM3 Implementation Details
Since the pretrained RA-CM3 model is not publicly available, we implement our own version following the methodology described in the original paper to serve as representative baseline for image-level attrivall-uggeent generation. Our implementation uses Janus-Pro as the backbone model to ensure fair comparison with our proposed methods, both approaches operate on the same foundation architecture.
We construct an image- retrieval database using the same CC12M S and JourneyDB 38 datasets employed for our patch-level retrieval database maintain consistency in the underlying data distribution. All images in the database are encoded into 512 dimensional vector representations using a pretrained CLIP 31 model. For each training instance our 50,000 sample training set, we retrieve the most relevant reference image by encoding the corresponding text prompt with the same CLIP model, extracting the [CLS] token as the text representation, and computing cosine similarity scores between the text representation and all image representations in the database. The image
22

---

with the highest similarity score is selected as the retrieved reference. Each retrieved image is then processed through the quantized autoencoder from Janus-Pre obtain image tokens (e),.... Uv] Z(0Ene( ()), which are subsequently encoded into 2048 dimensional vector representations in the language model's latent space using the image embedding and aligning layers in lanus-Pro. These retrieved image representations are concatenated with the text embeddings of the input prompts to form the augmented input for training the retrieval-enhanced model, which is the same training strategy used in RA-CM3.
During inference, given text prompt for image generation, we follow the same retrieval process used in training. The input prompt is encoded using the CLIP text encoder, and we compute cosine similarity with all images in the database to identify the most relevant reference image. The retrieved image is processed through the same pipeline to obtain its representation in the language model's s latent space. This representation is then prepended to the text prompt embedding to provide the model with both textual and visual context for generation. The augmented input is fed into the fine-tuned Janus-Pro model to generate the output image following the standard autoregressive generation procedure.
B.2 Show-o Implementation Details
Our patch-based autoregressive retrieval augmentation methods can be theoretically adapted to any model that generates images through discrete tokens. To demonstrate this generalizability. we implement both DAiD and FAiD on the Show-o (46) model, which generates images through a masked token decoding process rather than strict left-to-right autoregression. Show-o decodes multiple image tokens simultaneously at each time step by converting masked tokens specific image tokens based on a learned probability matrix. This fundamental difference in generation strategy necessitates several architectural adaptations to effectively incorporate our patch-based retrieval mechanisms while maintaining the model's 's inherent generation capabilities.
DAID on Show-o The implementation of DAiD on Show-o requires three key modifications to accommodate its non-autoregressivv generation strategy. First, instead of constructing retrieval queries
from upper-left neighboring patches as in autoregressive models, we utilize all eight surrounding
patches form the h-hop neighborhood representation for each target token position (i, This
comprehensive neighborhood encoding is computed as [V (i-1 V ((-1))) V ((-1)(+1)
V(GG-1) V ())++1) V ((++))=-) ((-1)))) : V (i++))(+ where missing positions are filled
with zero vectors O. Second, mitigate retrieval noise arising from sparse neighborhood information
in early time steps, we apply patch-level retrieval only during the final half of Show-o's 's decoding process when sufficient contextual information available. Third, since Show-o simultaneously predicts tokens for all patch positions at each time step rather than sequentially, we perform retrieval for all patch positions concurrently. At each qualifying time step f., for every patch position (i. j) in the partially generated image, we extract the eight-neighborhee representation the retrieval V
query and obtain the top-K K most similar patches ((.) (1.))
(1.))) from our database. We
E RRZZ using the same softmax formulation over retrieval distances as described in the main paper. These retrieval distributions are merged with Show-o's 's predicted distributions for each patch position using the weighted average D(ai) = (1 - A) D ((.3)
then construct position-specific retrieval distributions D (1.) retrieval
model + X- ((i3)
retrieval where controls the retrieval influence across all positions.
The adaptation of FAiD to Show-o involves both training and inference modifica- tions to accommodate the model's masked token generation process. During training, we prepare the training dataset by applying Show-o's noise injection process to generate intermediate noisy representations at each time step, which serve ground truth targets for the denoising process. For each training instance, we save these intermediate representations and apply patch-level retrieval to obtain relevant patches for all time steps. The training objective remains consistent with the standard Show-o formulation, but with augmented input representations that incorporate retrieved patch information. We insert FAiD modules into every L/6 decoder layers of Show-o's 's & model, where each module processes all patch positions simultaneously rather than focusing on a single next token. At each qualifying time step and for each FAiD-equipped layer 1, construct the 2D spatial representation H' E IR /VVNx/XX from the current hidden states and perform multi-scale feature smoothing for all patch positions. For cach position (i,j) and its corresponding retrieved patches
FAiD on Show-o
23

---

[h (( ). we apply the convolution operations {Conv2x2: Convax3: Dronxxxx.,,Coonxxxx conv
to capture contextual patterns at multiple scales. The refined representations are computed as h (1)) + Eg softmax(S2), h(,,),, where ((d))
bigg represents the output of the x g convolution for patch k at position (i. The final augmented representation for each position is calculated as hij ((+1)) = h' + Ahly + >x== $k
((.)), where Shiy represents the standard transformer layer
(k.5) are position- com- updates including self aiffetttention and forward components, and $ patibility scores computed through learned linear projections. During inference, procedure but apply retrieval and feature blending only during the final half of the generation time steps to ensure sufficient contextual information is available for effective patch integration.
follow the same
B.3 Training Setup
Training Datasets For model training, we utilize two large-scale image-caption datasets: CC12M [5] and Midjourney-v6 | From the training sets of these datasets, we randomly sam- ple a total of 50, 000 image-caption pairs (25, 000 from each dataset) to fine-tune our model. Each image is encoded into 576 patch features and corresponding image tokens with the same image tokenizer [39] employed in the Jamus-Pr Pro model. For each image patch, we further retrieve the top-K image tokens from retrieval database that exhibit similar neighborhood relationships. Conse- quently, each training instance comprises: (1) a textual image caption that serves as the conditioning input, (2) a sequence of 576 image tokens representing the ground-truth image, where each token is paired with K relevant image tokens retrieved from the database based on similar contextual features.
Training Details For the implementation of our FAiD approach, we fine-tune two pre-trained text- to-image generation models using the training dataset of 50K text-image pairs that we constructed. We select Janus-Pro- BB [9] and Show-o [46]
base models. The fine-tuning process is conducted on 4 NVIDIA 1100 (80GB) GPUs with a global batch size of 256 for a single epoch. We utilize the AdamW optimizer without weight decay, incorporating 10% linear warm-up schedule followed by a constant learning rate of 2e-4.
B.4 Evaluation Benchmarks and Metrics
To comprchensively evaluate our proposed methods, adopt multiple widely used benchmarks that assess different aspects of image generation quality:
GenEval [14 is benchmark designed to evaluate models' ability to understand and generate images based on specific attributes and relationships described in text prompts. It comprises multiple categories such single object generation, two-object composition, counting, colors, positioning, color attribution and so on. Performance is measured as the percentage of generated images that correctly align with the text descriptions.
DPG-Bench [18] (Detailed Prompt Generation Benchmark) evaluates how well image generation models handle detailed prompts with complex requirements, covering categories such as global image quality, entity generation, attribute accuracy, relationship modeling, and other complex generation tasks. Scores are reported as percentages.
For the Midjourney-300 benchmark 4p we employ three complementary metrics to evaluate the quality of generated images, including (1) Fréchet Inception Distance (FID) [17], which measures the statistical similarity between the distributions of generated and real images in the feature space of pre-trained Inception network; (2) CLIP-MMD (CMMD) (21), which measures the distance between real and generated images using CLIP embeddings and the Maximum Mean Discrepancy, and is specifically designed to better align with human perception of image quality and addresses several limitations of FID, including poor sample efficiency and incorrect normality assumptions; and (3) Fréchet Wavelet Distance (FWD) [1I. which measures the distance between real and generated images in the wavelet packet coefficient space. FWD captures both spatial and frequency information without relying pre-trained networks, making it domain-agnosii agnostic and robust to domain shifts across various image types. For all three metrics, lower scores indicate higher-qualing image generation, with both CMMD and FWD particularly effective in capturing distortions in generated images in ways that better correlate with human judgements.
1lava
24

---

C Experiment Results and Discussion
C.1 Accuracy of Patch-based Autoregressive Retrieval
To assess the effectiveness of our patch-level autoregressive retrieval mechanism, conduct a comparative analysis between the top-K re- trieved image tokens and the ground-truth to- kens to be generated. Specifically, we randomly sampled 000 instances from our training set, each comprising 576 image tokens and 576 x k retrieved tokens. To demonstrate the accuracy of the retrieved image tokens, for each ground- truth image token, also randomly sample vocabulary code non-relevant tokens. Us- ing the shared codebook, we transform im- age tokens into vector representations and com- pute the l2 distances between each ground-truth image token and its top-K retrieved counter- parts. Similarly, we also compute the mean of the l2 distance between each ground-truth token and the randomly sampled tokens. As shown in Figure [6 the distance between retrieved tokens and ground image tokens is signif- icantly smaller than the distance between ran- domly sampled tokens and ground-truth tokens. As & increases, the distance between the k-th retrieved token and the ground-truth token also increases, demonstrating the effectiveness of the retrieval approach and our assumption that image patches with similar neighbors usually exhibit inherent similarities.
1.983
Mean Valuee of Distance
1.53
1.51
1,505
1.506
1.310
1.500
1.512
1.512
1.515
1.500
1.50
1.493
1.48-
1.47
1.451
Random features
1.45
Top-10 Image
Figure 6: l2 distance between ground-truth tokens and top-10 retrieved tokens (blue line) compared to randomly sampled tokens (red dashed line). The curved arrow indicates broken y-axis that accom- modates the large gap between the retrieved token and the random token baseline.
C.2
Hyperparameter Optimization
14.06
23.08
14.75
Retrieval Weight (A)
5%
14.32
14.20
14.12
14.21
20
Level of Hop (h)
~
13.89
14.04
13.59
22
3
13.80
23.03
19.08
20
10%
18
16.34
16.60
16.39
16.63
12
13.83
13.13
14.26
18
123 23 13
13.98
13.94
13.78
16
15%
14.21
13.56
15.86
20.35
20.65
20.78
20.82
14
13.91
13.87
16.94
20
40
60
80
2
Retrieval Temperature (T)
Number of Blender Modules (b)
Figure Hyperparameter optimization results for DAiD and FAiD on FID scores. Left: FID scores for DAiD across different combinations of retrieval temperature and merging weight X.
Right: FID scores for FAiD across varying levels of hop and numbers of blender modules b. All experiments conducted the Midjourney-10 -O0K benchmark, with optimal configurations highlighted by red borders.
Both DAiD and FAiD require careful optimization of distinct sets of hyperparameters. For DAiD, we optimize the retrieval temperature 7 and merging weight which control the retrieval-based probability distribution sharpness and the balance between retrieval and model predictions, respec- tively. For FAiD, we focus on the level of hop (h) and number of blender modules (b), determining the spatial context incorporated during retrieval and extent of feature blending. To identify optimal
25

---

configurations, we conducted a systematic ablation study on the Midjourney-10K benchmark using Fréchet Inception Distance (FID) as the performance metric.
Figure presents the FID scores for DAiD across different combinations of and т, and for FAiD across varying levels of (h) and (b), where composite hop levels such "12" indicate combined use of multiple hop distances. Analysis of the DAiD results reveals that performance degrades as À increases, suggesting that modest integration of retrieval information enhances performance while excessive reliance impairs generative flexibility. The retrieval temperature T demonstrates less pronounced effects, though moderate value of 0.6 provides marginal benefits. For FAiD, configurations incorporating multiple hop levels generally outperform single hop levels, with the "12" configuration yielding optimal results. Regarding blender modules, an intermediate value consistently delivers the best performance, implying that moderate feature blending optimizes the incorporation of retrieved patches while avoiding both under-utilization and over-smoothing. Based on this analysis, we selected À = 0.05) and r = 0.6 for DAiD, and hop levels "12" with 2 blender modules for FAiD, achieving FIID scores of 14.12 and 13.13, respectively. These configurations effectively harness retrieval information while preserving the generative strengths of the underlying Janus-Pre model, as demonstrated by their superior performance on the benchmark.
D Limitations
While AR-RAG framework demonstrates strong performance across multiple benchmarks, several limitations should be acknowledged. First, our approach relies on discrete image tokenization and targets discrete token-based models, so it may be directly applied continuous diffusion models operating in latent spaces, Second, due to computational resource limitations, our retrieval database remains smaller than billion-scale databases. This limitation may introduce visual pattern biases, as the database may not fully capture the diversity of real-world visual patterns, potentially affecting the generation of underrepresented visual elements. Third, our implementation focuses exclusively on 2D image generation. While the underlying patch-based retrieval concept could theoretically extend to other structured generation tasks such 3D point cloud generation, we have not explored these applications.
E Broader impacts
We propose a novel retrieva-aaugmented approach to enhance existing image generation models. Our method is both highly efficient and readily adaptable to a wide range of applications. making it valuable for both academic research and industrial deployment. However, as our approach builds upon existing generative models, it may inherit their biases and could potentially produce inappropriate outputs in the absence of additional safety mechanisms. Furthermore, the large-scale retrieval database may contain unsafe undesirable content, which can be reflected in the retrieved image patches. To ensure safe deployment in real-world scenarios, additional safeguards and filtering measures are necessary mitigate these risks.
26


Reviews:
[{"review_id": "peHXv13aYA", "reviewer": "Reviewer_zZU3", "summary": "This paper tackles the problem of enhancing image generation by introducing Autoregressive Retrieval Augmentation (AR-RAG), which performs patch-level k-nearest neighbor retrievals at each generation step. Unlike previous static retrieval-based methods, AR-RAG dynamically incorporates context-aware visual references to improve generation quality. The authors propose two decoding strategies: DAiD, a training-free distribution-level augmentation, and FAiD, a fine-tuning method leveraging smoothed patch features. Experimental results on multiple benchmarks show that AR-RAG significantly improves object consistency, structural coherence, and performance on complex prompts.", "questions": "While the paper is of high quality overall, incorporating additional benchmark evaluations and deeper analyses, as discussed in the weaknesses section, could further improve its completeness and impact.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The idea of patch-based autoregressive retrieval augmentation is well motivated, effectively addressing several limitations of image-level retrieval methods such as over-copying from references and stylistic bias.\n2. The paper provides an in-depth exploration of the proposed paradigm through two well-designed implementations: (1) DAiD, a training-free method, and (2) FAiD, a parameter-efficient fine-tuning approach.\n3. The proposed method achieves state-of-the-art performance, and the experimental results convincingly demonstrate the effectiveness of each component.\n\nWeaknesses\n1. Additional benchmark evaluations could further highlight the method’s general effectiveness.\n- While the paper already evaluates the method on several standard benchmarks, including additional ones could strengthen the empirical support. Potential candidates include:\na) T2I-Benchmark [A], to assess improvements in image compositionality.\nb) RareBench [B], to evaluate the benefits of patch-based retrieval in handling rare concepts.\n\n2. Further ablation studies and analyses could enhance the paper’s completeness.\n- A qualitative visualization of retrieved patches during generation would help assess retrieval quality and provide insight into how patch-based retrieval aids compositional generation.\n- In addition to inference time, reporting GPU memory usage would offer a more comprehensive view of the computational overhead introduced by patch-based retrieval.\n- Investigating the effect of the hyperparameter λ in DAiD (Eq. 4) would be valuable. An ablation study on this parameter could clarify the trade-off between model-predicted generation and retrieval-based guidance.\n\n[A] T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, NeurIPS 2023\n\n[B] Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance, ICLR 2025", "was_revised": true, "final_justification": "I have thoroughly checked the authors’ rebuttals as well as the reviews from other reviewers. The responses sufficiently resolve my initial concerns on performance on other benchmarks and analysis on GPU memory. I appreciate the authors’ efforts and will keep my original recommendation of “accept.”", "rebuttal": "**Question:**  While the paper already evaluates the method on several standard benchmarks, including additional ones could strengthen the empirical support. Potential candidates include: a) T2I-Benchmark [A], to assess improvements in image compositionality. b) RareBench [B], to evaluate the benefits of patch-based retrieval in handling rare concepts.\n\n**Answer:** We thank the reviewer for these constructive suggestions. In response, we have conducted additional experiments on both RareBench and T2I-Bench, using the official prompts and evaluation protocols. The results are presented below.\nRareBench Results:\n| Models   | (single obj) Property | (single obj) Shape  | (single obj) Texture | (single obj) Action | (single obj) Complex | (multi obj) Concat | (multi obj) Relation | (multi obj) Complex |\n|----------|----------|--------|---------|--------|---------|--------|----------|---------|\n| RDM      | 68.00%   | 56.90% | 56.00%  | 44.00% | 64.50%  | 56.20% | 49.50%   | 39.50%  |\n| SDXL     | 60.00%   | 56.90% | 71.30%  | 47.50% | 58.10%  | 39.40% | 35.00%   | 47.50%  |\n| ImageRAG | 25.00%   | **86.50%** | **85.50%**  | 49.00% | 77.00%  | 57.50% | 36.00%   | 55.50%  |\n| Janus    | 53.50%   | 59.00% | 76.50%  | 53.50% | 78.00%  | 42.50% | 48.00%   | 50.50%  |\n| DAiD     | 64.00%   | 61.50% | 74.00%  | 54.00% | 78.50%  | 46.50% | 50.50%   | 54.50%  |\n| FAiD     | **74.00%**   | 62.00% | 78.50%  | **58.50%** | **79.00%**  | **67.00%** | **54.00%**   | **65.50%**  |\n\nT2I-Bench Results:\n\n| Models   | Color   | Shape    | Texture  | Numeracy |\n|----------|---------|----------|---------|----------|\n| RDM      | 0.666   | **0.555** | **0.538**  | 0.382    |\n| ImageRAG | 0.670   | 0.543    | 0.528   |  0.344    |\n| Janus    | 0.652   | 0.455    | 0.449   |  0.251    |\n| DAiD     | 0.630   | 0.459    | 0.464   |  0.245    |\n| FAiD     | **0.703** | 0.494    | 0.483   | **0.537** |\n\nAll models in above two tabels are the same models used in Tabel 1 in our paper including DAiD and FAiD. We adopt the official prompts and evaluation code for both benchmarks.\n\nAs observed on RareBench, our methods demonstrates substantial improvements over baselines, particularly for multi-object composition, indicating its effectiveness in generating rare or novel combinations not commonly found in the training data.\n\nOn T2I-Bench, our methods also outperform the baseline Janus model by a considerable margin. While our model does not surpass the diffusion-based RDM model in all evaluation sub-categories, we respectfully note that this may be partly attributable to the relatively lower performance of the original Janus architecture. Notably, our method outperforms all baselines by a substantial margin in the \"color\" and \"numeracy\" categories. We did not report results for 2d-spatial and 3d-spatial categories, as all evaluated models achieved a score of 0 on these sub-tasks.\n\n---\n\n**Question:**  A qualitative visualization of retrieved patches during generation would help assess retrieval quality and provide insight into how patch-based retrieval aids compositional generation.\n\n**Answer:**  We thank the reviewer for the excellent suggestion to visualize the retrieved patches during generation. In our visualization experiment, we take an image, randomly select a patch position, and retrieve the most relevant patch from the retrieval database conditioned on the surrounding patches. We then replace the selected patch with the retrieved one and observe that the retrieved patches integrate smoothly with their surrounding context.\n\nDue to conference policies, we are unable to submit additional PDF materials during the rebuttal phase. However, we will ensure that these visualizations are included in the revised version of the paper. Thank you again for this valuable suggestion.\n\n---\n\n**Question:**  In addition to inference time, reporting GPU memory usage would offer a more comprehensive view of the computational overhead introduced by patch-based retrieval.\n\n**Answer:**  Here is the breakdown of inference memory usage for our models. \n\nLoading Janus Pro: 4717MB\n\nPerforming a single image generation with Janus Pro: 5511MB\n\nLoading Janus Pro+FAiD: 5153MB\n\nPerforming a single image generation with Janus Pro+FAiD: 5949MB\n\nHosting Retrieval database: 77346MB\n\nAs shown, incorporating FAiD modules results in only a modest increase in memory usage during both model loading and inference (approximately 400 MB). While the retrieval database itself requires additional memory, it is stored separately and can be efficiently managed. Overall, our retrieval approach introduces limited overhead relative to the improved performance it provides.\n\n---\n\n**Question:**  Investigating the effect of the hyperparameter λ in DAiD (Eq. 4) would be valuable. An ablation study on this parameter could clarify the trade-off between model-predicted generation and retrieval-based guidance.\n\n**Answer:** We thank the reviewer for pointing out the importance of ablation on the hyperparameter λ in DAiD (Eq. 4). We apologize for not including this analysis in the main paper. As noted in Figure 7 in Appendix C.2 (Hyperparameter Optimization), we performed a hyperparameter search on λ to study its effect. In the revised version, we will ensure that this ablation study and the associated trade-offs are highlighted and discussed more prominently in the main text."}, {"review_id": "46UhWislW2", "reviewer": "Reviewer_Dnu4", "summary": "This paper introduces a novel method that performs retrieval augmentation during the autoregressive image generation process. For each patch, they retrieve likely relevant patches from a visual patch database given previously predicted tokens. They then incorporate these patches into the generation process using distribution augmentation, which biases the next-token distribution given the retrieved next patch, and feature augmentation (feature blending). The paper claims significant improvements on GenEval, DPG-Bench, and Midjourney-30k.", "questions": "I mostly want to see more compute controlled baselines. Namely, increase the number of parameters on Janus Pro until inference time matches +FAiD and fine-tune on the same dataset mixtures. Another baseline is just fine-tuning Show-o and Janus Pro on the dataset mixtures directly. As it is, it's hard to disentangle this method from the impact of compute and data. Another use case that I believe would strengthen the paper is the \"training free distribution alignment\" potential. Seems like including JourneyDB in the retrieval set improves Midjourney-30k with DAiD. Given that personalization is a popular RAG use case, I think this could be a useful supplementary experiment. Again, however, it would have to outperform just fine-tuning.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n+ Very creative strategy. This is the first method I've seen that incorporates multiple retrieval steps during the generative process for a single image. Brings to mind RAG-based iterative refinement methods. \n+ DAiD (distribution augmentation) strategy is simple and intuitive. Interested to see a simple baseline with DAiD by itself (no model, just D_retrieval).\n+ The paper is well-written. The method is complex and the paper does a good job of presenting it in easy to understand chunks. \n+ One potential advantage is the fact that DAiD is training-free. Not as touched-upon, but it could work for on the fly adaptation with a user provided DB. Results in Table 3 seem to support this use case.\n\n\n### Weaknesses\n\nThe primary weakness is that it is not clear from the stated results that this method is worth the compute and added complexity. (+DAiD) results are within margin of error for GenEval and DPG-Bench. (+FAiD) adds 20% extra parameters and 35% extra compute (on an L40, presumably this would get worse on faster GPUs since retrieval overhead would be a greater fraction of the forward pass). It's unclear if more parameters + a bit more training would yield the entirety of the gains from FAiD given equal inference time. \n\nFor Table 3, t2i generation on Midjourney-30k, I noticed that one of the retrieval datasets and fine-tuning datasets is JourneyDB, which is constructed with Midjourney images and is therefore more likely to be in distribution. I'd like to see if just fine-tuning with extra parameters on JourneyDB is enough to improve the Midjourney-30k scores as a baseline.", "was_revised": true, "final_justification": "I move my rating towards acceptance because the authors have provided compute-controlled baselines. Further their SFT with specific datasets provides insight into how much of the gain is from the retrieval dataset mixture versus the retrieval process itself.", "rebuttal": "We appreciate the reviewer's thoughtful feedback regarding computational efficiency and the source of performance improvements in our method. \n\n---\n\nTo demonstrate that the performance improvements stem from the retrieval mechanism rather than merely from distributional similarities between training and evaluation databases, we conducted the suggested experiments that fine-tune the baseline model on JourneyDB and our mixture dataset. The results are presented below:\n| models   | CMMD | FID | FWD | \n|----------|---------| ---------| ---------|\n| Janus 1B |    0.12      |     14.33     |     28.41     |\n| Janus 1B + DAiD |     0.11     |    9.15      |    28.00    |\n| Janus 1B + FAiD |      0.07    |    6.67      | 9.40 |\n| Janus 1B + JourneyDB SFT |     0.09     |   11.47      | 18.76 |\n| Janus 1B + Mixture-Data SFT |     0.12     |   11.94      | 16.96 |\n\nAs expected, fine-tuning on in-domain datasets (JourneyDB and the Midjourney training set) improves performance, with JourneyDB fine-tuning outperforming the mixture dataset approach. However, our FAiD method significantly outperforms both fine-tuning approaches across all metrics, demonstrating clear benefits from our retrieval mechanism beyond what can be achieved through domain-specific fine-tuning alone.\n\n---\n\nRegarding computational overhead, we would like to first clarify that: (1) Our retrieval process is optimized using the FAISS library with GPU acceleration, resulting in minimal overhead. As shown in Table 4, the DAiD method adds only 0.22% additional time compared to the vanilla Janus-Pro model. (2) The primary computational cost in FAiD comes from the additional convolutional operations in the decoder. This overhead is expected to decrease proportionally when running on faster GPU devices.\n\nFurthermore, we agree with the reviewer's suggestion to provide more explicit demonstrations of benefits versus costs. As suggested, we are conducting additional experiments with parameter-augmented baseline models. Specifically, we have added adapters to the vanilla Janus-Pro model and are fine-tuning it with the identical training setting. Due to time constraints, these experiments are still in progress. We commit to sharing these results during the discussion period and incorporating them in the next version of our paper."}, {"review_id": "Dau6v6Bb26", "reviewer": "Reviewer_FaiV", "summary": "This paper proposes a new image generation paradigm, Autoregressive Retrieval Augmentation (AR-RAG), which dynamically integrates fine-grained visual content by autoregressively incorporating k-nearest neighbor retrievals at the patch level. This approach avoids the issue of over-copying full-image information in existing image RAG methods. To implement this framework, the authors introduce DAiD, a training-free plug-and-play decoding strategy, and FAiD, a parameter-efficient fine-tuning method. Experiments on benchmarks such as Midjourney-30K, GenEval, and DPG-Bench validate the effectiveness of AR-RAG.", "questions": "please refer to the weakness part", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1, The validation motivation is reasonable. Compared with previous global Image RAG methods, AR-RAG utilizes region-of-interest (ROI) patch information more rationally.\n2. The writing is clear and easy to understand, facilitating comprehension of the proposed approach.\n\nWeakness:\n1. The inference speed is notably slow, hindering practical application. Unlike Image RAG, which requires only one retrieval, AR-RAG performs O(N²) retrievals, where N is the number of generated patches. This quadratic complexity significantly increases computational overhead.\n2. The experiments are conducted against relatively weak baselines. For example, the \"cat playing basketball\" issue in the teaser might not exist in the latest methods like FLUX, indicating that the comparative evaluation could be more rigorous.", "was_revised": true, "final_justification": "The authors have resolved my concern. I tend to borderline accept this paper", "rebuttal": "**Question:** The inference speed is notably slow, hindering practical application. Unlike Image RAG, which requires only one retrieval, AR-RAG performs O(N²) retrievals, where N is the number of generated patches. This quadratic complexity significantly increases computational overhead.\n\n**Answer:** We would like to clarify that the retrieval complexity of our method is O(N), not O(N²) as stated in the review. Specifically, for N patches, our approach performs N retrievals (i.e., one per image patch) rather than a quadratic number of retrievals. For each patch, we retain only the top 2,048 retrieved candidates. Furthermore, we employ the Faiss library to construct our retrieval database, which is highly optimized for large-scale and hierarchical nearest neighbor search. As demonstrated in Table 4 of our submission, the average inference time for Janus-Pro + DAiD to generate 100 images increases by only 0.22% compared to the Janus-Pro baseline and most of the addtional inference time is attributed to the retrieval process during inference. This negligible increase underscores the efficiency and practical feasibility of our retrieval method. We hope this clarification addresses the concerns regarding computational overhead.\n\n---\n\n**Question:** The experiments are conducted against relatively weak baselines. For example, the \"cat playing basketball\" issue in the teaser might not exist in the latest methods like FLUX, indicating that the comparative evaluation could be more rigorous. \n\n**Answer:** As suggested by the reviewer, we compare our method Janus-Pro + FAiD against two SOTA diffusion models including Flux-dev [1] and SD3-medium [2] on GenEval and MJHQ 30K. As reported in the Table below, these models consistently underperform relative to our proposed method, Janus-Pro + FAiD, despite their larger parameter sizes and pre-training on larger datasets.\n\n| Model                    | GenEval | MJHQ 30K |\n|--------------------------|---------|----------|\n| Janus-Pro + FAiD (1.3B)  | 0.78    | 6.67     |\n| Flux-dev [1] (12B)       | 0.67    | 10.15    |\n| SD3-medium [2] (2B)      | 0.62    | 11.92    |\n\n\nNotably, the GenEval benchmark specifically assesses the ability to generate objects with rare attributes and to compose images containing multiple, less commonly co-occurring objects. The comparatively lower performance of Flux-dev and SD3-medium on these benchmarks suggests that these models have limitations in handling complex generation tasks—such as modeling spatial relations, attribute binding, and multi-object composition—even in their latest versions. We believe this comparative analysis is both fair and rigorous, and it demonstrates the unique strengths of our approach.\n\n---\n\n[1] Black Forest Labs. Flux, 2024.\n\n[2]  Scaling rectified flow transformers for high-resolution image synthesis."}, {"review_id": "Xr6PqxbCIb", "reviewer": "Reviewer_L36Z", "summary": "This paper proposes a method that can retrieve image patches for image generation. Using the fact that the auto-regressive generation model (Janus-pro) utilizes a quantized image tokenizer, the algorithm finds the top-k relevant patches and modify the distribution from which each location is sampled. DaiD directly uses the image patches and FaiD softly integrates the retrieved patches to the model.", "questions": "No further questions.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This method is obviously novel and technically solid. I have seen many works that give reference images to the model but exploitation on image patches seems to be something new. This reminds me of PatchMatch algorithm which works really well on images with complex texture in Photoshop.\n- Injecting inductive bias to image generation models is usually very unstable and hard to handle, but this method is handling this problem with a clever method.\n\nWeaknesses\n- I really want to see if this method works on a problem setting such as zero-shot image generation. For example, generating a specific object or person often needs few images and LoRA does the rest. Auto-regressive models like Janus-pro are capable of doing the same task by accepting reference image but the performance is usually limited if the reference has been unseen during the training. I wonder if this method can cope with this setting and I am very sure this will add significant contribution to this paper.", "was_revised": true, "final_justification": "The authors resolve my comment about zero-shot generation and I am pleased that the result came out as I expected. I believe this work might be a quite important contribution for image generation models since it can retrieve per-image contribution, which might give a clue to solve the copyright issue for data providers. Therefore, I maintain my score as is.", "rebuttal": "We sincerely thank the reviewer for this insightful suggestion regarding zero-shot image generation and the importance of evaluating performance in such scenarios. In response, we conducted a targeted experiment to assess our method’s capabilities in this setting.\n\nSpecifically, we constructed an artificial retrieval database containing 30 images of Taylor Swift, collected from the internet, while ensuring no other human faces were included. The artificial retrieval database also contains 10,000 images from CC12M. (We filtered out images that contain human faces based on their captions) We then used the prompt “a woman is playing guitar on the stage” and generated 10 images each with both DAiD and FAiD. We observed that, when Taylor Swift’s images were present in the retrieval database, the generated faces in most images closely resembled her appearance. Conversely, after removing Taylor Swift’s images from the database and repeating the generation, the generated faces no longer displayed her features.\n\nThese results demonstrate that our retrieval-augmented approach can effectively leverage reference images for specific object or person generation, and is sensitive to the presence of such references in the retrieval set. We agree that further systematic exploration of zero-shot and few-shot settings is a promising direction and will add significant value to future versions of our work. We will make sure to include this new result in the revised version."}]

Author rebuttal:

Meta-review: The paper proposes a RAG framework for AR based image generation i.e. RAIG for autoregressive generation.  The key contribution is to perform patch level retrieval at each iterative stage using previously generated patches.

The paper has received unanimous final recommendations to accept the paper (2x A, 2x BA), following clarifications made in the rebuttal largely around additional baselines and to address the common reviewer voice around complexity concerns.  

zZUS had initial concerns about benchmarks and memory which were addressed. They are positive on the novelty and experimental sufficiency.

Dnu4 also had compute-cost related concerns that were addressed.  Additional baselines in the rebuttal converted the reviewer to a borderline accept.

FaiV much as Dnu4 had complexity and baselining concerns that were addressed.

L36Z was very positive and suggested some zero shot experiments that were mostly addressed in the rebuttal.

As all reviewers share a consensus view to accept, the AC recommendation is to accept.
Decision: Accept (poster)