Title: AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: retrieval augmented image generation

autoregressive image generation

discrete image tokens
Abstract: We introduce  Autoregressive Retrieval Augmentation (AR-RAG), a novel paradigm that enhances image generation by autoregressively incorporating k-nearest neighbor retrievals at the patch level.
Unlike prior methods that perform a single, static retrieval before generation and condition the entire generation on fixed reference images,  AR-RAG performs context-aware retrievals at each generation step, using prior-generated patches as queries to retrieve and incorporate the most relevant patch-level visual references, 
enabling the model to respond to evolving generation needs while avoiding limitations (e.g., over-copying, stylistic bias, etc.) prevalent in existing methods. To realize AR-RAG, we propose two parallel frameworks: (1) Distribution-Augmentation in Decoding (DAiD), a training-free plug-and-use decoding strategy that directly merges the distribution of model-predicted patches with the distribution of retrieved patches, and (2) Feature-Augmentation in Decoding (FAiD), a parameter-efficient fine-tuning method that progressively smooths the features of retrieved patches via multi-scale convolution operations and leverages them to augment the image generation process. We validate the effectiveness of AR-RAG on widely adopted benchmarks, including Midjourney-30K, GenEval and DPG-Bench, demonstrating significant performance gains over state-of-the-art image generation models.


Peer reviews:
comments: SUMMARY:
This paper proposes a method that can retrieve image patches for image generation. Using the fact that the auto-regressive generation model (Janus-pro) utilizes a quantized image tokenizer, the algorithm finds the top-k relevant patches and modify the distribution from which each location is sampled. DaiD directly uses the image patches and FaiD softly integrates the retrieved patches to the model.

STRENGTHS AND WEAKNESSES:
Strengths
- This method is obviously novel and technically solid. I have seen many works that give reference images to the model but exploitation on image patches seems to be something new. This reminds me of PatchMatch algorithm which works really well on images with complex texture in Photoshop.
- Injecting inductive bias to image generation models is usually very unstable and hard to handle, but this method is handling this problem with a clever method.

Weaknesses
- I really want to see if this method works on a problem setting such as zero-shot image generation. For example, generating a specific object or person often needs few images and LoRA does the rest. Auto-regressive models like Janus-pro are capable of doing the same task by accepting reference image but the performance is usually limited if the reference has been unseen during the training. I wonder if this method can cope with this setting and I am very sure this will add significant contribution to this paper.

QUESTIONS:
No further questions.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

comments: SUMMARY:
This paper proposes a new image generation paradigm, Autoregressive Retrieval Augmentation (AR-RAG), which dynamically integrates fine-grained visual content by autoregressively incorporating k-nearest neighbor retrievals at the patch level. This approach avoids the issue of over-copying full-image information in existing image RAG methods. To implement this framework, the authors introduce DAiD, a training-free plug-and-play decoding strategy, and FAiD, a parameter-efficient fine-tuning method. Experiments on benchmarks such as Midjourney-30K, GenEval, and DPG-Bench validate the effectiveness of AR-RAG.

STRENGTHS AND WEAKNESSES:
Strength:
1, The validation motivation is reasonable. Compared with previous global Image RAG methods, AR-RAG utilizes region-of-interest (ROI) patch information more rationally.
2. The writing is clear and easy to understand, facilitating comprehension of the proposed approach.

Weakness:
1. The inference speed is notably slow, hindering practical application. Unlike Image RAG, which requires only one retrieval, AR-RAG performs O(N²) retrievals, where N is the number of generated patches. This quadratic complexity significantly increases computational overhead.
2. The experiments are conducted against relatively weak baselines. For example, the "cat playing basketball" issue in the teaser might not exist in the latest methods like FLUX, indicating that the comparative evaluation could be more rigorous.

QUESTIONS:
please refer to the weakness part

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method that performs retrieval augmentation during the autoregressive image generation process. For each patch, they retrieve likely relevant patches from a visual patch database given previously predicted tokens. They then incorporate these patches into the generation process using distribution augmentation, which biases the next-token distribution given the retrieved next patch, and feature augmentation (feature blending). The paper claims significant improvements on GenEval, DPG-Bench, and Midjourney-30k.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n+ Very creative strategy. This is the first method I've seen that incorporates multiple retrieval steps during the generative process for a single image. Brings to mind RAG-based iterative refinement methods. \n+ DAiD (distribution augmentation) strategy is simple and intuitive. Interested to see a simple baseline with DAiD by itself (no model, just D_retrieval).\n+ The paper is well-written. The method is complex and the paper does a good job of presenting it in easy to understand chunks. \n+ One potential advantage is the fact that DAiD is training-free. Not as touched-upon, but it could work for on the fly adaptation with a user provided DB. Results in Table 3 seem to support this use case.\n\n\n### Weaknesses\n\nThe primary weakness is that it is not clear from the stated results that this method is worth the compute and added complexity. (+DAiD) results are within margin of error for GenEval and DPG-Bench. (+FAiD) adds 20% extra parameters and 35% extra compute (on an L40, presumably this would get worse on faster GPUs since retrieval overhead would be a greater fraction of the forward pass). It's unclear if more parameters + a bit more training would yield the entirety of the gains from FAiD given equal inference time. \n\nFor Table 3, t2i generation on Midjourney-30k, I noticed that one of the retrieval datasets and fine-tuning datasets is JourneyDB, which is constructed with Midjourney images and is therefore more likely to be in distribution. I'd like to see if just fine-tuning with extra parameters on JourneyDB is enough to improve the Midjourney-30k scores as a baseline.\n\nQUESTIONS:\nI mostly want to see more compute controlled baselines. Namely, increase the number of parameters on Janus Pro until inference time matches +FAiD and fine-tune on the same dataset mixtures. Another baseline is just fine-tuning Show-o and Janus Pro on the dataset mixtures directly. As it is, it's hard to disentangle this method from the impact of compute and data. Another use case that I believe would strengthen the paper is the \"training free distribution alignment\" potential. Seems like including JourneyDB in the retrieval set improves Midjourney-30k with DAiD. Given that personalization is a popular RAG use case, I think this could be a useful supplementary experiment. Again, however, it would have to outperform just fine-tuning.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of enhancing image generation by introducing Autoregressive Retrieval Augmentation (AR-RAG), which performs patch-level k-nearest neighbor retrievals at each generation step. Unlike previous static retrieval-based methods, AR-RAG dynamically incorporates context-aware visual references to improve generation quality. The authors propose two decoding strategies: DAiD, a training-free distribution-level augmentation, and FAiD, a fine-tuning method leveraging smoothed patch features. Experimental results on multiple benchmarks show that AR-RAG significantly improves object consistency, structural coherence, and performance on complex prompts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The idea of patch-based autoregressive retrieval augmentation is well motivated, effectively addressing several limitations of image-level retrieval methods such as over-copying from references and stylistic bias.\n2. The paper provides an in-depth exploration of the proposed paradigm through two well-designed implementations: (1) DAiD, a training-free method, and (2) FAiD, a parameter-efficient fine-tuning approach.\n3. The proposed method achieves state-of-the-art performance, and the experimental results convincingly demonstrate the effectiveness of each component.\n\nWeaknesses\n1. Additional benchmark evaluations could further highlight the method’s general effectiveness.\n- While the paper already evaluates the method on several standard benchmarks, including additional ones could strengthen the empirical support. Potential candidates include:\na) T2I-Benchmark [A], to assess improvements in image compositionality.\nb) RareBench [B], to evaluate the benefits of patch-based retrieval in handling rare concepts.\n\n2. Further ablation studies and analyses could enhance the paper’s completeness.\n- A qualitative visualization of retrieved patches during generation would help assess retrieval quality and provide insight into how patch-based retrieval aids compositional generation.\n- In addition to inference time, reporting GPU memory usage would offer a more comprehensive view of the computational overhead introduced by patch-based retrieval.\n- Investigating the effect of the hyperparameter λ in DAiD (Eq. 4) would be valuable. An ablation study on this parameter could clarify the trade-off between model-predicted generation and retrieval-based guidance.\n\n[A] T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, NeurIPS 2023\n\n[B] Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance, ICLR 2025\n\nQUESTIONS:\nWhile the paper is of high quality overall, incorporating additional benchmark evaluations and deeper analyses, as discussed in the weaknesses section, could further improve its completeness and impact.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}