Title: Optimizing Retrieval for RAG via Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Retrieval-augmented Generation

RAG

RL
Abstract: As retrieval-augmented generation (RAG) becomes more widespread, the role of retrieval is shifting from retrieving information for human browsing to retrieving context for AI reasoning. This shift creates more complex search environments, where relevance is difficult to pre-define. Existing retrievers rely on supervised fine-tuning (SFT) with human labels or synthetic data, resulting in static relevance that struggles to adapt to diverse RAG environments. To address this challenge, we propose R3, a Retrieval framework optimized for RAG through Reinforcement learning (RL). Specifically, we adopt an RL training paradigm that enables the retriever to explore and self-improve within given RAG environments, automating the learning process with minimal manual experimentation or tuning effort. Extensive experiments across diverse tasks demonstrate that \ours improves RAG performance by 5.2% over the original retriever and surpasses state-of-the-art retrievers by 4.9%, while achieving comparable results to LLM-augmented retrieval and RAG systems built on post-trained or instruction-tuned LLMs. It is both efficient and practical, requiring only 4 GPUs and completing training within a single day.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework called Open-RAG, which constructs contrastive learning supervision for retriever training by automatically determining whether a given Query-Document pair is a positive match in the context of RAG. This allows the retriever to better align with the downstream needs of the LLM-based QA task, rather than relying solely on semantic similarity, thereby improving task-specific RAG performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1 The paper presents an end-to-end mechanism to optimize the RAG system in accordance with the requirements of the downstream task. Specifically, it uses generation certainty as a signal to construct contrastive supervision for training the retriever. This is an intuitive, effective, and elegant approach.\n\n2 As an online, lightweight, and end-to-end optimization framework for RAG, the design of Open-RAG offers valuable engineering insights.\n\n3 Open-RAG only requires a one-time offline index construction over the entire corpus, significantly reducing system overhead.\n\n**Weaknesses:**\n\n1 The most novel aspect of this paper—its semi-parametric design—has already been proposed in prior work (SIDR), which diminishes the originality of the contribution.\n\n2 The methodological innovation appears limited. Steps 1 and 3 appear to be directly borrowed from SIDR, leaving only Step 2 as the novel component. As a result, the paper feels more like an application of SIDR to RAG, rather than a substantially new contribution.\n\nQUESTIONS:\n1 In Section 3.2, how is the evaluation conducted to construct the positive pool Dq+ and the negative pool Dq−? I guess it might be based on an exact match, but this should be clearly specified in the main text.\n\n2 As a core design choice of this work, the loss terms described in Section 3.4 are critical. However, I find their respective roles insufficiently explained. A more intuitive explanation would be helpful, especially regarding this confusing point: the component EBoT(q) does not participate in inference and appears only in the loss term. What is the rationale for including it in training?\n\n3 In Appendix D, what are the specific procedures for the IR setting and RAG setting respectively? The descriptions of “Unsupervised Pre-training,” “Supervised on MS MARCO,” “Supervised on NQ,” “Supervised on TriviaQA,” and “Pre-training + Supervised on Multiple Datasets” are unclear. Please clarify what each of these means in practice.\n\n4 In Section 4.1, the statement “Our experiments are conducted with 4 NVIDIA GPUs” should include the specific GPU model for reproducibility and transparency.\n\nLIMITATIONS:\nSee Questions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an end-to-end RAG framework that enhances performance by focusing solely on improving the retriever component. Positive and negative query-document pairs for contrastive learning are selected based on the likelihood of generating the final response. The study also highlights that relevance models used in traditional Information Retrieval (IR) often do not align well with RAG performance. Experimental results on five public RAG benchmarks show that the proposed method outperforms other compared approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces an end-to-end optimized RAG framework that enhances RAG performance by tuning the retriever component through online contrastive learning.\n\n2. It demonstrates that higher accuracy in traditional IR settings generally correlates with improved RAG performance for QA tasks, although this correlation does not extend to non-QA tasks.\n\n3. The proposed method outperforms other compared approaches on five public datasets.\n\nWeaknesses:\n1. The settings for the thresholds (\\tau_q^{+/-}) in line 197 are not clearly explained. These thresholds are crucial for determining positive and negative documents during online contrastive learning. The rationale for using the top 100 documents to establish these parameters is unclear. It would be beneficial to show the impact of varying these parameter settings on performance.\n\n2. An ablation study for the final loss function, as mentioned in line 224, is necessary. It would be insightful to observe the performance differences when different components are included in the objective loss. This analysis would help in understanding the contribution of each component to the overall effectiveness of the model.\n\nQUESTIONS:\nPlease see my questions in the weakness section. The final score may be changed based on the author response. My main concerns are the setting of thresholds and the ablation study on the objective loss.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies to apply the previously proposed sparse retrieval model, SiDR, to improving the end-to-end training efficiency of RAG systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper offers a method to improve the training efficiency of RAG by proposing a walk-around method without conducting computationally expensive indexing in RAG training. The proposed method is effective while using the retrieval model, SiDR.\nWeaknesses\n1. The biggest concern for me is that the comparison is a bit misleading. As I know, all the other models (e.g., RA-DIT and SELF-RAG) are not fine-tuned on the train set of the targeted datasets (e.g., NQ, TriviaQA, HotpotQA). This means the key point I can learn from the experiment is that fine-tuning RAG (either retriever or LLM) on the targeted dataset can help but the generalization capability is not confirmed. Correct me if I’m wrong. \n2. The previous study has proposed different RAG training strategies to address the issue of index updating for dense retrievers (See top-100 reranking and Query-side fine-tuning in Table 4 in [1]). Although the work provides another solution based on the previously proposed sparse retriever, SIDR. The motivation and contribution are a bit limited since the paper only builds upon a specific sparse retriever, SIDR.\n3. It seems that the obvious improvement is when applying the technique to pre-trained LLMs rather than instruction fine-tuned ones.\n\n[1] Atlas: Few-shot Learning with Retrieval Augmented Language Models; Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Tim  Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, Edouard Grave; arXiv:2208.03299\n\nQUESTIONS:\n1. Why there is only one-shot effectiveness shown in HotpotQA, while all the other datasets show both 1-shot and 10-shot effectiveness?\n\nLIMITATIONS:\nSee Weakness 2 and 3.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work highlights the importance of finetuning retrievers for RAG. They propose a method called OPEN-RAG that finetunes the retriever using contrastive pairs that are selected by running examples “end-to-end” through the full RAG pipeline. This approach shows improvement over other similar methods across standard RAG benchmarks such as NQ and HotPotQA. The authors also explore both online and offline forms of contrastive learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* This seems like quite a promising method. Finetuning the retriever is also elegant/straightforward and lightweight compared to other methods\n* The authors include ablation studies (online vs. offline, KL divergence vs contrastive loss). The appendix has extensive additional details and experiments\n\nWeaknesses:\n* It is surprising that offline only and online only alone result in suboptimal performance. The fact that the method relies on a mix of online and offline undermines the elegance of the approach. How much of this just depends on hyperparameters?\n* In Table 1, OPEN-RAG shows improvements across 3 different LLMs for NQ, Trivia And HotPotQA but not PubHealth or ARC-C. What is it about these two datasets?\n* Data quality usually plays a crucial role in finetuning. Additional experiments around data quality would make the paper stronger (Figures 3 and 4 are helpful)\n\nQUESTIONS:\nQ1: Many practitioners use models like ChatGPT, Claude and Gemini via APIs.How well will this method work? Presumably it can be adapted to API-based RAG systems? The offline computation of probability thresholds is quite clever (starting line 189). Is this something that wouldn’t be possible with APIs? \n\nQ2: In an API scenario, could you use an LLM to judge/rank relevance to pick positive and negative pairs instead of calculating probabilities? While it might be more costly, it might make the method more generalizable\n\nQ3: Does it matter if the training objective follows SiDR with the semi-parametric contrastive loss? Did you consider any other approaches (I guess KL divergence)?\n\nOther comments:\n* The name is very confusing, and does not convey any information about what the method actually does. OPEN-RAG sounds like it is RAG on an open set of questions. The method is also not end-to-end, since it is only optimizing/training the retriever. I really think the authors should consider changing the name to something that conveys a little more information. A good name goes a long way! And a bad name can cause a lot of unnecessary confusion. Even something as simple as “retriever-optimized RAG”\n* Section 1 line 89 states that OPEN-RAG offers “two forms of openness\" - both of these are silly and frankly distracting (it almost sounds like it was AI generated). I’m sure you can come up with a better name.\n* It would be helpful to include the preliminary study mentioned in line 45 in the main text and not in the appendix, if it is indeed very important\n* Typo line 243 “defualt”\n* Typo line 260 “while frozen the computation”\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-20"}