Title: Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Retrieval-Augmented Generation (RAG)

Test-time Scaling
Abstract: Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs). However, standard RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions. In this work, we reinterpret RAG as \textit{Retrieval-Augmented Reasoning} and identify a central but underexplored problem: \textit{Reasoning Misalignment}—the divergence between an LLM's internal reasoning trajectory and the evidential constraints provided by retrieval. To address this issue, we propose \textsc{AlignRAG}, a novel iterative framework grounded in \textit{Critique-Driven Alignment (CDA)}. We further introduce \textsc{AlignRAG-auto}, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations. At the heart of \textsc{AlignRAG} lies a \textit{contrastive critique synthesis} mechanism that generates retrieval-sensitive critiques while mitigating self-bias. This mechanism trains a dedicated retrieval-augmented \textit{Critic Language Model (CLM)} using labeled critiques that distinguish between evidence-aligned and misaligned reasoning. Empirical evaluations show that our approach significantly improves reasoning fidelity. Our 8B-parameter CLM improves performance over the Self-Refine baseline by \textbf{12.1\%} on out-of-domain tasks and outperforms a standard 72B-parameter CLM by \textbf{2.2\%}. Furthermore, \textsc{AlignRAG-auto} achieves this state-of-the-art performance while dynamically determining the optimal number of refinement steps, enhancing efficiency and usability. \textsc{AlignRAG} remains compatible with existing RAG architectures as a \textit{plug-and-play} module and demonstrates strong robustness under both informative and noisy retrieval scenarios. Overall, \textsc{AlignRAG} offers a principled solution for aligning model reasoning with retrieved evidence, substantially improving the factual reliability and robustness of RAG systems. Our source code is provided at \href{https://github.com/upup-wei/RAG-ReasonAlignment}{link}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ALIGNRAG, a novel framework that redefines Retrieval-Augmented Generation (RAG) as Retrieval-Augmented Reasoning. It addresses a key failure mode in RAG systems called Reasoning Misalignment—where the model's reasoning deviates from the retrieved evidence. ALIGNRAG uses a Critique-Driven Alignment (CDA) mechanism, powered by a Critic Language Model (CLM) trained via Contrastive Critique Synthesis (CCS). This CLM critiques and refines the reasoning process at test time, improving factual alignment and robustness.\n\nKey contributions:\n\nIdentification of reasoning misalignment as a core issue in RAG.\nIntroduction of CDA and CLM for test-time reasoning refinement.\nDemonstrated improvements over baselines like Self-RAG and Self-Refine.\nPlug-and-play compatibility with existing RAG systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nNovel Perspective: Reframing RAG as reasoning rather than just generation is conceptually impactful.\nTest-Time Optimization: CDA enables dynamic correction without retraining the base model.\nContrastive Critique Synthesis: Innovative training method that mitigates self-bias in critiques.\nEmpirical Rigor: Extensive experiments across 7 benchmarks and 3 model families.\nPlug-and-Play Design: ALIGNRAG integrates easily into existing RAG pipelines.\nOOD Robustness: Strong performance even under noisy or irrelevant retrieval conditions.\n\nWeaknesses / Limitations\nDependency on Retrieval Quality: Performance still degrades under extreme retrieval failure.\nComputational Overhead: Iterative refinement and CLM inference may increase latency.\nLimited Human Evaluation: Mostly automatic metrics; lacks qualitative human judgment.\nComplexity: The multi-step critique and refinement process may be harder to implement in production.\nGeneralization to Other Tasks: Focused on QA; unclear how well it transfers to other RAG applications like summarization or dialogue.\n\nQUESTIONS:\nHow does ALIGNRAG perform in real-time or latency-sensitive applications?\nCan the CLM be distilled or compressed for efficiency?\nHow does the system handle conflicting evidence in retrieved documents?\nCould this framework be extended to multi-modal RAG (e.g., vision-language)?\nWhat are the failure cases where ALIGNRAG underperforms compared to simpler baselines?\n\nLIMITATIONS:\nEffectiveness diminishes when none of the retrieved documents are relevant.\nMinor tuning may be required for optimal integration with different LLMs.\nThe critique model’s performance is still tied to the informativeness of the retrieved evidence.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to improve the accuracy of LLM responses by refining them within the RAG framework. Specifically, it involves CLM that critiques the generated responses to produce improved responses, and CCS that synthesizes critique data by contrasting good and bad responses for training CLM.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper is well-written and well-structured.\n\n2. The effectiveness of the proposed method is validated through extensive experiments and analyses.\n\n## Weaknesses\n\n1. Qualitative analysis is necessary to fully understand the effectiveness of the proposed method. For example, the proposed method shows performance improvement through refinement iterations even on relatively simple single-hop QA datasets such as NQ and TriviaQA. It would be helpful to include concrete examples demonstrating how the response is refined over the course of five iterations to eventually arrive at the correct response.\n\n    Furthermore, in Section 3, the paper discusses potential misalignments that can occur at each phase of RAG reasoning; illustrating these with actual examples of incorrect RAG responses would enhance clarity.\n\n2. It would be helpful to include InstructRAG [1] results in the analyses (Figures 2 and 4). Although InstructRAG is a training-time refinement method, it aims to ground RAG reasoning in the retrieved documents—similar in goal to the proposed method. Comparing these methods could offer valuable insights to readers. (It’s also worth noting that Self-Refine [2] may be at a disadvantage since it does not involve training.)\n\n3. (Minor) The novelty of the proposed method appears to be somewhat limited. Given the existence of numerous prior approaches focused on refining LLM responses, the core idea of the paper—generating critique data by contrasting good and bad answers—offers a relatively modest technical contribution.\n\n> [1] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales\n>\n> [2] SELF-REFINE: Iterative Refinement with Self-Feedback\n\nQUESTIONS:\n1. In Line 93, the paper claims that Self-RAG [1] requires architectural modifications. Could you elaborate on what specific modifications are being referred to?\n\n2. In Section 4.2.3 and Algorithm 2, it is described that the CLM is trained with good responses as part of the input (i.e., $y_{exp} \\in I_{critic}$). However, in Table 10, the good response appears to be part of the output. Table 10 seems to be correct, so it would be helpful to clarify this part.\n\n3. Was InstructRAG [2] trained using the same data described in Lines 223-224, as was the proposed method?\n\n4. (Minor) I’m curious how well the base model would perform if it were fine-tuned to generate the good responses $y_{exp}$ that match the correct answers.\n\n> [1] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\n>\n> [2] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRetrieval-augmented generation (RAG) enables knowledge-grounded LLMs but frequently exhibits reasoning misalignment, where inference drifts from retrieved evidence and yields factual errors. AlignRAG reframes RAG as Retrieval-Augmented Reasoning and embeds a Critique-Driven Alignment loop in which a dedicated Critic Language Model (CLM)—trained via contrastive critique synthesis to distinguish evidence-aligned from misaligned reasoning—generates targeted critiques at inference that steer the generator back to relevant documents . Without altering existing RAG pipelines, an 8B CLM delivers substantial gains over prior refinement methods and consistently outperforms a 72B baseline, all while maintaining robustness under noisy retrieval conditions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Principled Framework Design: Embedding the Critic Language Model into an iterative inference loop transforms RAG into a constrained optimization process. This design elegantly marries retrieval and generation phases and provides a clear mechanism for evidence-driven correction.\n- Plug-and-Play Compatibility: ALIGNRAG integrates seamlessly with existing RAG architectures without requiring modifications to the generator or retrieval components. This design choice maximizes practical applicability and lowers barriers to adoption.\n- Robustness: Experimental results on out-of-domain and noisy retrieval scenarios substantiate claims of improved stability, an essential quality for real-world deployments \n\nWeakness:\n- This paper lacks of the results of runing time. The iterative critique–generation loop incurs substantial computational overhead. Multiple forward passes through both CLM and generator can increase inference time, limiting usability in latency-sensitive contexts.\n- Effective deployment requires setting several hyperparameters, such as number of critique iterations, etc.—which may vary significantly across tasks, models and domains. Does this approach also require preparing a dedicated training dataset for each domain?\n- Table 1 shows that AlignRAG underperforms InstructRAG, yielding even lower scores. In Table 2, combining InstructRAG with test-time  alignment yields better performance—but by how much does the cost increase?\n\nQUESTIONS:\n- What are the results of runing time？\n- How does AlignRAG’s performance compare with that of IRCOT [1]？\n\n[1] Trivedi H, Balasubramanian N, Khot T, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions[J]. arXiv preprint arXiv:2212.10509, 2022.\n\nLIMITATIONS:\nIf the retrieval quality is insufficient, it will negatively impact AlignRAG’s performance.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper aims to improve RAG in the generation phase. It first analyzes a critical challenge in RAG, termed Reasoning Misalignment, referring to a disconnect between the model's reasoning process and the retrieved evidence. It then tries to understand the underlying cause of the problem by reconceptualizing RAG as Retrieval-Augmented Reasoning. Further, it proposes a framework named ALIGNRAG, which employs a Critic Language Model to dynamically correct misalignments during inference using retrieval-augmented critiques. Finally, extensive experiments across seven benchmark datasets and three model families are conduct to evalute the effectiveness of ALIGNRAG.

STRENGTHS AND WEAKNESSES:
The main strengths are summarized as follows:
1. The studied problem is interesting.

The main weaknesses are summarized as follows:
1. The analysis of the reasoning misalignment issue of RAG in Section 3 is not clear. It would be better if an intuitive example could be provided to illustrate the two characteristics of misalignment. Besides, I doubt whether there are truly three reasoning steps involved in RAG reasoning. The connection between this analysis the proposed solution seems weak.
2. The proposed method heavily rely on the construction of training data with complicated process. As shown in Equation (1), the quality label of retrived context and the supervision signal are difficult to obtain.
3. It would be better to include a case study in the experiments. The inference process is not straightforward for me.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03