Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization
Authors: Jiaqi Wei
Hao Zhou
Xiang Zhang
Di Zhang
Zijie Qiu
Noah Wei
Jinzhe Li
Wanli Ouyang
Siqi Sun
Venue: NeurIPS 2025 poster ()
Keywords: Retrieval-Augmented Generation (RAG)
Test-time Scaling
Primary area: applications

Abstract: Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs). However, standard RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions. In this work, we reinterpret RAG as \textit{Retrieval-Augmented Reasoning} and identify a central but underexplored problem: \textit{Reasoning Misalignment}—the divergence between an LLM's internal reasoning trajectory and the evidential constraints provided by retrieval. To address this issue, we propose \textsc{AlignRAG}, a novel iterative framework grounded in \textit{Critique-Driven Alignment (CDA)}. We further introduce \textsc{AlignRAG-auto}, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations. At the heart of \textsc{AlignRAG} lies a \textit{contrastive critique synthesis} mechanism that generates retrieval-sensitive critiques while mitigating self-bias. This mechanism trains a dedicated retrieval-augmented \textit{Critic Language Model (CLM)} using labeled critiques that distinguish between evidence-aligned and misaligned reasoning. Empirical evaluations show that our approach significantly improves reasoning fidelity. Our 8B-parameter CLM improves performance over the Self-Refine baseline by \textbf{12.1\%} on out-of-domain tasks and outperforms a standard 72B-parameter CLM by \textbf{2.2\%}. Furthermore, \textsc{AlignRAG-auto} achieves this state-of-the-art performance while dynamically determining the optimal number of refinement steps, enhancing efficiency and usability. \textsc{AlignRAG} remains compatible with existing RAG architectures as a \textit{plug-and-play} module and demonstrates strong robustness under both informative and noisy retrieval scenarios. Overall, \textsc{AlignRAG} offers a principled solution for aligning model reasoning with retrieved evidence, substantially improving the factual reliability and robustness of RAG systems. Our source code is provided at \href{https://github.com/upup-wei/RAG-ReasonAlignment}{link}.

Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization
Jiaqi We
Hao Zhou
Xiang Zhang Di Zhang
Zijie Qiu
Wei Wei
Jinzhe Li Wanli Ouyang²
Siqi Sun 25f
Zhejiang University 2 Shanghai Artificial Intelligence Laboratory
3 South China University of Technology
University of British Columbia Fudan University University of Hong Kong wei@zju. edu. edu. en, siqisunfuudan edu. cn
Abstract
Retrieval-augmente generation (RAG) has become a widely adopted paradigm for enabling knowledge-groundee large language models (LLMs). However, standard RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions. In this work, we reinterpret RAG as Retrieval-Augmentie central but underexplored problem: Reasoning Misalignment the divergence between an LLM's internal reasoning trajectory and the evidential constraints provided by retrieval. To address this issue, we propose ALIGNRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA). We further introduce ALIGNR AG-AUTO, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations. At the heart of ALIGNRAG lies contrastive critique synthesis mechanism that generates retrieval-sensinee critiques while mitigating self-bias. This mechanism trains dedicated retrieval-augmentt Critic Language Model (CLM) using labeled critiques that distinguish between evidence-alignes and misaligned reasoning. Em- pirical evaluations show that our approach significantly improves reasoning fidelity. Our 8B 3B-parameter CLM improves performance over the Self-Refine baseline by 12. % on out-of-domain tasks and outperforms standard 72B-parameeer CLM by 2.2%. Furthermore, ALIGNR AG-AUTO achieves this state-of state-of-the-arr performance while dynamically determining the optimal number of refinement steps, enhancing efficiency and usability. ALIGNRAG remains compatible with existing RAG chitectures as a plug-and aate-aaddplyyy module and demonstrates strong robustness under both informative and noisy retrieval scenarios. Overall, ALIGNRAG offers prin- cipled solution for aligning model reasoning with retrieved evidence, substantially improving the factual reliability and robustness of RAG systems. Our source code is provided at Github.
Reasoning and identify
Introduction
Large Language Models (LLMs) have significantly advanced natural language understanding and generation capabilities. Retrieval-Augmented Generation (RAG) [THIO) has emerged as a prominent paradigm for grounding LLM responses with external knowledge. However, RAG systems exhibit notable fragility, particularly when confronted with irrelevant or noisy retrieved evidence [I] [2].
Existing methods primarily rely static, training-time time optimizations, which are often insufficient to
"Equal contributions.
i Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

address the dynamic challenges of error propagation during inference [3 14]. We identify a critical, yet understudied, failure mode in RAG: reasoning misalignment- disconnect between the model's reasoning process and the retrieved evidence. Prior work has focused predominantly retrieval quality or generating more robust outputs, largely overlooking the explicit alignment of the reasoning steps with the provided evidence [13 7 18). While reflective approaches like Self- RAG [9] attempt error detection, they often necessitate architectural modifications or task-specific fine-tuning, limiting their generalizability.
improving
In this paper, we propose reconceptualizing RAG not merely as -aummeeed generation, but as Eetrieval-Aggmente Reasoning. We posit that RAG entails a structured reasoning process, typically involving stages such as (1) relevance assessment of retrieved documents, (2) mapping the query to specific points within the evidence, and (3) synthesizing evidence- evidence-integrated justifications. Reasoning misalignment occurs when breakdowns happen across these phases, for instance, when relevant evidence is retrieved but its content is not accurately integrated into the generated reasoning chain. These failure modes are pervasive and persist even with high-quality retrieval, remaining largely unaddressed by current methodologies.
To address reasoning misalignment, we introduce ALIGNRAG, a novel framework that employs Critique-Driven Alignment (CDA) to dynamically correct misalignments during inference using Setreval-auggeett critiques. Distinet from purppoe- generation refinement techniques [20 231. ALIGNRAC AG incorporates a contrastive critique synthesis mechanism. This mechanism is designed elicit evidence- in self-critical L.MSs (24]26] 21 20). This is achieved by training dedicated sarrivaal-augmenter Critic Language Model (CLM) on contrastive critiques. These critiques are generated by instruction- tuned LLMs and are guided by alignment signals derived from self- self-supervision or stronger external supervision. This paradigm breaks the circularity inherent in self-critical pipelines and specifically optimizes the CLM for evidence sensitivity, enabling it to reliably distinguish aligned from misaligned reasoning without propagating errors from potentially imperfect LLM feedback. At test time, ALIGNRAG iteratively refines the generated reasoning process by treating it as an optimizable artifact, transforming the RAG pipeline into an active reasoning system where critiques dynamically guide alignment with the retrieved evidence.
sridence-grounded grounded critiques and explicitly mitigate the self-bias commonly observed
To enhance the framework's practicality, we also introduce ALIGNR AG-NTO,, a more autonomous variant that eliminates the need for manual tuning of iteration counts. By training the CLM to predict a special '[oood'' token upon generating a satisfactory response, AL-GNRAG-AUTO implements a dynamic stopping mechanism at inference time. This allows the system to terminate the refinement loop as soon as the reasoning is aligned with the evidence, saving substantial computational resources while maintaining high accuracy. This "hands-free" approach makes our framework more robust, efficient, and readily deployable across diverse tasks without domain-specific adjustments.
Extensive evaluations across seven benchmark datasets and three diverse model families firmly establish ALIGNRAG's state-of-the-an (SOTA) performance, consistently surpassing existing meth- ods on a wide range of tasks. A key demonstration of its efficacy, driven by our critique learning strategy, is our 8B-parameter ALIGNR AG model outperforming self-refine approach by 12.1 % on out-of-domain (OOD) benchmarks and even much larger vanilla 72B- 22B-parameter CLM by 2.2%. ALIGNRAG's robustness shines under both informative and noisy retrieval scenarios, proving that when RAG retrieval falters, ALIGNR AG thrives. Moreover, its design as plug-and-play module ensures seamless integration into existing RAG pipelines without architectural modifications: for example, it enhanced InstruciRAG's OOD accuracy by 9.4% when applied to the Qwen2.5-14D 14BB model. These comprehensive evaluations underscore ALIGNRAG's superiority and versatility for Attrival-auggmente tasks, excelling in diverse retrieval conditions, ensuring high reasoning fidelity, and demonstrating strong generalization.
In summary, this paper makes the following key contributions: (1) We reconceptualize RAG as Retrieval-uummented Reasoning and identify Reasoning Misalignment as fundamental, understudied failure mode. (2) We introduce critique learning for RAG, novel pipeline for training CLMs to generate etrieval-augmented critiques while mitigating self-preference bias through contrastive synthesis approach. (3) We propose ALIGNR test-time framework that utilizes CDA steps to iteratively optimize the RAG reasoning process. We also present ALIGNR ALIGNRAG-AUTO an autonomous extension that dynamically determines the optimal number of refinement steps, enhancing efficiency and usability. (4) We provide extensive empirical validation demonstrating that ALIGNRAC achieves
2

---

Training Corpus Construction
Query
Retrieve
Not Relevant
Relevant
Relevant
TriviO34
Not Helpfol
Helifuul
Helpfot
Helpfol
Naturaliq4
Grandarity
Complete
Not Complete
Complete
Complete
(8) Contrastive Critique Synthesis
Preference
Prompt
Critique
Reasoning
identify the errors and hallucinations of the weak rationaie, and give your constructive criticism for...
LuM
Errors
Unexpected
Query
doc1..n
Expected
Hollucinations
Optimization
(C) Gritic LLM Training and Critique-Driven Alignment
Query: What the primary of the Great Fire of London 1666?
Lobel: The erimory couse was o fire that started in bakery on Pudding Lone.
Unexpected Response: ... rapidly due to the densely populoted area used in the buildings...
suggests
spread
Query
doc
Response
Critique
The does not cleurly that the from the bakery's oven the primory couse the fire
Critique:
Training
LLM
Unexpected Responser... The prinory couse of the Great Fire of London in 1665 fire that storted in bakery...
Expected Response
Output
Figure Overview of our ALIGNRAG Framework.
SOTA performance and significantly improves reasoning quality and robustness across various benchmarks and retrieval scenarios.
2 Reasoning Misalignment in RAG
RAG leverages external corpora 'D for grounded generation, with prior work largely focusing on improving retrieval [19 or generator robustness [3]. However, critical and underexplored challenge is the alignment of the model's intrinsic reasoning process with the specific evidential constraints from retrieved documents D. Unlike error propagation extended mathematical or code reasoning 14 27,, RAG failures often stem from inductive biases conflicting with external evidence, a qualitatively distinct problem.
We introduce reasoning misalignment as novel, RAG-specific failure mode. It occurs when the model's 's constructed reasoning path y deviates significantly from the information or relationships within retrieved evidence D, even when documents are relevant and contain the necessary facts. This distinct from factual errors, context-fres logical fallacies (28], or pure retrieval failures (P(D q) issues). Instead, it signifies breakdown in faithful evidential integration. Formally, misalignment is a structural deficiency the conditional distribution P(y q. D), where y is the generated reasoning for query y using D. It is characterized by: (1) Erosion of Evidential Priors: Reasoning inconsistent with statistical or semantic properties in DD such as P(span q, D) generated by the model differing significantly from the true salience. (2) Violation of Evidential Consistency: Deductive steps within contradict logical inferences derivable from D, i.e., I step; for some reasoning step step, E y.
We decompose RAG reasoning into three interdependent phases, each susceptible to misalignment despite ideal retrieval:
Phase I: Relevance Assessment. Misalignment occurs when the model fails to accurately gauge the relevance of specific text spans d, d D relative to the query g, effectively misestimating P(s relevant D) or assigning disproportionate weight to less relevant information 29
Phase 2: Query-Evidence Mapping. This phase is susceptible when the model struggles to correctly identify how elements q map onto information in DD representing a failure correctly derive evidential relationships R(g, D) from D based on [30].

---

Phase 3: Evidence-Integratee Synthesis. Misalignment here involves generating reasoning steps or conclusions step; E y that are not logically supported by the relevant evidence E, S D used for that step, violating the entailment E, - step;, creating inconsistencies when synthesizing information from multiple parts of 'D into coherent justification 31
This taxonomy highlights reasoning misalignment as a pervasive issue orthogonal to retrieval quality or basic factuality. Static prompting cannot dynamically correct these evidence-ground failures. To address this, we propose ALIGNRAG, a novel test-time framework enforcing evidential alignment via critique- guided alignment, offering a principled solution this fundamental RAG challenge.
3 Critique-Driven Alignment for Retrieval-Anggentted Reasoning
We present Critique-Driven Alignment (CDA), a novel test-time refinement framework designed to mitigate reasoning misalignment RAG. While conventional RAG pipelines often produce responses that partially or incorrectly reflect retrieved evidence, CDA introduces an explicit mechanism for identifying, diagnosing, and revising such failures via learned critic model. This section details our approach, outlining the problem formulation (<3.T), our structured training methodology for critique learning ($3.2). and the iterative test-time cest-time alignment process ($3.3)
3.1 Problem Setting
Given an input query q and set of retrieved documents = {di, dp our objective is to refine an initial response Yo -Mgen Mgen(1 (g, D) through iterative critique-informed updates from trained critic model Menuee To support the training of this critic model, we construct a critique supervision dataset:
S = {(yi az, Di, Ci, Yexp. Yunexp. uyunepp.)
(1)
where each instance contains the query qt, the ground-truth answer a,, the retrieved documents Di and vector c, = (r,, h, m.) orthogonal axes: relevance, helpfuiness, and completeness (related explicit links between query elements and evidence). For each instance, we include pair of responses (Yesp, Yunexp.) generated by an expert strong and a weak model, respectively. These response pairs, combined with qi and D; form a
133 capturing the quality of the retrieved context along
input representation Xpeet (g:, Di, Yexp,e Yu Using &pret, and potentially signals derived from a; and c; we synthesize the supervision signal ymneepp representing critique highlighting misalignments in Yunexp.3 relative to D, which is then used to train Meritie
3.2
Critic Training
3.2.1
Training Corpus Construction
To model the ambiguity and diversity inherent in real-world retrieval scenarios 35 we construct structured training dataset S = {(%, a, Dj, c) J.Y. where each instance includes query 9,, its gold answer a,, a retrieved document set D,, and context granularity vector c, e {0, 133. The vector encodes three orthogonal axes of contextual variation:
C₁ :r; hr, m,),,
where Relevance indicates the presence of relevant documents (derived from top-k retrieval results, augmented with irrelevant documents sampled from unrelated queries). Helpfulness h; is a binary label reflecting whether the document(s) contain answer spans corresponding to a, Completeness m, is a document-set-level binary label signifying whether D; collectively supports the full reasoning path required derive a;
To systematically simulate varied degrees of answerability, we define a multiple-tier contextual granularity hierarchy (Fig. AT), exposing critic models to diverse evidence configurations and enabling fine-graine supervision. The details of data construction could refer to Appendix A.3
3.2.2 Contrastive Critique Synthesis
We propose Contrastive Critique Synthesis (CCS), novel methodology generating evidence- grounded feedback by contrasting reasoning trajectories from language models with differing capabil- ities. Since LLMs produce sequences from conditional distributions, P(ylinput), self-critique risks

---

amplifying biases inherent in their own P.yl-). CCS counters this via dedicated Critic Language Model (CLM) trained on structured contrastive pairs Xnrr (g. D. exp Yunex)) This explicitly trains the CLM to identify misalignments w.r.t. D, learning Pentii (critiqu (pret) to capture deviations from evidential fidelity.
The CLM training employs pairwise generation protocol. It samples an unexpert response Yunexp Pweak (y)g, D) (prone to misalignment) and an expert Yexp ~ Pareng (y) q, D) (more aligned). To provide structured input for learning the conditional distribution of critiques, define a preference- augmented input tuple:
Xpref (g. D, YYes Yunexp);
which conditions the CLM's learning on both desired trajectories (Yexp) and common failure patterns
(Y "usexp) This contrastive formulation offers two key benefits for training an effective critic: (1)
constrains the learning objective for Pettii (pret) to focus on highlighting differences between
Yesp and Yunesp that are grounded in 2. promoting evidence sensitivity and faithfulness [36]; and (2)
it enables fine-grained diagnosis of specific misalignment types by analyzing path divergences [37]
The CLM learns (aniti(AAy
to generate a structured critique Ay Aymaxps strictly grounded D. Training targets aunesp are synthesized via a Critique Function F.:
Ay unexp F(Xpee)..
Mertti is trained to approximate F. The final critique format Ay umeexp is produced by an augmenta- tion operator G [output(MMann (Xnae() &espl which reformulates raw model output into constructive suggestions using Yesp as reference. This framework enables the critic to localize and explain misalignments, providing high-quality feedback for RAG alignment.
3.2.3 Critic LLM Training
To instantiate the CLM, we introduce Critique Fine-Tuning (CF7) 38 novel paradigm for training language models to produce constructive, sidence-groundee critiques. The goal is to transform base model Mbase into a proficient critic model Meritie using synthetic dataset of critiques C. Each training instance C, C is a tuple (q, D, "nnepp" Wynexpp Yexp).
CFT formulates critique generation maximizes the likelihood the model producing the correct critique Aynnep- conditioned on the full critique context Zeritic = (4. D, unexpp Yexp). Formally, the objective is:
conditional sequence generation task. The training objective
LCFT(0)
Σ C.EC
log pe (Pyyneess Ieritie),
(5)
where po is the probability distribution induced by the critic model Menties parameterized by This formulation enables the model to generate actionable, targeted feedback that improves downstream response quality. By decoupling critique generation from the target model's own outputs, CFT mitigates the self-preference bias commonly observed in iterative self-correction methods.
3.3 Critique-Driven Alignment Systems
To address reasoning misalignment in RAG inference, propose Critique-Driven Alignment (CDA). Unlike standard single-pass RAG generation, Yo = Mgen(q: D):
Yo = Mgen(g, D))
CDA reconceptualizes inference as an iterative optimization over latent reasoning space V. Each iteration leverages critiques to incrementally improve output alignment with retrieved evidence.
A learned critic Maric iteratively analyzes intermediate generations and provides critiques suggesting improvements. This yields a refinement trajectory:
Yo
CDA
CDA
CDA
Yr-
(7)
where each transition yt + Yt+1 is cattique-guidedd At step < T, the critic outputs an edit signal Ay, identifying issues in y, and proposing D-grounded revisions. Y4+1 is generated by augmenting
Mgn Mgen's 's input:
Ye+1 = Mgelly, 1 Ay,),
(8)
5

---

where @ denotes prompt augmentation with critique feedback. Ay, acts as a pseudo -randeet in discrete space, directing the generator toward D-aligned reasoning.
The final CDA output is the trajectory's terminal state:
Yfinal = CDA(q: D) = Yr-
(9)
This framework elevates alignment from static supervision to a dynamic iterative process at test-time, enabling demonstrably more reliable and evidence-grounded reasoning than existing RAG.
3.4 AlignRAG-autos Dynamic and Domain-General Alignment
We also develop AlignRAG-auto, an autonomous extension CDA that eliminates the need for manual iteration tuning and demonstrates strong cross-domain generalization. Whereas the baseline CDA requires specifying the maximum number of refinement steps T. AlignRAG-auto leverages a lightweight control mechanism that dynamically determines when alignment has been achieved. This improves both efficiency and robustness at deployment.
3.4.1 Training for Dynamic Refinement
We modify the training process for the Critic Language Model (CLM) to support dynamic inference. For each "unexpected" response Junexp: we first determine correctness relative to ground truth. Incorrect responses are labeled with [Bad] token, while correct ones are labeled with [Good]. The CLM is then trained generate the appropriate control token followed by structured critique:
po [Good/Bad]] Ay q, D, Yunesp)-
(10)
This
training objective equips the critic with both evaluative and corrective capacity. By conditioning critiques on explicit correctness judgments, the CLM learns to terminate refinement as soon as sufficient alignment achieved.
3.4.2 Dynamic Inference-Tinee Stopping
At inference, AlignRAG-aus proceeds iteratively as in CDA but introduces a dynamic stopping rule. the critic predicts [Good], the system halts refinement and accepts the current candidate:
Ve. (Mgnn @ Aye). if CLM outputs [Bad].
if CLM outputs [Good]
Me+1
(11)
This adaptive strategy avoids unnecessary iterations, saving compute while maintaining high fidelity to retrieved evidence. Empirically, we observe that only a subset of responses require multiple refinements, yielding substantial runtime reductions without compromising accuracy (see Table
4 Experiments
4.1 Experiment Setup
We evaluate method using three instruction-tuned backbones: Owen2.5-7BBInnttuee [39]. Swen2.5-14B-Instruu (39), and SasMAA.1-8B-Insse [40]. For simplicity, we refer to them Qwen2.5-7B. Qwen2.5-14B. and LLaMA3.1-8B.
Dataset. train strong critique generator, we construct 10K dataset by sampling 2K instances from each of five benchmarks: PopQA (41, TriviaQA 42 NaturalQuestions (43 2WikiMulti- hopQA 44], and ASQA [45). Furthermore, we evaluate our method on the same five in-domain benchmarks, along with two out-of-distribution (OOD) tasks, i.e., HotpotQA [46] and SQuAD [47]
Baselines. In our experiments, we compare our method against range of non-retrieval and retrieval- based baselines. For non-retrieval baselines, we include Chain-of-Thought (CoT) prompting [48 49 applied to models without retrieval augmentation. For standard RAG, we report performance from Vanilla Reasoning [3] 50], which performs step-by-step answer generation based on the retrieved passages. To assess the benefits of intermediate supervision, we include training-time refinement baselines such RetRobust 33 and InstructR [G[ Our main comparison is with these test- time refinement methods, as they share similar objectives. For test-time refinement, we evaluate Self-RAG [19] and Self-Refine, which iteratively revises outputs based on self- self-generated critique.

---

Table 1: Overall performance comparison of RAG systems employing various knowledge refinement strategies and reasoning configurations across five pneestion-answerin (QA) benchmarks. To ensure fair evaluation, all systems are tested under single-iteration test-time refinement setting. Results marked with are reproduced from [13]. Missing results in the original paper are denoted by "-". To highlight our method's impact of different model backbones, use the following color-coded notation for performance improvements: (A) represents the Qwen-2.5- -.nstruccc (A) represents the
Owen-2.5-Instie In5-Instr and (A) represents the -lama-3-Instrucee
Method Metric
NQ accuracy
MultiHopQA
TriviaQA
PApQA accuracy
ASQA str-em
accuracy
accuracy
Avg.
Baselings w/o Retrieval
Chain-of-thought
4H
0wen-2.5-lnstrukky
20.5
36.9
Qwen-2.5-lmstrruct in
48.1
49.3
72.8
25.4
31.6
45.4
42.1
41.9
61.8
26.9
25.1
40.0
Stundard RAG with Reasoning
Vanilla Reasoning
Qwen-2.55 Qwen-2.5-nnstrssk 5-Instrucc
60.22
44.7
73.2
63.7
42,8
56.9
Qwen-2.5 2.5-Lastructt
63.6
41.8
77.0
65.3
45.2
59.2
Llana-3. 3.Isuustuue
62.0
43.0
73.4
65.0
45,2
57.7
w/ Training- time Refinement
RetRobust 33
Llama-21p
39.6
$1.5
Slkana---Lnstruee
$4.2
51.7
71.5
56.5
10.5
55.5
InstructRAG [13
63.8
16.3
76.1
67.5
47.5
60.2
Qwwnn-2.5lasttrutt,
66.3
47.3
78.7
67.8
48.5
61.7
Llama- 3.1 I.sstustuuc
66.3
45.1
76.6
66.9
47.2
60.4
Test-time Refinement
Self-RAG [9)
Llama-2nu CLMM,,
42.4
35.9
68.9
55.8
30.0
46.6
Llama-22sa CLM
46.4
36.0
70.4
56.3
31,4
48.1
-inttu--nnstrucla, CLM
36,9
18.0
Self-Refine
2.en-255IIsstruuts. SELF
61.6(2)
45.0/A)
74.4(A)
(5.5(A)
(5.2(2)
58 (())
Owen-2.51lnstruztie + SELF
65.1(2))
(6.1(3)
78.01.)
67.0
47.3
60.7(3)
Shama-3.1-Insmmche SELF
61.4(2)
42.8(
(4.1(A))
((3))
(4.7(A)
57.8/A)
AlignRAG-fixed
Qwen-2.5 1.-ns2.--lsstruutt + CLMs.
65.9 (1 1.3%)
(9 4.5%)
77,8 3.4%)
65.4 (1 2,9%)
48,9 (1 3.7%)
62.1 (1 3,8%)
0wen-2.5. I.-stt,e CLMba
67.7 2.6%)
49.8 3.7%)
79.5 L.5%)
68.4 1.4%)
48.6 1.3%)
62.8 2.1%)
Llama- Insstrucct, CLMM.
65.3 3.9%%
4.2%%
2.9%)
0.4%)
47.1 (1 2.4%)
60.6 (1 2.8%)
Evaluation metrics. Following previous work [1I, we adopt the official correctness metric (str-em) for ASQA 45 and use accuracy for the other tasks, which measures whether the final generations
of the model align with the ground-truth 41 52
as the backbone and fine- tune it using LoRA for parameter--fffcient training. Moreover, the strong LLM we use to generate expected responses is 1.aMAA,1--B-Instuue and the weak LLM we use to generate unexpected responses is Qwen2.5500.5-IInstrue
Implementation Details. For the CLMM, we adopt
[39 We set the retrieval Top-K to 5 for each question.
4.2 Main Result
Table shows the overall performance of our method and the baselines in various families and sizes of the base model five benchmarks. And we provide all the additional results in Appendix [A.6]
First, compared to non-retrieval baselines such as Chain-of-Thougeee (CoT) prompting, all retrieval- augmented methods achieve significantly better performance, demonstrating the importance of incor- porating relevant external knowledge. Second, we observe further gains when applying training-time refinement methods. In particular, InstructRAC AG achieves strong performance across all backbones, outperforming Vanilla RAG by large margin, confirming the value of training refinement strategies.
Notably, ALIGNRAG achieves the best overall results on all three backbones compared to other test-time refinement methods. It surpasses Self-R RAG and Self- Relf-Refine by notable margins, achieving an average accuracy of 62.8% compared to 48.1 1% and 60.7%, respectively. The performance improvement consistent across all benchmarks, highlighting both the effectiveness and the strong generalization of our approach. This demonstrates that our rritique-driven alignment strategy can better guide the reasoning process and overcome the limitations of purely self-generated feedback.
Analysis of AlignRAG-aute The experimental results, detailed in Table 2 reveal a compelling comparison between the fixed-iteration and autonomous alignment strategies. Notably, AlignRAG- auto consistently achieves performance on par with, and in many cases slightly superior to, its
7

---

AlignRAG-fixed counterpart across all datasets and model sizes, For instance. on the NQ and ASQA benchmarks, AlignRAG-aan demonstrates clear improvements, suggesting its dynamic termination mechanism is highly effective.
the autonomous variant does not sacrifice accuracy for efficiency. In- stead, it demonstrates that the Critic Language Model is well-calibrated to dynamically deter- mine the optimal number of refinement steps, leading to robust performance without the need for manual hyperparameter tuning. This vali- dates AlignRAG-aus as more practical and efficient framework for cal-world deployment.
This is crucial finding:
Figure 2: Performance Comparison: AlignRAG- fixed (1 iter.) vs. AlignRAG-auto
AliguRAG-lise (1 iter.)
AlignRAG-auto
Dataset
7B
14B
8B
14B
PopQA TrivinQA
66.5
68.4
68.4
67.6
68.1
68.3
77.8
79.5
79.9
NQ MuliiMuliHPPAA
65.3
65.9
67.7
66.8
67.3
69.0
49.5
49.8
47.6
49.3
50.2
ASQA
47.1
48.9
48.6
48.8
49.6
49.8
4.3 Analysis
Note: All subsequent experimental analyses
are based on AlignRAG-fixed. For convenience, we'll use AlignRAG instead.
Generalization to OOD Scenarios. To assess the generalization capability of our method beyond the domains seen during training, we conduct out-of-distribution (OOD) evaluations
on two widely-used and challenging bench-
marks that are held out from the training set, i.e.
Average Performance (%)
100
HotpotQA [46] and SQuAD [47. This evalua- tion setting enables to examine how well the model transfers its reasoning and alignment abil- ities to novel domains. As shown in Figure 3 we compare ALIGNRAG with two baselines. ALIGNR AG consistently achieves the lowest performance drop across all backbones, outper- forming both baselines by large margin. For example, on LLaMA3. 1-8B, ALIGNRAG re- duces the performance drop to 32.4% compared
bution
57.9
61.5
40
41.4
20.4
20
Vanill RAG
Salf-Refiwe
Ours
Figure 3: Drop average OOD performance com- pared to average In-Domain performance.
to 41.6% for RefinR.fine. These results demonstrate that our CDA mechanism not only improves in-domain reasoning but also enhances robustness under domain shift.
Nundar Buntora
Tanderal Innias
Narter Brmbors
Nurbar clinniot
Narda O Taraeess
N.roe Cariion
X01
Inmoora
Ienturr
Number Inndors
N.roe
Figure Performance comparison of ALIGNRAG and Self-Refine across five refinement iterations.
Scalable Test-time Reasoning via Iterative Alignment Strategy. To assess test-time scalability of different refinement methods, we plot accuracy over five refinement steps across seven benchmarks, comparing the performance of ALIGNRAG with the Self-Refine baseline (Figure The curves reveal two notable trends. First, both methods generally benefit from iterative alignment, with accuracy improving on most tasks as the number refinement steps increases. This indicates that reasoning can scale with additional refinement steps. However, we occasionally observe slight degradation beyond certain point, which we attribute to potential noise accumulation or overcorrection during excessive iterations. Second, ALIGNRAG consistently outperforms Belf-Refine across all iterations

---

and benchmarks with notable margins. These findings demonstrate that ALIGNRAG not only enables scalable reasoning but also provides more stable and robust improvements.
When RAG Retrieval Falters, ALIGNRAG Thrives. performance degradation when retrieved doeu- ments lack pertinent answers, a prevalent yet insufficiently characterized failure mode termed Noisy Retrieval. Even in the absence of explicit answers within the retrieved corpus, ALIGN- RAG adeptly filters distractors to isolate under- lying reasoning signals while concurrently ac- tivating complementary parametric knowledge inherent to the base model.
RAG systems are susceptible to substantial
Psy04
Oum
Veria TAC
Thura (X0A
NeureOuerter
This dual strategy, as demonstrated in Figures 5a and [5]] empowers robust reasoning even der these adversarial conditions. Consequently, ALIGNR (Figure 5ab significantly outper- forms conventional methods like Vanilla RAG and Relf-Refine, which falter in such scenarios, by effectively transforming imperfect external retrievals and inherent model understanding into actionable insights.
(a) w/o answer.
(b) w/ answer.
Figure 5: ALIGNRAG performance under Noisy Retrieval (a) and Informative Retrieval (b) scenar- ios.
Superior Alignment Guidance Surpassing Strong LLM Baselines. We examine the ef- fectiveness of using a high-capacity Strong LLM, i.e., Qwen2.5-72B - in our Contrastive Critique Synthesis (CCS) framework. Specifi- cally, we compare three CLM supervision strate- gies under three backbones: (1) Vanilla Cri- tique Synthesis, where the CLM is trained with critiques directly generated by Qwen2.5-72B- Instruct, (2) directly using the 72B model as the CLM, and (3) our CCS-based CLM trained with critiques synthesized by the 72B Strong LLM. As shown in Figure 6] our method consistently outperforms vanilla critique synthesis, validating the benefit of introducing contrastive reasoning signals. Remarkably, our method even surpasses the directly supervised 72B CL.M in all backbones, suggesting that contrastive critique training enhances generalization and reduces reliance on model scale. These results demonstrate the strength of our approach in leveraging powerful LLMs for scalable and efficient CLM training.
Wall Ca kai CUN Bu
OMNRad 5-16
Figure 6: Comparison of different CLM supervi- sion strategies.
Integrate as a Plug-and-play Module into Exist- ing RAG Pipelines. To evaluate the generality and plug-and-play nature of our method, we integrate it into the InstructRAG framework across three back- bones. Table 2 reports the performance under both In-Domain (ID) and Out-of-Domain (OOD) evalua- tion. We observe consistent improvements in both familiar and unseen distributions. The variant with alignment significantly outperforms the original In- structRAG (13), demonstrating that our method can be incorporated into existing RAG pipelines in a zero-modee poro-modification, test-time manner, highlighting its strong compatibility and practical utility.
Table 2:
Combination of training-tine time (In- StructRAG) and test-time alignment.
Method
ID (avg.)
OOD (avg.)
Qwen2.5-78
InstructRAG
39,5
2.5
w/ Alignment
61.5
2.0%)
30.1 (1 2.1%)
w/ Alignment
63.0 3.5%)
31.7 (1 3.7%)
Qwen2.5.14R
InstructRAC
GIT
24.9 (TS)
w/ Alignment
62.5 (? 0.8%)
33.4 (1 8.5%)
w/ Alignment'
63.9 2.2%)
34.3 (1 9.4%)
LLMMA3 1-88
InstructRACi
60.4
28, TA))
w/ Alignment
61.7 (1 1.3%)
29.4 (1 1.0%
Alignment
61.9 (7 1.5%)
30.5 (1 2.1%)
4.4 Ablation Study
'CLM trained with critiques synthesized by Qwen2.5572B-Innste

---

We perform an ablation study seven benchmarks to iso- late the effect of the Critic Language Model (CLM) and our contrastive critique synthesis (CCS). Specifically, we com-
N Aworge Peromanne (N)
pare four settings: (1) Vanilla RAG, which generates with- out critique; (2) RAG Frozen CLM, where pretrained LLaMA3. 1-BB serves as an untrained critic; (3) RAG + Trained CLM, where the CLM is fine-tuned on critiques over weak LLM responses, but without contrastive signal; (4) RAG Trained CLM (CCS), our full method with contrastive critique synthesis.
Vealla
Proee CLU
Transa cLd ccc
LLa
Qwan2667B
Owent 5-14B
Figure 7: Ablation study on the CLM and our CCS.
Figure 7 shows that even a frozen CLM consistently im- proves over Vanilla RAG, confirming the utility of auxiliary critique. Training the CLM with weak LLM critiques yields further gains by capturing recurring error patterns. The largest improvements come from CCS, which explicitly contrasts expert and weak responses. For example, on Qwen2.5-7B, CCS raises average accuracy from 49.6% to 52.9%. Complete results are presented in Table (Appendix).
Related Work
enhances LLMs by grounding generation in external knowledge. While significant efforts have focused on improving retrieval accuracy [60-64] or training better generators (85)68), a persistent challenge is the presence of noisy irrelevant retrieved content. To mitigate this, recent methods filter or denoise context, predominantly via training-lime optimizations [15 66H42), such as [3] self-supervised denoising. However, these approaches are limited by their static nature and fail to address dynamic eтrоr propagation during inference [14]
RAG [53 59]
InstructR
AG's AG's
Even with accurate retrieval, the generated reasoning may deviate from the evidence. We identify this reasoning misalignment as a critical, underexplored RAG failure mode. Prior attempts like Self- RAG [99 introduce special tokens to control reasoning but require architectural modifications. In contrast, our novel critique-Drived Alignment (CDA) is a test-time method that dynamically realigns reasoning with evidence without modifying the base model architecture, offering plug-and-pi play alternative to training-heavy variants. Furthermore, unlike general self-refinement techniques (7375), propose novel critique learning paradigm training dedicated Critic Language Model (CLM) specifically for evidence-gee -rruunddd critiques. Crucially, this paradigm explicitly mitigates the self-bias
inherent in self-critiquing L.LMs 26 21 20). Unlike external verification methods [76] 77 that
act as post-hoc filters, our trained CLM actively optimizes the evidence- evidence-grounded reasoning process as an artifact during inference, enabling dynamic alignment with retrieved knowledge.
Conclusion
We present ALIGNRAG, an iterative framework that reframes Retrieval-Augmentee Generation as Retrieval-Augmented Reasoning to tackle the overlooked challenge of Reasoning Misalignment. Its key innovation, Critique-Driver Alignment (CDA), uses specialized Critic Language Model (CLM)-trained via contrastive critique synthesis-to boost evidence sensitivity and correct reasoning errors at inference. ALIGNRAC sets new -tate-of-the-art. This principled approach advances the reliability and faithfulness of retrieval- -aggeeted systems.
Acknowledgements
This project was fully supported by the Shanghai Artificial Intelligence Laboratory (S.S.). This work was done during Jiaqi Wei's internship at Shanghai Artificial Intelligence Laboratory.
10

---

References
[1] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- -uumeettee generation for mnowledge-intensing NLP tasks. In Proceedings of NeurlPS, 2020.
[2] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning. pages 9929-3938. PMLR, 2020.
[3] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmentt generation for wwwpwllede-intensiiee NLP tasks. Advances in Neural Information Processing Systems, 33:9459-94477 2020.
[4] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43, 2023.
[5] Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don't trust chatgpt when your question is not in english: a study of multilingual abilities and types of Ilms, arXiv preprint 230.::23005.6339 2023.
[6] Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah Smith, Yejin Choi, Kentaro Inui, et al. RealTime QA: What's 's the answer right now? Advances in Neural Information Processing Systems, 36, 2024.
[7] Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang. Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang. Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen tau Yih, and Xin Luna Dong. CRAG - compre RAG benchmark. arXiv preprint arXiv:2406.04744 2024,
[8] Xiang Zhang, Juntai Cao, Jiaqi Wei, Yiwei Xu, and Chenyu You. Tokenization constraints in Ilms: A study of symbolic and arithmetic reasoning limits, arXiv preprint arXiv:2505.14178. 2025.
[9] Shuyan Zhou, Uri Alon, Frank Xu, Zhengbao Jiang, and Graham Neubig. DocPrompting: Generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2022.
[10] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.0m 095/6, 2025.
[11] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmenned black-box language models. arXiv preprint arXiv:2301.12652, 2023.
[12] Hongjin Su. Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang. Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. BRIGHT: A realistic and challenging benchmark for reasoning-intensinn retrieval. arXiv preprint arXiv:2407.12888 2024.
[13] Zhepei Wei, Wei-Lin Chen, and Yu Meng, Instructrag: Instructing eettrivall-uggeeeee genera- tion via setf-synthesized rationales. arXiv preprint arXiv:2406. 13629, 2024.
[14] Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266, 2025.
[15] Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. RAG vs Fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint 200.0:2401.08446, 2024.

---

[16] Jinghan Zhang, Xiting Wang. Weijieying Ren, Lu Jiang, Dongjie Wang, and Kunpeng Liu. RATT: A thought structure for coherent and correct LLM reasoning. arXiv preprint arXiv2446600746, 2024.
[17] Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. RAFT: Adapting language model domain specific RAG, arXiv preprint 1rXiv:4403.10131 2024.
[18] Zhengren Wang, Jiayang Yu, Dongsheng Ma, Zhe Chen, Yu Wang, Zhiyu Li, Feiyu Xiong, Yanfeng Wang, Linpeng Tang, Wentao Zhang. et al. Rare: eetrieval-augmente reasoning modeling. arXiv preprint arXiv:2503.23351 2025.
[19] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023.
[20] Qingjie Zhang, Han Qiu, Di Wang. Haoting Qian, Yiming Li, Tianwei Zhang, and Minlie Huang. Understanding the dark side of Ilms' intrinsic self-correction. arXiv preprint arXiv:2412.14959 2024.
[21] Ting Wu, Xuefeng Li, and Pengfei Liu. Progress or regress? self-improvement reversal in post-training, arXiv preprint arXiv:2407.0500 00000, 2024.
[22] Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebac7, and Jan Leike. Llm critics help catch Ilm bugs. arXiv preprint arXiv:2407.00215 2024.
[23] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXtv:2406.07496. 2024.
[24] Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Pride and prejudice: Lim amplifies self-bias in self-refinement. arXiv preprint arXiv:2402. 2024.
[25] Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in llm-as-a-judge. arXiv preprint arXhv:2410.21879, 2024.
[26] Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in Ilm-as-a-judge. arXiv preprini arXiv:2502.0155 2025.
[27] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thouge for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025.
[28] Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for ol-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884, 2024.
[29] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval Retreeval-augmentte black-box language models. In Proceedings of NATCL-HLT, pages 8371-8384, 2024.
[30] Bingyu Wan, Fuxi Zhang, Zhongpeng Qi, Jiayi Ding, Jijun Li, Baoshi Fan, Yijia Zhang, and Jun Zhang. Cognitive-aligned document selection for retrieval-augmenned generation. arXiv preprint arXiv:2502.11770 2025.
[31] Yan Weng, Fengbin Zhu, Tong Yc, Haoyan Liu, Fuli Feng, and Tat-Seng Chua. Optimizing knowledge integration in vetrieval-augmeente generation with self-selection. arXiv preprint 20X.v22502.06148, 2025.
[32] Juntai Cao, Xiang Zhang, Raymond Li, Chuyuan Li, Chenyu You, Shafiq Joty, and Giuseppe Carenini. Multi2: Multi-agent test-time scalable framework for multi-document processing. arXiv preprint arXiv:2502.20592, 2025.
12

---

[33] Ori Yoran, Tomer Wolfson, Oni Ram, and Jonathan Berant. Making retrieval- augmented language models robust irrelevant context. In The Twelfth International Conference on Learning Representations, 2024.
[34] Zhilin Zhang, Xiang Zhang, Jiaqi Wei, Yiwei Xu, and Chenyu You. Postergen: Aesthetic-aware paper-to-poster generation via multi-agent Ilms. arXiv preprint arXiv:2508. 17188, 2025.
[35] Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. Enhancing noise robustness of retrieval- augmented language models with adaptive adversarial training. arXiv preprint arXiv:2405. 209788 2024.
[36] Xiang Zhang, Juntai Cao, Jiaqi Wei, Chenyu You, and Dujian Ding. Why does your cot prompt (not) work? theoretical analysis of prompt space complexity, its interaction with answer space during cot reasoning with Ilms: A recurrent perspective. arXiv preprint arXiv:2503.10055 2025.
[37] Shuai Zhao, Linchao Zhu, and Yi Yang. Learning from reference answers: Versatile language model alignment without binary human preference data. arXiv preprint NXXIv:2504.09895, 2025.
[38] Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuming: Learning to critique is more effective than learning to imitate. arXiv preprint arXiv:2501.11700 2025.
[39] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rXiv:2412.15115. 2024.
[40] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[41] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pages 9802-9822, 2023.
[42] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611 1, 2017.
[43] Tom Kwiatkowski. Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[44] Xanh Ho, Anh-Khoo Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. 2020.
[45] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. ASQA: Factoid questions meet long-fore answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273-8288, 2022.
[46] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint vrXiv:1809009600 2018
[47] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1606.0600.005200 2016.
[48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24833 2022.
13

---

[49] Xiang Zhang, Muhammad Abdul- Mageed, and Laks VS Lakshmanan. Autoregressivet chain of thought= recurrent: Recurrence's 's role in language models' computability and a revisit of recurrent transformer. arXiv preprint wrXiv:2409.09239, 2024.
[50] Xiang Zhang, Bradley Hauer, and Grzegorz Kondrak. Improving hownet-based chinese word sense disambiguation with translations. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 4530-4536, 2022.
[51] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 0465-6488, 2023.
[52] Timo Schick, Jane Uwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zeitlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Phirty-sevente Conference on Neural Information Processing Systems, 2023.
[53] SU Hongjin, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirteenth International Conference on Learning Representations, 2024.
[54] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. preprint arXiv:2312 10997, 2023.
generation for large language models: A survey, arXiv
[55] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206-2240. PMLR, 2022.
[56] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404. 161.30600., 2024.
[57] Tianyu Fan, Jingyuan Wang, Xubin Ren, and Chao Huang. Minirag: Towards extremely simple retrieval-augmenttd generation. arXiv preprint arXiv:2501.061 13,1,,100 2025.
[58] Huatong Song. Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592 2025.
[59] Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, al. Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint Sidi:2200.00225: 2024.
[60] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6789, 2020.
[61] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Sub- ramanian, Evelina Bakhturina, Mohammad Shocybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference Learning Representations, 2024.
[62] Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust RAG against retrieval corruption. arXiv preprint 2rXi::2405.15556 2024.
[63] Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. Poisoning retrieval corpora by injecting adversarial passages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13764-13775, 2023.
14

---

[64] Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. PoisonedRAG: Knowledge poi- soning attacks aetrietll-ugmmented generation of large language models. arXiv preprint arXiv:2402.07867, 2024.
[65] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. RAGCache: Efficient knowledge caching for carieval-augmented generation. arXiv preprint arXiv:2404. /2457, 2024.
[66] Xiang Zhang. Senyu Li, Ning Shi, Bradley Hauer, Zijun Wu, Grzegorz Kondrak, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Cross-modal consistency in multimodal large language models. arXiv preprini 1rXiv:2411.09273. 2024.
[67] Yuhan Liu, Hanchen Li, Kuntai Du, Jiayi Yao, Yihua Cheng, Yuyang Huang, Shan Lu, Michael Maire, Henry Hoffmann, Ari Holtzman, et al. CacheGen: Fast context loading for language model applications. arXiv preprint arXiv:2310.072400 2023.
[68] Chenyu You, Haocheng Dai, Yifci Min, Jasjeet S Sekhon, Sarang Joshi, and James S Duncan. Uncovering memorization effect in the presence of spurious correlations. Nature Communica- tions, 16(1):5424, 2025.
[69] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval. arXiv preprint arXiv:2401.18059 2024.
[70] Puyuan Liu, Xiang Zhang, and Lili Mou. A character-level length- algorithm for non- autoregressive sentence summarization. Advances in Neural Information Processing Systems, 15:29101-29112 2022.
[71] Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, and Ge Yu. Rankcot: Refining knowledge for aetrievll-augment generation through ranking ch thoughts. arXiv preprint arXiv:2502. 17888, 2025.
[72] Zhicheng Lee, Shulin Cao, Jinxin Liu, Jiajie Zhang, Weichuan Liu, Xiaoyin Che, Lei Hou, and Juanzi Li. Rearag: Knowledge-guine reasoning enhances factuality of large reasoning models with iterative retrieval augmented generation. arXiv preprint arXiv:2503.21720 2025.
[73] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with seld-fke Advances in Neural Information Processing Systems, 36:46534-46594, 2023.
[74] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8633 8652,, 2023.
[75] Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang, Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, arXiv preprint arXiv:2406.072 07394, 2024.
[76] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint 2rXiv:2402.06457, 2024.
[77] Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang, Zenan Zhou, and Wentao Zhang. Mm-verify: Enhancing multimodal reasoning with chain-of-thoughe verification. arXiv preprint arXiv:2502. 13383, 2025.
[78] Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. FlashRAG: A modular toolkit for efficient retrieval-augeee aaugenel- generation research. arXiv preprint arXiv:2405.133: 13576, 2024.
[79] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 01255728-537444 2023.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions, such as reconceptualizing RAG as Retrieval Augmentte Reasoning, identifying Reasoning Mis- alignment, proposing AlignRAG with its Critique-Driig Driven Alignment (CDA) and contrastive critique synthesis mechanism. These claims are supported by the experimental results presented, particularly in Section and summarized in the conclusion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
16

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: While it uses established mathematical formulations for its components, does not present theoretical results in the form of theorems requiring formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
17

---

Justification: Section 5.1 "Experiment Setup" and Appendix A ("Experimental Details") provide extensive details. Appendix A.2 further provides hyperparameters for CLM fine- tuning (e.g., learning rate, batch size, optimizer, LoRA configurations). This level of detail should allow for high degree of reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper explicitly states that the code for the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (nttps://nips.co for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc/publll//
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
18

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 (Experiment Setup) and Appendix A (Experimental Details, especially A.2 Implementation Details and A.3 Training Corpus Construction) provide comprehensive details data sources and splits (standard benchmarks used, and the construction of the new critique dataset detailed), hyperparameters for CLM training (e.g.. learning rate, batch size, LoRA rank, alpha, dropout), and the optimizer used (AdamW).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While the paper presents extensive empirical comparisons across multiple benchmarks, each primary experiment conducted single time obtain the final results due to cost consideration.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
19

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Appendix A.2 details the implementation details. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::/neuripps http:///ww..cc/tacccssii........
Question:
Answer: [Yes]
Justification: The research focuses improving the factual reliability and reasoning align- ment large language models RAG systems using publicly available datasets and models. There no indication of human subject experimentation, data privacy violations, other ethical concerns that would conflict with the NeurlPS Code of Ethics. The aim is to enhance model performance in a way that could lead more trustworthy Al.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
The paper does propose the release of new large-scale pretrained language models or datasets scraped from the web that would typically carry high risk for misuse. The primary new model component is fine-tuned Critic Language Model (CLM) and specific critique dataset. Therefore, safeguards for high-risk veneral-purpon model releases are not directly applicable.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and baseline models are properly cited, with their respective licenses acknowledged in the text.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
asseet are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithcodd com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
21

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces a new model architecture and provides open access to the code via a link the abstract. is assumed that code repository includes sufficient documentation alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research described does involve crowdsourcing or direct experiments with human subjects for data collection or evaluation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects; therefore, IRB approval or discussion of risks to participants is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
The LLM usage is declared in the experimental setups.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer our LLM policy http::///eerrrpss.cccooorrrrrees for what should or should not be described.
23

---

A Appendix
A.1 Linitations
While ALIGNRAG represents a sigmificant step forward in retrieval-augmentee reasoning through critique- gotlueegueedd optimization, several limitations remain. Although the framework improves robust- ness to noisy or partially irrelevant retrieved content, its effectiveness may diminish under extreme retrieval failure, specifically, when none of the retrieved documents are relevant the query. In such cases, the quality of rerrieval-augmenter critiques deteriorates, as they are inherently dependent on the informativeness and accuracy of the retrieved evidence. Consequently, the CLM's capacity to steer the generator toward faithful reasoning may be constrained. The interaction between retrieval quality and the model's corrective ability remains an open area for further exploration. Moreover, despite being designed as a plug-and-play module, optimal integration of ALIGNR may require minor tuning depending on the generator LLM and the underlying RAG architecture. Standardizing adaptation protocols for deploying the CLM across diverse pipelines could enhance its case of adoption and generalizability. Despite these limitations, ALIGNRAG establishes robust foundation for improving evidence alignment in RAG systems and opens promising avenues for future research.
A.2 Additional Implementation Details
Retrieve Setup. We use the Wikipedia corpus provided by [78] as the default external knowledge source for retrieval. We evaluate our method on seven diverse QA benchmarks spanning multiple task types, including standard factoid QA, nulti-hop reasoning, and long-form generation. PopQA [41], TriviaQA 42], NaturalQuestions 43), and SQuAD [4] fall under standard factoid QA, where models answer factual questions based on Wikipedia web-based evidence.
PopQA focuses on entity-centric questions derived from structured knowledge bases, testing factual recall over encyclopedic content.
TriviaQA contains trivia-style questions authored by enthusiasts, each paired with multiple distant-supervisce evidence documents.
NaturalQuestions presents real user queries issued to Google Search, with answers extracted from Wikipedia, simulating realistic search behavior.
ASQA 45 is a long-form QA benchmark focused on ambiguous questions with paragraph- level answers.
2WikiMuldHopPAA
44 and HotpotQA 46 are multi-hop QA datasets that require rea-
soning over multiple passages. 2WikiMultiHopQA evaluates compositional reasoning across two Wikipedia articles, while HotpotQA incorporates both supporting and distracting sentences, encouraging interpretable multi-step reasoning.
SQuAD is widely used extractive QA dataset where answers are short spans from Wikipedia passages.
Following the setup in InstructRAG [13] we adopt dataset-specific retrievers for each query: Contriever-MS MARCO for PopQA and TriviaQA. DPR for NaturallQuestions, GTR for ASQA, and BM25 for 2WikiMultiHopQA, For HotpoiQA and SQuAD, we adopt the e5-base-v? encoder. By default, we retrieve the top most relevant documents from Wikipedia corpus for each question.
Training Details. We tne-tune our models using the LoRA method on 2 NVIDIA A100 GPUs, each with 80GB of memory. The fine-tuning process conducted over 2 epochs with learning rate of le-5 using the AdamW optimizer and employs a per-device batch size of 16, leveraging gradient accumulation to handle larger effective batch sizes. We set the LoRA--peccis follows: tora_rank 16 and fora_alphaa = 64, ensuring efficient adaptation to downstream tasks. The sequence cutoff length 6144 tokens, with warmup ratio of 0.1 applied to stabilize training. Additionally, we utilize 6f16 (brain floating point) precision to reduce memory usage and accelerate training while maintaining numerical stability.
hyperparameters as
A.3 Training Corpus Construction Details
To systematically simulate varying degrees of answerability, we introduce novel four-tier Contextual Granularity Hierarchy (Figure which forms the basis for structured aontext-aware critique learning.
24

---

This hierarchy is designed to expose critique models to a broad spectrum of evidence conditions, thereby facilitating fine-gre ane-grained supervision under explicitly controlled scenarios.
The hierarchy is defined along three orthogonal dimensions of contextual variation- helpfulness, and completeness-anne and comprises the following four levels:
Hierarchy-I: 1: Not Relevant, Not Helpful, Not Complete. We sample 200 instances per benchmark, where the context is randomly selected from evidence retrieved for unrelated questions. These contexts are neither topically relevant nor contain partial answers, thus offering no utility in addressing the question.
Hierarchy-2. Relevant, Not Helpful, Not Complete. We sample 400 instances per benchmark, in which the retrieved context comprises the top-5 documents relevant to the question but lacking any content that supports a correct answer. Although relevant, these contexts remain unhelpful and incomplete.
Hierarchy-3 & 4: Relevant, Helpful, Not Complete Complete. We sample 1,400 instances per benchmark where the context is both relevant and helpful, containing either partial (no single document provides complete answer) or complete answer-supporting information. To capture varying levels of difficulty, we categorize queries into five tiers based on the number of documents that individually contain supporting evidence (ranging from 5). Easier queries correspond to higher number of such documents. We sample 400, 400, 200, 200, and 200 instances across these five levels, respectively.
This hierarchical corpus introduces a novel fine-grained supervision signal for training critique models, enabling more nuanced understanding of answerability and evidence quality in generation.
A.4 Critic LLM Training via CPO
In addition to Critique Fine- Fine-Tuming (CFT), we introduce novel training paradigm for critique language models, termed Critique Preference Optimization (CPO). CPO extends the Direct Preference Optimization (DPO) framework (79] to the domain of critique generation, enabling preference-based alignment of critique models with respect to human-quality judgments.
For each training example, we construct a pair of candidate critiques: a rejected critique A generated by a weaker model Mweak. and an accepted critique A from stronger model Matrong- The critic model Mantie is then optimized to prefer the stronger critique over the weaker one using ranking-basn based objective:
Po Po(Ayiteny g. D, Vunexp))
]]
Cero -Ee logo
B Blog
(12)
po Dynnexp
g. D, Yunexp)
where (.) denotes the sigmoid function, ,8 is temperature parameter controlling preference sharp- ness, and py is the conditional likelihood of a critique under the model. Importantly, the conditioning includes the stronger generation Yuteap to ground the critique in high-quality reference behavior.
This training strategy represents a novel application of preference optimization the critique generation setting. It allows the model to learn fine- ine-grained distinctions in critique quality and improves alignment with human preferences, surpassing traditional supervised learning approaches in adaptability and scalability.
A.5 Pseudo-code of Novel Algorithms for Critique-Aware Learning
To promote clarity and reproducibility, present formalized pscudo-code for the core contributions of our framework, highlighting novel procedures for critique generation, fine-tuning, and align- ment. These algorithmic components reflect our key innovations in critique-aware generation and optimization.
Algorithm Iintroduces Contrastive Critique Synthesis, a novel mechanism that elicits actionable critiques by contrasting outputs from a weak and a strong model. This facilitates the identification of failure modes in weaker generations using preference-informed critique models. Algorithm 2 Critique Fine-Tuning (CFT), formalizes supervised learning regime using synthetic critiques and structured input templates to fine-tune a base model toward producing useful critiques.
25

---

we present Critique Preference Optimization (CPO), which extends the Di- rect Preference Optimization (DPO) framework to critique generation. This formulation enables preference-based alignment of critique models using pairs of more and less preferred critiques. Lastly, Algorithm I describes Critique-Driven Alignment (CDA), a novel iterative refinement procedure that integrates critique signals into the generation loop, producing responses that are successively improved based on model-generated feedback.
In Algorithm 3
Collectively, these algorithmic components define unified, modular framework for critique-aware alignment, marking novel contribution controllable and preference-aligee language model training.
26

---

Algorithm CONTRASTIVE CRITIQUE SYNTHESIS (Novel critique generation via response com-
parison)
Require: Input query y, contextual grounding D, weak model M Mweak strong model Mstrongg critique model Meritic
Ensure: Generated critique Dyucex for weak model output
1: Buncxp t- --weakk4 (q. D)
> Generate suboptimal response
2: Dexp MMstnnng (g. D)
Generate preferred response
3: Aprel < (q, D, Mexp, Vumexp)
Construct preference-informe input Generate contrastive critique
4: Yunexp f- Meritie (Krret)
5: Yunexp <- G Wynneep Mexp)
> Refine critique with improvement guidance
6: return Dyneep
Algorithm 2 CRITIQUE FINE-TUNING (CFT): Supervised adaptation via synthetic critiques
Require: Base model Mweaks synthetic dataset C, template Teritie learning rate 7). epochs N Ensure: Critique-aware model Mentic
MMritii < -Meak
Initialize from weak model
2: for epoch = to N do
3:
for each (g, D, Yunexps Yunesps Yesp) E do
4:
Teritic (u. D. Yunexp: Mexp)
Compose critique context & Predict critique D Compute NLL loss Update model
5:
Yuuues po (- Teritie)
6:
LCFT log po Yunexp Teritic)
7:
4 0 V CCFT
8: return Maritie
Algorithm 3 CRITIQUE PREFERENCE OPTIMIZATION (CPO): Alignment via pairwise critique preferences
Require: Queries {4}, contexts {D}, weak model Mweak strong model Mstrong, initial model Mentiic temperature
Ensure: cefffrence-aligneen critique model Mettie for each (q. D) do
2:
Aunees f- Mweak (q. D)
Infer less less- pesssereeerdd critique
3: A f- -Mstrong (g, D)
Infer preferred critique
4: P (Auuncpp: A
Construct preference pair
5: for epoch 1 to N do
6: for each = (Ay D do
Compute preference loss Looo 8: 0 - V Lpo
7:
Direct Preference Optimization loss Update parameters
9: return Meriti
Algorithm CERTIQUEE-DRIVEN ALIGNMENT (CDA): Iterative refinement via model-generated critique signals
Require: Query q, document set D = {di,-. ) generation model M Myen- critique model Meritie
iterations T
Ensure: Refined, critique-aligned response yexp
1: Yo MMgnn( (q. D)
> Initial retrieval-augmented generation
2: for = to 7 do
3: Dult +- Menie q, D)
I Critique current response > Refine using critique
4: Vi+1 t Mgeal (Ve + Ay1, q, D)
5: Yeep f D/
Final critique-aware output
6: return Vexp
27

---

A.6 Additional Experiment Results
In this section, we present additional experimental results to provide a comprehensive understanding of the proposed method and its performance under various conditions.
Generalization to Out-of- Disttribution Data.
To supplement the OOD generalization results in Figure 3 [able/Opprovieer complete breakdown of ID and OOD performance across benchmarks and backbones. While the main text reports average performance drops between ID and OOD settings, the detailed analysis reveals that ALIGNRAG reduces the OOD drop significantly (e.g., from 40.3 to 32.2 on Qwen2.5-7B and from 44.0 to 33.1 on LLaMA3.1 1-B) compared to
Additionally, ALIGNRAG achieves substantial absolute gains on OOD datasets (e.g., +12.4 on HotpotQA and +11.5 on SQuAD for Qwen2.5-7B), demonstrating improved generalization capabilities under domain shifts. These results confirm that the proposed CDA-based alignment strategy enhances model robustness across distributions without overfitting to the training data.
Robustness under Retrieval Quality Variance. To evaluate robustness under varying retrieval conditions, we compare Vanilla RAG, SELF-REFINE, and ALIGNRAG in two retrieval scenarios: (Informative) and (Noisy). Figure DareeSSunnn summarizess the results. In the Noisy scenario, where noisy or misleading retrieval often causes reasoning misalignment, ALIGNR AG consistently outperforms the baselines. For example, on NaturalQuestions (e.g.. +5.6 on (wen2.55-7B) and 2WikiMultiHopQA (c.g., +3.9 on LLaMA3.1-88 achieves the largest margins over SELF- SELF-REFINE. Even in the Informative scenario, where retrieved documents are highly relevant, ALIGNRAG demonstrates superior accuracy (e.g.. +1.9 on ASQA and +4.7 on 2WikiMultHHoQAA using Qwen2.5-14B) These results illustrate that ALIGNRAG enhances reasoning robustness across a wide range of retrieval quality levels.
two tasks particularly sensitive to retrieval Quality-ALINRRAA
Integration into Existing RAG Pipelines. To assess the plug-an and--addaa compatibility of our align- ment strategy, we integrate it into the INSTRUCTRAG framework across three backbones and evaluate its performance seven benchmarks. The detailed results, provided in Figure g reveal that our alignment approach consistently improves accuracy, both for in-domain datasets (e.g.. PopQA, Triv- iaQA) and OOD datasets (e.g., SQuAD, HotpotQA). Notably, the improvements pronounced on challenging datasets such as SQuAD (+10.2 on Qwen2.5-14B) and HotpotQA (+8.6 on Qwen2.5-14DB when leveraging the 72B model in our CCS pipeline. These results demonstrate that our method can be seamlessly incorporated into existing RAG pipelines, enabling substantial test-time improvements without requiring modifications the model architecture training objectives.
particularly
Superior Alignment Guidance Surpassing Strong LLM Baselines. To validate the effectiveness of contrastive critique supervision when guided by 72B model, we report full results across seven QA benchmarks in Table 3 Our method consistently outperforms both vanilla critique synthesis and direct CLM supervision. Notably, the improvements are more pronounced on complex datasets such as HotpotQA and SQuAD, where our method yields gains of up to +9.2 and +6.0 on Qwen2.5-705 respectively, over direct supervision. These results confirm that contrastive critique signals distilled from a stronger model can significantly enhance the generalization ability of smaller CLMs.
Different Training Strategies for CLM. To compare training strategies for the CLM, Table evaluates our proposed Critique Fine-tuning (CFT) approach against Critique Preference Optimization (CPO) A.4 Across three backbones and seven benchmarks, CFT consistently outperforms CPO, particularly on retrieval-sensitive and 00D-heavy tasks such as HotpotQA and SQuAD, For example, on Qwen2.5-14B, CFT raises the average accuracy from 51.0 to 53.6 and improves performance on SQuAD from 20.2 to 26.6. Similarly, on LLaMA3.1 1-B. CFT achieves a +4.0 gain in average performance and a +10.0 improvement on SQuAD. These results underscore the superiority of preference-based critique generation over preference-basee output generation for CLM training, particularly in retrieval-intenss contexts.
Ablation on CLM and Contrastive Critique Synthesis. To supplement the high-level ablation analysis, Table 4 presents detailed results for seven benchmarks under four Critic Language Model (CLM) configurations. While the main text reports averaged scores on seven benchmarks, the table provides detailed results. Introducing a frozen CLM yields noticeable gains over Vanilla RAG (e.g.. +3.8 PopQA and +3.9 on ASQA for Qwen2.5-7B), confirming the utility of auxiliary critique.
28

---

WAAgmeett
Algiemon (728)
Owerd2.577
OwerdS-148
LLaMAS11448
03
Tesach
Addl
Pupia
Theeot
Roodd
AsEA
Hatt
trges
thrach
Figure 8: Details of evaluation result of InstructRAG w/o and w/ Alignment method three backbones across seven benchmarks.
Qawar 5-78
Ownnn 5-14B
LI LLAMAS 1-88
PopdA
PspOx
POzDA
13.3 151
Natua Caeckots
2N1-414 GS
ZAR HH
(a) w/O answer.
2w072.5-73
Owan2 6-14B
LLaMA31-88
PapOA
Per
ASO
20 pl
++
ZAMMLEHESAA
Nat.maG.sstoona
(b) w/ answer.
Figure 9: Performance of different methods under Unanswerable (a) and Answerable (b) retrieval conditions. Each radar chart reports the average performance across three instruction-tuned backbones on five benchmarks.
Further training of the CLM amplifies these benefits, particularly for OOD datasets such as SQuAD and HotpotQA. Notably, our contrastive critique synthesis (CCS) achieves the best performance on nearly all benchmarks, including +2.2 gain on MultiHopAA and +4.7 gain on SQuAD for Qwen2.5- 14B. These results demonstrate that contrastive alignment is crucial for generating retrieval-sensinee critiques, leading consistent and robust improvements across diverse QA scenarios.
29

---

Table 3: Detailed results of performance comparison of different trained CLM when using 72B
model supervise.
PopQA (acc)
TriviaQA (acc)
NQ
MultiHopQA (acc)
ASQA (em)
HotpotQA (acc)
SQuAD (acc)
Method
Avg.
(acc)
Vamilla Critique Synthesis
Qwen2.5-7B
66.6
76.2
65.4
46.0
47.9
25.2
17.9
49.3
Qwen2.5-148
67.5
78.9
66.8
47.5
48.7
29.6
21.2
51.5
LLaMA3.1 1-B
66.6
75.7
64.4
43.7
47.1
22.9
13.2
47.7
Direct CLM (72B)
Qwen2.5 2-
66.4
78.7
66.8
46,9
49.3
24.7
50.1
Qwen2.5-14B
68.6
79.3
67.8
49.3
26.8
19.7
51.3
LLaMA3.1-8B
67.1
78.9
66.4
45.7
49.1
24.2
15.9
49.6
Oury
Qwen2.5-7B
66.0
77.5
66.4
49.9
48.6
33,9
24.2
52.4
Qwen2.5-144
66.9
79.4
68.6
50.7
49.9
35.5
25.4
53.8
LLaMA3.1
66.6
77.0
66.3
49.6
48.2
32.0
20.7
51.5
Table 4: Details ablation study on the CLM and CCS. Frozen CLM refers to a vanilla LLaMA3.1- 8B used as the critic. CCS refers our proposed contrastive critique synthesis.
PopQA (acc)
TriviaQA (acc)
NQ
MuliiHopQA
ASQA (em)
HotpotQA (acc)
SQUAD (acc)
Method
Avg.
(acc)
(acc)
Vanilla RAG
QwenZ.5-P7
63.7
73.2"
60.2
44.7
42.8
18.5
5.0
44.6
Owen2.5-144
65.3
77.0
63.6
44.8
45.2
23.3
12.6
47.4
I.AMA3.1-80
65.0
73.4
62.0
43.0
45.2
17.1
6.1
44.5
w/ Prozen CLM
Qwen2.5-7H
67.5
7.511
62.5
45.4
46.7
20.6
13.7
47.3
Qwen2.5-
68.0
78.0
65.1
25.3
15.6
49.5
LLaMA3.1-88
66.1
74.1
61.4
42.8
44.7
18.8
8.7
45.2
w/ Trained CLM
Qwen2.5-7B
66.9
76.1
64.0
46.7
46.5
26.8
19.9
49.6
Qwen2.5-1a4
67.1
78.4
65.7
47.6
48.8
30.0
21.9
51.4
LLaMA3.1
65.1
74.3
62.7
42.8
46.5
25.2
47.6
w/ Trained CLM, w/ CCS (Ours)
Qwen2.5 5-7B
68.4
77.8
65.9
49.3
48.9
33.7
26.1
32.9
Qwen2.5 5-14B
68.4
79.5
67.7
49.8
48.6
34.8
26.6
53.6
I.L.MA31--88 3-888 66.5
77.0
65.3
47.0
47.1
32.2
22.8
51.1
Table Overall performance comparison of Critic Language Model using different training methods.
PopQA
TriviaQA (acc)
NQ (acc)
MultiHopQA
ASQA (em)
HotpotQA (acc)
SQuAD (acc)
Method
Avg.
(ace)
(acc)
Qwen2.5-7B
CPO
66..
76.3
63.35
46.3
47.1
25.6
17.3
48.9
Ours
68.4
77.8
65.9
49.5
48.9
33.7
26.1
52.9
Owen2.5-147
CPO
67.5
77..5
666.T
47.4
47.7
29.5
20.2
5TT
Ours
68.4
79.5
67.7
49.8
48.6
34.8
26.6
53.6
LLaMA3.1-8R
CPO
66.4
75.0
62.7
44.0
-4.77
23.5
12.8
47.1
Ours
66.5
77.0
65.3
47.0
47.1
32.2
22.8
51.1
30

---

Table 6: Drop average hottof-Distributtin performance compared to average In-Domain per- formance across three instruction-tuned backbones. Lower values indicate better generalization capability.
PopQA (acc)
TriviaQA (acc)
NQ (acc)
MultiHlpQAA (acc)
ASQA (em)
HotpotQA (acc)
SQuAD (ace)
Method
Avg.
Avg.
Drop.
Owen2.5-78
Vanilla RAG
63.7
7322
60.2
44.7
42.8
56.9
1835
'.0""
1338
43.1
Self-Reline
74.4
45.2
58.3
21.3
14.6
40.3
AlignRAG
68.4
77.8
65.9
49.5
48.9
62.1
33.7
26.1
29.9
32.2
QwenZ. 2.122.5.44 548
Vanilla RAG 65.3
77.0
63,6
44,8
45.2
59.2
23.3
12.6
18.0
41.2"
Self-Refine
67.0
78.0
65.1
46.1
47.3
60.7
24.4
16.0
20.2
40.5
AlignRAG
68.4
79.5
49.8
48.6
62.8
34.8
26.6
30.7
32.1
LLaMA3.
Vanilla RAG
65.0
73.4
62.0
43.0
45.2
$7.7
17.1
6.1
11.6
46.1
Self-Retine
66.1
74.1
61.4
42.8
44.7
57.8
18.8
8.7
13.8
44.0
AlignRAG
66.5
77.0
65.3
47.0
47.1
60.6
32.2
22.8
27.5
33.1
31

---

A.7 Prompt Templates
Critique Synthesis Prompt. We propose novel structured pipeline for generating targeted feedback to train critic models, systematically deriving critiques from contrasting outputs of large language models (LLMs). To ensure the critiques are both consistent and informative, we introduce a preference- augmented impul as a key component in the critique generation process. This approach is grounded in the use of pairwise comparisons of reasoning paths, which provides two core innovations. First, it constrains the output space of the critique language model (CLM), ensuring consistency and minimizing noise during critique generation [36]. Second, it generates high-quality reasoning traces that facilitate the creation of constructive, fine-grained feedback. The pairwise-path formulation central to this framework: by contrasting the reasoning processes underlying Y Yunexp (unexpected response) and Yexp (expected response), the CLM synthesizes critiques that directly inform model supervision. This is exemplified in Tab. 7 rationale generation) and Tab. fab.99e ( (for critique generation). This structured methodology not only enhances the quality of the generated critiques but also ensures they are targeted, actionable, and aligned with the requirements of improving weaker models.
Critique Learning Prompt. To further advance critique generation, we introduce the concept of critique learning, where the objective is to generate critique, denoted Aymexp- that captures the divergence between expected and unexpected responses while incorporating user-defined preferences. As part of this framework, we present a novel Critique Fine-Tuning (CFT) prompt (see Tab. for details) designed to optimize the learning process for critique generation. Additionally, we explore an alternative training strategy, Critique Preference Optimization (CPO), which explicitly aligns critique generation with user-defined preference signals (see Tab, I for the corresponding prompt). These prompts, tailored for critique learning, establish principled mechanism for training models to generate preference- alignned critiques.
We introduce a novel framework, Eritique-driven Alignment (CDA), to address reasoning misalignment in retrieval -aggeeened generation (RAG) systems. CDA reimagines the RAG inference process as a discrete-time dynamical system operating over la- tent reasoning space y. Within this framework, the inference process is iteratively refined by meta-reasoning module Menitie- which critiques intermediate outputs and proposes targeted improve- ments. This iterative refinement produces a sequence of progressively improved responses, ensuring reasoning alignment.
Critique-driven Alignment Prompt.
CDA leverages three distinct prompt types to structure the refinement pipeline effectively:
Rationale Generation: Using the rationale generation template (see Tab. 12p. the system generates an initial explanation or reasoning chain to support the initial response Yo- This rationale serves as the foundation for critique generation subsequent steps.
Critique Generation: Using the critique generation template (see Tab. [3]. the meta- reasoning module Menue identifies reasoning gaps or inconsistencies in the intermediate response y, based on the rationale and provides an actionable critique Ayr-
Using the refinement generation template (see Tab. 14), the system incorporates the critique Ay, into the generation process to produce the refined response Y1+1- This ensures that the updated response aligns with the critique feedback while maintaining coherence and relevance to the original query
Refinement Generation:
By iteratively applying these three prompts, the CDA framework introduces systematic and con- trolled refinement process that enhances reasoning alignment and response quality over successive iterations. This novel paradigm ensures that critiques are not only actionable but also effectively integrated into the refinement process to achieve consistent improvements in reasoning accuracy.
32

---

Table 7: Rationale generation prompt template for critique synthesis 13
Rationale Generation for Critique Synthesis
Input: Read the following documents relevant to the given question: {question}
Document [1] (Title: ...); {contents)
Please identify documents that are useful to answer the given question: "[question)" and explain the contents lead to the answer: [answer).
of the documents is aligned with the answer, in that case, you have explain the only based on your own knowledge, without referring the provided information.
(task-specific instruction)
Output: (rationale)
Table 8: Task-specific instruction used in rationale generation prompt 13
Task-specific Instruction for Rationale Generation
ASQA:
Note that the question may ambiguous and have multiple correct answers. Make sure your response includes all correct answers and provides clear reasoning details followed by concise conclusion.
Note that the question mainly asks about the object entity that holds certain relation- ship with the given subject entity. There may he multiple correct answers. Make sure your response includes all correct answers and provides clear reasoning details followed by a concise conclusion.
PopQA:
TriviaQA Natural Questions / 2WikiMultiHopOA:: Note that the question may be composi- tional and require intermediate analysis to deduce the final answer. Make your response is grounded and provides clear reasoning details followed by concise conclusion.
Table 9: Critique generation prompt template for critique synthesis.
Critique Generation for Critique Synthesis
Input: Read following documents relevant to the given question: [question]
Document [1] (Title: ...): {contents)
Here is the given weak rationale: r.tio.aaaionale)... Here is the given gold rationale: [gold_rattee soodd_rationale).
First, explain how the gold rationale leads the answer step by step.
Then, identify the errors and hallucinations of the weak rationale, and give constructive criticism for improving the weak rationale to be more aligned with the gold rationale.
Output: (critique)
33

---

Table 10: Augmented critique generation prompt template for critique fine-tuning (CFT).
Augmented Critique Generation for Critique Fine-tuning (CFT)
Input: Read the following documents relevant to the given question: {question]
Document [1] (Title: ...); {contents)
Here is the given weak rationale: (weak_rationale)
Please identify the weaknesses and ballucinations of the rationale, and give constructive criticism for improving the weak rationale.
Output:
The critique for the rationale is: (critique). The better rationale should be: (gold rationale).
Table 11: Critique generation prompt template for critique preference optimization (CPO).
Augmented Critique Generation for Critique Preference optimization (CPO)
Input: Read the following documents relevant the given question: {question]
Document [1] (Title: :):: {contents)
Here is the given weak rationale: (weak_ rationalto
Please identify the weaknesses and hallucinations of the rationale, and give constructive criticism for improving the weak rationale.
Chosen: The critique for the rationale is: (weak_ ccrck__ritique). Rejected: The critique for the rationale is: (gold_critique),
Table 12: Rationale generation prompt template for Critique-driven Alignment.
Rationale Generation for Critique-driven Alignment
Input:
Read the following documents relevant the given question: {question)
Document [1] (Title: :..): [contents)
Please identify documents that useful to answer the given question: "(question)", and explain how the contents lead the answer: [answer).
Output: (rationale)
A.8
Case Study
To provide concrete illustration of the reasoning misalignment issues our framework addresses, we present a series of case studies. These examples demonstrate how failures can occur at each of the three distinct phases of retrieval-augmented reasoning- Relevance Assessment, Query-Evidence
34

---

Table 13: Critique generation prompt template for Critique-driven Alignment.
Critique Generation for Critique-drive driven Alignment
Input: Read the following documents relevant to the given question: {question]
Document [I] (Title: ...); {contents)
Here is the given weak rationale: -rcak_ratiooale).
Please identify the weaknesses and ballucinations of the rationale, and give constructive criticism for improving the weak rationale.
Output: (critique)
Table 14: Refinement generation prompt template for Critique-driven Alignment.
Refinement Generation for Critique-driven Alignment
Input: Read following documents relevant to the given question: {question]
Document [1] (Title: :..): (contents)
Here the given weak rationale: (weak rationale). Here is the given critique: critique.
Please correct the weak rationale based on the critique, and write better rationale to explain how the contents lead the answer.
Output: [refinement)
Mapping, and Evidence- Integrated Synthesis even when the initial retrieval is successful. We also include failure analysis of our own model, ALIGNRAG, to highlight its limitations.
Wvidence-Integrated
A.8.1 Illustrating Reasoning Misalignment
Case Misalignment in Relevance Assessment. This case demonstrates a failure in the initial reasoning phase, where the model incorrectly dismisses highly relevant evidence. The retriever successfully finds a document containing the correct answer, but the generator's internal relevance assessment fails, causing it to discard the evidence and claim the information is unavailable. This highlights that successful retrieval is insufficient if the model cannot recognize the value of retrieved content.
Table 15: An example of misalignment in Phase (Relevance Assessment).
Failure Mode: Relevance Misjudgment
Question
Who is the mother of Mary in Islam?
Golden Answer
Hannah
Retrieved Evidence
Document explicitly states, "Hannah, the mother Mary..." "The given documents are not relevant the question." [Bad] - Model failed to recognize clearly relevant evidence.
Initial Response
Critic Evaluation
Refined Response
"Hannah mentioned as the mother of Mary the provided context."
35

---

Case 2: Misalignment in Query-Evidence Mapping, In this scenario, the model fails during the mapping phase. Although multiple documents are retrieved, the model latches onto a document with vague thematic association (Document 1) while ignoring another document (Document 5) that contains the precise, explicit answer. This mapping failure demonstrates the challenge of aligning the specific query with the most salient evidence span, a critical step for accurate, grounded generation.
Table 16: An example of misalignment in Phase 2 (Query-Evidence Mapping).
Failure Mode: Evidence Mapping Failure
Question
What name given to very long forward pass in football made in desperation?
Golden Answer
Hail Mary
Retrieved Evidence
Document directly defines the "Hail Mary pass" in this context.
Initial Response
The model cites Document 1, which discusses related football terms but ignores the direct definition in Document 5.
Critic Evaluation Refined Response
[Bad] Model missed the most direct and salient piece of evidence. Correctly identifies "Hail Mary" using the evidence from Document 5.
Case 3: Misalignment in Synthesis. This case illustrates a failure at the final synthesis stage. The model correctly identifies and internally processes the relevant document containing the answer. However, it fails to integrate this crucial piece of information into its final generated output. The evidence understood but ultimately omitted, rendering the response incomplete and unhelpful. This shows that even with perfect retrieval and mapping, the synthesis process itself can be point of failure.
Table 17: An example of misalignment in Phase 3 (Synthesis).
Failure Mode: Synthesis Error
Question
Who was the director of Alexander?
Golden Answer
Oliver Stone
Retrieved Evidence
Document states, "It was directed by Oliver Stone..." The model discusses the movie but fails state the director's name, despite having access to the information.
Initial Response
Critic Evaluation
[Bad] Information was present in the evidence but was omitted in the final output.
Refined Response
"The only director identified in the provided documents is Oliver Stone."
A.8.2 Failure Analysis of ALIGNRAG
While ALIGNRAG is designed to correct the misalignments above, it is not without its own failure modes. The most critical weakness arises when the initial retrieval is incomplete or fails provide any relevant evidence. In such cases, the Critic Language Model (CLM) may attempt to "over-correct" the initial response by injecting factual knowledge from its own parameters, which not grounded in the provided context. This can lead to factual drift and produce refined response that is still incorrect but for a different reason.
A.9 Broader Impact
The AlignRAG framework promises positive societal impact by enhancing the factual reliability and evidence- grrnnndd reasoning of LLMs, potentially leading to more trustworthy AI systems in areas like education, research, and complex decision-supporr. thereby reducing the spread of unsupported or misaligned information. This can empower users with more accurate and verifiable information. However, challenges and risks must be acknowledged: reliability, even if improved, could lead to over-reliance by users and reduction in critical scrutiny. Furthermore, the critique mechanism itself, while aiming for better alignment, might inadvertently
increased perception of
36

---

Table 18: representative failure case for ALIGNRAG.
Failure Mode: Over- Cerr-Corenttion due to Weak Retrieval
Question
When was the first 10 dollar bill made?
Golden Answer
1861
Retrieved Evidence
No retrieved document explicitly mentioned the year 1861. "The first 10 dollar bill was issued in 1911." (Incorrect) [Bad] The critique proposed the year 1914, likely based the
Initial Response
Critic Evaluation
CLM's internal knowledge rather than the provided evidence.
Refined Response
"The first $10 bill was made in 1914." (Incorrect)
absorb or amplify subtle biases present in the data used for training the Critic Language Model or in the "expert" examples used for contrastive synthesis if not meticulously curated and audited. Therefore, the responsible development and deployment of such advanced RAG systems necessitate ongoing research into robust bias detection and mitigation techniques, ensuring diversity in training data and retrieved evidence, and promoting digital literacy encourage critical user engagement with Al-generated content.
37


Reviews:
[{"review_id": "HkzcnjiE7O", "reviewer": "Reviewer_UcqY", "summary": "This paper aims to improve RAG in the generation phase. It first analyzes a critical challenge in RAG, termed Reasoning Misalignment, referring to a disconnect between the model's reasoning process and the retrieved evidence. It then tries to understand the underlying cause of the problem by reconceptualizing RAG as Retrieval-Augmented Reasoning. Further, it proposes a framework named ALIGNRAG, which employs a Critic Language Model to dynamically correct misalignments during inference using retrieval-augmented critiques. Finally, extensive experiments across seven benchmark datasets and three model families are conduct to evalute the effectiveness of ALIGNRAG.", "questions": "Please refer to weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strengths are summarized as follows:\n1. The studied problem is interesting.\n\nThe main weaknesses are summarized as follows:\n1. The analysis of the reasoning misalignment issue of RAG in Section 3 is not clear. It would be better if an intuitive example could be provided to illustrate the two characteristics of misalignment. Besides, I doubt whether there are truly three reasoning steps involved in RAG reasoning. The connection between this analysis the proposed solution seems weak.\n2. The proposed method heavily rely on the construction of training data with complicated process. As shown in Equation (1), the quality label of retrived context and the supervision signal are difficult to obtain.\n3. It would be better to include a case study in the experiments. The inference process is not straightforward for me.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns.", "rebuttal": "We sincerely thank you for your critical feedback and insightful suggestions. We agree that the analysis in Section 3 would be significantly clearer with concrete examples and that a detailed case study is essential for understanding the inference process. \n\n> *The analysis of the reasoning misalignment issue of RAG in Section 3 is not clear. It would be better if an intuitive example could be provided to illustrate the two characteristics of misalignment. Besides, I doubt whether there are truly three reasoning steps involved in RAG reasoning. The connection between this analysis the proposed solution seems weak.*\n\nThis is a very fair point, and we appreciate the opportunity to clarify our thinking.\n\nThe three-phase breakdown we propose—(1) Relevance Assessment, (2) Query-Evidence Mapping, and (3) Evidence-Integrated Synthesis—is not intended as a rigid procedural pipeline, but rather as a diagnostic framework for analyzing reasoning alignment in RAG systems. While models may not explicitly follow these stages, many leading reasoning-centric RAG approaches (e.g., InstructRAG [1], Self-RAG [2]) implicitly adopt similar structures to guide retrieval-augmented generation. For instance, `InstructRAG has demonstrated that structured reasoning can effectively stimulate LLM capabilities in RAG settings`. Similarly, AlignRAG has shown that such structured processes enhance both the reliability and explainability of answers—qualities that are especially important in high-stakes domains like medical and clinical RAG systems.\n\nThis framework allows us to systematically identify distinct failure modes, even when the retrieval step itself is accurate. More importantly, it serves as the foundation for our proposed solution: `our Critic Language Model (CLM) is explicitly trained to recognize and generate critiques linked to failures at each of these reasoning stages.` By diagnosing whether a response suffers from weak relevance assessment, flawed evidence mapping, or poor synthesis, the CLM can guide the generation process toward more accurate and evidence-grounded outputs.\n\nTo further illustrate the utility of this framework, we will include detailed case studies in the appendix that show how real-world errors correspond to these phases of reasoning.\n\n**Case 1: Misalignment in Phase 1 — Relevance Assessment**\n\n| **Aspect** | **Details** |\n|:-----------------------|:-----------------------------------------------------------------------------|\n| **Question**           | *Who is the mother of Mary in Islam?*                                       |\n| **Golden Answer**      | Hannah |\n| **Initial Response**   | \"The given documents are not relevant...\"                                   |\n| **Retrieved Evidence** | Document 2 explicitly states “Hannah, the mother of Mary”                    |\n| **Critic Evaluation**  | `[Bad]` — Model failed to recognize clearly relevant evidence                |\n| **Refined Response**   | \"Hannah is mentioned as the mother of Mary...\"                              |\n| **Failure Mode**       | **Relevance misjudgment** led to discarded correct evidence                  |\n| **Root Cause**         | Poor signal interpretation in Phase 1                                        |\n| **Takeaway**           | Even strong retrieval fails if relevance is not properly assessed            |\n\n**Case 2: Misalignment in Phase 2 — Query-Evidence Mapping**\n\n| **Aspect**  | **Details** |\n|:-----------------------|:-----------------------------------------------------------------------------|\n| **Question**           | *What name is given to a very long forward pass in American football made in desperation?* |\n| **Golden Answer**      | Hail Mary |\n| **Initial Response**   | Cited Document 1, ignored Document 5                                         |\n| **Retrieved Evidence** | Document 5 directly defines \"Hail Mary pass\"                                 |\n| **Critic Evaluation**  | `[Bad]` — Model missed most direct evidence                                  |\n| **Refined Response**   | Correctly identifies \"Hail Mary\"                                             |\n| **Failure Mode**       | **Mapping failure** — missed clear span-to-query alignment                   |\n| **Root Cause**         | Salient match ignored during evidence mapping                                |\n| **Takeaway**           | Query-to-doc matching must prioritize direct matches over vague associations |\n\n**Case 3: Misalignment in Phase 3 — Synthesis**\n\n| **Aspect** | **Details** |\n|:-----------------------|:-----------------------------------------------------------------------------|\n| **Question**           | *Who was the director of Alexander?*                                        |\n| **Golden Answer**      | Oliver Stone  |\n| **Initial Response**   | Read correct doc but failed to state answer                                 |\n| **Retrieved Evidence** | Document 5: “It was directed by Oliver Stone…”                              |\n| **Critic Evaluation**  | `[Bad]` — Info was present but omitted in output                            |\n| **Refined Response**   | “The only director identified is Oliver Stone.”                             |\n| **Failure Mode**       | **Synthesis error** — failed to integrate known evidence                     |\n| **Root Cause**         | Incomplete summarization of relevant facts                                  |\n| **Takeaway**           | Strong evidence is useless if the synthesis step fails to surface it         |\n\n[1] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales\n\n[2] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\n\n> *The proposed method heavily rely on the construction of training data with complicated process. As shown in Equation (1), the quality label of retrived context and the supervision signal are difficult to obtain.*\n\nWe appreciate you raising this important concern about the complexity of our training data construction. We recognize the need to be clearer on this point. While the process is multi-faceted, we would like to emphasize that it is designed to be **`fully programmatic and self-supervised`**, avoiding any need for expensive manual annotation beyond the ground-truth answers already present in the benchmark datasets.\n\nWe will clarify how both key components of our training data are generated automatically and don't not require additional human labeling:\n\n1.  **Automated Generation of Context Quality Labels (`c_i`):** You are correct to ask how the quality labels for retrieved context (`c_i`) are obtained. This process is fully automated. Following standard practices in RAG evaluation, we programmatically check for the presence of the ground-truth answer strings within the retrieved documents to label them for \"helpfulness\" and \"completeness.\" This allows us to systematically create a diverse training corpus representing various retrieval scenarios without any manual effort.\n\n2.  **Automated Generation of Supervision Signals (`Δy_unexp,i`):** The core supervision signal—the critique (`Δy_unexp,i`) itself—is also generated automatically via our novel **Contrastive Critique Synthesis (CCS)** mechanism. By contrasting the outputs of \"weak\" and \"strong\" models for the same input, we create a high-quality and consistent supervisory signal. This self-distillation approach is a key advantage, as it `does not require human labeling` and is more `scalable`. It also constrains the learning objective, focusing the critic on identifying evidence-based misalignments rather than stylistic preferences. Furthermore, this signal is improvable; as we show in Section 5.3 (\"Superior Alignment Guidance Surpassing Strong LLM Baselines\"), using a stronger model to generate these critiques further boosts performance.\n\nIn summary, the entire data construction pipeline is automatic, scalable, and reproducible. We will revise Section 4.2.1 to more clearly articulate these automated steps and allay any concerns about the practicality of our method.\n\n> *It would be better to include a case study in the experiments. The inference process is not straightforward for me.*\n\nThis is an excellent suggestion. To make our `critique-and-refine` loop clear, we will add the following step-by-step case study to the main paper.\n\n**Query:** \"what is the shape of the ohio flag called?\"\n\n**Retrieved Evidence:** Multiple documents stating the flag is a \"non-rectangular,\" \"swallowtail,\" and \"burgee.\"\n\n**Step 1: Initial Generation (Weak Rationale)**\nThe generator model first produces a flawed rationale:\n> *\"The documents are not relevant. They talk about Ohio's flag, but not its shape. \"*\n\nThis is a **Phase 1 (Relevance Assessment)** failure. The model has dismissed highly relevant documents.\n\n**Step 2: Critique Generation**\nAlignRAG's Critic Language Model (CLM) takes the query, evidence, and the weak rationale as input and produces a targeted critique:\n> *\"The critique for the rationale is: [Bad]. The weak rationale fails to acknowledge clearly stated, relevant evidence. It dismisses all documents as irrelevant even though they explicitly mention the flag's shape, using terms like 'swallowtail' and 'burgee' ...\"*\n\n**Step 3: Refined Generation**\nGuided by the critique, it produces a corrected, evidence-aligned rationale:\n> *\"Documents 2–5 consistently describe the Ohio flag as a 'swallowtail' and a 'burgee,' both of which are names for a non-rectangular or V-shaped flag. Therefore, the shape of the Ohio flag is called a swallowtail.\"*\n\n---\n\n> *Paper Formatting Concerns: Line 205: repeated equation. Line 232: missing space between contents and citations. Line 234: grammar mistake, \"revises\" -> \"revise\"*\n\nThank you for spotting these. We will correct them in the next version.\n\n**Please let us know if you have any further questions—we’d be happy to address them. Thanks again for your time and thoughtful review!**"}, {"review_id": "x5C6BjUZGP", "reviewer": "Reviewer_Kguz", "summary": "Retrieval-augmented generation (RAG) enables knowledge-grounded LLMs but frequently exhibits reasoning misalignment, where inference drifts from retrieved evidence and yields factual errors. AlignRAG reframes RAG as Retrieval-Augmented Reasoning and embeds a Critique-Driven Alignment loop in which a dedicated Critic Language Model (CLM)—trained via contrastive critique synthesis to distinguish evidence-aligned from misaligned reasoning—generates targeted critiques at inference that steer the generator back to relevant documents . Without altering existing RAG pipelines, an 8B CLM delivers substantial gains over prior refinement methods and consistently outperforms a 72B baseline, all while maintaining robustness under noisy retrieval conditions.", "questions": "- What are the results of runing time？\n- How does AlignRAG’s performance compare with that of IRCOT [1]？\n\n[1] Trivedi H, Balasubramanian N, Khot T, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions[J]. arXiv preprint arXiv:2212.10509, 2022.", "limitations": "If the retrieval quality is insufficient, it will negatively impact AlignRAG’s performance.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Principled Framework Design: Embedding the Critic Language Model into an iterative inference loop transforms RAG into a constrained optimization process. This design elegantly marries retrieval and generation phases and provides a clear mechanism for evidence-driven correction.\n- Plug-and-Play Compatibility: ALIGNRAG integrates seamlessly with existing RAG architectures without requiring modifications to the generator or retrieval components. This design choice maximizes practical applicability and lowers barriers to adoption.\n- Robustness: Experimental results on out-of-domain and noisy retrieval scenarios substantiate claims of improved stability, an essential quality for real-world deployments \n\nWeakness:\n- This paper lacks of the results of runing time. The iterative critique–generation loop incurs substantial computational overhead. Multiple forward passes through both CLM and generator can increase inference time, limiting usability in latency-sensitive contexts.\n- Effective deployment requires setting several hyperparameters, such as number of critique iterations, etc.—which may vary significantly across tasks, models and domains. Does this approach also require preparing a dedicated training dataset for each domain?\n- Table 1 shows that AlignRAG underperforms InstructRAG, yielding even lower scores. In Table 2, combining InstructRAG with test-time  alignment yields better performance—but by how much does the cost increase?", "was_revised": true, "final_justification": "The response has adequately addressed my concerns.", "rebuttal": "We sincerely thank you for your insightful suggestions. Below, we provide a point-by-point response to your concerns. **Note: The results below correspond to a single iteration of AlignRAG due to time constraints.**\n\n> *Effective deployment requires setting several hyperparameters, such as number of critique iterations, etc.—which may vary significantly across tasks, models and domains. Does this approach also require preparing a dedicated training dataset for each domain? What are the results of runing time？*\n\nWe appreciate this thoughtful observation regarding the practicality and usability of the framework. Your comment directly inspired the development of a more autonomous and robust variant: `AlignRAG-auto`. This version eliminates the need for manual iteration tuning and demonstrates strong generalization without requiring domain-specific data.\n\n**Training for Dynamic Refinement:** We modified the training process for our Critic Language Model (CLM). For each initial \"unexpected\" response (`y_unexp`), we first determine its correctness. Incorrect responses are labeled as `[Bad]`, while correct ones are labeled as `[Good]`. The CLM is then trained to generate the appropriate `[Good]` or `[Bad]` token followed  by critique.\n\nThis enables AlignRAG-auto to operate `dynamically at inference time`: it evaluates each response and terminates the refinement loop as soon as the CLM emits a `[Good]` token, saving substantial compute. This dynamic stopping mechanism removes the need for pre-defined iteration counts and makes the system more efficient and hands-free.\n`As the table below illustrates, this approach yields high accuracy while refining only a subset of samples.`\n\n**Refinement Ratio for AlignRAG-auto (% of samples requiring refinement)**:\n| Model| PopQA| TriviaQA | NQ | 2WikiMultiHopQA | ASQA | avg|\n| :--------------------|:------|:--------|:------|:----------|:------|:------|\n| AlignRAG-auto-8B | 63.0 | 50.1 | 53.3 | 56.6 | 54.8 | 55.5 |\n| AlignRAG-auto-7B | 66.3 | 49.7 | 56.9 | 55.1 | 55.7 | 56.8 |\n| AlignRAG-auto-14B | 61.0 | 47.7 | 49.5 | 55.0 | 49.9 | 52.6 |\n\n**Performance Comparison**:\n| Method| PopQA | TriviaQA | NQ| 2WikiMultiHopQA | ASQA | avg|\n|--------------------|-------|----------|-------|------------------|-----------|--------|\n| AlignRAG-8b | 66.5  | 77.0  | 65.3  | 47.0   | 47.1   | 60.4  |\n| AlignRAG-7b | 68.4  | 77.8 | 65.9  | 49.5   | 48.9   | 62.1  |\n| AlignRAG-14b | 68.4  | 79.5  | 67.7  | 49.8   | 48.6   | 62.8  |\n| AlignRAG-auto-8b | 67.6 | 77.6 | 66.8 | 47.6  | 48.8  | 61.3 |\n| AlignRAG-auto-7b | 68.1 | 78.1  | 67.3 | 49.3  | 49.6  | 62.5  |\n| AlignRAG-auto-14b | 68.3 | 79.9  | 69.0 | 50.2   | 49.8   | 63.0  |\n\nImportantly, AlignRAG-auto does not require domain-specific training. The CLM is trained on a diverse dataset spanning five benchmarks, and as shown in Figure 2 and Table 6, AlignRAG demonstrates strong generalization and robustness to out-of-distribution (OOD) settings. This supports the claim that the CLM captures general principles of evidence-based reasoning and alignment, not domain-specific patterns.\n\n`We are grateful for this question, as it pushed us to develop the AlignRAG-auto variant, which we believe makes our framework more robust, efficient, and practical. We will be adding this to the final version of our paper. `\n\n> *This paper lacks of the results of runing time. The iterative critique–generation loop incurs substantial computational overhead. Multiple forward passes through both CLM and generator can increase inference time, limiting usability in latency-sensitive contexts.*\n\nThank you for highlighting this critical consideration. We agree that computational efficiency is vital for real-world adoption.\n\nAlignRAG indeed makes a deliberate `trade-off`, prioritizing correctness and reliability—especially important in high-stakes applications (e.g., clinical or medical RAG)—at the cost of increased compute. However, your feedback prompted us to quantify and reduce this cost.\n\nAs described above, AlignRAG-auto introduces a `\"refine only when necessary\"` approach. The CLM bypasses refinement for responses it deems correct, reducing the average computational load significantly. Across benchmarks, `only ~55% of samples require refinement`—without sacrificing accuracy.\n\nTo provide concrete results, we estimated the computational cost in terms of TFLOPs for a 7B model. The table below shows that AlignRAG-auto, especially when combined with optimizations like `Prefix-caching` [1], offers a compelling balance of performance and efficiency, bringing the cost much closer to that of non-iterative methods while delivering superior accuracy.\n\n**Estimated Computational Cost (7B Model)**:\n\n| Method | Accuracy | TFlops |\n| :---------------------------- | :------------ | :---------- |\n| Vanilla RAG | 56.9 | 13.2 |\n| Self-Refine | 58.3 | 35.6 |\n| AlignRAG | 62.1 | 78.7 |\n| AlignRAG w/ Prefix cache | 62.1 | < 46.8 |\n| AlignRAG-auto w/ Prefix cache | 62.5 | < 25 |\n\n`Future Optimizations:` The computational cost can be further mitigated by standard **engineering practices**. Techniques like **Prefix-caching** (which we've estimated in our tests), **predictive decoding**[2], and **optimized inference engines** like vLLM are complementary to our approach and can substantially reduce latency in production environments.\n\nIn summary, while iterative refinement carries an inherent cost, our new **AlignRAG-auto** variant offers a highly effective solution that preserves state-of-the-art performance while significantly cutting the average computational overhead, making the framework much more practical for real-world use. We will add this analysis to the appendix.\n\n[1] ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition\n\n[2] Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding\n\n> *Table 1 shows that AlignRAG underperforms InstructRAG, yielding even lower scores. In Table 2, combining InstructRAG with test-time alignment yields better performance—but by how much does the cost increase?*\n\nWe appreciate this careful reading. In fact, this combination is exactly where AlignRAG shines.\n\nAs Table 2 shows, AlignRAG adds significant value even when applied to strong training-time baselines like InstructRAG. For example, `applying AlignRAG to the Qwen2.5-14B-Instruct model yields a 9.4% gain in OOD accuracy`—a substantial improvement. This demonstrates that training-time methods alone are not sufficient to fully resolve reasoning misalignments. AlignRAG complements such models by correcting errors at inference, providing targeted refinement based on the actual retrieved evidence. \n\nWith the introduction of AlignRAG-auto, prefix cache and more optimizations, this significant performance gain is now achievable with a much more manageable computational budget, as detailed in our previous response.\n\n> *How does AlignRAG’s performance compare with that of IRCOT.*\n\nThis is a very relevant baseline, and we thank the reviewer for the suggestion. IR-CoT and AlignRAG represent two different, powerful strategies for improving RAG systems.\n* `IR-CoT` focuses on `improving retrieval quality` by interleaving search and generation.\n* `AlignRAG` focuses on `improving reasoning quality` by refining the final output after retrieval is complete.\n\nTo address this, we ran new experiments using Qwen2.5-7B-Instruct, limiting IR-CoT to 3 iterations and top-2 documents per round:\n\n| Method  | PopQA | TriviaQA | NQ   | 2WikiMultiHop | ASQA | Average |\n| :----------------------- | :---- | :------- | :--- | :------------ | :--- | :------ |\n| IR-CoT | 51.4  | 74.1 | 58.9 | 49.6  | 38.8 | 54.6   |\n| AlignRAG-auto | 68.1 | 78.1  | 67.3 | 49.3  | 49.6 | 62.5  |\n| IR-CoT + AlignRAG | 57.5  | 79.0   | 66.9 | 56.5  | 47.0 | 61.4   |\n\nAs expected, IR-CoT performs strongly on multi-hop tasks where its iterative retrieval is most beneficial. AlignRAG demonstrates superior overall performance by focusing on the crucial final step of aligning reasoning with the provided evidence.\n\nMost importantly, the frameworks are complementary. The `IR-CoT + AlignRAG-auto` result shows that our method can take the high-quality documents found by IR-CoT and further enhance the final output by correcting reasoning misalignments. This shows that test-time alignment is a valuable and distinct contribution, even for systems with advanced retrieval strategies. We will add this comparison to the paper.\n\n> *If the retrieval quality is insufficient, it will negatively impact AlignRAG’s performance.*\n\nWe agree completely. Our work aims to maximize the utility of whatever information is retrieved.\n\nOur experiments in **Figure 4 (\"When RAG Retrieval Falters, AlignRAG Thrives\")** were designed specifically to test our framework's robustness under imperfect retrieval conditions. The results show that AlignRAG is significantly more resilient to `noisy retrieval`—a very common real-world scenario—than existing baselines. It excels at filtering out distractor information and correctly utilizing relevant snippets when they are present.\n\nHowever, as noted in Appendix A.1, the framework’s effectiveness diminishes in cases of extreme retrieval failure, where no relevant information is present for the critic to reason over. We will make this limitation more explicit in the revised manuscript.\n\n**We thank the reviewer for the careful and detailed reading. Please don’t hesitate to reach out with further questions!**\n\n**Thank you again for your time!**"}, {"review_id": "kVN9oQOLeE", "reviewer": "Reviewer_sEZn", "summary": "This paper proposes a method to improve the accuracy of LLM responses by refining them within the RAG framework. Specifically, it involves CLM that critiques the generated responses to produce improved responses, and CCS that synthesizes critique data by contrasting good and bad responses for training CLM.", "questions": "1. In Line 93, the paper claims that Self-RAG [1] requires architectural modifications. Could you elaborate on what specific modifications are being referred to?\n\n2. In Section 4.2.3 and Algorithm 2, it is described that the CLM is trained with good responses as part of the input (i.e., $y_{exp} \\in I_{critic}$). However, in Table 10, the good response appears to be part of the output. Table 10 seems to be correct, so it would be helpful to clarify this part.\n\n3. Was InstructRAG [2] trained using the same data described in Lines 223-224, as was the proposed method?\n\n4. (Minor) I’m curious how well the base model would perform if it were fine-tuned to generate the good responses $y_{exp}$ that match the correct answers.\n\n> [1] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\n>\n> [2] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well-written and well-structured.\n\n2. The effectiveness of the proposed method is validated through extensive experiments and analyses.\n\n## Weaknesses\n\n1. Qualitative analysis is necessary to fully understand the effectiveness of the proposed method. For example, the proposed method shows performance improvement through refinement iterations even on relatively simple single-hop QA datasets such as NQ and TriviaQA. It would be helpful to include concrete examples demonstrating how the response is refined over the course of five iterations to eventually arrive at the correct response.\n\n    Furthermore, in Section 3, the paper discusses potential misalignments that can occur at each phase of RAG reasoning; illustrating these with actual examples of incorrect RAG responses would enhance clarity.\n\n2. It would be helpful to include InstructRAG [1] results in the analyses (Figures 2 and 4). Although InstructRAG is a training-time refinement method, it aims to ground RAG reasoning in the retrieved documents—similar in goal to the proposed method. Comparing these methods could offer valuable insights to readers. (It’s also worth noting that Self-Refine [2] may be at a disadvantage since it does not involve training.)\n\n3. (Minor) The novelty of the proposed method appears to be somewhat limited. Given the existence of numerous prior approaches focused on refining LLM responses, the core idea of the paper—generating critique data by contrasting good and bad answers—offers a relatively modest technical contribution.\n\n> [1] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales\n>\n> [2] SELF-REFINE: Iterative Refinement with Self-Feedback", "was_revised": true, "final_justification": "The parer is well-written and effectively validates the effectiveness of the proposed method. However, I am not fully convinced about how the proposed method robustly achieves such performance improvements. Therefore, after considering its novelty, the comprehensiveness of baselines, and other factors, I have made the final decision of \"Borderline accept.\"  I would like to state that my overall evaluation of this paper is positive.", "rebuttal": "We sincerely thank you for your detailed and constructive feedback. Your suggestions have helped us clarify our contributions and strengthen our analysis. **Note: The results below correspond to a single iteration of AlignRAG due to time constraints.**\n\n> *Qualitative analysis is necessary to fully understand the effectiveness of the proposed method ...*\n\nWe fully agree—qualitative examples are essential to highlight where and how AlignRAG improves reasoning. In response, we will include a new subsection featuring detailed case studies. One such example is shown below:\n\n**Case: Misalignment in in Query-Evidence Mapping**\n\n| **Aspect** | **Details** |\n|------------------------|------------------------------------------------------------------------------|\n| **Question**  | *What name is given to a very long forward pass in football made in desperation?* |\n| **Golden Answer**  | Hail Mary |\n| **Initial Response**   | Cited Document 1, ignored Document 5  |\n| **Retrieved Evidence** | Document 5 directly defines \"Hail Mary pass\"|\n| **Critic Evaluation**  | `[Bad]` — Model missed most direct evidence |\n| **Refined Response**   | Correctly identifies \"Hail Mary\" |\n| **Failure Mode**       | **Mapping failure** — missed clear span-to-query alignment                   |\n| **Root Cause**         | Salient match ignored during evidence mapping                                |\n| **Takeaway**           | Query-to-doc matching must prioritize direct matches over vague associations |\n\nDue to rebuttal word limits, we include one example here. Additional cases can be found in our responses to reviewers pwHC and UcqY. Many thanks for this suggestion!\n\n> *It would be helpful to include InstructRAG results in the analyses (Figures 2 and 4). Although InstructRAG is a training-time refinement method, it aims to ground RAG reasoning... Comparing these methods could offer valuable insights to readers.*\n\nThis is an excellent suggestion. Providing a direct comparison with a strong training-time refinement method like InstructRAG within our analytical figures offers a much richer context. We agree this is a valuable addition for our readers and have revised our analysis accordingly.\n\nAs you correctly noted, this also provides a fairer comparison for test-time methods like Self-Refine. Below are the updated results for Figure 2 (OOD Generalization) and Figure 4 (Robustness to Retrieval Quality), now including InstructRAG.\n\n**Revised Analysis for Figure 2: OOD Generalization**\n\n| Method | In-Domain| OOD | Drop|\n| :--- | :--- | :--- | :--- |\n| Self-Refine | 58.9 | 17.3 | 41.6 |\n| InstructRAG | 61.8 | 27.0 | 34.8 |\n| **AlignRAG** | **61.8** | **29.4** | **32.4** |\n\nThis data, which will be integrated into Figure 2, reveals that `AlignRAG demonstrates the strongest generalization to OOD tasks`, experiencing the smallest performance drop among all methods. This highlights that our critique-driven approach learns more robust and transferable reasoning alignment principles.\n\n**Revised Analysis for Figure 4: Robustness to Retrieval Quality**\n\n* **Informative Retrieval (with answer):**\n\n| Method | PopQA | TriviaQA | NQ | 2WikiMultiHopQA | ASQA |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Self-Refine | 92.6 | 94.8 | 86.8 | 76.6 | 80.1 |\n| InstructRAG | 94.0 | 95.2 | 87.9 | 80.3 | 82.5 |\n| **AlignRAG** | **94.7** | **96.2** | **89.1** | **81.9** | **83.3** |\n\n* **Noisy Retrieval (without answer):**\n\n| Method | PopQA | TriviaQA | NQ | 2WikiMultiHopQA | ASQA|\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Self-Refine | 4.9 | 17.1 | 9.7 | 33.2 | 34.4 |\n| InstructRAG | 4.0 | 19.9 | 12.9 | 34.6 | 35.0 |\n| **AlignRAG** | **5.4** | **23.5** | **15.5** | **36.8** | **36.7** |\n\nThis analysis, which will form the basis of a revised Figure 4, shows that `AlignRAG consistently outperforms both Self-Refine and InstructRAG in both informative and noisy retrieval scenarios`. The performance gains are especially pronounced in noisy settings, validating our framework's ability to thrive when retrieval quality falters.\n\nThank you again for this valuable suggestion; we will incorporate these updated figures and analyses into the final manuscript.\n\n> *(Minor) The novelty of the proposed method appears to be somewhat limited... the core idea of the paper—generating critique data by contrasting good and bad answers—offers a relatively modest technical contribution.*\n\nWe appreciate the reviewer's perspective and would like to clarify what we believe are the key novelties of our work, which, taken together, form a significant contribution. While iterative refinement is an established concept, our work introduces several innovations specifically designed to tackle a critical, understudied failure mode in RAG systems:\n\n1. `Identifying the Core Problem:` We are the first to formally identify, define, and diagnose **Reasoning Misalignment** as a distinct RAG failure mode. Our work moves the focus beyond just improving retrieval or generation fluency to explicitly aligning the model's reasoning steps with retrieved evidence.\n\n2. `A Novel, Bias-Mitigating Training Method:` We propose a highly effective method for training a specialized, retrieval-aware Critic Language Model (CLM). Our Contrastive Critique Synthesis (CCS) mechanism uses a form of model **self-distillation** by contrasting outputs from \"weak\" and \"strong\" models to generate supervisory signals. This approach is novel because it trains the critic to be explicitly evidence-sensitive and mitigates the \"self-preference\" bias common in prior self-correction work. This method is so effective that our 8B CLM, trained via CCS, outperforms a much larger 72B model used directly as a critic.\n\n3. `A Practical, Efficient Framework:` Inspired by reviewer Kguz's feedback during this cycle, we further developed **AlignRAG-auto**, which introduces a dynamic, controllable refinement loop that significantly enhances computational efficiency by learning when to stop iterating. This makes our framework far more practical than prior iterative methods.\n\n4. `Comprehensive Empirical Proof:` We provide extensive validation of AlignRAG's effectiveness, demonstrating not only SOTA performance but also superior robustness in noisy retrieval scenarios and plug-and-play compatibility with existing SOTA frameworks like InstructRAG.\n\nWe believe these contributions—identifying a core problem, proposing a novel bias-mitigating training strategy, engineering an efficient solution, and providing robust empirical proof—collectively represent a substantial and valuable advancement for the RAG paradigm.\n\n> *Question 1: In Line 93, the paper claims that Self-RAG requires architectural modifications. Could you elaborate on what specific modifications are being referred to?*\n\nThank you for this request for clarification.The architectural modifications in Self-RAG refer to the **expansion of the model's vocabulary** to include special tokens (e.g., `[RETRIEVE]`, `[CRITIQUE]`, `[Relevant]`). The base generator model must then be explicitly fine-tuned to both generate and interpret these new tokens to control the retrieval and generation process.\n\nBy contrast, AlignRAG `requires no architectural or vocabulary changes`. The base model is used as-is, with alignment introduced through a separate, modular CLM using standard prompting. We will clarify this in the paper to better highlight the simplicity and modularity of our approach.\n\n> *Question 2: In Section 4.2.3 and Algorithm 2, it is described that the CLM is trained with good responses as part of the input... However, in Table 10, the good response appears to be part of the output. Table 10 seems to be correct...*\n\nThank you for your sharp-eyed reading and for identifying this inconsistency. You are absolutely correct. Table 10 reflects our actual implementation, and the description in Algorithm 2 and Section 4.2.3 was erroneous.\n\nFor training, the critique context `I_critic` is `(q, D, y_unexp)`, and the model is trained on a target sequence that concatenates both the critique `Δy_unexp` and the better rationale `y_exp`. This encourages the CLM to learn the mapping from a flawed response to a high-quality, corrected one. We will correct Algorithm 2 and the surrounding text to be consistent with this implementation. We sincerely appreciate you pointing out this error.\n\n> *Question 3: Was InstructRAG trained using the same data described in Lines 223-224, as was the proposed method?*\n\nYes. To ensure a fair and controlled comparison, we trained the InstructRAG baseline using the `exact same 10K dataset mixture` that we used to train our Critic Language Model. We will state this explicitly in the experimental setup section to make our commitment to a fair comparison clear.\n\n> *Question 4 (Minor): I’m curious how well the base model would perform if it were fine-tuned to generate the good responses y_exp that match the correct answers.*\n\nThis is a very insightful question that gets to the heart of a powerful baseline. This approach is conceptually very similar to the **InstructRAG** framework itself. InstructRAG works by generating synthetic rationales (`y_exp` in our terminology) and then fine-tuning the base model on this data.\n\nCrucially, our results in Table 2 show that applying AlignRAG on top of InstructRAG yields further significant gains (e.g., a **+9.4% boost in OOD accuracy** for the Qwen2.5-14B model). This demonstrates that our critique-driven framework provides a necessary, complementary mechanism to identify and correct these remaining errors. We will add a note to the paper to clarify this conceptual link.\n\n**We sincerely thank the reviewer again for highlighting these points—your feedback has helped us significantly improve the completeness and clarity, let us know if you have further questions after reading our responses!!**"}, {"review_id": "31HmD8FmYv", "reviewer": "Reviewer_pwHC", "summary": "This paper introduces ALIGNRAG, a novel framework that redefines Retrieval-Augmented Generation (RAG) as Retrieval-Augmented Reasoning. It addresses a key failure mode in RAG systems called Reasoning Misalignment—where the model's reasoning deviates from the retrieved evidence. ALIGNRAG uses a Critique-Driven Alignment (CDA) mechanism, powered by a Critic Language Model (CLM) trained via Contrastive Critique Synthesis (CCS). This CLM critiques and refines the reasoning process at test time, improving factual alignment and robustness.\n\nKey contributions:\n\nIdentification of reasoning misalignment as a core issue in RAG.\nIntroduction of CDA and CLM for test-time reasoning refinement.\nDemonstrated improvements over baselines like Self-RAG and Self-Refine.\nPlug-and-play compatibility with existing RAG systems.", "questions": "How does ALIGNRAG perform in real-time or latency-sensitive applications?\nCan the CLM be distilled or compressed for efficiency?\nHow does the system handle conflicting evidence in retrieved documents?\nCould this framework be extended to multi-modal RAG (e.g., vision-language)?\nWhat are the failure cases where ALIGNRAG underperforms compared to simpler baselines?", "limitations": "Effectiveness diminishes when none of the retrieved documents are relevant.\nMinor tuning may be required for optimal integration with different LLMs.\nThe critique model’s performance is still tied to the informativeness of the retrieved evidence.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nNovel Perspective: Reframing RAG as reasoning rather than just generation is conceptually impactful.\nTest-Time Optimization: CDA enables dynamic correction without retraining the base model.\nContrastive Critique Synthesis: Innovative training method that mitigates self-bias in critiques.\nEmpirical Rigor: Extensive experiments across 7 benchmarks and 3 model families.\nPlug-and-Play Design: ALIGNRAG integrates easily into existing RAG pipelines.\nOOD Robustness: Strong performance even under noisy or irrelevant retrieval conditions.\n\nWeaknesses / Limitations\nDependency on Retrieval Quality: Performance still degrades under extreme retrieval failure.\nComputational Overhead: Iterative refinement and CLM inference may increase latency.\nLimited Human Evaluation: Mostly automatic metrics; lacks qualitative human judgment.\nComplexity: The multi-step critique and refinement process may be harder to implement in production.\nGeneralization to Other Tasks: Focused on QA; unclear how well it transfers to other RAG applications like summarization or dialogue.", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your thorough review and insightful questions. Your feedback has helped us clarify key points and inspired valuable new analyses that significantly strengthen the paper. **Note: The results below correspond to a single iteration of AlignRAG due to time constraints.**\n\n> *Retrieval Quality: Performance still degrades under extreme retrieval failure.*\n\nThis is an excellent observation. The quality of retrieved evidence is a fundamental limitation for any RAG system. AlignRAG is designed to make the most of whatever context is available—especially in common but difficult \"noisy retrieval\" scenarios.\n\nTo quantify this, we evaluated AlignRAG in cases where retrieved documents were topically relevant but lacked a direct answer (see Section 5.3). Results show that AlignRAG significantly outperforms baselines in these settings, demonstrating a stronger ability to filter distractors and reason effectively.\n\n**Performance on Noisy Retrieval (Average across three backbones)**\n| Method| PopQA | TriviaQA | NQ| 2WikiMHQA| ASQA| Average|\n|:-------------|:-----:|:--------:|:----:|:---------:|:-----:|:-----------:|\n| Self-Refine | 4.9 | 17.1 | 9.7  | 33.2 | 34.4  | 19.86 |\n| **AlignRAG** | **5.4**| **23.5**| **15.5** | **36.8**| **36.7** | **23.58** |\n\nWe also acknowledge in Appendix A.1 that performance degrades under extreme retrieval failure (i.e., no relevant evidence). This is an important boundary condition, and we will highlight it more clearly as a promising area for future work.\n\n> *Computational Overhead: Iterative refinement and CLM inference may increase latency.* \n\nWe appreciate this concern. AlignRAG introduces additional computation in exchange for higher factual reliability—particularly valuable in high-stakes scenarios.\n\nIn response to reviewer feedback, we developed `AlignRAG-auto`, a dynamic variant where the CLM first assesses the quality of the initial response and then decides whether refinement is necessary. `This enables the system to skip refinement for ~55% of cases, reducing average latency while preserving accuracy`.\n\nFurthermore, this overhead can be mitigated by standard engineering optimizations complementary to our framework, such as `prefix-caching`[1], speculative decoding, and optimized inference engines like vLLM. Prefix-caching is especially effective for iterative tasks like ours, making AlignRAG much more viable for latency-sensitive applications.\n\nTo provide concrete data on latency, we estimated the computational cost (TFLOPs) and found that `AlignRAG-auto`, especially `when combined with standard optimizations like Prefix-caching`, offers a compelling balance of performance and efficiency.\n\n**Estimated Computational Cost (7B Model)**\n|Method|Accuracy| TFlops|\n|:---| :---:| :---:|\n|Vanilla RAG| 56.9 | 13.2 |\n|**AlignRAG-auto**| **62.5** | **< 25.0** |\n\nThis analysis, which we will add to the appendix, demonstrates that the framework's overhead is manageable and can be made practical for many applications.\n\n[1] ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition\n\n> *Limited Human Evaluation: Mostly automatic metrics; lacks qualitative human judgment.*\n\nWe agree that human judgment would further strengthen our findings. Due to time and resource constraints, large-scale human studies were out of scope, but we took two concrete steps to address this:\n\n1.  **Manual Case Studies:** We manually examined samples across benchmarks and presented a representative case in the main text to demonstrate how AlignRAG improves both reasoning paths and final answers.\n2.  **LLM-as-a-Judge Evaluation:** To provide a more scalable and robust qualitative assessment, `we used GPT-4o-mini as an impartial judge to score the quality of the final generated responses (refer to [1])`. The results (scored 1-100) on the Qwen2.5-7B-instruct model are shown below:\n\n| Method | PopQA | TriviaQA | NQ | 2WikiMHQA | ASQA | **Average** |\n|:-------------|:-----:|:--------:|:----:|:---------:|:-----:|:-----------:|\n| InstructRAG  | 54.6  | 71.3     | 65.0 | 12.8      | 53.5  | 51.44   |\n| Self-Refine  | 51.5  | 71.7     | 51.4 | 13.1      | 55.9  | 48.72   |\n| **AlignRAG-auto** | **54.9**  | **76.1**     | **54.5** | **19.2**      | **63.2**  | **53.58**   |\n\nThese results confirm that AlignRAG delivers higher-quality, more reliable reasoning, even when judged qualitatively.\n\n[1] You Don’t Need That Much Data to Train a Search Agent via RL.\n\n> *Complexity: The multi-step critique and refinement process may be harder to implement in production.*\n\nA valid concern. Despite its multi-step nature, AlignRAG is modular and requires `no architectural changes` or special tokens in the base LLM. It operates cleanly at test time, making integration straightforward with minimal engineering effort.\n\n> *Generalization to Other Tasks: Focused on QA; unclear how well it transfers to other RAG applications like summarization or dialogue.*\n\nThank you for this insightful suggestion. To evaluate transferability, we conducted new experiments on multiple-choice QA (`PubMedQA`, `BioASQ`) and open-domain summarization (`WoW`). Results on Qwen2.5-7B-instruct are shown below:\n\n**Performance on New Tasks (Qwen-7B)**\n| Method | PubMedQA| BioASQ| WoW (llm-as-judge) |\n|:-----------------|:--------:|:------:|:----:|\n| Qwen-7B | 44.4 | 46.0 | 33.0 |\n| **+AlignRAG-auto** | **54.0** | **56.1** | **42.1** |\n\nThese gains demonstrate that AlignRAG's critique-driven alignment extends well beyond QA and generalizes effectively to other RAG applications.\n\n> *How does ALIGNRAG perform in real-time or latency-sensitive applications?*\n\nThank you for these important practical questions. Our primary strategy for near-real-time performance is the AlignRAG-auto variant, which we detailed in the \"Computational Overhead\" response. `By dynamically skipping the refinement step for ~55% of inputs`, it drastically cuts average latency while maintaining high accuracy, making it suitable for many latency-sensitive scenarios.\n\n> *Can the CLM be distilled or compressed for efficiency?*\n\nYes, the CLM can be efficiently compressed, and this is a key strength of our Contrastive Critique Synthesis (CCS)** method. As shown in Figure 5 and discussed in Section 5.3 (\"Superior Alignment Guidance...\"), our framework is designed for this. Our results demonstrate that an `8B CLM trained via CCS using guidance from a 72B model actually outperforms the 72B model used directly as a critic by 2.2%`.\n\nThis proves that not only *can* the CLM be distilled, but that our CCS method is exceptionally effective at creating small, powerful critic models that are both efficient and more effective than their larger teachers.\n\n> *How does the system handle conflicting evidence in retrieved documents?*\n\nThis is an important, forward-looking question. The current CLM is optimized for improving alignment and coherence based on input evidence, but it is not yet explicitly trained for conflict resolution. This is a valuable direction for future work, and we will note this limitation in the revised manuscript.\n\n> *Could this framework be extended to multi-modal RAG?*\n\nThis is a fantastic idea. We believe the core principles of critique-driven refinement can be extended to multi-modal settings. While out of scope for this work, we plan to explore this in future research and will mention it in our discussion.\n\n> *What are the failure cases where AlignRAG underperforms compared to simpler baselines?*\n\nThank you for this thoughtful suggestion. We will include a new `\"Failure Analysis\"` subsection in the appendix. A representative failure example is shown below:\n\n| **Aspect**| **Details**|\n|:-----------------------|:-----------------------------------------------------------------------------|\n| **Question**  | *When was the first 10 dollar bill made?* |\n| **Golden Answer** | 1861 |\n| **Initial Response**   | \"The first 10 dollar bill was issued in 1911.\"|\n| **Retrieved Evidence** | No document explicitly mentioned 1861 |\n| **Critic Evaluation**  | `[Bad]` — Critique proposed 1914 based on CLM prior, not evidence|\n| **Refined Response** | \"The first $10 bill was made in 1914.\" |\n| **Failure Mode** | **Over-correction** due to missing evidence; CLM injects unsupported facts|\n| **Root Cause** | Incomplete retrieval → Critic relies on internal knowledge → factual drift|\n| **Takeaway** | AlignRAG can degrade accuracy when retrieval is weak and hallucination risk is high |\n\nThis illustrates a key limitation: without sufficient context, the CLM may inject unsupported corrections. Addressing such cases remains an important area for future improvement.\n\n> *Limitations: Effectiveness diminishes when none of the retrieved documents are relevant. Minor tuning may be required for optimal integration with different LLMs ...*\n\nAs discussed above, the framework's effectiveness may decrease in cases of \"extreme retrieval failure.\"\n\nIntegration and Tuning: AlignRAG was designed as a plug-and-play module, emphasizing broad compatibility and minimal tuning overhead. Our experiments confirm this goal: AlignRAG consistently improves performance across three instruction-tuned backbones—`Qwen2.5-7B`, `Qwen2.5-14B`, and `LLaMA3.1-8B`.\n\nTo further validate its plug-and-play nature, we integrated AlignRAG with strong baselines like `InstructRAG` and `IRCoT`, without modifying the base models. For example, using AlignRAG with InstructRAG yielded a +9.4% gain in OOD accuracy on Qwen2.5-14B, along with consistent improvements across all models and benchmarks.\n\nWhile further fine-tuning could offer incremental gains, our results demonstrate that AlignRAG is effective as a zero-modification, test-time module—making it practical for existing RAG pipelines. We appreciate the suggestion and plan to evaluate more models in future work.\n\n**We thank the reviewer for providing valuable feedback and we welcome further discussion if any of your concerns are not answered here!**"}]

Author rebuttal:

Meta-review: The paper introduces AlignRAG that reconceptualizes RAG as Retrieval-Augmented Reasoning. The central problem identified is Reasoning Misalignment — a failure where model reasoning drifts from retrieved evidence. To address this, the authors propose a critique-driven alignment loop. The critique model generates rationales and iteratively refines them to better align with evidence. Experiments across 7 benchmarks and three LLM families show that AlignRAG improves factual accuracy, robustness to noisy retrieval, and even outperforms larger models.

All reviewers agreed that the problem is important and the solution is technically sound. Reviewers praised the conceptual novelty (pwHC, Kguz), practical design (plug-and-play, modular), and robustness results. During rebuttal, the authors provided concrete case studies (addressing UcqY, sEZn), runtime and efficiency analyses including AlignRAG-auto (addressing Kguz, pwHC), and new baseline comparisons (addressing sEZn, Kguz, pwHC). 

Most concerns were addressed during rebuttal and all reviewers agree the paper has meet the acceptance bar.
Decision: Accept (poster)