KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG

## Abstract

Abstract Retrieval-Augmented Generation (RAG) enables large language models (LLMs) to access broader knowledge sources, yet factual inconsistencies persist due to noise in retrieved documents—even with advanced retrieval methods. We demonstrate that enhancing generative models’ capacity to process noisy content is equally critical for robust performance. In this paper, we present KARE-RAG (Knowledge-Aware Refinement and Enhancement for RAG), which improves knowledge utilization through three key innovations: (1) structured knowledge representations that facilitate error detection during training, (2) Dense Direct Preference Optimization (DDPO)—a refined training objective that prioritizes correction of critical errors, and (3) a contrastive data generation pipeline that maintains semantic consistency while rectifying factual inaccuracies. Experiments show our method significantly enhances standard RAG pipelines across model scales, improving both in-domain and out-of-domain task performance without compromising general capabilities. Notably, these gains are achieved with modest training data, suggesting data-efficient optimization is possible through targeted learning strategies. Our findings establish a new direction for RAG improvement: by improving how models learn to process retrieved content, we can enhance performance across diverse inference paradigms. All data and code will be publicly available on Github.

## 1 Introduction

Retrieval-Augmented Generation (RAG) Lewis et al. (2020); Shi et al. (2023) has emerged as a useful approach for knowledge-intensive tasks, enabling large language models (LLMs) to retrieve relevant information and generate informed responses. However, retrieval often brings noisy, irrelevant, or conflicting informationGao et al. (2023); Yoran et al. (2023); Wu et al. (2024); Xu et al. (2024a, b); Longpre et al. (2022); Liu et al. (2024a), causing factual inconsistencies. While existing work primarily addresses this through improved retrieval methodsJiang et al. (2023); Gao et al. (2023); Trivedi et al. (2022a); Mao et al. (2024); Ma et al. (2023); Jeong et al. (2024) or post-processing pipelinesYu et al. (2023); Xu et al. (2023); Fang et al. (2024), recent studies reveal an equally critical challenge: optimizing generation models to reliably process retrieval contextsSingh et al. (2021); Lin et al. (2023); Asai et al. (2023); Li et al. (2024a)

To optimize the generation model, a natural approach involves constructing instruction-tuning datasets from final answers and applying supervised fine-tuning (SFT) to the generation module Lin et al. (2023); Asai et al. (2023). However, the inherent complexity of RAG tasks makes models prone to overfitting training signals while suffering from catastrophic forgetting. The demonstrated effectiveness of Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. (2022) and Direct Preference Optimization (DPO) Rafailov et al. (2024) in aligning large language models has consequently spurred interest in adapting these preference-based methods for RAG’s generation module. Recent studies further validate DPO’s advantages in RAG scenarios Li et al. (2024a), showing its ability to provide stable fine-tuning through pairwise preference learning.

Nevertheless, significant challenges hinder DPO’s broader adoption in RAG systems. The multi-stage nature of RAG architectures Gao et al. (2023), comprising at least document parsing and answer generation in each retrieval cycle, presents fundamental difficulties. While DPO training utilizes pairwise samples, data constructed directly from final answers fails to adequately capture the nuances of this intricate processing pipeline, often requiring prohibitively large datasets to achieve satisfactory results Li et al. (2024a). Additionally, DPO methods suffer from low sampling efficiency during training. Generating high-quality positive examples through conventional sampling and prompt adjustment proves particularly challenging, especially for smaller models, further compounding training difficulties.

To address these issues, we develop KARE-RAG(nowledge-Aware Refinement and Enhancement for RAG)—a training framework that teaches models to better utilize. Our main contributions are as follows:

Structured Knowledge Supervision Mechanism. We introduce graph-based intermediate representations that transform noisy documents into structured knowledge graphs during training. This creates explicit learning objectives for distinguishing relevant content, addressing the limitation of end-to-end approaches that directly optimize final answers Lin et al. (2023). The structured format provides verifiable supervision signals, enabling models to learn noise-resistant information processing patterns.

Dense Direct Preference Optimization (DDPO). We deploy an enhanced version of DPO incorporating token-level importance weighting, which dynamically prioritizes correction of critical knowledge discrepancies. Through contrastive analysis of sample pairs where minimal representation differences determine answer correctness, DDPO teaches robust information filtering strategies that generalize to standard RAG pipelines Rafailov et al. (2024).

Automated Contrastive Data Generation. We develop an LLM-powered pipeline that generates high-quility training pairs while preserving semantic-structural consistency through three-stage self-verification: 1) Error localization, 2) Targeted correction, and 3) Validation filtering. This eliminates reliance on human annotation and overcomes the positive sample scarcity problem inherent to complex RAG workflows, ensuring models learn substantive knowledge discrimination rather than superficial pattern matching.

Through comprehensive empirical evaluation, our approach demonstrates statistically significant improvements across both in-domain and out-of-domain benchmarks, evidencing enhanced robustness in knowledge utilization and superior generalization capabilities. Furthermore, we systematically evaluate multiple knowledge representation formats—including knowledge graph, keypoint structure, and summary—revealing that structured representations yield substantially better performance compared to unstructured summarization baselines. Crucially, the method maintains efficacy across varying model scales, achieving consistent performance gains without compromising baseline capabilities on general language understanding tasks.

Figure: Figure 1: Illustration of our KARE-RAG Method. The left side of the image illustrates the differences between our method and Vanilla RAG, while the right side outlines the general workflow of our data construction process.
Refer to caption: extracted/6506396/figures/KARE-RAG.drawio.png

## 2 Related Work

LLMs possess a robust in-context learning capabilityDong et al. (2022); Ram et al. (2023), enabling them to handle more complex tasks. Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating external knowledge retrievalKarpukhin et al. (2020); Xiong et al. (2020), improving performance in tasks like question answeringTrivedi et al. (2022a) and fact verification Lewis et al. (2020). It can reduce the model’s propensity for hallucination and increase the reliability of the model’s outputsKandpal et al. (2023); Xu et al. (2023); Luo et al. (2023). However, retrieval often brings noisy or conflicting information, causing factual inaccuraciesLongpre et al. (2021); Xu et al. (2024a); Liu et al. (2024a).

To mitigate the impact of noise on generation effectiveness, some studies have introduced additional steps to enhance retrieval precision. For instance, several methods employ query rewrite techniques to increase the similarity between the query and the relevant textsWang et al. (2023); Mao et al. (2023), some others prompt LLMs to summarize query related informations into noteYu et al. (2023).

Optimizing the RAG process to enhance generation outcomes is also a highly popular direction. Many current efforts are exploring various methods to construct training data for training retrieval models to improve retrieval accuracyLin et al. (2023); Shi et al. (2023). Self-RAG trains models to learn to retrieve relevant documents on demandAsai et al. (2023). DDR constructs a multi-level training framework based on the final generation outcomes, simultaneously training both retrieval and generation models, enabling LLMs to learn to handle conflicts between intrinsic knowledge and retrieved documentsLi et al. (2024a).

Reinforcement learning is a commonly used algorithm in the optimization of large models. Due to the complexity of the PPO algorithmSchulman et al. (2017), current work predominantly relies on the Direct Preference Optimization(DPO) algorithmRafailov et al. (2024), which aligns the outputs of LLMs with human preferences to optimize large models. There are also numerous efforts to further refine the DPO algorithm to enhance training effectiveness. RLHF-VYu et al. (2024) increases the weight of tokens in the differing parts of positive and negative examples during training, improving the training outcomes for visual models. Step-DPOLai et al. (2024) focuses on step-by-step optimization for long-chain reasoning processes. Additionally, some studies have explored the relationship between DPO and SFT LossLiu et al. (2024b), demonstrating that SFT Loss can serve as a regularization term to stabilize the training effects of DPO.

Knowledge Graphs (KG)Hogan et al. (2021) are widely used in NLP tasks such as question answering, reasoningChen et al. (2020), and fact verification Tong et al. (2019). A significant advantage of knowledge graphs is their structured nature, which can effectively reduce noise in natural language texts, benefiting the RAG scenario. Consequently, numerous RAG works based on knowledge graphs have emergedPeng et al. (2024); Edge et al. (2024); Hu et al. (2024); Mavromatis and Karypis (2024). Current research on knowledge graphs primarily focuses on utilizing them to encode text libraries to enhance retrieval effectivenessLi et al. (2023); Huang et al. (2023); Li et al. (2024b), or integrating GNN with LLMs in the generation phase to improve output qualityGao et al. (2024); He et al. (2024); Jiang et al. (2022). These methods typically require preliminary processes to construct static knowledge graphs, making the overall workflow relatively complex.

## 3 Method

In this section, we introduce the Knowledge-Aware Refinement and Enhancement for RAG(KARE-RAG) method, shown in Figure [1](https://arxiv.org/html/2506.02503v1#S1.F1). We First introduce the Knowledge-Aware RAG(KA-RAG) pipeline, which decompose the generation process of RAG into three step to explicitly demonstrate how the model organizes and utilizes retrieved documents (left part of Figure [1](https://arxiv.org/html/2506.02503v1#S1.F1)). Then, we introduce the DDPO(Dense Direct Preference Optimization) training method that optimizes the model to focus more on the nuanced differences between data pairs. Finally we will introduce our data generation pipeline(right part of Figure [1](https://arxiv.org/html/2506.02503v1#S1.F1)).

### 3.1 KA-RAG Pipeline

In a standard RAG (Retrieval-Augmented Generation) process, the LLM is given a query $q$ and a set of retrieved documents $D=\{d_{1},\dots,d_{n}\}$ relevant to $q$ to generate answer to the given query. The retrieved documents invariably contain noise—irrelevant passages, conflicting information, or partial matches—that propagates through the generation process. This noise contamination systematically degrades output quality, introducing factual inaccuracies and information gaps in the final responses.

To explicitly demonstrate how the model organizes question-relevant information from the documents and generates answers, we decompose the generation process into three distinct stages in our experiments. The first stage structured knowledge representation, the second stage focuses on CoT (Chain-of-Thought) reasoning, and the final stage generates the answer based on the reasoning chain. The process can be formated as follow:
Knowledge Organization

$$ $y_{\mathrm{KG}}=\operatorname{LLM}(\mathrm{Instruct_{KG}},q\oplus D)$ (1) $$

CoT

$$ $y_{\mathrm{CoT}}=\operatorname{LLM}(\mathrm{Instruct_{CoT}},q\oplus y_{\mathrm {KG}})$ (2) $$

Generation

$$ $y_{\mathrm{Gen}}=\operatorname{LLM}(\mathrm{Instruct_{Gen}},q\oplus y_{\mathrm {CoT}})$ (3) $$

where $\oplus$ denote the concatenation operation, $\mathrm{Instruct_{KG}},\mathrm{Instruct_{CoT}},\mathrm{Instruct_{Gen}}$ are prompts specifically designed to address the three corresponding stages.

In the knowledge organization phase, we formalize the extracted information as structured graph representations. This graph-based formalism, inspired by knowledge graph architectures, provides three key representational advantages: (1) explicit node-edge relationships that enforce logical consistency, (2) discrete knowledge units that enable precise error localization, and (3) topological constraints that prevent information entanglement. The resulting structured representation serves as an intermediate knowledge substrate that is both human-interpretable and machine-tractable, establishing a robust foundation for subsequent processing stages.

### 3.2 Dense Direct Preference Optimization

Direct Preference Optimization (DPO) aligns language models with human preferences through pairwise comparison of preferred and dispreferred outputs. The method optimizes the relative likelihood of positive versus negative examples using a Bradley-Terry preference model formulation Rafailov et al. (2024). However, two fundamental limitations emerge when applying standard DPO to knowledge organization tasks: First, its uniform token-weighting scheme proves inefficient for lengthy, structured outputs where discriminative signals concentrate in sparse critical segments (e.g., entity relationships in knowledge graphs). Second, the reward maximization objective forces artificial divergence between semantically similar pairs, causing both sequence probabilities to depress simultaneously, which destabilizes training dynamics and reduces sample efficiency.

To address these issues, we adopted a variant of the DPO algorithm proposed Yu et al. (2024), known as Dense Direct Preference Optimization (DDPO). To make things clearer, we briefly review the DPO algorithm. The reward function for a specific output $y$ of input $x$ is represented as follows:

$$ $r(x,y)=\beta\log\frac{\pi_{*}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z (x)$ (4) $$

Where $\beta$ is a constant hyperparameter, and $Z(x)$ is a partition function. $\pi_{ref}(y|x)$ is the base model we want to optimize, and kept fixed during training. $\pi_{*}(y|x)$ is the model we actually updated. Then we can get the DPO loss:

$$ $\displaystyle\mathcal{L}_{\mathrm{DPO}}$ $\displaystyle=-\mathbb{E}_{(x,y^{+},y^{-})}\biggl{[}\log\sigma\Bigl{(}r(x,y^{+ })-r(x,y^{-})\Bigr{)}\biggr{]}$ (5) $\displaystyle=-\mathbb{E}_{(x,y^{+},y^{-})}\biggl{[}\log\sigma\biggl{(}\beta \biggl{(}\log\frac{\pi_{*}(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}$ $\displaystyle\quad-\log\frac{\pi_{*}(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)} \biggr{)}\biggr{)}\biggr{]}$ $$

DPO algorithm treats different token with uniform weight, and the score can be calculated as follow:

$$ $\log\pi(y\mid x)=\sum_{y_{t}\in y}\log p(y_{t}\mid x,y_{<t})$ (6) $$

$y_{t}$ is the $t$-th token of the response $y$. To ensure that the model pays more attention to the modified tokens $y_{c}$ compared to the unmodified tokens $y_{u}$ during optimization, we introduce a weighting mechanism that assigns higher weights to the $y_{c}$ tokens when computing the score. The modified score calculation formula is as follows:

$$ $\displaystyle\log\pi(y\mid x)$ $\displaystyle=\sum_{y_{t}\in y_{u}}\log p(y_{t}\mid x,y_{<t})$ (7) $\displaystyle+\gamma\sum_{y_{t}\in y_{c}}\log p(y_{t}\mid x,y_{<t})$ $$

$\gamma$ is a hyperparameter utilized to modulate the weight of tokens within $y_{c}$ when computing the score, while the weight of tokens in $y_{u}$ remains constant at 1.

While DDPO demonstrates effectiveness in addressing key limitations of standard DPO, we observe that the simultaneous reward reduction phenomenon persists in certain cases. To mitigate this effect, we integrate supervised fine-tuning (SFT) loss as a regularization term, following recent theoretical insights Liu et al. (2024b). The complete loss function combines these components as follows:

$$ $\mathcal{L}=\mathcal{L}_{\mathrm{DPO}}-\alpha\sum_{t=1}^{T}\log p(y_{t}\mid x, y_{<t})$ (8) $$

$\alpha$ is a hyperparameter that governs the weighting of the SFT loss within the overall loss function.

### 3.3 Data Generation

The inherent capabilities of large models are generally robust. Therefore, the negative examples ($y^{-}_{\mathrm{KG}}$) are mostly structurally coherent but may contain localized inaccuracies. A fundamental challenge arises when sampling both positive examples using the same base model ($\mathrm{LLM_{Gen}}$). Although theoretically capable of producing corrected outputs, the model’s stochastic generation process introduces unintended variations in two critical dimensions: (1) knowledge sequencing patterns and (2) surface-level phrasing. These variations become particularly pronounced in lengthy outputs, artificially inflating the perceived differences between positive ($y^{+}_{\mathrm{KG}}$) and negative pairs beyond their substantive errors.

Our framework addresses this through targeted refinement. By employing an advanced model ($\mathrm{LLM_{Exp}}$) to systematically edit $y^{-}_{\mathrm{KG}}$ while preserving its structural backbone, we ensure contrastive pairs differ only in critical error regions. This design guarantees minimal semantic divergence—essential for the model to learn error correction patterns rather than superficial variations. As detailed in Algorithm [1](https://arxiv.org/html/2506.02503v1#alg1), the workflow prioritizes structural fidelity through three core mechanisms: 1) Error localization, 2) Context-aware patching, and 3) Consistency validation, making it particularly effective for complex knowledge organization tasks.

Figure: Algorithm 1 KARE-RAG Data Construction

## 4 Experimental Settings

**Table 1: Overall Performance of KARE-RAG comparing to different baseline methods. The best result in each block is highlighted. The last two columns shows the average improvement of EM and F1 of current method compares to Vanilla RAG baseline.**
| Train<br>Method | In Domain | Out Of Domain |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Musique | NQ | HotpotQA | 2WikimultihopQA | PopQA | WebQ | Average Gain |  |  |  |  |  |  |  |  |  |
| EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) |  |  |
| Llama-3.2-3B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | 4.47 | 9.89 | 33.2 | 45.59 | 27.89 | 37.75 | 11.9 | 22.24 | 34.32 | 41.6 | 17.96 | 35.14 |  |  |  |
| \hdashline   Vanilla(DPO) | 6 | 16.18 | 25.2 | 39.45 | 14.06 | 28.04 | 8.79 | 20.22 | 32.86 | 40.19 | 19.59 | 35.55 | -4.95 | -3.77 |  |
| KARE(SFT) | 2.73 | 7.29 | 20.06 | 30.46 | 21.73 | 31.35 | 17.07 | 23.77 | 29.52 | 35.8 | 14.76 | 30.05 | -4.43 | -6.18 |  |
| KARE(DPO) | 5.05 | 13.08 | 32.82 | 46.35 | 26.94 | 38.78 | 14.65 | 24.93 | 38.32 | 45.47 | 19.29 | 36.54 | 1.35 | 1.95 |  |
| KARE(DDPO) | 5.59 | 13.28 | 33.21 | 46.56 | 27.62 | 39.59 | 15.5 | 25.53 | 38.29 | 45.52 | 19.44 | 36.97 | 1.76 | 2.37 |  |
| Llama-3.1-8B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | 6 | 12.37 | 34.64 | 48.3 | 29.52 | 40.66 | 14.94 | 24.89 | 35.43 | 44.1 | 16.83 | 35.33 |  |  |  |
| \hdashline   Vanilla(DPO) | 7.82 | 19.18 | 25.94 | 41.81 | 17.58 | 33.59 | 9.14 | 18.56 | 36.36 | 43.78 | 22.78 | 38.86 | -3.91 | -3.34 |  |
| KARE(SFT) | 4.88 | 10.46 | 33.62 | 44.1 | 23.69 | 32.77 | 24.37 | 29.39 | 35.88 | 40.73 | 24.16 | 36.13 | 2.07 | -2.03 |  |
| KARE(DPO) | 7.73 | 15.75 | 37.4 | 50.57 | 32.26 | 43.83 | 20.98 | 29.92 | 40.27 | 47.39 | 20.42 | 38.09 | 3.99 | 3.3 |  |
| KARE(DDPO) | 8.02 | 15.75 | 37.86 | 50.84 | 32.36 | 44.29 | 21.35 | 30.11 | 40.88 | 47.77 | 19.83 | 37.89 | 4.18 | 3.52 |  |
| Qwen2.5-14B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | 6.66 | 14.58 | 32.67 | 47.01 | 30.64 | 42.31 | 19.73 | 29.5 | 36.38 | 43.98 | 18.36 | 35.11 |  |  |  |
| \hdashline   Vanilla(DPO) | 9.68 | 19.84 | 12.63 | 24.37 | 19.22 | 30.37 | 24.55 | 31.85 | 16.3 | 21.35 | 13.68 | 26.65 | -10.28 | -12.66 |  |
| KARE(SFT) | 6.08 | 14.12 | 32.23 | 45.7 | 28.32 | 40.56 | 26.67 | 34.32 | 36.57 | 43.86 | 16.39 | 33.9 | 0.48 | 0.09 |  |
| KARE(DPO) | 7.57 | 16.58 | 34.99 | 49 | 32.84 | 45.47 | 23.43 | 32.4 | 38.12 | 45.65 | 19.64 | 36.64 | 2.25 | 2.25 |  |
| KARE(DDPO) | 8.32 | 17.48 | 36.19 | 49.96 | 33.96 | 46.45 | 25.14 | 33.72 | 38.58 | 46.09 | 20.03 | 36.95 | 3.22 | 3.05 |  |

## 5 Evaluation Results

In this section, we first compare the overall performance of our method with several baseline models. Subsequently, we conduct ablation experiments and analyses to further validate the effectiveness of our approach.

### 5.1 Main Results

**Table 2: Ablation Study. The test of different knowledge organization format under Vanilla RAG Pipeline.**
| Format | In Domain | Out Of Domain |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Musique | NQ | HotpotQA | 2WikimultihopQA | PopQA | WebQ |  |  |  |  |  |  |  |
| EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) |  |
|  | 6.0 | 12.37 | 34.64 | 48.3 | 29.52 | 40.66 | 14.94 | 24.89 | 35.43 | 44.1 | 16.83 | 35.33 |
| Graph | 8.02 | 15.75 | 37.86 | 50.84 | 32.36 | 44.29 | 21.35 | 30.11 | 40.88 | 47.77 | 19.83 | 37.89 |
| Keypoints | 7.36 | 16.31 | 34.73 | 49.06 | 30.07 | 42.93 | 18.04 | 29.06 | 41.24 | 48.38 | 19.19 | 37.75 |
| Summary | 5.67 | 14.2 | 32.77 | 47.11 | 24.43 | 37.44 | 11.43 | 24.4 | 39.22 | 46.72 | 17.77 | 35.55 |

The overall performance evaluation, as presented in Table [4](https://arxiv.org/html/2506.02503v1#S4), demonstrates that models trained with our KARE-RAG framework achieve consistent and reliable improvements across both in-domain and out-of-distribution tasks. Specifically, the DDPO-trained Llama-3.1-8B model exhibits average gains of +4.18% EM and +3.52% F1 over the Vanilla RAG baseline on out-of-domain tasks. Notably, our framework maintains strong scalability across different model architectures: the Llama-3.2-3B-Instruct model achieves +1.76% EM/+2.37% F1 average improvement, while Qwen2.5-14B-Instruct model shows gains of +3.22% EM/+3.05% F1. These improvements are particularly significant as they are attained without any modifications to the standard Vanilla RAG inference pipeline, underscoring the robustness and generalizability of our training approach.

Enhanced Knowledge Utilization through KARE-RAG Training. Our analysis reveals critical limitations in conventional training methods. While standard DPO training yields improved in-domain performance, it demonstrates unstable results on out-of-domain tasks (with average metrics showing degradation), suggesting susceptibility to overfitting and inadequate learning of document filtering strategies. Similarly, supervised fine-tuning (SFT) proves unreliable, particularly for smaller models like Llama-3.2-3B-Instruct, which suffers a substantial performance drop (-4.43% EM). In contrast, KARE-RAG trained models maintain consistent improvements across all evaluation scenarios, indicating genuine enhancement in information filtering and utilization capabilities.

Importance of Token-Level Weighting. The better performance of DDPO compared to standard DPO validates the importance of our token-level weighting mechanism. This approach enables more precise focus on discriminative features between positive and negative examples, leading to measurable improvements in model performance during inference. The consistent gains across different architectures and task domains further confirm the effectiveness of this design choice.

Figure: Figure 2: The Improvement of EM metrics under Vanilla RAG Pipeline with different knowledge organization format.
Refer to caption: extracted/6506396/figures/format_hist.png

### 5.2 Analysis

In this section, we conducted ablation experiments and analyses to further validate the effectiveness of our method.

Impact of Different Knowledge Organization Formats. We compare the effects of various intermediate information organization formats on training effectiveness to emphasize the importance of structured representation. Experiments on Llama-3.1-8B-Instruct were conducted using two additional formats: a semi-structured keypoints format (one key point per line) and an unstructured summary format (organizing information into a summary). Specific prompts are provided in the Appendix [A.6](https://arxiv.org/html/2506.02503v1#A1.SS6)

As illustrated in Table [2](https://arxiv.org/html/2506.02503v1#S5.T2) and Figure [2](https://arxiv.org/html/2506.02503v1#S5.F2), the structured graph representation leads to substantial improvements in both in-domain and out-of-domian tests, while the semi-structured keypoints format shows moderate gains. The unstructured summary format even shows a decline in performance after training.

The results strongly suggest that structural inductive biases should be incorporated into RAG training objectives when possible, as they provide clearer learning signals for knowledge-intensive tasks while maintaining compatibility with standard inference pipelines.

**Table 3: The results demonstrate the effectiveness of the KARE-RAG-trained model across various RAG generation pipelines, with the final two columns showing its average EM and F1 score improvements over the baseline (untrained) model for each pipeline.**
| Inference<br>Pipeline | Train<br>Method | In Domain | Out Of Domain |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Musique | NQ | HotpotQA | 2WikimultihopQA | PopQA | WebQ | Average Gain |  |  |  |  |  |  |  |  |  |
| EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) |  |  |
| Llama-3.2-3B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla |  | 4.47 | 9.89 | 33.2 | 45.59 | 27.89 | 37.75 | 11.9 | 22.24 | 34.32 | 41.6 | 17.96 | 35.14 |  |  |
| KARE-RAG | 5.59 | 13.28 | 33.21 | 46.56 | 27.62 | 39.59 | 15.5 | 25.53 | 38.29 | 45.52 | 19.44 | 36.97 | 1.76 | 2.37 |  |
| \hdashline<br>Knowledge<br>Aware |  | 4.39 | 10.7 | 25.57 | 37.33 | 21.34 | 31.76 | 15.6 | 21.92 | 32.27 | 38.46 | 17.42 | 35.2 |  |  |
|  | KARE-RAG | 6.99 | 15.8 | 28.51 | 40.72 | 28.85 | 38.79 | 26.67 | 33.45 | 36.96 | 42.24 | 18.65 | 36.68 | 5.49 | 5.44 |
| \hdashline<br>Chain Of<br>Thought |  | 5.01 | 11.8 | 28.03 | 41.01 | 24.0 | 35.43 | 14.15 | 21.07 | 32.92 | 39.99 | 16.98 | 34.98 |  |  |
|  | KARE-RAG | 7.24 | 16.8 | 29.31 | 42.73 | 26.41 | 39.03 | 22.57 | 30.69 | 36.23 | 43.42 | 16.14 | 35.64 | 2.92 | 3.81 |
| \hdashline<br>Chain Of<br>Note |  | 4.18 | 10.63 | 29.15 | 42.42 | 22.04 | 32.84 | 10.19 | 17.43 | 30.83 | 38.34 | 15.94 | 34.59 |  |  |
|  | KARE-RAG | 6.21 | 15.16 | 30.44 | 44.45 | 25.06 | 37.87 | 16.87 | 25.8 | 34.69 | 42.71 | 15.94 | 35.42 | 2.97 | 4.13 |
| Llama-3.1-8B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla |  | 6 | 12.37 | 34.64 | 48.3 | 29.52 | 40.66 | 14.94 | 24.89 | 35.43 | 44.1 | 16.83 | 35.33 |  |  |
| KARE-RAG | 8.02 | 15.75 | 37.86 | 50.84 | 32.36 | 44.29 | 21.35 | 30.11 | 40.88 | 47.77 | 19.83 | 37.89 | 4.18 | 3.52 |  |
| \hdashline<br>Knowledge<br>Aware |  | 7.57 | 15.13 | 33.07 | 46.03 | 30.2 | 41.17 | 18.54 | 25.78 | 38.79 | 44.76 | 20.13 | 37.86 |  |  |
|  | KARE-RAG | 11.42 | 20.53 | 36.21 | 49.36 | 33.98 | 46.01 | 30.61 | 37.92 | 41.92 | 47.55 | 22.1 | 39.95 | 4.82 | 5.04 |
| \hdashline<br>Chain Of<br>Thought |  | 8.52 | 16.4 | 35.63 | 48.48 | 31.11 | 42.45 | 16.71 | 23.52 | 37.71 | 44.35 | 20.47 | 38.8 |  |  |
|  | KARE-RAG | 10.34 | 19.28 | 37.22 | 50.28 | 34.4 | 46.82 | 23.88 | 31.5 | 40.38 | 46.96 | 22.0 | 39.63 | 3.25 | 3.52 |
| \hdashline<br>Chain Of<br>Note |  | 6.83 | 14.47 | 35.06 | 48.61 | 29.05 | 40.7 | 17.37 | 24.2 | 36.06 | 43.32 | 19.44 | 38.7 |  |  |
|  | KARE-RAG | 9.18 | 18.44 | 37.55 | 50.73 | 33.11 | 45.34 | 22.56 | 30.44 | 39.36 | 46.3 | 21.26 | 40.07 | 3.37 | 3.47 |
| Qwen2.5-14B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla |  | 6.66 | 14.58 | 32.67 | 47.01 | 30.64 | 42.31 | 19.73 | 29.5 | 36.38 | 43.98 | 18.36 | 35.11 |  |  |
| KARE-RAG | 8.32 | 17.48 | 36.19 | 49.96 | 33.96 | 46.45 | 25.14 | 33.72 | 38.58 | 46.09 | 20.03 | 36.95 | 3.22 | 3.05 |  |
| \hdashline<br>Knowledge<br>Aware |  | 9.02 | 17.67 | 36.75 | 50.15 | 35.91 | 48.25 | 24.51 | 30.88 | 39.2 | 45.95 | 19.88 | 37.82 |  |  |
|  | KARE-RAG | 11.09 | 20.78 | 37.19 | 50.46 | 37.49 | 50.56 | 34.65 | 42.12 | 40.65 | 47.3 | 20.72 | 38.95 | 2.89 | 3.27 |
| \hdashline<br>Chain Of<br>Thought |  | 9.02 | 17.04 | 37.99 | 51.16 | 34.8 | 46.66 | 21.15 | 28.08 | 38.28 | 45.35 | 19.78 | 37.48 |  |  |
|  | KARE-RAG | 10.72 | 21.63 | 37.34 | 51.07 | 36.88 | 50.22 | 29.61 | 38.37 | 39.42 | 46.87 | 18.8 | 37.36 | 2.01 | 3.03 |
| \hdashline<br>Chain Of<br>Note |  | 8.36 | 16.81 | 35.31 | 49.25 | 32.92 | 45.54 | 21.37 | 30.61 | 36.31 | 44.26 | 17.77 | 36.2 |  |  |
|  | KARE-RAG | 9.27 | 19.55 | 36.49 | 50.49 | 35.04 | 48.39 | 28.05 | 37.61 | 37.99 | 45.65 | 18.36 | 37.58 | 2.45 | 2.77 |

## 6 Conclusion

Our work presents KARE-RAG, a training-based approach for enhancing RAG systems. KARE-RAG introduces structured knowledge organization as an intermediate learning objective and employs DDPO to provide fine-grained optimization through dynamic weighting, enabling models to develop improved information filtering and integration capabilities. This training strategy enhances information processing robustness while maintaining full compatibility with standard RAG pipelines and existing enhancement techniques. Comprehensive experiments demonstrate consistent improvements in handling retrieval contexts across multiple benchmarks. KARE-RAG proves effective across different model scales and requires no architectural changes during deployment, suggesting training-based improvements can serve as a practical complement to existing RAG enhancement approaches.

## Limitations

While our method demonstrates strong empirical performance, several implementation characteristics warrant discussion.

Representation Format Scope. The current framework primarily utilizes graph-structured representations, which have shown strong experimental performance for modeling entity relationships. While this approach proves effective, we acknowledge that alternative structured formats—such as hierarchical trees for taxonomic knowledge or temporal sequences for event-based information—may offer complementary advantages and represent promising directions for future investigation.

Data Construction Pipeline. Although our refinement process handles the majority of cases effectively, certain challenging samples remain difficult to correct due to inherent limitations in the refinement model’s capabilities. This suggests two potential improvement avenues: (1) integration of dedicated reasoning modules for enhanced error verification, and (2) development of specialized refinement models targeting specific error patterns. These directions may further improve the pipeline’s robustness while maintaining its current efficiency.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*.
- Asai et al. (2023)
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023.
Self-rag: Learning to retrieve, generate, and critique through self-reflection.
*arXiv preprint arXiv:2310.11511*.
- Berant et al. (2013)
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.
Semantic parsing on freebase from question-answer pairs.
In *Proceedings of the 2013 conference on empirical methods in natural language processing*, pages 1533–1544.
- Chen et al. (2020)
Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020.
A review: Knowledge reasoning over knowledge graph.
*Expert systems with applications*, 141:112948.
- Dong et al. (2022)
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022.
A survey on in-context learning.
*arXiv preprint arXiv:2301.00234*.
- Edge et al. (2024)
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024.
From local to global: A graph rag approach to query-focused summarization.
*arXiv preprint arXiv:2404.16130*.
- Fang et al. (2024)
Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. 2024.
Trace the evidence: Constructing knowledge-grounded reasoning chains for retrieval-augmented generation.
*arXiv preprint arXiv:2406.11460*.
- Gao et al. (2024)
Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, and Dongsheng Li. 2024.
Two-stage generative question answering on temporal knowledge graph using large language models.
*arXiv preprint arXiv:2402.16568*.
- Gao et al. (2023)
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023.
Retrieval-augmented generation for large language models: A survey.
*arXiv preprint arXiv:2312.10997*.
- He et al. (2024)
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024.
G-retriever: Retrieval-augmented generation for textual graph understanding and question answering.
*arXiv preprint arXiv:2402.07630*.
- Hendrycks et al. (2020)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*.
- Ho et al. (2020)
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020.
Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.
*arXiv preprint arXiv:2011.01060*.
- Hogan et al. (2021)
Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. 2021.
Knowledge graphs.
*ACM Computing Surveys (Csur)*, 54(4):1–37.
- Hu et al. (2021)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*.
- Hu et al. (2024)
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2024.
Grag: Graph retrieval-augmented generation.
*arXiv preprint arXiv:2405.16506*.
- Huang et al. (2023)
Yongfeng Huang, Yanyang Li, Yichong Xu, Lin Zhang, Ruyi Gan, Jiaxing Zhang, and Liwei Wang. 2023.
Mvp-tuning: Multi-view knowledge retrieval with prompt tuning for commonsense reasoning.
In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 13417–13432.
- Jeong et al. (2024)
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024.
Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity.
*arXiv preprint arXiv:2403.14403*.
- Jiang et al. (2022)
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2022.
Great truths are always simple: A rather simple knowledge encoder for enhancing the commonsense reasoning capacity of pre-trained models.
*arXiv preprint arXiv:2205.01841*.
- Jiang et al. (2023)
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.
Active retrieval augmented generation.
*arXiv preprint arXiv:2305.06983*.
- Jin et al. (2024)
Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024.
[Flashrag: A modular toolkit for efficient retrieval-augmented generation research](https://arxiv.org/abs/2405.13576).
*CoRR*, abs/2405.13576.
- Kandpal et al. (2023)
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.
Large language models struggle to learn long-tail knowledge.
In *International Conference on Machine Learning*, pages 15696–15707. PMLR.
- Karpukhin et al. (2020)
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.
Dense passage retrieval for open-domain question answering.
*arXiv preprint arXiv:2004.04906*.
- Kwiatkowski et al. (2019)
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.
Natural questions: a benchmark for question answering research.
*Transactions of the Association for Computational Linguistics*, 7:453–466.
- Lai et al. (2024)
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024.
Step-dpo: Step-wise preference optimization for long-chain reasoning of llms.
*arXiv preprint arXiv:2406.18629*.
- Levy et al. (2017)
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.
Zero-shot relation extraction via reading comprehension.
*arXiv preprint arXiv:1706.04115*.
- Lewis et al. (2020)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
*Advances in Neural Information Processing Systems*, 33:9459–9474.
- Li et al. (2023)
Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. 2023.
Graph reasoning for question answering with triplet retrieval.
*arXiv preprint arXiv:2305.18742*.
- Li et al. (2024a)
Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu, Zhiyuan Liu, et al. 2024a.
Rag-ddr: Optimizing retrieval-augmented generation using differentiable data rewards.
*arXiv preprint arXiv:2410.13509*.
- Li et al. (2024b)
Zhuoyang Li, Liran Deng, Hui Liu, Qiaoqiao Liu, and Junzhao Du. 2024b.
Unioqa: A unified framework for knowledge graph question answering with large language models.
*arXiv preprint arXiv:2406.02110*.
- Lin et al. (2021)
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.
Truthfulqa: Measuring how models mimic human falsehoods.
*arXiv preprint arXiv:2109.07958*.
- Lin et al. (2023)
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023.
Ra-dit: Retrieval-augmented dual instruction tuning.
In *The Twelfth International Conference on Learning Representations*.
- Liu et al. (2024a)
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a.
Lost in the middle: How language models use long contexts.
*Transactions of the Association for Computational Linguistics*, 12:157–173.
- Liu et al. (2024b)
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. 2024b.
Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer.
*arXiv preprint arXiv:2405.16436*.
- Longpre et al. (2021)
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.
Entity-based knowledge conflicts in question answering.
*arXiv preprint arXiv:2109.05052*.
- Longpre et al. (2022)
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2022.
[Entity-based knowledge conflicts in question answering](https://arxiv.org/abs/2109.05052).
*Preprint*, arXiv:2109.05052.
- Luo et al. (2023)
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. 2023.
Sail: Search-augmented instruction learning.
*arXiv preprint arXiv:2305.15225*.
- Ma et al. (2023)
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023.
Query rewriting for retrieval-augmented large language models.
*arXiv preprint arXiv:2305.14283*.
- Mallen et al. (2022)
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
*arXiv preprint arXiv:2212.10511*.
- Mao et al. (2023)
Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. 2023.
Large language models know your contextual search intent: A prompting framework for conversational search.
*arXiv preprint arXiv:2303.06573*.
- Mao et al. (2024)
Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2024.
Rafe: Ranking feedback improves query rewriting for rag.
*arXiv preprint arXiv:2405.14431*.
- Mavromatis and Karypis (2024)
Costas Mavromatis and George Karypis. 2024.
Gnn-rag: Graph neural retrieval for large language model reasoning.
*arXiv preprint arXiv:2405.20139*.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744.
- Peng et al. (2024)
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024.
Graph retrieval-augmented generation: A survey.
*arXiv preprint arXiv:2408.08921*.
- Rafailov et al. (2024)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36.
- Ram et al. (2023)
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.
In-context retrieval-augmented language models.
*Transactions of the Association for Computational Linguistics*, 11:1316–1331.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*.
- Shi et al. (2023)
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.
Replug: Retrieval-augmented black-box language models.
*arXiv preprint arXiv:2301.12652*.
- Singh et al. (2021)
Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. 2021.
End-to-end training of multi-document reader and retriever for open-domain question answering.
*Advances in Neural Information Processing Systems*, 34:25968–25981.
- Team (2024)
Qwen Team. 2024.
[Qwen2.5: A party of foundation models](https://qwenlm.github.io/blog/qwen2.5/).
- Tong et al. (2019)
Peihao Tong, Qifan Zhang, and Junjie Yao. 2019.
Leveraging domain context for question answering over knowledge graph.
*Data Science and Engineering*, 4(4):323–335.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*.
- Trivedi et al. (2022a)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a.
Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.
*arXiv preprint arXiv:2212.10509*.
- Trivedi et al. (2022b)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b.
[Musique: Multihop questions via single-hop question composition](https://arxiv.org/abs/2108.00573).
*Preprint*, arXiv:2108.00573.
- von Werra et al. (2020)
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020.
Trl: Transformer reinforcement learning.
[https://github.com/huggingface/trl](https://github.com/huggingface/trl).
- Wang et al. (2023)
Liang Wang, Nan Yang, and Furu Wei. 2023.
Query2doc: Query expansion with large language models.
*arXiv preprint arXiv:2303.07678*.
- Wang et al. (2024)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.
*arXiv preprint arXiv:2406.01574*.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837.
- Wu et al. (2024)
Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024.
How easily do irrelevant inputs skew the responses of large language models?
*arXiv preprint arXiv:2404.03302*.
- Xiao et al. (2023)
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.
[C-pack: Packaged resources to advance general chinese embedding](https://arxiv.org/abs/2309.07597).
*Preprint*, arXiv:2309.07597.
- Xiong et al. (2020)
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.
Approximate nearest neighbor negative contrastive learning for dense text retrieval.
*arXiv preprint arXiv:2007.00808*.
- Xu et al. (2023)
Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023.
Recomp: Improving retrieval-augmented lms with compression and selective augmentation.
*arXiv preprint arXiv:2310.04408*.
- Xu et al. (2024a)
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2024a.
Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks.
In *Proceedings of the ACM on Web Conference 2024*, pages 1362–1373.
- Xu et al. (2024b)
Zhipeng Xu, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Chaojun Xiao, Zhiyuan Liu, Ge Yu, and Chenyan Xiong. 2024b.
[Activerag: Autonomously knowledge assimilation and accommodation through retrieval-augmented agents](https://arxiv.org/abs/2402.13547).
*Preprint*, arXiv:2402.13547.
- Yang et al. (2024)
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024.
Qwen2 technical report.
*arXiv preprint arXiv:2407.10671*.
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
*arXiv preprint arXiv:1809.09600*.
- Yoran et al. (2023)
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023.
Making retrieval-augmented language models robust to irrelevant context.
*arXiv preprint arXiv:2310.01558*.
- Yu et al. (2024)
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024.
Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13807–13816.
- Yu et al. (2023)
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023.
Chain-of-note: Enhancing robustness in retrieval-augmented language models.
*arXiv preprint arXiv:2311.09210*.

## Appendix A Appendix

### A.1 Training Settings for Vanilla(DPO)

Due to the inclusion of knowledge organization and CoT processes in the training data constructed by the KARE-RAG method, it is not suitable for direct use in Vanilla(DPO). Therefore, we made certain adjustments to the data collection method and reconstructed a batch of training data.

Data Generation. The data construction process is simpler compared to KARE-RAG, with the overall workflow illustrated in Algorithm [2](https://arxiv.org/html/2506.02503v1#alg2). The volume of data is slightly larger than that of KARE-RAG, with approximately 4k data points for different models. We also attempted to obtain the same amount of training data as the KARE-RAG method through sampling for experiments, but the differences in experimental results were not significant. Consequently, we ultimately did not sample the training data.

Implementation Details. When training with standard DPO alone, the process is prone to instability. Therefore, in our experiments, we also incorporated SFT Loss as a regularization term. We trained for one epoch with a learning rate set to 5e-5 and the weight of the SFT Loss set to 0.1. We also experimented with training using DDPO, but since the positive and negative examples obtained from Vanilla RAG were relatively brief, it did not yield satisfactory results.

### A.2 Computational Efficiency

Our framework demonstrates strong practical efficiency in both data construction and model training phases. For data generation, the pipeline leverages GPT-4o-mini’s capability to perform targeted refinements, keeping the construction cost around $20. The training process employs parameter-efficient LoRA adapters combined with DeepSpeed-Zero3 optimization, enabling efficient fine-tuning on a single A100-40G GPU. Specifically, training completes in under 1 hour for 3B models and within 4 hours for 14B models.

Figure: Algorithm 2 Vanilla RAG Data Construction

### A.3 Evaluation on Non-QA Tasks

**Table 5: Evaluation results on non-QA tasks: multiple-choice question dataset TruthfulQA and slot filling dataset Zero-shot RE. The best result in each model group is highlighted.**
| Train Method | TruthfulQA | Zero-shot RE |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Rouge-1(%) | BLEU(%) | Precision(%) | Recall(%) | F1(%) |  |
| Llama-3.2-3B-Instruct |  |  |  |  |  |
|  | 13.54 | 6.09 | 47.79 | 54.35 | 49.06 |
| KARE-RAG | 18.78 | 7.75 | 48.69 | 62.0 | 51.53 |
| Llama-3.1-8B-Instruct |  |  |  |  |  |
|  | 14.8 | 5.49 | 49.48 | 60.16 | 51.87 |
| KARE-RAG | 17.76 | 7.42 | 56.48 | 64.55 | 58.29 |
| Qwen2.5-14B-Instruct |  |  |  |  |  |
|  | 18.98 | 7.26 | 49.37 | 68.05 | 53.32 |
| KARE-RAG | 20.08 | 8.68 | 50.14 | 68.95 | 54.1 |

To comprehensively evaluate the generalization capability of KARE-RAG beyond question answering, we conduct additional experiments on two representative non-QA tasks: multiple-choice question dataset TruthfulQA and slot filling dataset Zero-shot RE. The results in Table [5](https://arxiv.org/html/2506.02503v1#A1.T5) demonstrate consistent performance gains across these diverse task formats.

The TruthfulQA benchmark assesses models’ ability to reject misleading but statistically plausible answers. Our findings reveal consistent improvements across all model sizes, the structured knowledge training helps models better recognize factual inconsistencies. We can observe similar improvement in the result of Zero-shot RE task. These results substantiate that KARE-RAG’s advantages extend beyond its original QA design context, regardless of output format, providing evidence for its general applicability to knowledge-intensive NLP tasks.

Figure: (a) Reward/accuracy
Refer to caption: extracted/6506396/figures/train_rewards_accuracies.png

### A.4 Ablation study on the needs for SFT Loss

In this section, we validate the role of incorporating SFT Loss during the preference learning training process through a series of comparative experiments and analyses. As shown in Table [A.5](https://arxiv.org/html/2506.02503v1#A1.SS5), incorporating both DPO Loss and SFT Loss significantly improves the model’s training effectiveness and stability on OOD tasks, compared to using DPO Loss alone. Standard DPO Loss amplifies the reward difference between positive and negative examples, increasing the likelihood of sampling positive examples (Equation [5](https://arxiv.org/html/2506.02503v1#S3.E5)). Figures [3(a)](https://arxiv.org/html/2506.02503v1#A1.F3.sf1) and [3(b)](https://arxiv.org/html/2506.02503v1#A1.F3.sf2) show that standard DPO behaves as expected, with reward/accuracy approaching 1 and reward/margin increasing. However, due to the minimal differences between positive and negative examples and the lack of regularization in DPO, the model often overfits, causing both reward/chosen and reward/rejected to decrease (Figure [3(c)](https://arxiv.org/html/2506.02503v1#A1.F3.sf3)). This indicates that the model outputs fewer positive examples than the untrained model, leading to suboptimal training. Including SFT Loss mitigates this issue, keeping reward/chosen positive, maintaining reward/accuracy near 1, and preventing overfitting.

### A.5 Evaluation on IRCOT pipeline

We further validate our method on IRCOT (Iterative Retrieval Chain-of-Thought), a more complex RAG framework that requires multi-round retrieval and reasoning. As shown in Table [7](https://arxiv.org/html/2506.02503v1#A1.T7), KARE-RAG demonstrates consistent improvements across this challenging setup. The Llama-3.1-8B model shows particularly strong gains (+10.6% average accuracy), proving our training method’s effectiveness even when handling intricate multi-step retrieval scenarios. These results confirm that the knowledge refinement capabilities learned by KARE-RAG generalize well to sophisticated RAG architectures requiring iterative information gathering and reasoning.

**Table 6: Performance Evaluation on Non-QA Tasks across Model Scales. Results demonstrate KARE-RAG’s effectiveness on TruthfulQA (multiple-choice QA) and Zero-shot RE (slot filling) tasks. The best result in each model group is highlighted.**
| Inference<br>Pipeline | Train<br>Method | In Domain | Out Of Domain |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Musique | NQ | HotpotQA | 2WikimultihopQA | PopQA | WebQ | Average Gain |  |  |  |  |  |  |  |  |  |
| EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) | EM(%) | F1(%) |  |  |
| Llama-3.2-3B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla |  | 4.47 | 9.89 | 33.2 | 45.59 | 27.89 | 37.75 | 11.9 | 22.24 | 34.32 | 41.6 | 17.96 | 35.14 |  |  |
| \hdashline<br>Knowledge<br>Aware |  | 4.39 | 10.7 | 25.57 | 37.33 | 21.34 | 31.76 | 15.6 | 21.92 | 32.27 | 38.46 | 17.42 | 35.2 | -2.61 | -3.53 |
|  | DDPO | 6.99 | 15.8 | 28.51 | 40.72 | 28.85 | 38.79 | 26.67 | 33.45 | 36.96 | 42.24 | 18.65 | 36.68 | 2.87 | 1.91 |
|  | DDPO w/o SFT | 6.45 | 16.61 | 23.33 | 37.53 | 23.2 | 36.35 | 24.49 | 35.14 | 31.98 | 39.1 | 14.12 | 33.82 | -1.63 | -0.08 |
| Llama-3.1-8B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla |  | 6 | 12.37 | 34.64 | 48.3 | 29.52 | 40.66 | 14.94 | 24.89 | 35.43 | 44.1 | 16.83 | 35.33 |  |  |
| \hdashline<br>Knowledge<br>Aware |  | 7.57 | 15.13 | 33.07 | 46.03 | 30.2 | 41.17 | 18.54 | 25.78 | 38.79 | 44.76 | 20.13 | 37.86 | 1.87 | 0.46 |
|  | DDPO | 11.42 | 20.53 | 36.21 | 49.36 | 33.98 | 46.01 | 30.61 | 37.92 | 41.92 | 47.55 | 22.1 | 39.95 | 6.69 | 5.5 |
|  | DDPO w/o SFT | 10.01 | 19.77 | 32.03 | 44.96 | 28.57 | 41.67 | 30.74 | 39.67 | 36.06 | 42.11 | 20.17 | 39.22 | 3.24 | 2.87 |
| Qwen2.5-14B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla |  | 6.66 | 14.58 | 32.67 | 47.01 | 30.64 | 42.31 | 19.73 | 29.5 | 36.38 | 43.98 | 18.36 | 35.11 |  |  |
| \hdashline<br>Knowledge<br>Aware |  | 9.02 | 17.67 | 36.75 | 50.15 | 35.91 | 48.25 | 24.51 | 30.88 | 39.2 | 45.95 | 19.88 | 37.82 | 3.69 | 3.03 |
|  | DDPO | 11.09 | 20.78 | 37.19 | 50.46 | 37.49 | 50.56 | 34.65 | 42.12 | 40.65 | 47.3 | 20.72 | 38.95 | 6.58 | 6.3 |
|  | DDPO w/o SFT | 10.47 | 21.14 | 32.56 | 46.85 | 35.03 | 49.66 | 32.95 | 43.01 | 38.12 | 46.04 | 15.6 | 34.83 | 3.3 | 4.5 |

### A.6 KA-RAG Pipeline Prompts

In this section, we present all the prompts utilized in our experiments. For Vanilla RAG, we directly used the standard prompt template provided by FlashRAGJin et al. (2024) without any modifications. Table [8](https://arxiv.org/html/2506.02503v1#A1.T8) contains the prompts for the KA-RAG Pipeline when employing a Knowledge Graph as the intermediate representation for knowledge organization. Tables [9](https://arxiv.org/html/2506.02503v1#A1.T9) and [10](https://arxiv.org/html/2506.02503v1#A1.T10) display the prompts for the scenarios using Keypoints and Summary, respectively. Table [11](https://arxiv.org/html/2506.02503v1#A1.T11) showcases the prompts we employed during the data construction phase. When constructing training data for Vanilla(DPO) in Appendix [A.1](https://arxiv.org/html/2506.02503v1#A1.SS1), we use the same prompts in Table [11](https://arxiv.org/html/2506.02503v1#A1.T11) to check answerability.

**Table 8: KA-Pipeline Prompt(Graph Format)**
| Knowledge Organization |  |
| --- | --- |
| System Prompt | You are a helpful AI assistant that are good at extracting crucial information from documents which are helpful for answering a given question. For a given question, focus on identifying the key entities, their attributes, and their relationships that are directly relevant to generating an accurate answer. Only include entities and attributes that are crucial for understanding and forming the answer, and avoid unnecessary details.<br>Your response must include the following keys and strictly adhere to the exact structure without any additional text before or after the keys:<br>Entities:<br>- [Entity 1] (Attributes: [Attribute 1, Attribute 2, …])<br>- [Entity 2] (Attributes: [Attribute 1, Attribute 2, …])<br>…<br>Relationships:<br>1. [Entity 1] -> [Relationship] -> [Entity 2]<br>2. …<br>Important Note:<br>1. Do not include any extra commentary or unnecessary details in the response.<br>2. Closely follow the structure provided above and ensure that the response is concise and directly addresses the question.<br>3. Don’t provide the answer to the question. Instead, focus on extracting the key entities, attributes, and relationships that are essential for answering the question accurately. |
| \hdashline<br>User Prompt | Question: {question}<br>Documents: {reference}<br>Knowledge Graph: |
| CoT |  |
| System Prompt | You are a helpful AI assistant that are good at doing reasoning on the knowledge graph to answer a given question. For a given question and relevant knowledge graph, provide the step by step reasoning process to derive the answer. Make sure the reasoning steps are logical, coherent, and directly related to the question and the information in the knowledge graph. |
| \hdashline<br>User Prompt | Question: {question}<br>Knowledge Graph: {$y_{\mathrm{KG}}$}<br>Reasoning Steps: |
| Generation |  |
| System Prompt | You are a helpful AI assistant that are good at generating final answers based on the reasoning steps provided for a given question.<br>Important Notes:<br>1. Make sure the final answer is accurate, concise, and directly addresses the question.<br>2. Only give me the answer and do not output any other words. |
| \hdashline<br>User Prompt | Question: {question}<br>Reasoning Steps: {$y_{\mathrm{CoT}}$}<br>Answer: |

**Table 9: KA-Pipeline Prompt(Kepoints Format)**
| Knowledge Organization |  |
| --- | --- |
| System Prompt | You are a helpful AI assistant that are good at extracting crucial information from documents which are helpful for answering a given question.For a given question, please thoroughly analyze the list of documents (given as strings) and extract the most<br>relevant key points that directly answer the question.Ensure that each key point is directly supported by evidence found within the documents, and avoid unnecessary details.<br>Your response must include the following keys and strictly adhere to the exact structure without any additional text before or after the keys:<br>Key Points:<br>1. [Key Point 1]<br>2. [Key Point 2]<br>…<br>Important Note:<br>1. Do not include any extra commentary or unnecessary details in the response.<br>2. Closely follow the structure provided above and ensure that the response is concise and directly addresses the question.<br>3. Don’t provide the answer to the question. Instead, focus on extracting the key points that are essential for answering the question accurately. |
| \hdashline<br>User Prompt | Question: {question}<br>Documents: {reference}<br>KeyPoints: |
| CoT |  |
| System Prompt | You are a helpful AI assisitant that are good at doing reasoning on the keypoints to answer a given question.For a given question and relevant keypoints, provide the step by step reasoning process to derive the answer.Make sure the reasoning steps are logical, coherent, and directly related to the question and the information in the keypoints. |
| \hdashline<br>User Prompt | Question: {question}<br>Keypoints: {$y_{\mathrm{Key}}$}<br>Reasoning Steps: |
| Generation |  |
| System Prompt | You are a helpful AI assistant that are good at generating final answers based on the reasoning steps provided for a given question.<br>Important Notes:<br>1. Make sure the final answer is accurate, concise, and directly addresses the question.<br>2. Only give me the answer and do not output any other words. |
| \hdashline<br>User Prompt | Question: {question}<br>Reasoning Steps: {$y_{\mathrm{CoT}}$}<br>Answer: |

**Table 10: KA-Pipeline Prompt(Summary Format)**
| Knowledge Organization |  |
| --- | --- |
| System Prompt | You are a helpful AI assistant that are good at extracting crucial information from documents which are helpful for answering a given question.For a given question, you need to extract a note that is directly relevant to generating an accurate answer.<br>Important Note:<br>1. Don’t directly provide the answer to the question. Instead, focus on extracting the relevant information that is essential for answering the question accurately and formulating a note.<br>2. The length of the response is limited, so make sure to include only the most relevant information. |
| \hdashline<br>User Prompt | Question: {question}<br>Documents: {reference}<br>Note: |
| CoT |  |
| System Prompt | You are a helpful AI assisitant that are good at doing reasoning on the note to answer a given question.For a given question with relevant note, provide the step by step reasoning process to derive the answer.Make sure the reasoning steps are logical, coherent, and directly related to the question and the information in the note. |
| \hdashline<br>User Prompt | Question: {question}<br>Note: {$y_{\mathrm{Note}}$}<br>Reasoning Steps: |
| Generation |  |
| System Prompt | You are a helpful AI assistant that are good at generating final answers based on the reasoning steps provided for a given question.<br>Important Notes:<br>1. Make sure the final answer is accurate, concise, and directly addresses the question.<br>2. Only give me the answer and do not output any other words. |
| \hdashline<br>User Prompt | Question: {question}<br>Reasoning Steps: {$y_{\mathrm{CoT}}$}<br>Answer: |

**Table 11: Data Construction Prompts**
| Check Answerability |  |
| --- | --- |
| System Prompt | You are a helpful AI assistant that is very good at judging whether the answer can be derived from the documents for a given question. You will be given a question, a set of documents, and golden answers. Please determine whether any of the golden answers can be derived from the documents for the given question.<br>If any of the golden answers can be derived from the documents, the judgement should be "True". If none of the golden answers can be derived from the documents, the judgement should be "False". Do not output any explanation, only output the judgement. |
| \hdashline<br>User Prompt | Question: {question}<br>Documents: {reference}<br>Golden Answers: {golden_answers}<br>Judgement: |
| Refine Prompt |  |
| System Prompt | You are a professional AI assistant that is very good at refining the flawed knowledge graph. The knowledge graph is extracted from the documents for a given question so that it can be helpful for answering the question. But the current knowledge graph may contain incorrect information, redundant information, or lack critical information, making it impossible to deduce the correct answer for the given question. You will be given a question, a set of documents, the flawed knowledge graph and golden answers. Your task is to add, remove, or modify the content in the knowledge graph to make it accurate and relevant to answering the question. Make sure that the refined knowledge graph’s content is both relevant to answering the question and entirely derived from the provided document.<br>Important Notes:<br>1. The new knowledge graph should be refined based on the flawed knowledge graph, don’t start from scratch.<br>2. The refined knowledge graph should be of the same format as the flawed knowledge graph.<br>3. Do not directly add the golden answers to the knowledge graph, the refined knowledge graph should be derived from the documents.<br>4. Do not output the explanation of your changes, only output the refined knowledge graph! |
| \hdashline<br>User Prompt | Question: {question}<br>Documents: {reference}<br>Flawed Knowledge Graph: {$y_{\mathrm{KG}}^{-}$}<br>Golden Answers: {golden_answers}<br>Refined Knowledge Graph: |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: There some typos and the citation formats should be better adjusted. Besides the questions mentioned above, I also have the following question
- As shown in Table 1, the in-domain performance is consistently worse than the baseline. Is there an explanation? How would people address that if they only want to achieve better in-domain performance (which can be common in some real applications)?
soundness: 3
strengths: - This paper proposes to train an LLM with RL by optimizing its ability to extract structured knowledge for QA, which seems to be a valid direction for RAG in general.
- Experiments are done on multiple datasets with different backbone models, and the results show strong performance on OOD test sets specifically
- Detailed analysis and ablation studies show the effectiveness of proposed method
summary: This paper proposes knowledge-aware refinement for RAG, which is to train an LLM to learn to first structured knowledge (e.g., a knowledge graph) from text before generating a final answers. The training method is developed based on DDPO where the negative samples are obtained from those with sub-optimal quality of knowledge structure and more attention are put on the tokens corresponding to modified tokens.
weaknesses: - The baseline selection is limited. It would be better to compare with other prompt-based and especially graph-based RAG methods to show the value of knowledge refinement
- Because the prompt schema is rather complicated, it would be better to report the cost and token consumption of the proposed method and how it compares with baselines
- The paper lacks some real case studies to show what a good quality knowledge structure can be induced by the model and how it helps the final answering.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: The training primarily relies on the Musique dataset, but the paper does not justify this choice or examine cross-dataset generalization. Could the authors provide results on alternative training–testing configurations—for example, training on PopQA and evaluating on Musique, or the reverse，to better assess the robustness and transferability of the proposed method across different domains?
soundness: 2
strengths: + The paper proposes a practical approach that trains models to generate intermediate structured representations for optimizing RAG, effectively addressing the sparse supervision problem in end-to-end methods like DPO.
+ The method shows strong data efficiency and OOD generalization, suggesting that the model learns a robust strategy for knowledge organization rather than mere memorization.
+ The trained model incurs no additional inference overhead, making it easy to integrate into existing RAG systems.
+ The Refinement Pipeline uses a teacher model to generate minimally different contrastive pairs, providing high-quality signals for DDPO training.
summary: The paper presents KARE-RAG, a training strategy that refines structured knowledge after retrieval to enhance the robustness and generalization of Retrieval-Augmented Generation (RAG) systems. The method introduces knowledge-aware sampling, which organizes retrieved documents into intermediate structured representations prior to generation, and a refinement pipeline that leverages a large language model to correct factual errors in negative samples while preserving structural coherence. Additionally, the paper proposes Dense Direct Preference Optimization (DDPO), extending standard DPO through token-level weighting to emphasize critical knowledge components during training. Experiments on multiple benchmarks to demonstrate consistent gains in robustness and out-of-domain generalization, achieved without modifying the standard RAG inference pipeline.
weaknesses: - The refinement stage relies heavily on a more advanced expert LLM to generate high-quality positive samples. While this improves data quality, it raises concerns about novelty and practical independence, as the performance gain may largely depend on the capabilities of the external expert model.
- The training primarily uses the Musique dataset, but the paper does not provide sufficient justification for this choice or explore cross-dataset generalization. It would strengthen the work to test setups such as training on PopQA and evaluating on Musique, or vice versa, to better assess the method’s robustness across different domains.
- The paper improves the generation of structured “graph-like” text representations but does not perform explicit reasoning over knowledge graph connections. This distinction should be clarified, as the current formulation may conflate improved structural formatting with genuine graph-based reasoning.
- Since Musique is a multi-hop QA dataset with overlapping conditions and complex logical relations, it is notable that keypoint-based representations outperform graph-based ones on this dataset. This suggests that even after refinement, the proposed graph structure may still lose critical contextual information and may not be universally effective across all reasoning scenarios.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - How to design specific structured representation. 
- Mabye limitations of automated sample generation. The sample refinement pipeline relies on advanced LLMs for error correction, which may be affected by the performance of the underlying LLMs. The sample quality in extremely complex scenarios is not explained.
- Indomain is not better than DDR based on Llama3-3b/8b, and Qwen-14b.
soundness: 3
strengths: High data efficiency: Achieves robust performance with only a small amount of training data, addressing the traditional reliance on large-scale high-quality datasets.
Strong practicality: Does not alter the standard RAG inference pipeline, incurs no additional computational overhead, and can be seamlessly integrated into existing systems.
Excellent generalization and compatibility: Delivers superior OOD (out-of-distribution) performance, supports multiple structured representations (e.g., knowledge graphs, key-point structures), and ensures stable performance gains across different model scales.
summary: Interesting research paper. Existing RAG systems struggle with multi-hop reasoning and conflicting information. Traditional optimization methods (e.g., SFT, DPO) require substantial high-quality training data and lack sufficient supervision for intermediate knowledge processing steps. Providing a good solution. Proposes KARE-RAG, which enables models to learn information discrimination from limited data through structured knowledge representation, automated refined sample generation, and the DDPO training strategy. Outperforms baselines on both in-domain and out-of-domain benchmarks, requires no modifications to standard RAG inference pipelines, is compatible with various model scales, and will make related data and code publicly available.
weaknesses: - How to design specific structured representation. Heavily relies on carefully designed knowledge representation structures, whose versatility and adaptability to diverse task scenarios have not been fully verified.
- Mabye limitations of automated sample generation. The sample refinement pipeline relies on advanced LLMs for error correction, which may be affected by the performance of the underlying LLMs. The sample quality in extremely complex scenarios is not explained.
- Room for expansion in complex task adaptation: Although it mentions handling multi-hop reasoning and conflicting information, the upper performance limit in scenarios with ultra-large-scale knowledge sources and ultra-high-complexity queries remains unclear.
- Indomain is not better than DDR based on Llama3-3b/8b, and Qwen-14b.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses
soundness: 2
strengths: 1.The framework design is clear and easy to follow. It shifts the learning goal of “how to use retrieved evidence” from optimizing only the final answer to learning structured intermediate knowledge, which helps reduce the sparse supervision problem in end-to-end DPO.

2.The experimental results look good, and the ablation studies are fairly complete.
summary: This paper proposes KARE-RAG, which first builds a “knowledge-aware” intermediate representation in RAG, then uses an LLM expert to iteratively refine the student model’s incorrect knowledge representation. Finally, it applies DDPO at the token level, focusing only on the key difference regions between positive and negative samples to improve factual consistency and robustness. The results look decent when compared with existing methods.
weaknesses: 1.The main innovation lies in verifying whether the LLM Expert’s refinements are correct. The verification process uses both the gold answer y₍gnd₎ and the retrieved docs D. But since y₍gnd₎ is also used to judge whether the refinement is “correct,” the model could end up overfitting to the human answer format instead of truly learning a general knowledge verification logic.

2.The expert model is trusted completely without any external calibration or cross-checking. I think that’s an unsafe and weak assumption in the experimental design.

3.The paper doesn’t report important metrics about the expert, such as quality, cost, number of refinement steps, success rate, discarded samples, or processing time per sample.

4.The algorithm looks elegant in theory but seems hard to scale. Any system that claims “end-to-end optimization” but relies on multiple LLM refinement loops must report throughput and actual cost.

5.The baselines are outdated. Although the related work mentions Self-RAG and Graph-RAG/G-Retriever—both highly relevant to structured knowledge use—the main table only compares with Vanilla, RA-DiT, and DDR, missing stronger baselines in the same space.

6.The paper should also include current top reasoning LLMs (like DeepSeek, GPT-5, Claude 3.5, Gemini 1.5) as baselines to better show the method’s advantage.

7.From about 20k samples in the MuSiQue dataset, the pipeline only produced 2,400 usable pairs, showing low sample-generation efficiency.

8.The figures and layout still need polishing, especially Figure 1.


Decision: Reject