Resisting Contextual Interference in RAG via Parametric-Knowledge Reinforcement
Authors: Chenyu Lin
Yilin Wen
Du Su
Hexiang Tan
Fei Sun
Muhan Chen
Chenfu Bao
Zhonghou Lv
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models ; Retrieval-Augmented Generation ; Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Knowledgeable-R1, a reinforcement-learning framework that explicitly trains large language models to use parametric knowledge (PK) to resist contextual interference while still exploiting external context when it is reliably helpful.

Abstract: Retrieval-augmented generation (RAG) improves performance on knowledge-intensive tasks but can be derailed by wrong, irrelevant, or conflicting retrieved text, causing models to rely on inaccurate evidence and cascade errors. We propose Knowledgeable-R1, a reinforcement-learning framework that explicitly trains large language models to use parametric knowledge (PK) to resist contextual interference while still exploiting external context when it is reliably helpful. Knowledgeable-R1 introduces a joint sampling scheme that generates paired responses with and without retrieval, and learns both local advantages (within each decoding regime) and global advantages under the same input to quantify when to ignore misleading context versus adopt it. We employ an asymmetric advantage transformation that amplifies exploratory behaviors toward parametric knowledge. Experiments show that Knowledgeable-R1 significantly improves robustness and reasoning accuracy in knowledge conflict scenarios and general RAG scenarios, outperforming SOTA baselines by +22.89% in counterfactual scenarios, and without degradation when the retrieved context is fully accurate.

Published as a conference paper at ICLR 2026
RESISTING CONTEXTUAL INTERFERENCE IN RAG VIA
REINFORCEMENT
Chenyu Lin 13+
Yilin Wen'*
Du Su²
Hexiang Tan? Zhonghou Lyu't
Fei Sun?
Muhan Chen'
Chenfu Bao
Baidu Inc. Institute of Computing Technology, Chinese Academy of Sciences
Naanaii University
"Tsinghua University
chenyul Sepuulinede In@mail nankai edu. cn, (sudu, sunfei, tanhexiang21 jeict..ee......... (wenyilin, chenmuhan@1, baocheenuu,Lvzhoongouu bcf25@mais . enu.cnn
com
ABSTRACT
Retrieval-augmented generation (RAG) improves performance on knowledge- intensive tasks but can be derailed by wrong, irrelevant, or conflicting retrieved text, causing models rely on inaccurate evidence and cascade errors. We propose Knowledgeable-Ri a reinforcement-llarringg framework that explicitly trains large language models to use parametric knowledge (PK) resist contextual interference while still exploiting external context when it is reliably helpful. Knowledgeable-R introduces a joint sampling scheme that generates paired re- sponses with and without retrieval, and learns both local advantages (within each decoding regime) and global advantages under the same input to quantify when to ignore misleading context versus adopt it. We employ an asymmetric advantage transformation that amplifies exploratory behaviors toward parametric knowledge. Experiments show that Knowledgeable-t reasoning accuracy in knowledge conflict scenarios and general RAG scenarios, outperforming SOTA baselines by +22.89% in counterfactual scenarios, and with- out degradation when the retrieved context is fully accurate. Datasets and code are available https: / /g thub. http:////yyy...///////oollaaaaa
significantly improves robustness and
INTRODUCTION
Retrieval-augmentte generation (RAG) has become a prominent approach for enhancing large language models (LLMs) by integrating contextual knowledge, thereby mitigating hallucinations and reducing factual errors (Nakano et al., 2021; Gao et al., 2023). However, recent studies indicate that when contextual knowledge introduced, LLMs can become overly reliant on this external information, suppressing their internal parametric knowledge. This phenomenon, known as context dominance, is particularly evident under conditions of noisy, counterfactual, or internally inconsistent evidence (Su et al., 2024; Xie et al., 2024b; Shi et al., 2023). Conflict-focuset evaluations confirm that LLMs often adopt incorrect retrieved statements even when their parametric knowledge is correct and can lag behind retrieval- ffree reasoning when retrieval is imperfect (Bi et al., 2025a; Wang et al., 2025a; Wen et al., 2024). These findings highlight an imbalance in how models utilize knowledge.
A central challenge in Retrieval- Augmented Generation (RAG) systems is determining when to rely on contextual knowledge (CK) and when to revert to parametric knowledge (PK), as well as how to integrate the two in stable and faithful manner. 1) Prompting approaches help guide the model to validate or filter the context while combining it with parametric knowledge, which improves the coherence of the output (Ding et al., 2024; Cheng et al., 2024; Press al., 2023; Wang al., 2023; He et al., 2024; Wang et al., 2025a). 2) Decoding- approaches, such as those proposed by (Bi et al., 2025b), adjust the token distribution during generation to mitigate conflicts between external context and parametric knowledge. While effective in some scenarios, both prompting and decoding methods add computational complexity and lack generalizable decision rule for managing diverse contextual situations. 3) Fine-tuning methods, such as Self-RAG (Asai et al., 2024) and InFO-RAG (Xu et al., 2024), train LLMs to implicitly learn decision rules for knowledge utilization. However, these methods often require complex data annotation pipelines, which can limit flexibility and scalability.
Equul contribution.
CCoresponding author.

---

Published as a conference paper at ICLR 2026
Query (q): What the capital of France?
Comeect
37.3
Ouns
Pamuecrrc Knuwlalge
Sumple Injentry]
GIRO
P(ats)
3,1
RAG
25.4
only
Probabiliy Distibution,
Joat
Sumpling
7 Marimdaan Plylo. Reseordlyy
Winng
Policy
IIM
Sample
Optimizaion
Trajesoors
P
Coneenuull Knowledge
Rewand--
Context (c): capital France is Lyon.
Probability Distribrrion
Reward
Probability Distrbution
Figure 1: Overview of Knowledgeable-RR) which integrates parametric (blue) and contextual (pink) sampling with policy optimization to ensure robust reasoning under misleading context. Green and red blocks indicate correct and incorrect responses, respectively. The inset chart displays accuracy on the ConFiQA-ME adversarial scenario (S2), where Knowledgeable (37.3%) %) significantly outperforms GRPO w/ RAG (19.7%), query-only prompting (25.8%) and RAG prompting (8. 1%); see Table 2.
When contextual knowledge appears reliable but is actually incorrect, LLMs should ignore it and de- cide whether fall back on parametric knowledge. This behavior is rare but crucial, as current LLMs tend to rely more on context when faced with conflicting or misleading information. Reinforcement learning (RL) can assist by adjusting the probability rankings of outputs, encouraging the LLM to explore these rare but critical decisions, such as when to fall back on parametric knowledge (Stiennon et al., 2020; Ramamurthy et al., 2023). Therefore, we believe RL has the potential to help LLMs learn how to effectively balance and utilize both parametric and contextual knowledge. Recently, models like OpenAI-ol (Jaech et al., 2024) and DeepSeek-R. (Guo et al., 2025b) have employed RL techniques such PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024) to enhance LLMs' logical reasoning and problem-solving abilities through experience and feedback. However, in RAG systems, these methods are limited by their sampling space and objectives, which leads to primary focus on context-aware reasoning while overlooking the importance of incorporating parametric knowledge.
The goal of this work is to develop a reinforcement learning framework that explores both con- textual and parametric knowledge after retrieval input. We call our method Knowl edgeable-RI because it is knowledge-awan reasoning approach that uses parametric knowledge to mitigate contextual interference the RAG system. A conceptual overview our framework is presented in Figure 1. Specifically, Knowledgeable e- enables LLMs to (1) explore both parametric and contextual knowledge through joint sampling, (2) distinguish the relative merits of these two knowl- edge types using locally and globally defined advantages, and (3) mitigate penalties for less likely parametric outputs by applying an adaptive asymmetric advantages transformation. We conducted experiments across five scenarios, demonstrating that Knowl mwowleggeable-RR outperforms range of RÁG approaches by a significant margin. By effectively utilizing both parametric and contextual knowledge, Knowledgeable-t achieving +22.9 % improvement over GRPO baselines in counterfactual scenarios (Table 3) and 30.47% improvement original RAG prompting, while maintaining strong performance in reliable contexts.
shows significant improvements in adversarial context settings,
2 RELATED WORK
2.1
LIMITATIONS OF EXISTING RAG ROBUSTNESS METHOOSS
Recent studies establish that L.MMs can store substantial parametric knowledge (PK), but their ability to use external contextual knowledge (CK) degrades under noisy inputs. Many studies have analyzed this phenomenon and found that may be caused by position biases (Liu et al., 2024; Hu et al., 2025) or interference when PK and CK conflict (Farahani et al., 2024). These observations highlight the need for methods that can dynamically retain reliable PK when CK is misleading, while still leveraging CK when beneficial.
significant thread in RAG research focuses on robustness to irrelevant or adversarial passages. Methods include noise-injected training (Yoran et al., 2024), attention masking (Fang al., 2024),
2

---

Published as a conference paper at ICLR 2026
and adversarial filtering (Shi et al., 2024; Cohen-Wang Wang et al., 2024). Evaluation benchmarks now systematically stress confficting evidence scenarios (Wang et al., 2025b; Tan et al., 2024). However, these approaches largely operate the passage level through pre-retrieval filtering post-retrieval weighting, lacking a mechanism to actively suppress harmful context during generation. This limitation is particularly acute when misleading information appears plausible, models tend to over-prioriin over-prioritize context even when it contradicts their parametric knowledge (Farahani et al., 2024; Gao et al., 2023).
2.2
REINFORCEMENT LEARNING FOR DECISION POLICIES IN LLM REASONING
Beyond supervised fine-tuning, reinforcement learning (RL) has emerged as a powerful paradigm for shaping model behavior without extensive human annotation. Recent work demonstrates RL's effectiveness in guiding reasoning processes, such as generating internal rationales (Zelikman et al., 2024; Li et al., 2025b) or stabilizing long chain-of-thought reasoning (Yu et al., 2025; Chen et al., 2025; Cheng al., 2025; Jin et al., 2025; Li et al., 2025a; Song et al., 2025). However, these methods primarily focus on reasoning structure rather than on parametric and contextual knowledge-aware learning for LLMs themselves.
3 METHOD
Our objective is to enable a single language model to appropriately leverage parametric knowledge (PK) versus contextual knowledge (CK) in retrieval-augmented generation (RAG). The key challenge lies in the varying reliability of retrieved context, which can be helpful, redundant, or misleading. An ideal LLM should:
Select the most accurate answer from its parametric knowledge when no context is provided
(purametric-only correctness).
Select the most accurate answer from its contextual knowledge when context is provided (context-
aware correctness).
Fall back on its correct parametric knowledge when faced with conflicting or noisy context, ensuring robustness to misleading information (robusiness to misleading context).
To achieve this, we propose a multi-ohjective reinforcement learning framework that trains L.MS to simultaneously optimize these three goals.
3.1 TASK DEFINITION
We formulate the problem as token-level reinforcement learning task. Let be the vocabulary, and let prompt be the input sequence. The prompt be of two types: p, containing only the query q. or p', containing both the query g and a retrieved context passage c. At each decoding step t, the model (parameterized by 0) produces a probability distribution over V based the prompt and previously generated tokens Octi
T oze) softmax(fozz, oc())
The goal is optimize 0 such that the generated sequence o (oI...., OT) maximizes the expected reward, which reflects answer correctness and appropriate knowledge utilization.
3.2 THREE SAMPLING/DECODIN POLICIES.
Let denote the query-only prompt and p' denote the query+context prompt. As we illustrate in Section 1, GRPO uses a single sampling policy during LLM training, making it difficult for the LLM to explore parametric-awarr answers in the combined retrieval input . contrast, our method defines three distinct policies for each query 9 over the next-token distributions:
To(0: p, 0ct),
To p', d'z)) to p' p(()
Notation. We use o to denote a token sequence generated when the current policy is conditioned on (query-only input), and of to denote token sequence generated when the current policy is
3

---

Published as a conference paper at ICLR 2026
conditioned on p' (query+context input). We write oct for the prefix up to step t-1, and for the token at step t. With these, the three policy types have the following inputs/outputs and intended behaviors:
PK (Parametric): input p = query, output (answer from parametric knowledge).
CK (Context-aware): input p' query+context, output (answer using context). RPK (Robust-PK): input p' = query+context, output o (answer consistent with PK).
CK and RPK share the same input y' but target different behaviors: CK exploits reliable context; RPK stays on the PK trajectory when context is noisylconflicting PK and RPK share the same oulput type but are trained under different conditioning (without vs. with context). At inference, the model does not explicitly switch controllers; the learned token distributions encode when to follow context or implicitly revert to PK.
Clarification on RPK sampling. RPK does generate an independent answer. For each query, we first sample a PK trajectory opk = (of ..... ...,. .op.) from xp(- p..) (query-only). In the RPK branch, the same token sequence oPk used as the larget, but evaluated under the query+context prompt p': at each step t, we compute (o | p', p2xt)) and maximize its log-probability under the RPK reward. In other words, RPK re-evaluates the PK answer under p' and encourages the model maintain PK tokens even when misleading context is present.
Table 1: I/O and behavior of the three decoding policies.
Policy type Input prompt
Output type
Intended behavior
PK
query
from parametric knowlodge
Parametric-only
CK
tceneet
E answer using context
Use context when helpfill
RPK
query4context answer consistent with PK
Fallback answer under misleading context
3.3 ADVANTAGE CALCULATION WITH LOCAL AND GLOBAL NORMALIZATION
To effectively balance the three objectives, we introduce a novel advantage calculation scheme that combines within-objective and cross-objective comparisons. The components are:
Local Advantage writhin-policy, within-input): compares trajectories that share the same input and the same output policy type.
Global Advantage (same input state, cross-output): compares CK and RPK trajectories under p/ in a unified pool, deciding whether to use context or fall back to PK-style decoding.
The glohal advantage mechanism enables effective differentiation hetween group trajectories, ensuring that rewards can still capture cross-source preferences (CK vs. RPK) when all trajectories within group are uniformly good or bad. This maintains meaningful feedback for different knowledge types.
Notation. Rewards R(-) are computed per trajectory based answer correctness (e.g., EM). Let rewards be RRck for PK trajectories 0 nt(: p..), Rg. for CK trajectories of 7 v/, and R, for RPK trajectories 0; ~ # v), Define the global (same-input) pool under p' as Up = (1ękk) (REPK).
To address the "overcorrection" problem, when CK is reliable but slightly differs from PK (e.g.. when factual details like new capital city are updated), the LLM should trust CK more due to its timeliness. Thus, we design distinct advantages for PK, CK, and RPK.
PK (Parametric). The advantage of PK is defined as A, = Apk-local, since PK relies on query-only input:
R.ck mcan ((RPk))
A pk-locall
std([RPp)) +
This term encourages the query-only parametric answer to be as accurate as possible.
CK (Context-aware). The advantage of CK is A = A Ack-local + globbll combining both local and global terms:
4

---

Published as a conference paper at ICLR 2026
Rg mean( (11gg))))) std ((5(() +
Rok mean (Upr)
Agk-local
A A
Agk-global
std(Ul,) ) +
RPK (Robust-PK under Context). The advantage of RPK is defined as A, Aglobal focuses solely on the global comparison under the same input p':
RPK
R2 mean(4p')
Ag Aglobal
std(U(p)) +
Summary. PK has a local advantage, ensuring accuracy based on the query input. CK combines both local and global advantages, prioritizing context when both knowledge types are correct, as context is frequently updated and more reliable. RPK has only global advantage, comparing PK's performance in the same input p' to maintain as a fallback when context is misleading. This balance ensures the model uses context effectively without over-relying it when PK is more reliable.
3.4 KNOWLEDGE BALANCE MODULATION
During training, context-aware (CK) paths often get higher rewards than robust parametric knowledge (RPK) paths due to the usefulness of retrieved contexts, creating a bias towards context. This can make the model rely too much CK and weaken its ability to use parametric knowledge in noisy or conflicting situations. RPK paths, reducing the penalty when PK-based decoding is slightly worse than context-following, ensuring parametric knowledge remains viable fallback.
address this, we introduce an asymmetric advantage transformation for
We define a modulation function T that transforms the RPK advantages Ag:
A BBAA, A.
if > 0, if A S O,
T(A;;
where 3 E [0.01, 1] is modulation coefficient. When B < 1, negative advantages (indicating poor performance of PK decoding) are reduced, making the model less sensitive occasional mistakes when relying on parametric knowledge.
To dynamically balance the exploration of parametric and contextual knowledge, we adapt 3 during training based the relative performance of CK and RPK trajectories. Let B denote the current mini-batch of training examples. We compute the total advantage for each knowledge type:
Sok Σ Al.
Sipk+
Σ iEBB
A. Sipk- Σ A.
jEB,
iEBrpk-
where Be is the set of CK trajectories, Bepk+ is the subset of RPK trajectories with positive advantages, and Bypk- the subset with non-positive advantages.
We then update B to maintain balance between the two knowledge sources:
Sack 5ck--Spk++ Stpkt, 0.01,
8 - clip
Srpk-
The update rule adjusts the penalty coefficient 3 based on the performance gap between context-aware (CK) and robust parametric knowledge (RPK). When CK outperforms RPK, 33 decreases, reducing penalties for negative RPK advantages and encouraging more exploration of RPK. As the gap narrows, increases, making RPK training more cautious.
When context is highly beneficial (i.e., Srpk- is much larger than Srpk+), reducing penalties for poor RPK performance (3 decreases) allows the model to keep RPK as fallback option. This method, similar reward shaping (Ng et al., 1999; Devlin et al., 2011), ensures that parametric knowledge stays usable. also resembles focused exploration techniques (Schulman et al., 2015; Espeholt et al., 2018) that prevent forgetting learned behaviors. While this introduces some bias in the gradient estimates, helps balance the model's 's tendency to over-rely on helpful context from the training data, allowing it to resist misleading context when needed.
5

---

Published as a conference paper at ICLR 2026
3.5 POLICY OPTIMIZATION
We adopt PPO-style updates with clipping. For each trajectory, we compute the probability ratio between the current and old policies:
PK (o PP 0i,ct)
=olo Ip,, divce Tbaa (o, pl, "ger))
#o(0 p', 0i,et)
CK
RPK
F. Tead (oi, P; 0i,<1)"
te (os p', 0c,c1)
where oz) are the corresponding prompts and tokens.
The objective for each component is defined as follows:
Tikk
lo.1
JPK 1
min
PK A,, clip(rih, 1- - ())
Tlp
1of1
JcK 1
min [C A₁, clip(r CK jtt c- 11
(1)
nick
JAPK
1
min
fi
RPK T(A:), clip(r..t RPK 1 --,, 1 T(,))
Tlpk
The total objective is the weighted sum of the individual objectives:
(0) Apk AckJoK + Arpk
with Apk = Ack Appk = 1.0 in our experiments. We use = 0.2 for clipping.
The goal of our method is optimize 0 to maximizes J(0), the full training procedure is outlined in Algorithm in Appendix L.
4 EXPERIMENTS
We evaluate whether the model can leverage parametric knowledge to produce accurate answers despite external contextual interference, and remain robust when context knowledge is correct.
4.1 EXPERIMENTAL. SETUP
We test robustness under five contextual conditions with increasing difficulty: Scenario (S1): Correct contextual knowledge (Section 4.2.1); Scenario II (S2): Adversarial contextual knowledge (Section 4.2.1); Scenario 1II (S3): Self-conflicting contextual knowledge (Section 4.2.1); Scenario IV (S4): Irrelevant contextual knowledge (Section 4.2.1): Scenario V (S5): Partially relevant contextual knowledge (Section 4.2.1). These scenarios reflect real-world retrieval situations where irrelevant conflicting context mislead generation.
4.1.1 REASSNING INSTRUCTION AND BASELINES
We evaluate with Qwen2.
Qwen2. I7B-Instrcc,, Swen2.5-14B 4B-IIstruct and
Llama3. 1-88- Instruct. Baselines include query-only prompting, RAG prompting, Astute-RAG (a prompting method for imperfect retrieval (Wang et al., 2025a)), CK-PLUG (Bi et al., 2025c), SFT, and GRPO (Guo et al., 2025a) with RAG. We evaluate using exact match (EM) (Jin al., 2025). To ensure fairness, all methods use the same total training data and are trained for one epoch. For GRPO and Knowledgeable-Ri we use a global batch size of 128, a rollout batch size of 32, rollout temperature of and learning rate of 10 All experiments same system prompt for both training and inference, where the inference input combines the query and context. For further experiment details, see Appendix J, and K.
H100 GPUs, using the
Baseline clarifications. CK-PLUG (Bi et al., 2025c) a plug-and and -layp decoding method that detects knowledge conflicts via confidence gain and adjusts token probabilities for conflict spans, enabling

---

Published as a conference paper at ICLR 2026
fine-graineds control parametric vs. contextual reliance. GRPO w/ RAG trains only the CK objective with query+context rollouts (no PK/RPK branches). using the same number of total rollouts Knowledgeable--11. SFT is strong fine-tuning baseline trained the same mixture of good/bad context data as Knowledgeable-RR with explicit conflct-handllngg instructions. Our method splits rollouts into PK, CK, and RPK groups; roughly half the rollout budget is allocated query-only PK trajectories, making the two methods directly comparable.
4.1.2 DATASETS
We introduce KnowQA to test methods across the five scenarios. First, we test parametric knowledge using query-only prompts (Appendix H), where correct answer means the model has the relevant PK. Then, we create contextual inputs using the top-5 retrieved paragraphs. This method suitable for knowledge-intensivee tasks. KnowQA includes scientific, factual, and commonsense questions, organized by scenario:
Correct and Wrong Context. We adapt PC (PC-QA, PC-MR, PC-MC) and NC (NC-QA, NC-MR, NC-MC) from ConFiQA (Xie et al., 2024a) for supporting vs. opposing evidence. QA stands for single-hop QA, MR for multi-hop reasoning, and MC for multiple conflicting inputs. We check the correctness of the context by looking for made-up sources. ConFiQA is counterfactual-rettie retrieval benchmark with three sub-datasets (QA/MR/MC).
Conflict Context. We create SC from ConFiQA using conflicting evidence and test whether models can handle contradictions by showing both positive and negative examples together. Irrelevant Context. We use ExplainPE, (Wen et al., 2024), a medical QA dataset from the Chinese National Licensed Pharmacist Examination. It tests the impact of knowledge mismatches on model performance, where adding retrieved documents can lower accuracy compared query-only. Partly Irrelevant Context. We use HotpotQA (Yang et al., 2018), 2WikiMultHHopQA (Ho et al., 2020), and MuSiQue (Trivedi et al., 2022), which include valid evidence mixed with distractors.
4.1.3 RETRIEVAL SETUP
We use prrasee-provided contexts without any additional retriever. The contexts (including both correct and incorrect passages) are directly taken from the benchmark construction, without re- ie-nddxxing or re-retrieval from a larger corpus. We adopt the top-5 paragraphs provided by these benchmarks, preserve their original order, and only truncate when the backbone model's maximum length is exceeded. For ConFiQA training, correct and incorrect contexts are mixed in equal proportion.
4.2 EXPERIMENTAL RESULTS
We evaluate Knowledgeable-RI in various contexts and find that it effectively selects reliable evidence while ignoring misleading or irrelevant information, especially when parametric knowledge (PK) sufficient. This leads to significant improvements challenging interference settings (S2- S5) without harming performance in correct contexts (S1). Consistent results are observed across models of different sizes (3B, 7B, 8B, 14B), with detailed results for Qwen2. 5-78-Innstrue and Llama3. 1 in Tables 2 and 3, and similar patterns for other models (see Appendix M). Our experimental analysis highlights three key strengths of Knowledgeable-
Robustness to Incorrect Context. Significant improvements in adversarial (S2), conflicting (S3), and irrlevant (S4) settings demonstrate the method's s ability to handle unreliable context.
Preservation of Correct Context Benefits. No performance drop occurs when the context is accurate (S1), keeping competitive with context- baselines.
Effective Context Selection. Large gains on PK-anseeralle questions, particularly in mixed- context scenarios (S5), highlight the method's 's ability filter relevant from irrelevant information.
4.2.1 PERFORMANCE ACROSS CONTEXTUAL SCENARIOS
Scenario I: Correct Contextual Knowledge (S1). When given accurate evidence, Knowledgeable-RI keeps the performance improvements from correct context. As shown in Table 2 (left columns), for PC-QA and 75.51% % on PC-MC, staying competitive with the best baseline (GRPO w/ RAG). Similarly, for Llama3. -B--Instruct it reaches 80.03% % PC-QA and 80.24% on PC-MC.
Knowledgeable-Rl achieves 80.90% on
7

---

Published as a conference paper at ICLR 2026
Table 2: Overall accuracy across five contextual scenarios: correct (S1), wrong (S2), conflict (S3), irrelevant (S4), and partly-irrelever (S5). Best results are in bold, second best are underlined. The "improve" " row shows gains over RAG prompting.
Correct (81)
Wrong (S2)
Cunflict (S3)
Ircelevant (84)
Purtly Irvelevant (55)
Method
PCRMR
PC-MC
PC-QA
NC
NC
NC-QA
SC
ExplainPE
HotPoiQAA
MuliHHopQA
2Wiki
Musique
Owen2.5-78-1ae S..----Instt
Query-oabl only promating
27.72%
24.56%
31.67%
25.93%
25.821%
32,3115
29.57%
64.45%
20.90%
25.54%
4.36%
RAG prompting
68.68%
66.39%
74.35%
13.47%
8.00%
11.31%
59,50%
02.21%
22.53%
641%
CK PLUG (Bi al., 20256)
64.69%
66.55%
78.06%
11.62%
8.06%
7.92%
55.00%
55.0016
22.74%
24.76%
6.25%
Astune (Wang al., 2025a)
65.51%
66.05%
77.62%
12.799
7.07%
10.34%
56,74%
17.87%
20.35%
SFT
71.95%
27.70%
74.70%
24.92%
21.05%
21.97%
68.50%
66.6016
30.14%
32.20%
11.75%
GRPO RAG enwwledgesble-R1
77.56%
77.36%
50.133
26.948
19.74%
26.01%
75.3358
66.5015
27.93%
33.95%
11.29%
25.0811
75.51%
90.90%
43.94%
37.34%
29.40%
76.33%
67.57%
31.45%
37.52%
12,04%
improve
+9.41%
49.12%
+6.54%
430.47%
+29 +29.299
418.05%
+15.9256
+5.36%
+11.0995
414.99%
45.638
Llana3. I-BB-Instruct
Query-only promsttng
29.375
26,
30.93%
27.10%
27.63%
42.655
32.088
43.26%
20.60%
21.025
RAG promping
64.85%%
61.999
76.42%
22.50%
16.28%
24.39%
61.17%
39.1618
24.44%
25.500%
8.19%
PLG (Bi 20250)
54.79%
58.45%
60.71%
12.1255
9.05%
17.29%
31,5455
22.355
24.635%
5.25%
Astuse (Wing al, 2025a)
65. 841
64.855%
77.97%
1796
9.225%
17.291
59,8346
40.1418
1.0518
30.264
9.64%
72.88%
79.225
84%
42.595
35.53%
86%
47.17%
$3.59%
13 33.3665
GRPO wi RAG
78.05%
79.7318
82.62%
41 41.58%
35.69%
39.2618
76.58%
47,5618
34.84%
$3.2214
16.59%
Knowl wnwoableelllleeo
73.76%
50.34%
50.03%
55,999
41.12%
44.59%
23,67%
49.61%
37.06%
45.37%
649
improve
+8.91%
+18.25%
+3.61%
432.49%
+24.845%
+19.71%
+15.4156
+10.45%
+12.6226
+21.87%
+5.30%
These results show that learning to manage noisy contexts doesn't 't reduce the use of reliable evidence, addressing a key concern in robust RAG systems.
Scenario II: Adversarial Contextual Knowledge (82). This scenario highlights the weaknesses of naive RAG and the benefits of our method. When the context adversarial, RAG prompt- ing causes huge drop in performance- to 13.47 %/8.06 %/11.311% (NC-MR/MC/QA) on Qwen2. 5-7--nnrruut much lower than the query-only baseline. Knowledgead @nowledgeable-R1 greatly im- proves this, increasing performance by +30.47/+29.288 /9/ 18.09 percentage points and outperforming GRPO w/ RAG (c.g.. 43.94% vs. 26.94% on NC-MR). The consistent gains of +32.449/424.844-11.7 on Llama3. --8 Instruct suggest that the method effectively learns to ignore misleading evidence and rely on parametric knowledge when needed.
Scenario III: Self-Confticting Contextual Knowledge (S3). When the context has internal con- tradictions, edgwledgeable-R1 performs competitively: 76.33% % compared to 75.33% for GRPO w/ RAG on Qwen2. 1-7B-Instrrcc and 73.67% % compared to 76.58% on 1lama3. 1-8B. Though the improvements are small, the consistent gains across models show stable handling of conflicting information. Resolving contradictions within the context seems difficult for all methods, indicating an area for future improvement.
Scenario IV: Irrelevant Contextual Knowledge (S4). In the ExplainPE benehmark, where the context is completely unrelated, we see the typical "RAG-hurt" effect: query-only (64.45%) performs better than RAG prompting (62.21% %) on Qwen2.5-78-Ime 1-78-Innstruct. Knowledgeable-RI achieves 67.57%, actively rejecting irrelevant context and outperforming both methods. This shows that the method can recognize when context isn't helpful for the query.
Scenario V: Partially Relevant Contextual Knowledge (S5). When valid evidence is mixed with distractors, Knowl shooleddgeable-Re performs strongest or nearly strongest. On Qwen2. 5-75---7--- II--Intt it achieves 31.45% % on HotpotQA, 37.52% on 2WikiMuliHHopAA, and 12.04% on MuSiQue, with significant gains over RAG prompting. Notably, the strong performance on 2WikiMulttHopQA and MuSiQue is achieved without fine-tuning on these datasets, showing that the method can generalize to new evidence types and distractor patterns beyond the training data (HotpotQA). This ability to generalize across datasets is especially useful for real-world applications, where the evidence may differ from the training data.
4.2.2 ANALYSIS ON PARAMETRIC- CRRAMETRIC-KKNWLEDDE ANSWERABLE SUBSET
We define the parametric-knowlled answerable subset questions where the model provides correct answers using query-only prompting, showing that it has enough parametric knowledge. When focusing on this subset, all methods improve, but Knowledgeable-Ri shows the biggest gains when the context is wrong or mixed (Table 3). Specifically, on NC-MR, NC-MC, and NC-QA, our method achieves an average improvement of 22.89% over GRPO w/ RAG.
8

---

Published as a conference paper at ICLR 2026
Table Accuracy on the parametric- -onoerrii---oeeeeeeee answerable subset across all scenarios, Best results are in bold, second best are underlined.
Currel (81)
Wrong (82)
Conflict
Irrelevant (54)
Partly Terelevant (88)
Method
PCMM
PC-MC
PC-QA
NC-MR
NC-MC
NC-QA
SC
ExplainPP
HotPeQQA
2Wik MuhillopQA
Musique
Quan2 5-75-Iratruct
promptig
86.99%
12.475L
78.50%
87.08%
41.159
20.75%
CK-PLUG 202300
79.76%
83.56%
95.65%
29.87%
18:47%
18.50%
82.876
77.58'
55.3015
42. 76776
20.759
87.67%
31.17%
24.50%
81.18%
73.0596
41.73%
34.50%
20.755
GRPO wt RAG
93.21%
93.15%
91.29%
37.61%
46.5013
62,59%
93.58%
$4.39%
06.56%
55 55.42%
45,2915
Knowl edgeable-R1
05,83%
98.21 %
07.83%
89.61 9
79.62%
66.00%
96.07%
83.18%
78.49%
66.72%
66,8%%
Improve
+10.12%
+8.22%
+3.81%
-57.1415
+59.5 57%
+37.50%
+8.99%
0.4615
+29 6516
+25.5668
+46.23%
Llano3.1 1-88-
RAG prompiing
50. 50.8058
87.745 745
96 06:085
52.175%
$7.5098
48.48%
87.019
67.99%
55. 1696
90.9256
$4,00%
CK-PLUG(Bi 20256)
82.58%
32.58%
67.111%
49.15%
42.79'5
16.118
Astree (Wiing ot sl., 3025g)
01.01%
89.09%
97.41%
35.40%
20.349%
$9.39'
83.129
20.2096
3.125)
52.25%
$5.575
GRPO
94.19%
66.1443
61.748
Knowledgeable-R1
94.38%
93.55%
98.71%
88.8216
75,60%
$1.82%
96.62%
78.10%
$3.55
$1.69'
71.14%
-4.99%
+5.81%
+1.73%
-36.685
+38. 10%6
+33.34%
+9.61%
+10.15%
+28.39%
+41.77%
+26.24%
Table Ablation study of Knowledgeable-Ri components. TITE: Both parametric and contextual knowledge correct; TIFE: parametric correct, contextual wrong; FTTE: parametric wrong, contextual correct; FIFE: Both wrong. Best results are in bold, largest performance drops are underlined.
PC-MR & NC-MR
PC-QA NC-QA
Method
FITE
Avg.
Avg.
Anowledgeable-4
95.21%
79.62%
69.06%
22.62%
65.40%
97.83%
66.00%
73.05%
11.93%
62.12%
Multi-ohjective and multi-sumpling strategy
KnowLedgeable--RP RI- JPK
95.89%
75.80%
69.06%
21.51%
64.12%
97.83%
66.00%
72.04%
9.79%
61 61.30%
Knowledgeable-- J.K JAPK (GRPO)
93.15%
46.50%
72.20%
10.42%
55.60%
97.28%
62. 50%
72.04%
8.59%
60.07%
Locul and global advantages
Knowl odgeable-st R1- A tocal
96.58%
82.80 80%
66.82%
18.63%
64.69%
97.28%
64.90%
68.77%
9.07%
59.91%
Knowl wwweledeebbl-RR A gtontt
93.15%
66.88%
71.75%
14.41%
60.95%
97.28%
54.00%
73,80%
7.64%
58.18%
Knowledge halonce modulation
Knowl Knowlldggaah edgeab I--R1- adape ?
93.15%
$2.23%
72.65%
10.20%
56.95%
96.745
53.50%
74.56%
6.21%
58.02%
For Qwen2. 5-7B-Instruct, Knowledgeable-R1 achieves 89.61 99.619//99622%/666.00 and GRPO w/ RAG (57.61 76/46.50%//2.550-5 )) Consistent trends are observed for Llama3. 1 1-BB-Instruct These results confirm that our method effectively identifies when parametric knowledge is reliable, allowing it to ignore conflicting information while maintaining stable performance in correct contexts (S1).
on NC-
MR/MC/QA, significantly outperforming RAG prompting (32.47 /tt.//99975......
Significant improvements are observed in Scenario V (partially relevant context). On HotpotQA, knowledgeable achieves 78.49% vs. 66.54% (GRPO w/ RAG) using Qwen2.5-7B- 5-7B-Instrui and 83.55 % vs. 73.17% % using Llama3. Instruct This indicates that the learned policy effectively identifies useful evidence while filtering out distractions.
4.2.3 ABLATION STUDIES
We conduct ablation studies on Qwen2.5 5-7B- IBBIInsrucc to evaluate the contribution of each compo- nent (Table 4). The largest performance drops occur in TIFE (parametric correct, context incorrect) scenarios, highlighting the importance of each component in handling misleading context.
Multi-objective and Multi-sampling Strategy. We assess PK, CK, and RPK policies. Removing the parametric objective (Jpk) results in moderate drops, particularly in FIFE scenarios (-1.11% % for MC, -2. 14% for QA). The most significant degradation occurs when the relative parametric reward (JRPk) ) is removed, causing catastrophic losses in TIFE scenarios (-33.12% for MC, -3.50% for QA). This confirms Jnrk is crucial for robustness under contextual interference.
Local and Global Advantages. Removing Ack-toret causes significant performance drops in context-ane antweebbll scenarios, while removing A ckk-glon glohat reduces TIFE performance. This suggests local advantages boost respective types (e.g., CK performs better when context is correct), while

---

Published as a conference paper at ICLR 2026
global advantages optimize the overall knowledge balance. Further ablation in Appendix B shows the framework is robust to the local-global ratio choice.
Knowledge Balance Modulation. Adaptive 3 modulation is vital; fixed reduces TIFE per- formance by 27.39% (MC) and 12.50% (QA). The drop in FIFE scenarios further highlights the effectiveness of adaptive balancing when both knowledge sources are unreliable. Together, these components work synergistically to enable robust reasoning in challenging scenarios.
4.3 ADDITIONAL ANALYSIS
In this section, we investigate performance trade-offs, robustness, and the dynamic behavior of our framework.
Knowledge Balance. A critical component of Knowledgeable-RR is the adaptive 3 modulation. As shown in Table 5, the adaptive balancing scheme significantly outperforms any single fixed 3 value across different datasets without requiring per-dataset tuning. Fixed values often create a trade-off: low 3 might favor parametric knowledge but hurt contextual reasoning, while high / does the opposite. Dynamically adjusting this balance based on prediction confidence ensures robust handling of conflicting information. Furthermore, during training, 3 rapidly converges to stable regime (stabilizing at 0.01 within 8 steps on ConFiQA, see Appendix A), indicating that the policy quickly learns to shield parametric knowledge from unreliable contexts.
Table 5: 3 sensitivity study: fixed vs. adaptive. Adaptive 3 achieves best-or-near-bes across tasks without per-dataset tuning.
3
MC-avg
QA-avg
0.01
56.03%
52.83%
0.2
52.74%
52.53%
0.5
51.19%
52.63%
0.8
48.22%
51.75%
1.0
48.05%
52.52%
Adaptive 3 (ours) 56.43%
54.85%
Robustness to Data and Weights. A common concern is whether the model overfits to the "noise ratio" of the training data. However, even when trained on 99% correct context (only 1% bad), Knowledgeable-P still outperforms GRPO on S2 scenarios (Appendix E), proving it learns genuine decision boundary rather than relying on dataset statistics. Additionally, regarding the S1 (correct context) performance gap compared to GRPO w/ RAG, ablations show this is tunable via JCK weight. Increasing its weight (e.g., ratio 1:2:1) closes the gap while preserving most $2 advantages (Appendix C), demonstrating controllable Pareto front between contextual compliance and parametric safety.
CONCLUSION
In this work, we presented Knowledgeable-Ri a principled reinforcement learning framework that systematically addresses the challenges of knowledge integration in suriival--uugmenee generation. By employing joint sampling strategy and asymmetric policy optimization, our approach enables models to effectively distinguish between reliable external context and internal parametric knowl- edge. Extensive benchmarking across diverse scenarios highlights that thowledgeabl e-R1 achieves superior robustness against contextual interference, particularly in adversarial and conflicting settings, while preserving competitive performance on standard RAG tasks. These results underscore the potential of reinforcement learning for calibrating knowledge utilization. Future work will explore the scalability of this approach to more complex multi- source retrieval environments and investigate the integration of diverse exploration cues to further enhance model reasoning and grounding.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We confirm that this work aligns with the ICLR Code Ethics. We have considered potential ethical aspects, including the use of public datasets in compliance with their licenses, the broader societal impact our research, and potential biases our methodology. To the best of our knowledge, this work raises immediate ethical concerns, and we declare conflicts of interest.
REPRODUCIBILITY STATEMENT
To facilitate the reproducibility our work, we have made the following available: (1) the source code of our algorithm, which conducts training using the VERL framework (Sheng et al., 2025), along with supplementary materials; (2) complete experimental configurations in J. Additionally, detailed account of the data pre- pre-processing procedures for the datasets in Section 4.1.2 can be found in
LIMITATIONS
Our method introduces reinforcement learning framework to address knowledge conflicts in large language models (LLMs), focusing on the automatic integration of parametric and contextual knowledge through sampling and policy optimization. In Sections S3 and S5. evaluate scenarios with mixed-context types. However, the errors in the context from Section have not been analyzed in sufficient detail. For example, when the amount of conflicting evidence changes (e.g.. incorrect result vs. incorrect out of 5 retrieved), it's unclear how the sensitivity of Knowledgeable-R to parametric knowledge is affected. Future research should explore more complex scenarios and conduct experiments to understand how the model responds to different levels of conflict and evidence reliability.
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. Int. Conf. Learn. Represent,, 2024.
Baolong Bi, Shaohan Huang. Yiwei Wang. Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, and Shenghua Liu. Context-DPO: Aligning language models for fottfussait In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proc. Annu. Meet. Assoc. Comput Linguist., pp. 10280-10300, July 2025a. ISBN 979-8-89 88778-8-77--55555
Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. Parameters vs. context: Fine-grained control of knowledge reliance language models. arXiv preprint arXiv:2503. 15888 2025b.
Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. Parameters context: Fine-grained control of knowledge reliance in language models, 2025c.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.088 0906666677. 2025.
Sitao Cheng, Liangming Pan, Xunjian Yin, Xinyi Wang, and William Yang Wang. Understanding the interplay between parametric and contextual knowledge for large language models. arXiv preprint arXiv:2410.08414 2024.
Zhoujun Cheng, Richard Fan, Shibo Hao, Taylor W Killian, Haonan Li, Suqi Sun, Hector Ren, Alexander Moreno, Daqian Zhang. Tianjun Zhong, et al. K2-think: A parameter-effiiiet reasoning system. arXiv preprint arXiv:2509.0760 2025.
Benjamin Cohen- Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. Contextcite: Attributing model generation to context. Adv. Neural Inform. Process. Syst., 958877, 2024.
11

---

Published as a conference paper at ICLR 2026
Sam Devlin, Daniel Kudenko, and Marek Grzeš. An empirical study of potential-bass based reward shaping and advice in complex, multi-agent systems. Advances in Complex Systems, 14(02): 251-278, 2011.
Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models. arXiv preprint arXiv:2402. 106/2, 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learne architectures. In Inf. Conf. Mach. Learn., pp. 1407-1416, 2018.
Yucheng Fang. Ruochen Wang, Kun Qian, Yansong Feng, Diyi Yang, and He He. Enhancing noise robustness of retrieval- augmented language models via RAAT. In Proc. Annu. Meet. Assoc. Comput Linguist. 2024.
Mojtaba Farahani, Orion Weller, Dawn Lawrie, Benjamin Van Durme, and Benjamin Van Durme. Deciphering the interplay between parametric and contextual knowledge in language models. In Conf. Empir. Methods Nat. Lang. Process. 2024.
Yunfan Gao, Yum Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, et al. Ketrievallauugmenter generation for large language models: A survey. CoRR, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in LLMs through reinforce- ment learning. Nature, 645(8081):633 -38,633,, sep 2025a.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025b.
Bolei He, Nuo Chen, Xinran He, Lingyong Yan, Zhenkai Wei, Jinchang Luo, and Zhen-Hua Ling. can improve retrieval augmented generation. In Conf. Empir. Methods Nat. Lang. Process., pp. 10371-10393, 2024.
Retrieving, rethinking and revising: The
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Int. Conf. Learn. Represent., pp. 6609-6625, 2020.
Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in Ilm agents via incremental multi-turn interactions. arXiv preprint arXiv:2507.05257, 2025."
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang. Hamed Zamani, and Jiawei Han. Search-rl: Training Iims reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095 2025.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic enhanced large reasoning models. Conf. Empir. Methods Nat. Lang. Process., 2025a.
Zhong-Zhi Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, al. From system to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. /7419, 2025b.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 11:157-173, 2024.
12

---

Published as a conference paper at ICLR 2026
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question crestion-answering with human feedback. arXiv preprint arX1v:2112.09332,222 2021.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Int. Conf. Mach. Learn., volume 99, pp. 278-2877 1999.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In Int. Conf. Learn. Represent. 2023.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Int. Conf. Mach. Learn., pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.064477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseckmath: Pushing the limits of mathematical reason- ing in open language models. arXiv preprint ar Xiv:2200.003300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In EuroSys Proc. European Conf. Comput. Syst. pp. 1279 1297, 2025.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Int. Conf. Mach. Learn., volume 202, pp. 31210-31227, 23-29 Jul 2023.
Yunxiao Shi, Xing Zi, Zijing Shi, Haimin Zhang, Qiang Wu, and Min Xu. Enhancing retrieval and managing retrieval: A four-m synergy for improved quality and efficiency in rag systems. In Front. Artif. Intell. Appl., pp. 2258 2265.22652222 IOS Press, 2024.
Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher++: Incentivizing the dynamic knowledge acquisition of iims via reinforcement learning. arXiv preprint 10.i::2505..70000 2025.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Adv. Neural Inform. Process. Syst., 33:3008 00.0008-021,,,, 2020.
Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in Ilm. In Adv. Neural Inform. Process. Syst., volume 37, pp. 103242-103268, 2024.
Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? In Proc. Annu. Meet. Assoc. Comput Linguist., pp. 6207-6227, 2024.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Trans. Assoc. Comput. Linguist., 10:539-5555 554, 2022.
Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ô Ank. Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. Proc. Annut Meet. Assoc. Comput Linguist., 2025a.
13

---

Published as a conference paper at ICLR 2026
Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Retrieval-augmente generation with conflicting evidence. In Conference on Language Modeling, 2025b.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In Int. Conf. Learn. Represent, 2023.
Yilin Wen, Zifeng Wang, and Jimeng Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. In Proc. Annu. Meet. Assoc. Comput Linguist., pp. 10370- 10388, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In Int. Conf. Learn. Represent, 2024a.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models knowledge conflicts, 2024b.
Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen. Xueqi Cheng, and Jie Zhou. Unsupervised information refinement training of large language models for retrieval-augmenter generation. In Proc. Annu. Meet. Assoc. Comput Linguist., pp. 143-145, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D) Manning. Hotpoiqa: A dataset for diverse, explainable multi-hop question answering. In Conf. Empir. Methods Nat. Lang. Process., pp. 2369-23388 2018
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmennte language models robust to irrelevant context. Int. Conf. Learn. Represent, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-STaR: Language models can teach themselves to think before speaking. In Int. Conf. Learn. Represent,, 2024.
A TRAINING DYNAMICS OF 3
In our adaptive scheme, 8 acts as a dynamic gate that modulates the influence of the relative parametric reward. As shown in Table 6, on the CoMFiQA-MC dataset, 33 undergoes a rapid descent during the initial stages of training, dropping from 0.23 and stabilizing at very low value (0.01). This rapid convergence is significant: it indicates that the reinforcement learning process quickly identifies the inherent unreliability of the external context in adversarial scenarios. By lowering 3, the model effectively increases the penalty for deviating from its correct internal parametric knowledge when pressured by misleading context. This "calibration" process happens surprisingly fast, suggesting that the model's internal representations already contain the necessary cues to distinguish between high-confidence parametric facts and low-quality external interference.
Table 6: 33 value over training steps (ConFiQA-MC). Rapid convergence near-optimal regime that protects PK as fallback.
Step
0.23
2
0.15
4
0.07
6
0.10
8
0.01
14

---

Published as a conference paper at ICLR 2026
B LOCAL VS. GLOBAL ADVANTAGE WEIGHT ABLATION
We ablate the local:global advantage combination ratio to confirm robustness to this hyperparameter. The "local" advantage (Ack-locul) focuses on optimizing the policy relative the average of its own rollouts for a specific query-context pair, encouraging the model to find the best reasoning path within that specific evidence. In contrast, the "global" advantage (A4k-globel) penalizes the model based on its performance across the broader distribution of samples. Results ConFiQA (Qwen2.5-7B) show that the I:1 combination is near-optimal. This suggests that the model benefits equally from source- spurce-specific optimization and overall distributional stability. The fact that performance remains stable across the 1:1 and 1:2 ratios indicates that Knowl thowledgeable-R is not overly sensitive to the exact weighting of these signals, provided both present balance specialized reasoning with general robustness.
Table Looall advantage ratio ablation. Combining both is better than either alone; the exact ratio matters little.
Local:Global
Setting
Avg. score
1:0
local-only
60.95
0:1
global-only
64.69
1:1
local+global
65.40
local+global
65.20
C SI PERFORMANCE GAP AND COMPONENT WEIGHTING
key discussion arises regarding the performance gap on S1 (correct context) compared to standard GRPO w/ RAG baseline. To investigate this, we performed sweep over the weights of our three objectives (JPK, JJck JJPK)) As shown in Table 8, the SI performance is extremely sensitive to the weight of the contextual objective JCK. By increasing its relative weight to a 1:2:1 ratio, can almost entirely close the gap with the bascline 78.04%% vs. 77.36%) while still maintaining a massive lead in the adversarial S2 scenario (32.73% 19. (..7%)). This confirms that there is tunable "Pareto front" between absolute mastery reliable context and robustness to unreliable context. We chose the 1:1:1 ratio our default because offers more conservative "safety first" profile, prioritizing the prevention of catastrophic failures in adversarial settings.
Table 8: Ablation of JrK:JCK:JRPK weights on ConFiQA-MC (Qwen2.5-7B). Higher JCK weight closes the S1 gap.
JPK:JcK:JaPK
S1 (correct ctx)
$2 (adversarial)
1:2:1
78.04%
32.73%
1:1:1 (ours)
75.51%
37.34%
0:1:1
75.68%
35.53%
0:1:0
(GRPO w/ RAG)
77.36%
19.74%
D QUERY-ONLY VS. CONTEET BEHAVIOR
We compare query-only and query+context inference for both GRPO w/ RAG and Knowledgeable- ConFiQA-MC (Qwen2.5-7B) under S1 (correct context) and S2 (wrong con- text). As shown Table 9, snowledgeable exploits high-quality context (S1: 6444475.%875 and also substantially outperforms GRPO w/ RAG when context is wrong (S2: 43.94% 26. 26.94%), demonstrating that the learned policy actively attenuates misleading context rather than ignoring blindly.
15

---

Published as a conference paper at ICLR 2026
Table 9: Accuracy under query-only vs. query+context inference (ConFiQA-MCC, Qwen2.5-7B). Our method benefits from context when it is correct and resists it when wrong.
Method
(correct ctx)
S2 (wrong ctx)
Avg.
Query-only (base)
27.72%
25.92%
26.82%
RAG prompting
65.68%
13.47%
39.58%
ERPO-RAG (query-only)
32.48%
31.48%
31.98%
GRPO-RAG (context)
77.56%
26.94%
52.25%
Ours (query-only)
35.64%
36.70%
36.17%
Ours (context)
75.08%
43.94%
59.51%
SENSITIVITY TO GOOD/BAD CONTEXT RATIO
common concern in RL for RAG is whether the model just learns the "noise ratio" of the training data. To test this, we trained Knowl @oowledgeall-R11 on data where bad contexts were extremely rare (only 1%). As Table 10 shows, Knowl showledgeable-Ri still manages improve S2 performance signifi- cantly compared to GRPO w/ RAG. This is crucial finding: it suggests that Knowledgeable- not just learning a naive prior that says "the context is often wrong, but is instead learning genuine decision boundary that triggers parametric knowledge whenever the context is perceived as unreliable. Even small number adversarial signals during training are sufficient to ground the model's 's reasoning process in its own parametric knowledge.
Table 10: Sensitivity to good:bad context ratio in training (ConFiQA-MR, Qwen2.5-7B), Even with 99% good context, Knowledgeable-RR improves S2 resistance.
Good:Bad Ratio
Method
S1 (correct ctx)
S2 (wrong ctx)
RAG
65.68%
13.47%
99%:1%
GRPO w/ RAG
73.60%
14.31%
Knowledgeable-R1
72.11%
17.34%
RAG
65.68%
13.47%
75%:25%
GRPO RAG
74.92%
20.03%
Knowledgeable-R1
73.76%
29.29 %
RAG
65.68%
13.47%
50%:50% (default)
GRPO w/ RAG
77.56%
26.94%
Knowledgeable-R1
75.08%
43.94%
F PARTIAL-CONFLICT SENSITIVITY
We construct test sets varying the number of misleading passages (1, 8) while keeping the query and gold answer fixed. Knowledgeable-t consistently outperforms GRPO w/ RAG across all levels of context adversariality
Table 11:
Accuracy vs. number of misleading passages. Knowledgeable-R1 maintains a substantial lead across all levels. (ConFiQA-MC, Qwen2.5-7B)
# misleading passages
2
GRPO RAG
26.94%
26.60%
25.76%
25.42%
Knowledgeable-Ri (ours)
43.94% %
43.10%
42.26%
40.07%
G CONTEXT-DEPENDREE DIAGNOSTICS
To verify that robustness arises from the leamed PK/CK arbitration rather than coincidence, evaluate on subsets stratified by knowledge correctness and report two explicit diagnostic metrics on test set where the context is designed to be wrong:
16

---

Published as a conference paper at ICLR 2026
Factuality-ssore (w.rt. context): This metric measures the linguistic and semantic alignment between the model's 's answer and the provided context. A high score when context is wrong indicates that the model has been "led astray" by the evidence. Lower is better in this scenario.
KL-score: This measures the KL divergence between the output probability distributions generated with vs. without context. It acts as a structural measure "contextual reliance." " KL-score is low, the model is essentially ignoring the context and relying on its parametric prior. Lower is better when context known to be mislcading.
As seen in Table 12, Knowledgeable achieves significantly lower scores on both metrics, proving has learned actively decouple its reasoning from unreliable external inputs.
Table 12: Context-dependence diagnostics on wrong-context set (ConFiQA-MCC) Lower factuality-score and KL-score confirm Knowledgeable-Rl relies less on misleading context.
Method
Factuality-score 4
KL-score |
GRPO w/ RAG
0.40
213.73
Knowledgeable-R1 (ours)
0.16
168.83
To further rule out that robustness is purely inherited from the base model, we stratify by whether RAG prompting gives the correct answer (Table 13). On the subset where RAG prompting always fails, our method achieves 32.72% far ahead of SFT (18.73% and GRPO (19.92%), indicating the improvement stems from the learned PK/CK arbitration mechanism.
Table 13: Accuracy stratified by RAG-prompting outcome (ConFiQA-MC, wrong-context test set). Our method gains the most the hardest subset where RAG prompting always fails.
Method
RAG-answer correct
RAG-answer wrong
RAG prompting
100%
0%
SFT
90.27%
18.73%
GRPO w/ RAG
96.61%
19.92%
Knowledgeable-R1 (ours)
95.25
32.72% %
H PROMPT DESIGNED IN Knowledgeable-Ri AND BASELINES
To assess the performance of Knowledgeable--R we compared against range of different baseline methodologies:
Prompt-based Methods:
These approaches include query-only prompting, Retrieval-Augmented Generation (RAG prompting), and Astute RAG (Wang et al., 2025a).
Decoding-based Methods: We experiment with CK-PLUG (Bi et al., 2025c).
fine-tuning-based Methods: We evaluate Knowledgeable-R1 against GRPO (Guo al., 2025b) with RAG prompting (GRPO w/ RAG).
These baselines encompass broad spectrum of retrieval-enhanced and fine-tuning methods. To ensure fair comparison, we standardized the contextual knowledge sources, training datasets, and Large Language Models (LLMMs) used across all methods.
17

---

Exclude any Published as a conference paper at ICLR 2026
RAG prompting
(search_rest results)
You helpful assistant. After the asks question, you first think carefully and then give the answer.
When responding, please keep the following points in mind:
The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively.
Output your final answer directly between the tag <answer> </answer> without any intermediate steps. Here an example:
user's question: what is the capital of China?
<think> reasoning process here </think>
<answer> Beiling </answer>
Question: {question]
Astute RAG prompt
Generate a document that provides accurate and relevant information to answer the given question. If the information is unclear uncertain, explicitly stale 'I don't know" avoid any hallucinations.
Question: [question)
Document:
Task: Consolidate information from both memorized documents and externally retrieved documents in response the given question.
For documents that provide consistent information, cluster together.
For documents with conflicting information, separate them into distinct documents. irrelevant information.
Question: (question)
Context: [context)
Provide consolidated documents JSON format:
["comtent"" : consoofidaledccmmenn "source" [ """dcidd"","commsisccc""""" "conssistevvy, roupp" "groupid")
Task: Answer the question using consolidated information from both internal and external documents. Initial Context: (initial context)
Consolidated Context: [consolidated context)
After the user asks a question, you first think carefully and then give the answer. When responding.
please keep the following points in mind:
Answer the question using consolidated information from both internal and external documents. enclosed within <think> </think> and <answer> </answer> tags,
The reasoning process and respectively.
Output your final answer directly between the tag <answer> </answer> without any intermediate steps. If the user gives a multiple choice question, your answer must be single option A or B or C or D or E Here is example:
Question:
what the capital of China?
<think> reasoning process here </think
<answer> BeiJing <lanswer>
Now, you should
user's question. After answer user's question, you should stop generate. Here
is user's question:
Question:
[question]
18

---

Published as a conference paper at ICLR 2026
CK-PLUG prompt
You are helpful assistant. After the user asks question, you MUST direct give the final answer. When responding, please keep the following points in mind:
Output your final answer directly hetween the tag <answer> </answer> without any intermediate steps.
You must directly output your final answer and don't output another things.
Stop your output after final answer
Here example:
user's question: what the capital of China?
<answer> Beiling </answer>
Retrieved information:
(retrieved information]
Question:
[question]
GRPO w/ RAG & RR- (CK)
are helpful assistant. After the user asks question, you first think carefully then give the answer.
When responding, please keep the following points in mind:
The reasoning process and answer enclosed within <think> </think> and <answer> </answer> tags, respectively.
Output your final answer directly between the tag <answer> </answer> without any intermediate steps. Here is an example:
user's question: what is the capital of China?
<think> reasoning process here </think>
<answer> BeiJing </answer>
Retrieved information:
(retrieved information)
Question: (question]
Knowledgeable-RRe (PK)
You are helpful assistant. After the user asks question, you first think carefully and then give the answer.
When responding, please keep the following points in mind:
reasoning process answer are enclosed within <thinks </think> and <answer> </answer> tags, respectively.
Output your final answer directly between the tag <answer> </answer> without any intermediate steps. Here is example:
user's question: what the capital of China?
<think> reasoning process here </think>
<answer> BeiJing </answer>
Question: [question]
SFT baseline prompt
You are helpful assistant. After the user asks a question, you MUST directly give the answer.
When responding, please keep the following points in mind:
Please answer with reference to the context, but if the context conflicts with your internal knowledge, prioritize your internal knowledge.
Output your answer directly between the tag <answer> </answer> without any intermediate steps.
Here is an example:
Question:
what the capital of China?
<answer> BeiJing </answer>
Question:
(question]
19

---

Published as a conference paper at ICLR 2026
I INFERENCE INPUT IN EXPERIMENTS
Inference Input in Experiments
You are helpful assistant. After the user asks question, you first think carefully and then give the answer.
When responding, please keep the following points in mind:
The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively.
Output your final answer directly between the tag <answer> </answer> without any intermediate steps.
Here example:
user's question: what the capital of China?
<think> reasoning process here </think>
<answer> BeiJing </answer>
Retrieved information:
(retrieved information)
Question:
(question]
J EXPERIMENTAL SETUP
Datasets. In our comprehensive evaluation of Knowledgeable-R1, we have considered a diverse range of benchmark datasets that represent various challenges of reasoning and knowledge, cat- egorized into five scenarios in 4.1.2. The specific datasets are: Multi-hop Question Answering with HotpotQA, 2WikiMultiHoPAA, and Musique; Knowledge Conflict Question Answering with ConFiQA, where external knowledge retrieval contradictions necessitate internalized knowledge for judgement, and ExplainPE, a medical knowledge multiple-choice dataset that tests noise robustness by introducing three tiers of noise levels based on the number of non-matching external documents retrieved. For ConFiQA, we randomly add correct and incorrect contexts to data for training in the same proportion. Our aim is to test Knowledgeable-RIi capability handle complex scenarios like single-hop answer, multi-hop reasoning and answer choice selection under different levels of external error or noisy context.
Training. We conduct experiments with Qwen2.5 5-3B-Instruct, Qwen2. and Llama3. 1-8B- Instruct All datasets, except HotpotQA, 2Wiki- MultiHopQA, and Musique, are trained and evaluated on their respective training and test sets. The ConfiQA dataset includes three subsets: MR, MC, and QA, which similarly trained and evaluated on their respective sets. For HotpotQA, 2WikiMultiHopQA and Musique, we limit the number of retrieved documents to 5 (the official setting uses 20) ensure incomplete document evidence during training. At test time, we evaluate on the validation sets of these datasets.
Training stability. Knowledgeable-R1 remains stable across all experiments: losses, rewards, and value estimates evolve smoothly across 200+ training runs covering both GRPO w/ RAG and Knowledgeable-R configurations. No collapse or divergence was observed, indicating that the additional PK/RPK objectives introduce extra instability on top of the standard GRPO training loop.
Retrieval setup. We use dataset-provides contexts without any additional retriever. The contexts (including both correct and incorrect passages) are directly taken from the benchmark construction, without re-indexing or re-retrieval from larger corpus. We adopt the top-5 paragraphs provided by these benchmarks, preserve their original order, and only truncate when the backbone model's maximum length is exceeded.
Computational Efficiency. KnowLedgeable-RR uses CK + 16 RPK trajectories (vs. GRPO's 32). With shared forward passes, per-epoch time is 1h 38m 1h 19m for GRPO modest ~24% overhead for significantly larger robustness gains.
For both GRPO-basee baselines and Knowledgeable-R1, rollout batch size of 32, a rollout temperature of 1, and a learning rate of 10 10-6 All experiments conducted on 8 H100 GPUs, with consistent system prompt for both training and inference.
global batch size of 128,
20

---

Published as a conference paper at ICLR 2026
K EVALUATION
For RAG generation capability. We use Exact Match (EM) as the evaluation metric following (Jin al., 2025). Evaluation is conducted the test validation sets of all datasets to assess both in-domain and out-of-domain performance. For integrating parametric/contextuall knowledge task, adopted EM and self-defined metrics evaluate the model's ability utilize parametric and contextual knowledge.
L METHOD TRAINING PROCEDURE
Algorithm Knowledgeable-R!: Policy Optimization for Knowledge Exploration
Current policy me. old policy Tea. dataset D. training steps Stepmax parametric knowledge prompt- ing rollout number Tykk, contextual knowledge prompt rollout number nick clip parametric advantage transformation function ( (-), parametric knowledge prompt p. contextual knowledge prompt p'
Input:
Output: Updated policy TTo
1: for s = 1 to Stepmax do
2: Sample batch (p) D
Sample batch (p) ~
4: Sample for " from Rew (0 p)
Ppp
Parametric knowledge rollouts
5: Sample {or 101 from T (o' pp)
Contextual knowledge rollouts
Compute advantages A A', and A, for rollouts
7:
Define JrK as the parametric knowledge based-policy objective: loal
To (or.e |p, or,ce)
To (or, 1p, ol.ct)
JPK
min
Ar, clip
- -1-,,1+ A
1pk
Tasu (or, p. o,,c))
Thu (or, Ps 02,0))
8:
Define Jok the contextual ledge based-policy objective:
1611
1
me (o',s p'o0)
To (dbis pl. o')
Jck
min
To (o),, pp.'', o' 1/, clip
te 43
Tow (0),e pp, p' o'j,ee)
j=1 1=1
9:
Define JAPK as the parametric knowledge reasoning policy objective with contextual knowledge input
JAPK
Apt ozl
min
Te (or.. p', ov.ca) T(A;), clip
Iaa (or, p', o..c))
1011-,,11 1, -1
T(A.)
Tipk
Th. (oi, p', oi,ct)
THa (0i, 1/ or,)))))
10:
Combine JPK, Jck, JAPK to form the updated objective function [()::
T(0) = ApkJPK + AaxJck + Arp.JRPK
11:
0 1-0++OJ(0
Update current policy parametric
12:
Boua 8
b Update policy parametric
13:
end for
M ADDITIONAL EXPERIMENT RESULTS
We validated the performance improvement our method across models of varying sizes with 3B, 7B, 14B. As shown in the Table 14, our approach achieved significant enhancements compared to RAG prompt in all five scenarios. also showed considerable improvement over GRPO w/RG when external knowledge contained errors or incomplete information, and a slight improvement when external knowledge served as interference. This demonstrates the effectiveness and versatility of our method.
N USE OF LLMS
Large language models (LLMs), specifically GPT-5 and DeepSeek-R1, were used solely as a supple- mentary tool during the preparation of this work for tasks such as polishing the writing. The authors are solely responsible for the entire research conception, technical direction, scientific content, and interpretation of results. The LLMSs were employed only to assist the presentation and clarity of the manuscript.
21

---

Published as a conference paper at ICLR 2026
Table 14: Overall accuracy across five contextual scenarios: correct (S1). wrong (S2), conflict (S3). irrelevant (S4), and partly-irreevvnt (S5). Best results are in bold, second best are underlined. The "improve" row shows gains over RAG prompting.
Correct (S1)
Wrong (S2)
Conflict (83)
Irrelevant (84)
Partly Irrelevant (S5)
Method
PC MR
PC MC
QA
NC MR
NC MC
NC QA
ExplaiPPE
HotPoRA
2Wiks MuluHoppAA
Musique
Query only prompting
30.05%
2h.52%
35.80%
30,6416
28.95%
39.42%
30,9258
70,80 80%
25.35%
26.69%
5.671
promptng
GRPD oi RAG
73.60%
74.66%
78.86%
18.599
31.749
36.519
74.00%
34.71%
39.095
14.73%
I
14.00%
Improve
+7.43%
+11 32%
+3.515
+24.99%5
+20,01%
+10.01%
+10.65%
+14.00%
+19.34%
+7.09%
-e-nsstruct
Query-only prompting
27.728
34.65%
31,679
25.09%
35.80%
32.311%
20.87%
64.45%
20.9072
25.549
4.16%
prompting
Astore (Wingg et al., 2025p)
65.519
66.09%
77.62%
12.79%
7.00%
10.341%
$4.2006
74%
17,87%
20.35%
6.79%
GRPO RAG Kioe-e
77.56%
30.399
33,05
25.08%
75.51%
80.90%
43.94%
37.34%
29.40%
76.35%
67.57%
31.45%
37.52%
12.04%
+9.15
+9.12%
+6.545
-29.2915
+5.35%%
+14.9%
+5.63%
Qven2 5-30 ----snttuuct
Query prompting
23.92%
53.325
RAG prompting
55.12%
52.8%
59.35%
13:476
8.72%
6.79%
51,25%
42,19%
14.7116
1216
3,6415
62.055
GRPO wi RAG
70.96%
70.61%
80.03%
19.87%
15.13%
19. 19.30%
66.679%
$3.61.5
24.7898
35.571
9.06%
Knowlesgssble-R!
66.17%
$4.69%
improve
+11.05%
+6.93%
+19.11%
+15.32%
+19.9%
+15.15%
+1.37%
22


Reviews:
[{"review_id": "XGrE7qYB9Q", "reviewer": "Reviewer_GFpR", "summary": "This paper proposes Knowledgeable‑R1, a reinforcement‑learning framework for RAG that explicitly trains a single LLM to answer from parametric knowledge, PK when no context is provided or use contextual knowledge, CK when retrieved evidence is reliable, and fall back to PK when the retrieved context is misleading. The method samples paired trajectories for the same question under three “policies”: PK (query‑only), CK (query+context), and RPK (query+context but encouraged to stay on the PK answer). It then computes advantages with two normalizations: local (within the same policy) and global (CK vs. RPK under the same input), plus an asymmetric advantage transformation with a dynamic coefficient β that down‑weights negative advantages for RPK to preserve PK as a viable fallback. Optimization uses PPO‑style clipping.\nThe paper evaluates five scenarios—accurate, adversarial, self‑conflicting, irrelevant, and partially‑relevant contexts—on Qwen2.5 and Llama‑3.1. It shows large gains when context is wrong or mixed, while remaining competitive when context is correct. For example, on adversarial context with Qwen2.5‑7B, NC‑MR improves from 13.47% (RAG prompting) to 43.94% (Knowledgeable‑R1), and on partly‑relevant context, HotpotQA rises from 20.36% (RAG prompting) to 31.45% (Knowledgeable‑R1).", "questions": "Against which baseline and which aggregation is the reported 23% gains computed? Please align the abstract with Table 2 and report confidence intervals or paired significance tests. \n\nWhat retriever, indexing corpus, and query formulations were used? How are top‑k passages truncated and ordered?\n\nTraining uses three rollout types; what is the token and wall‑clock overhead vs. GRPO w/ RAG\n\nCan you please report context-dependence metrics(corresponding to Weakness 3)\n\nCan you add experiments that compare the actual performance under query-only and query+context settings?(corresponding to weakness 4)\n\nPlease plot β over training and provide a simple sensitivity study?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles “context dominance” in RAG and designs RL signals that explicitly arbitrate between CK and PK at the same input state (global advantage), which is the right granularity for conflict resolution. The three‑policy setup is well‑motivated and neatly summarized\nThe combination of local vs. global advantages and an asymmetric transformation for RPK is a coherent way to reward using context when both sources agree or CK is right (timeliness) and protect PK exploration when CK is wrong. The β adaptation rule is simple and effective.\nThe performance compared with baselines is promising.", "weaknesses": "Retrieval details are not clear, like what retriever, indexing corpus, and query formulations were used.\n\nThe headline gains mostly compare to RAG prompting (e.g., +30.47/+29.28/+18.09 on Qwen2.5-7B NC-MR/MC/QA; Table 2). Against GRPO w/ RAG, improvements are smaller (e.g., 43.94 vs 26.94 = +17.00). How the “23% over GRPO” figure is aggregated should be clarified. \n\nThe paper relies on exact match accuracy but does not directly quantify context dependence like answer stability when contexts are removed/perturbed. Whether the robustness comes from mechanism (ignoring misleading passages) or coincidence (answering correctly despite them) remains unclear.\n\nThe paper claims that the model implicitly decides to use RAG context or not, but the default inference template feeds query+context, lacking the comparison of query-only and query + RAG context.\n\nThe paper reports that  fixed β will degrade performance but does not include further analysis like the sensitivity of β", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ErgV1aZgRS", "reviewer": "Reviewer_72rj", "summary": "Targets contextual interference in RAG and proposes Knowledgeable-R1 with joint sampling, local/global advantages, and asymmetric β-shaping to balance PK vs. context use. Shows strong robustness under adversarial/contradictory/irrelevant contexts; in S1 (correct context) it’s slightly below GRPO w/ RAG. Missing pieces: finetuning baselines (e.g., Self-RAG, InFO-RAG) and sensitivity ablations (fixed vs. adaptive β, local+global advantage weights, and J_PK/J_CK/J_RPK weighting) to diagnose the S1 gap.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper squarely targets the challenge of contextual interference—conflicts between context prompts and parametric knowledge—and proposes Knowledgeable-R1, which improves robustness via joint sampling, local/global advantage design, and an asymmetric advantage transformation (reward shaping).", "weaknesses": "1. Lack Finetuning baselines. Please include finetuning methods cited in Related Work (e.g., Self-RAG, InFO-RAG) as baselines for a fair comparison.\n2. β scheduling. Compare fixed β settings ({0.2, 0.5, 0.8, 1.0}) versus the adaptive β scheme to demonstrate the necessity of adaptivity.\n3. Advantage composition. Validate whether combining local + global advantages is necessary (e.g., ablations varying their relative weights).\n4. S1 performance gap. In settings with correct context (S1), performance is generally below GRPO w/ RAG. This may stem from the weighting among J_PK, J_CK, J_RPK, or from the advantage formulation. Please discuss and, if possible, provide experiments to diagnose these factors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "StxIe6jEBA", "reviewer": "Reviewer_6P9g", "summary": "The study proposes Knowledgeable-R1—a reinforcement learning (RL) framework. It explicitly trains large language models (LLMs) to leverage parametric knowledge (PK) for resisting contextual interference, while still utilizing external context when it is reliably beneficial.\n\nExperimental results show Knowledgeable-R1 significantly boosts robustness and reasoning accuracy in both knowledge conflict scenarios and general RAG scenarios. It outperforms state-of-the-art (SOTA) baselines by 23% in counterfactual scenarios, with no performance degradation when retrieved context is fully accurate.", "questions": "1. The experimental section should include additional discussions on training costs (e.g., training duration, GPU resource consumption) and training stability.\n\n2. To better demonstrate the necessity of the complex RL framework, it is recommended to add a stronger SFT baseline. For instance, conduct SFT on the same dataset (containing both good and bad context) used for RL, and use specific instructions such as: \"Please answer with reference to the context, but if the context conflicts with your internal knowledge, prioritize your internal knowledge.\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a highly practical and critical issue in Retrieval-Augmented Generation (RAG) systems: \"context interference\" or \"Context Dominance\". Specifically, when Large Language Models (LLMs) encounter retrieved context that is erroneous, irrelevant, or conflicting, they tend to over-rely on this context while ignoring their internal, more accurate Parametric Knowledge (PK).\n\n2. It designs a multi-objective Reinforcement Learning (RL) framework. The most crucial design within this framework is the introduction of three sampling strategies, with the Robust-PK (RPK) strategy being particularly notable. Given input in the form of \"query + context\", this strategy trains the model to generate answers consistent with PK—effectively making the model ignore the context. This creates a dynamic competition within the model between \"utilizing context\" and \"resisting context\".\n\n3. \"Knowledge Balance Modulation\" (detailed in Section 3.4) is an ingenious technical element. Through an asymmetric and adaptive transformation of the advantage function (via coefficients), it effectively prevents the RPK strategy from being overshadowed by the Context-aware (CK) strategy— which is \"usually more useful\"—during training. This ensures the model retains the ability to fall back on PK when confronted with \"occasional\" erroneous context.", "weaknesses": "1. The method introduces significant training overhead. It requires maintaining and optimizing three distinct strategic objectives (PK, CK, RPK), calculating complex \"local + global\" advantages, and finally implementing asymmetric modulation. This is far more complex in both implementation and computation compared to standard Supervised Fine-Tuning (SFT) or GRPO with RAG.\n\n2. As mentioned in Section 3.2 of the paper: \"During inference, the model does not explicitly switch controllers; the learned token distribution... implicitly... falls back to PK.\" This description is overly vague. Are these three strategies (PK, CK, RPK) merely \"scaffolding\" used during training, with only one model produced in the end? If so, how does this single model arbitrate between these three conflicting behaviors during inference?\n\n3. The method appears to heavily rely on the presence of samples with \"bad context\" (S2-S5) in the training data. Appendix C mentions that an \"equal proportion\" of correct and incorrect context was randomly added to ConFiQA. If 99% of the training dataset consists of \"good context\" (S1), can the method still effectively learn the ability to \"resist\" (erroneous context)? While adaptivity may alleviate this issue, the paper lacks a sensitivity analysis of the proportion of \"good/bad\" context in the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRIo3k9UKA", "reviewer": "Reviewer_C4dV", "summary": "To address the tendency of models to disregard **parametric knowledge (PK)** when the retrieved context in retrieval-augmented generation is **noisy, misleading, or contradictory**, the authors propose **Knowledgeable-R1** — a **reinforcement learning (RL)** framework that explicitly trains LLMs to resist contextual interference while still leveraging useful retrieved information.\n\nTraining follows a **GRPO-style** setup with three policies:\n\n* **PK:** query-only (parametric knowledge)\n* **CK:** query + context (context-aware)\n* **RPK:** query + context, but decoding along the PK trajectory to test robustness", "questions": "- Why don't you include KL-regularizer $\\lambda_{KL} D_{KL}[\\pi_\\theta \\| \\pi_{ref}]$ in your total objective to avoid drifting too far from the reference policy, same as in [1], for example?\n- What does the barplot in the top-right corner (the one with values 37.3, 19.7, 8.1 and 25.8) of Figure 1 mean? \n\n[1] Xu, Tingqiao, et al. \"DyKnow-RAG: Dynamic Knowledge Utilization Reinforcement Framework for Noisy Retrieval-Augmented Generation in E-commerce Search Relevance.\" arXiv preprint arXiv:2510.11122 (2025).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Well-motivated problem.** The work addresses a practical and underexplored limitation of RAG systems — their lack of robustness to noisy contexts.  \n\n**Strong empirical validation.** Results hold across two model families, four model sizes, and five datasets, showing consistent robustness gains under four interference conditions while preserving accuracy on clean contexts.  \n\n**Comprehensive ablations.** Each design choice (joint sampling, local/global advantages, adaptive β) is empirically justified, with clear performance drops when omitted.\n\n**Novelty.** Unlike prior approaches that dealt with noisy contexts (e.g., [1], [2], [3]), this framework explicitly operationalizes the PK–CK trade-off in a reinforcement learning manner rather than using prompt engineering or fine-tuning. It closely resembles [4], although that work is quite recent and may not have been available when this paper was written.\n\n[1] Asai, Akari, et al. \"Self-rag: Self-reflective retrieval augmented generation.\" NeurIPS 2023 workshop on instruction tuning and instruction following. 2023.\n\n[2] Yoran, Ori, et al. \"Making retrieval-augmented language models robust to irrelevant context.\" arXiv preprint arXiv:2310.01558 (2023).\n\n[3] Wang, Fei, et al. \"Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models.\" arXiv preprint arXiv:2410.07176 (2024).\n\n[4] Xu, Tingqiao, et al. \"DyKnow-RAG: Dynamic Knowledge Utilization Reinforcement Framework for Noisy Retrieval-Augmented Generation in E-commerce Search Relevance.\" arXiv preprint arXiv:2510.11122 (2025).", "weaknesses": "**Key method part is not well-explained**. It is not quite clear for me how RPK sampling works. in line 152 RPK policy is defined as $ \\pi (o_t \\mid p^{\\prime}, o_{<t})$ - output is conditioned on previously generated tokens $o_{<t}$ and $p^{\\prime} =$ query + context. But in the next paragraph, the authors say that $o_t$ is conditioned on query-only input  $p$: \"we use $o$ to denote a token sequence generated when the current policy is conditioned on $p$ (query-only input)\". Then the authors say that $o$ means possibly two different things - output of PK (\"answer from parametric knowledge\") and output of RPK (\"answer consistent with PK\"). Could you please elaborate what is the difference between these two and maybe use different variables for them? What is input and output when each new token in RPK is generated? I think it is very important as RPK is one of the key differences of Knowledgeable-R1 from GRPO + RAG baseline.\n\n**Heuristic nature of β-modulation.** The adaptive penalty scaling is empirically tuned, with limited theoretical justification. A more formal grounding or ablation across β dynamics would strengthen the argument.  \n\n**Limited exploration of partial-conflict granularity.** The paper notes this in its limitations — sensitivity analysis on varying proportions of misleading passages would be valuable.  \n\n**Narrow task domain.** Evaluation focuses on QA-style tasks; it remains unclear whether the learned robustness generalizes to open-ended generation, dialogue, or summarization settings.\n\n**Confusing writing:**\n\n- Baselines are not well explained in Section 4.1.1:  \n  - CK-Plug is omitted.  \n  - GRPO with RAG is not clearly described — the difference from Knowledgeable-R1 is unclear.  \n  - Key explanations appear only in the appendix.  \n- Lines 187–209 — indices *i*, *j*, *k* are unclear. Do they denote rollout indices?  \n  - The distinction between them is not specified — could they be unified under a single index?\n- Font size in Tables 2 and 3 is too small to be properly readable.\n\n**Typos**\n- In Figure 1, the text “Probability distribution” overlaps with the image border.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)