LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
Authors: Guanzheng Chen
Michael Qizhe Shieh
Lidong Bing
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Long Context
Reinforcement Learning with Verifiable Rewards
Primary area: foundation or frontier models, including LLMs
TL;DR: RLVR for improving long-context capabilities.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding—the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.

Published as a conference paper at ICLR 2026
LONGRLVR:
LONG-CONTELL REINFORCEMENT LEARNING RE-
QUIRES VERIFIABLE CONTEXT REWARDS
Guanzheng 1..
Michael Qizhe Shieh 1.3
Lidong Bing?
National University of Singapore Miro AI 3 absolute AI
gc. c.c.chenêu. edu, chchees nus. edu. sg, aadong.bingeshanda..m
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly ad- vanced the reasoning capabilities of Large Language Models (LLMs) by opti- mizing them against factual outcomes. However, this paradigm falters in long- context scenarros, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contexiual grounding- ability to find and reason over exter- nally provided information. We identify a key reason for this failure: a reward based solely the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, ren- dering learning intractable. To overcome this bottleneck, we introduce LongR- LVR to augment the sparse answer reward with dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the cor- grounding information, providing robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long- context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and bench- marks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is wawprrralaaabsssuutttt//o/ggolo
available
INTRODUCTION
Reinforcement Learning with Verifiable Rewards (RLVR) Lambert et al. 2024 Guo et al. 2025) has emerged as pivotal paradigm in advancing the reasoning capabilities of Large Language Mod- els (LLMs). By rewarding verifiable outcomes, RLVR effectively steers LLMs to explore diverse reasoning pathways for achieving factually accurate and logically sound solutions. This paradigm has recently propelled LLMs, such DeepSeek-RI (Guo al. (025), achieve expert-level rea- soning ability in domains like mathematics and programming (Guo et al. 20255 Jaech et al., 2024] Kimi et al. 2025 Huang & Yang, 2025). The remarkable success of RLVR on complex reasoning makes it never more compelling for applying to the next frontier: enabling LLMs to explore and rea- son over vast external environment to unlock broader intelligence (Zhang et al. 2025). However, the interaction of LLMs with such environments necessitates processing extensive external information, which poses significant challenges on their long-context capabilities.
Effective long-context reasoning typically hinges upon robust contextual grounding: the abil- ity to accurately retrieve and synthesize information from external documents (Wan et al. 2025). Yet, recent studies (Yue et al.) 2025; Wen et al.) 2025) suggest that RLVR pri- marily sharpens the internal knowledge that LLMs have already acquired during pretraining. This may limit the efficacy of RLVR for enhancing the long-context capabilities of LLMs.
Work done during the internship of Guanzheng Chen at MiroMind Al. 1 "Coresponding Author.

---

Published as a conference paper at ICLR 2026
As shown in Figure 1 when applying naive RLVR with outcome-only rewards for fi- nal answers upon long-context training, the model's contextual recall score (measured by retrieving reference chunks identifiers as de- tailed in Figure 2) quickly stagnates. This plateau relevant retrieval directly creates a ceiling for answer accuracy. thus halting overall learning progress on training rewards.
0.45
LongRLVR (solid)
Naive RLVR (dashod)
0.40
Accuracy Reward
0.5
Confextunl Rocall
0.35
0.50
-0.3
0.25
0.20
-0.2
In this work, we introduce LongRLVR to ad- dress the bottleneck of naive RLVR on long- context training. We first formally prove that the outcome-only reward causes vanishing gradients for the contextual grounding, ren- dering the learning to become sparse and in- tractable for long sequences. To address this, LongRLVR incorporates context reward into outcome-only rewards to augment the sparse learning signal on contextual grounding. Specifically, for each rollout, steer the model generate the grounding chunk identifiers from the long context before achieving the final answer (see Figure 2}. These identifiers will be compared with ground- truth counterparts to access a verifiable reward. By explicitly rewarding the model for extracting relevant evidences, we provide dense learning signal that mitigates the vanishing gradient issue. Therefore, our LongRLVR overcomes the bottleneck of long-context RLVR training and allows both contextual recall and answer accuracy to improve continuously throughout training (see Figure I).
0.15
-0.1
Training Step
Figure 1: The accuracy reward and contextual recall of naive RIVR and LongRLVR on the training data.
To support the training of LongRLVR, we develop comprehensive data synthetic pipeline that pro- duces high-quality long-context crestion-answering data annotated with the necessary grounding chunks. We validate effectiveness through extensive experiments on I.LMMA331 (Dubey et al., 2024) and Qwen2.5 (Yang et al. 20255 models across challenging long-context benchmarks such as RULER (Hsich et al. 2024). LongBench v2 (Bai et al. 2024b), and LongReason (Ling et al.) 2025). Our method consistently and significantly outperforms the outcome-only RLVR baseline. For instance, LongRLVR largely catapults the score of Qwen2.5-14B-IN. across all benchmarks (73.17 - 88.90 on RULER-QA, 40.2 46.5 on LongBench v2, and 73.55 + 78.42 on LongRea- son). By successfully training models to ground their reasoning in provided context, LongRLVR not only overcomes the limitations of conventional RLVR but empower these models with remarkable long-context reasoning abilities comparable with, and even superior to, state-of-the-art reasoning models such Qwen3 (Qwen 2025 series.
2
METHOD
In this section, we introduce LongRLVR to remedy the limitations of RIVR in long-context tasks. We first present an explicit grounding formulation for long-context RLVR in 22. Next, 22. we formally prove that outcome-only rewards lead to vanishing gradient problem for this ground- ing process. To solve this, we introduce our verifiable context reward, presenting its theoretical foundation in I and a practical F-score-based implementation in 2.2.2 Finally, we detail the synthetic data generation pipeline that enables this approach in $2.4
2.1 RLVR ON LONG CONTEXTS: AN EXPLICIT GROUNDING FORMULATION
The standard RLVR framework aims optimize policy To X, X,Q) that generates answer y given a context X and a question Q. The objective is to maximize the expected verifiable reward Tans (v), which typically evaluates the correctness of the final answer:
Jans (0) (xx.Q)~D [Ey-ro(plx.g) (ram (v)]
(1)
This formulation, while effective for tasks where reasoning relies parametric knowledge, ignores two distinct processes in long-context scenarios: (1) contextual grounding, the act of identifying the relevant subset of information within X. and (2) answer generation, the act of synthesizing an answer from the grounded information. When the context X is extensive, the grounding process becomes non-trivial yet remains implicit within the monolithic policy To(y) X,Q).
2

---

Published as a conference paper at ICLR 2026
Here, refactor the policy explicitly model these two stages. Let the long context X be seg- mented into a set of N chunks, C fci..... CN J. the Long-contex: policy should jointly involve grounding and answering to identify subset of selected chunks Z C C and final answer y. This process is modeled as a factorized distribution:
He (y, Z x) = gnd
7 X.Q)
ans 8 (y x,
(2)
Grounding Head
Answer Head
The Grounding Head is responsible for contextual grounding, selecting the evidence Z required to answer the question. The Answer Head then conditions on this selected evidence to produce the final output y-
2.2 THE VANISHING GROUNDING GRADIENT WITH OUTCOME-ONLY REWARDS
We now formally analyze the learning dynamics of the factorized policy (Eq. 2 when optimized solely with the final answer reward, Yans We will demonstrate that this outcome-only signal insufficient for learning the grounding head (mg))) ), creating fundamental bottleneck for long- context reasoning.
Our analysis is based on a common property of long-context reasoning tasks: a correct solution often requires synthesizing complete set of prerequisite evidence. Partial information, while help- ful, typically yields lower reward. That said, an LLM may occasionally answer correctly from a subset of G or from alternative supporting evidence. This structure motivates the following formal assumption.
Assumption (Sparse Answer Reward), Let G S C' be the ground-truth set of essential evidence chunks. There exists a non-negative, monotone set function f 2 - Rzo with f(0) = O such that the expected unswer reward conditioned on the selected set Z depends only on which ground-truth chunks are present:
El
Z
po f(ZnG),
(3)
where pa is a baseline reward from partial or spurious evidence. This form allows different chunks in G to have different importance and credits arbitrary subsets G.
To analyze the gradient, we introduce logit s, for each chunk ej C' and denote by 2j = 1cc E Z} its selection indicator. Let Pj Pro E Z) Eolzy] be the marginal selection probability under the grounding policy, we can derive the proposition below.
Proposition 1 (Vanishing Gradients for Grounding). Under Assumption 7 and the grounding pa- rameterization in Eq. (9) the gradient of the expected answer reward with respect to the logit for any essential chunk Cj is:
V,, E[[ras]] Cov( f( (Z)G), 24) - p,(1 -=)(= 2,-1-
Let (T) A FITU (CT denote the marginal gain of chunk c, for any T C G (cgh and assume (T) S 8, for some constant 5, > Define the activation event for Oj
- {Z 4, ((Zn)) \ {c²))= > 0).
i.e. the event that the rest of the prerequisite evidence that makes oj useful is already present in Z. Then
0 x
E(ran] I py(1-p;) p,)8, Pr(E,).
(See proof in Appx. A.2
Proposition shows that the learning signal for selecting any single required chunk is scaled by (r, the probability that all of the other prerequisite evidence that interacts with ey has already been selected. In challenging long-context tasks where correctly answering the question requires combining many pieces of implicit evidence, this activation event extremely unlikely under the initial RLVR policy: single rollout must simultaneously include large subset of G before Cj can receive positive credit. Consequently, the answer-only gradient for ej is suppressed by the tiny factor Pro(E,) and becomes effectively zero for many ground-truth chunks early in training. Once these gradients vanish due to small standard deviation of context rewards (Razin et al.) 2023), the grounding head is non-trivial to increase the selection probability of the corresponding evidence, causing contextual recall to stagnate and inducing the platcau in training reward observed in Figure [
3

---

Published as a conference paper at ICLR 2026
2.3 LONGRLVR: LEARNING WITH A VERIFIABLE CONTEXT REWARD
To surmount the vanishing gradient problem introduced in 2..2 we propose augmenting the sparse, outcome-only reward with a direct, dense signal that supervises the grounding head. The core the incorporation of verifiable context reward, Texx, which provides granular learning signal for the contextual grounding process.
2.3.1 THEORETICAL FOUNDATION
We begin by defining general class of context rewards as any function that increases whenever an additional ground-truth chunk in G' is correctly selected, i.e., reward that is monotone in the set Z n G' rather than only in its cardinality. Different chunks may contribute different amounts. For analytical tractability, we consider a simple additive form that assigns a (possibly distinct) weight to each ground-truth chunk:
Texx (Z.G)=
Σ ap 1fck Z}. ONEG
where Or > controls the contribution of chunk Cs. This formulation ensures the policy receives positive feedback for each relevant chunk selects, irrespective of whether the complete evidence set G is recovered.
The final reward in the LongRLVR framework is a linear combination of the answer and context rewards:
Trotal (y. Z) = Tans (3) + Texx (Z,
(5)
then prove this general structure is sufficient to provably resolve the vanishing gradient problem.
Proposition 2 (Non- Vanishing Grounding Signal). For the total reward Tinal = Tams + Tax with Tax (cx(Z, = Ensegr Og {ck }), the gradient of the expected total reward with respect to the logit 8y for any essential chunk ej E G is (see proof in Appx. A.3
rroual
V. V(raa]r + ta Var(z,) +
Σ Og Cov 2z)
From Pane
ezEG
From Fas
In particular, combining this with Proposition 7 shows that the answer-only term at most P5 85) (r)(E;), while the context term always contains the dense component aj Var(z;) ag (1 -p;) that not multiplied by (Io(E,). If the grounding policy tends to select related chunks together (so that (ov(2k,2) 23) 2 0 for k # j). the cross-covariance terms further strengthen this signal.
The second term in Proposition 2 thus provides a dense learning signal for each chunk that is in- dependent of the rare activation event E,, preventing the gradient from vanishing even when the answer-only component is negligible. This theoretical foundation establishes that rewarding inter- mediate grounding steps-at the level of actual chunks rather than just outcome correctness is sound and effective strategy for overcoming the learning bottleneck in long-context RLVR.
The second term provides a dense learning signal for each chunk that is independent of the joint suc- cess probability 4. preventing the gradient from vanishing. This theoretical foundation establishes that rewarding intermediate grounding steps sound and effective strategy for overcoming the learning bottleneck in long-context RLVR.
2.3.2 A PRACTICAL INSTANTIATION: THE MODULATED F-SCORE REWARD
While general formulation guarantees non-vanishing gradient, a well-designed, normalized reward is crucial for stable and effective training. A naive metric like recall G) is insuffi- cient, it would incentivize degenerate policy of selecting all available chunks. A practical reward must balance the retrieval of correct evidence (recall) with the avoidance of irrelevant information (precision).
4

---

Published as a conference paper at ICLR 2026
<long context>
<CHUNK_1> 1> Marie Curie born Warsaw, Poland... she moved Paris pursue higher education... </CHUNK_1>
<CHUNK_2> The Curies' early research inspired by Henri Becquerel's s 1896 discovery... </CHUNK_2>
<CHUNK_N> In December 1898, they announced the discovery of second ele ment, "radium," ".. </CHUNK.N> </long.context>
Question:
Where was Marie Curie born and what was the second radioactive element she co-discovered?
Output: <think
</think>
<useful chunks> <CHUNK 1>, <CHUNK N </useful chunks>
<answer>
Marie Curie was born Warsaw, Poland, and the second radioactive element she co-discovered was radium. </answer>
Figure 2:
Data format for LongRLVR training. The model is tasked retrieve useful chunks from the long context before generating the final answer. These chunk identifiers are utilized to derive verifiable context rewards.
To this end, adopt the F3-score the measure of grounding quality. The F3-score is the weighted harmonic mean of precision and recall:
Precision G, Recall (Z, G)
F) B2)
(6)
(B2 (recision(Z,G)) + Recall G)
where 3 is parameter that allows us to weigh recall more heavily than precision (e.g., B ensuring the model is primarily incentivized to gather all necessary evidence.
2),
To create synergistic effect between grounding and final answer accuracy, we formulate our context reward as modulated combination the Fg-score and the answer reward:
Fex (y, Z,G) = (z G) + (1 - n) Tans (3) (Z,
where 7) E [0, 1] is a blending byperparameter. This reward structure has two key components: (1) Unconditional Grounding Reward (1) F3): This provides dense, stable reward for selecting correct evidence, ensuring the grounding head always receives a learning signal. (2) Synergistic Success Reward ((1 7) ans -Fg): This component reward for high-quality grounding is unlocked only upon generating a correct answer. It incentivizes the model treat accurate grounding as a means correct final answer, unifying both objectives and preventing the policy from perfecting grounding in isolation.
as synergistic gate, ensuring that the full
With our proposed context reward, the final LongRLVR objective is to maximize the expected total reward over the data distribution and the stochastic policy:
J (0) = ((x.Q.G)>D [[.2)))=T [Fans(y) + Tax (y. 2, G.G)]]
(8)
This objective be optimized using standard policy gradient algorithms such PPO and GRPO.
To facilitate the computation Tax. we design the policy to first generate list of identifiers for the selected chunks (Z) before generating the final answer (1)) illustrated in Figure
2.4 SYNTHETIC DATA GENERATION FOR GROUNDED QA
Training LongRLVR necessitates a specialized dataset comprising tuples of (X, Q, G.y), where G the ground-truth set of evidence chunks from context X essential for answering question with

---

Published as a conference paper at ICLR 2026
answer y. As such datasets are exceedingly rare, developed the automated pipeline detailed in Algorithm[]] to produce high-fidelity, challenging QA pairs with precise grounding annotations. This pipeline is crucial for the direct supervision of the contextual grounding mechanism in model.
Algorithm Synthetic Data Generation Pipeline for Grounded QA
1: Input: A collection of long documents X.
Output: A filtered dataset D {(X, Q. 9))
3: for each document X X do
4: // Step 1: Semantic Clustering and Evidence Identification
5: Partition X into set of text chunks C = {(i,..... CV
6: Emhed all chunks into dense vector space using sentence encoder.
7: Apply a density-based clustering algorithm to the embeddings to form thematic clusters
[Ki, )......
8: // Step 2: Per-Cluster QA Generation and Scoring
9: Initialize of best-per-cluster candidates, Sace
0.
10:
for each cluster K, IC do
11:
Generate Candidates: Prompt generator LLM with the content of K; to synthesize candidate
tuples {(Qs, Vi, G5)
12:
Crucially, the LLM itself identifies the necessary evidence Gj S K; for each QA pair.
13:
Score Candidates: For each candidate tuple, verifier L.M to assign quality score sj based on question clarity, answer fidelity, and evidence necessity.
14:
Intra-Cluster Selection (Stage I): Identify the candidate (Qi v7 GT) with the highest score st within the cluster.
15:
Add the highest-scoring tuple (Qi, yi Gf Sax. // Step 3: Inter-Cluster Selection and Finalization (Stage 2)
16:
17:
Select the tuple (Q" v" G*) from Saxe that has the overall highest score, hreaking ties randomly. Add final, document-best tuple Qx,", """""""""" ,,",""
yy"") ) to dataset
19:
return
This automated, multi-stage pipeline enables the scalable creation of challenging long-context QA examples with the explicit evidence annotations required to compute our verifiable context reward.
EXPERIMENTAL SETUP
3.1 IMPLEMENTATION DETAILS
Data Curation. To train our model, we constructed a large-scale, high-quality dataset of 46K long-context question-anssering pairs with explicit grounding annotations. We sourced documents from book, arXiv, and code domains, filtering for lengths between 8K and 64K tokens. Following the pipeline detailed in Alg. [] we first identified semantically coherent clusters of text segments within each document. For each document, we then used a powerful generator model, Qwen3-235B- A22B (Qwen 2025). to create multiple candidate QA pairs, with each answer grounded in specific evidence segments. To ensure the highest quality, the same model used as judge score the correctness and evidence relevance of each pair. A two-stage rejection sampling process selected the single best QA pair per document, and we applied strict final filter, retaining only pairs with quality rating above 9 out of 10. See more details in Appx. AB
Training Details. We train three models: LLaMA-3.1 1--8A Qwen2.5-7B-IM, and Qwen2.5- 14B- IM with RLVR implemented by naive Group Relative Policy Optimization (GRPO) (Shao al.) 2024). Crucially, before training of each model, we exclude easy questions for which its answer upon full long context rated or higher by a Qwen3-A235B-A222
judge. For the RL training, we the AdamW optimizer with constant learning rate of le-6 and 5-step linear warmup. During rollouts, we use a prompt batch size of 512 and sample responses per prompt, with a maximum context length of 64K and response length of 4096. We train all models for epoch 46K crafted data. For hyperparameters, we set as 0.1 and 3 as 2 in Eq.
'lll models refer the instruct version.

---

Published as a conference paper at ICLR 2026
Table 1: The evaluation of models on long-context benchmarks. The metric in all benchmarks is accuracy. The best score across all models is highlighted in green, and the second-best is in red. Additionally, the best score within each trained model comparing among SFT, RLVR, and our LongRLVR is bolded.
RULER-QA
LongBench
LongReason
Model
32K
64K
128K
AVG
Short
Medium
Long
Overall
32k
64k
128k
Avg.
LLaMA-3.1-708
70.4
64.2
47.6
60.73
36.2
45,0
34.0
25.9
61.16
63.30
48.30
57.59
Qwen2.5-72B- YaRN
66.9 54.5
47.2
56.20
43.5
48.9
40.9
43.5
74.27
74.53
69.48
72.76
Qwen3-8B (Thinking)
86.5
84.0
81.8
84.10
43.3
28.8
32.4
37.6
77.23
71.28
65.99
71.50
Qwen3-14B (Thinking)
91.2
89.0
82.6
87.60
51.7
42.3
38,9
44.9
80.86
77.08
74.56
77.50
QwenLong--LI-2BB
89.0
77.0
72.4
79.47
53.3
34.4
33.3
41.0
84.13
83.63
75.06
80.94
L.LMA-3.1-8B
65.8 63.7
58.8
62.77
34.4
31.6
21.3
30.4
51.45
49.94
46.53
49.31
-SFT
68.4 65.3
60.4
64.70
36.1
28.4
28.7
31.2
50.88
49.11
48.87
49.62
-RLVR
72.0 68.8
62.6
67.80
35.6
31.2
29.6
32.4
49.87
49.62
49.37
49.62
LongRLVR
85.5 76.5
80.33
30.7
36.2
51.89
56.80
53.23
Qwen2.5-7B-IM
70.5 66.0
58.5
65.00
37.8
31.2
28.7
33.0
66.75
66.25
66.36
66.45
-SFT
72.4 64.2
56.8
64.47
36.7
32.6
28.7
33.2
68.
66.83
66.62
67.36
-RLVR
74.4 68.5
57.8
66.90
37.2
29.3
30.6
32.4
70.78
69.02
68.01
69.27
LongRLVR
82.5 76.5
78.67
45.6
35.8
38.6
80.35
79.47
77.83
79.22
Qwen2.5-114-I1N
90.6 70.6
64.4
75.20
51.7
34.0
33.3
40.2
75.44
71.79
73.42
73.55
-SFT
88.0 66.5
62.2
72.23
48.9
34.9
33.3
39.6
74.18
70.03
69.27
71.16
-RLVR
86.3 69.0
64.2
73.17
48.3
36.7
31.5
39.8
74.06
71.91
71.03
72.33
LongRLVR
95.4 87.8
88.90
55.6
43.3
46.5
81.23
77.96
76.07
78.42
3.2 EVALUATION PROTOCOL
Baselines. We compare LongRLVR against two controlled baselines: Supervised Fine-Tuning (SFT) and naive RLVR (GRPO). All methods are applied to LLaMA-3. 1-8B, Qwen2.5-7B-IM, and Swen2.5-14B-IM. using the same synthetic training data. To contextualize performance, also report scores for leading open-source models (LLaMA-3.1 1--0B, Qwen2,5-72B, Qwen3 series) and specialized long-context baseline, QwenLong-LLI322 (Wan al. 2025). The context windows of Qwen2.5-72B and Qwen3 models are extended to 128K using YaRN (Peng et al. 2023). while Qwen3 models are evaluated their thinking mode.
Benchmarks, We evaluate all models on three challenging long-context QA benchmarks: (1) RULER-QA QA (Hsieh al. 2024): A synthetic benchmark testing multi-hop reasoning over arbitrary context length. We focus on this QA task with the lengths of 32K, 64K, and 1 128K. (2) LongBench v2 (Bai al. 2024b): A realistic multi-choice QA benchmark on documents up to 128K tokens. Standard baselines are evaluated with CoT, while models that output reasoning steps (ours and the Qwen3 series) are evaluated on their final answer. (3) LongReason (Ling et al. 2025); A synthetic multi-choice benchmark designed for controllable evaluation of long-context reasoning. We evaluate the lengths of 32K, 64K, and 128K.
4 RESULTS AND ANALYSES
4.1 MAIN RESULTS
we present the comprehensive evaluation of LongRLVR against various baselines. The results reveal the exceptional effectiveness of our approach, which analyze through two critical comparisons: (1) against naive SFT and RLVR baselines to demonstrate consistent and substantial gains, and (2) against superior L.MSs establish its competitiveness.
Table ]
Consistent and substantial gains over naive SFT and RLVR. LongRLVR consistently and sub- stantially outperforms both SET and naive RLVR when applied to the same base models with identi- cal training data. This is established across different model families (LLaMA and Qwen) and scales (7B, 8B, and 14B), confirming the general applicability of our approach. For instance, LLngRLVR achieves large gains over naive RLVR across all benchmarks and models: for 0-ee2.5-14B-11

---

Published as a conference paper at ICLR 2026
LongREVR Atssettoolyy only
AnturrA
0.955
Contermaa Recall
LangBead
0.20
0.5
12
10
43
10
20
16 20
36
Training Step
Training Step
Training Step
Figure 3: Study on reward components. The answer-only model suffers from stagnating contextual recall, which caps its final performance. The context-only model excels at recall but fails to achieve accurate rewards. By synergizing both signals, 20on2.5-7B-IM-LLnghhe most stable performance on the LongBench v2 benchmark, proving that both rewards are essential.
achieves the best and
(e.g., 46.5 vs. 39.8 on LongBench v2), Qwen2.5-7B-IM (e.g., 38.6 vs. 32.4 on LongBench v2), and LLL-MA-3.1-8B (e.g., 36.2 32.4 LongBench v2). The consistency of these large gains provides strong evidence that LongRLVR effectively remedies the fundamental limitations of naive RLVR on long-context scenarios by directly supervising the contextual grounding process. In ad- dition, the superiority to SFT demonstrates the potential of RLVR as compelling post-training approach for incentivizing long-context capabilities.
Comparable superior L.Mss. Beyond outperforming direct RLVR, LongRIVR elevates LLMs to a exceptional performance tier, enabling them to surpass much larger conventional models and rival the latest specialized reasoning LLMs. First, our LongRLVR demonstrates remarkable pa- rameter efficiency against larger, conventional LLMs. Our Sowen2.5-7BIIM model (79.22 on Lon- gReason) significantly outperforms both the 3LaMA-3.1-70BB (57.59) and the Qwen2.5-72B-Yaa (72.76). Similarly, our 14B model (46.5 on LongBench v2) even surpass the performance of the 72B model, showcasing the ability to instill powerful long-context reasoning capabilities in much smaller parameter footprint. Second, LongRLVR empower conventional base models with excep- tional long-context reasoning abilities that compete with and even surpass specialized models. No- tably, Qwen2.55 11102255--4-BB11 1-B-14BBBB 1M, trained with LongRLVR, outperforming the newer Qwen3-14BB 14B (88.90 vs 87.60 on RULER- QA, 78.42 vs 77.50 on LongReason) which benefits from a more advanced backbone and post-training strategy. Moreover, which derives from the reasoning model, R1-Distilled Sestilled-Qwen-322 trained with long-context RLVR. This demonstrates the significant effectiveness method to unlock superior long-context reasoning for non-reasoning LLMs.
14B model is comparable the much larger
QwenLong-LIise
4.2
IMPACT OF REWARD COMPONENTS
In Figure we demonstrate that our LongRLVR overcomes the bottleneck of outcome-based RLVR by incorporating verifiable context rewards. To isolate the impact of the context reward, in Figure 3 we compare the training of Qwen2.5-7B. 1M with the full LongRLVR against using answer-onf and context-only (F3 score in Eq. (7) rewards, respectively. The results confirm our central hypothesis that contextual recall of answer-only baseline quickly stagnates, thus creating a hard perfor- mance ceiling on both the training reward and the downstream task. Conversely, the model trained with context-only reward, despite involving a flat answer reward, shows rapid initial performance gains the LongBench v2 benchmark. This demonstrates that mastering contextual grounding is a foundational capabilities that directly boosts long-context reasoning. However, without the final answer reward to steer reasoning toward correct outcome, its downstream performance eventually degrades. While our LongRLVR succeeds by synergizing both signals, hence achieving continually improved training answer reward and downstream tasks performance.
4.3 IMPACT OF DATA QUALITY
We study the impact our two data quality strategies in our data synthetic pipeline: (1) using re- jection sampling to select high-quality generated QA pairs, and (2) filtering out easy questions. We ablate these choices using the Qwen2.5-7B- model and report the overall score on LongBench v2. The results shown in Figure 4 First, Figure (left) shows that rejection sampling quality is
8

---

Published as a conference paper at ICLR 2026
Owert 5-1
HaMaAci114 1-41
Qwee2 LLMMA
LongBench 12 (Overal))
LangBeach
LongB
14
0.3
0.35
0.5
0.75
1.0
04 06
126
#CChaank per Document
(a) Effect of blending factor f)-
(b) Effect of F-score parameter [3. (c) Robustness to chunk number.
Figure 5: Ablation studies key hyperparameters for LongRLVR. We analyze the overall per- formance on LongBench v2 while varying (a) the blending factor 17 in the context reward, (b) the F-score parameter 33 and (c) the number of chunks per document. Results reported for both Qwen2.5-7B and LLaMA-3.1-8BB
critical. Using the best-rated samples achieves our top score of 38.6, which degrades significantly with median (36.6) and worst-rated (34.8) samples. Second, Figure 4 (right) analyzes our filtering strategy. Our default method filtering only easy questions proves most effective. Crucially, filter- ing out hard questions is highly detrimental, causing performance to plummet to 35.8, nearly low as applying no filtering at all (35.6). This suggests that these challenging examples are essential for enhancing the complex reasoning ability required for congecontext tasks.
4.4 ABLATION STUDIES ON HYPERPARAMETERS
We further conduct ablation studies to analyze key hyperparameters in our method, with re- sults shown in Figure 5 (I) Blending Fac- tor 17. This factor balances the unconditional grounding reward (Fa) and the synergistic re- ward (rams F3). Figure 5a shows that perfor- mance peaks small, non-zero value (n) 0.1). A purely synergistic reward (1) 0) is suboptimal because the initial learning signal is too sparse. Conversely, purely unconditional reward (n) = 1) decouples grounding from the final goal of producing correct answer, hence leading to inferior effectiveness. (2) F-score Parameter B. The ,9 parameter trades off re- call and precision the grounding reward, As shown Figure 5b] performance is optimal at 3 = 2. This moderately prioritizes recall, which critical for complex reasoning where failing to retrieve a single essential piece of ev- idence can be catastrophic. A lower encour- ages overly conservative policy that fails to retrieve all necessary chunks, while a higher 3 incentivizes retrieving too much irrelevant context, which complicates the final reasoning step. (3) Robustness to Number of Chunks. Figure 5c demonstrates that LongRLVR is remarkably robust the number of chunks per document, maintaining high performance from 16 to 128 chunks per document during evaluation. This a significant practical advantage over traditional retrieval sys- tems, which are often highly sensitive chunking strategy. This robustness indicates that the model learns a flexible semantic grounding policy rather than relying on surface-level features, allowing it identify relevant information regardless of how it is segmented.
40
LongBench v2 (Overall)
38
345
Sample Best
Sample Median
Sumple Warst
Filter Easy Filerr Hard
Filerr Both
No fittring
Figure 4: Data quality ablation on LongBench v2. Left:
The effect of rejection sampling qual-
ity. Right: The effect of different data filter- ing strategies. High-quality, challenging data is shown to be most effective. Results are reported on

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Reinforcement Learning with Verifiable Rewards (RLVR). Reinforcement Learning with Ver- ifiable Rewards (RLVR) has emerged as powerful paradigm for enhancing the reasoning of LLMs by rewarding models based on deterministic, ground-truth outcomes like passing unit tests or deriv- ing a correct solution (Lambert et al. 2024; Guo al. 2025). This approach has propelled models to expert-level (e.g.. IMO-level mathematics) performance on complex, self-contained reasoning tasks such as mathematics and programming (Guo et al. [025] Jaech et al.] 2024 Kimi al. 2025
Huang & Yang] 2025). In these settings, the primary challenge is to refine the model's s internal, parametric knowledge to discover correct chain of thought (uue et al.) 2025 Wen et al. 2025).
However, the efficacy of this outcome-only reward structure is limited in long-context scenarios, where success hinges first on identifying relevant evidence from a vast external input-a process term contextual grounding (Wan et al. 2025). Wang et al. (025) incorporate retrieval reward in RLVR for the appearance of correct context in thinking process. Our work directly addresses this gap by introducing verifiable reward for the intermediate grounding process itself.
Long Context Alignment. Previous studies successfully extended model context windows
through methods like Rotary Position Embedding (RoPE) scaling (Su et al. (002 Chen al. 2023] Peng et al., 2023) An et al. (024). Yet, the extended models with long context windows often fail to reliably use the information in applications. To solve this, long-context alignment becomes crucial to unlock the model's 's latent capabilities by post training, which includes long-context SFT (Bai et al.]
2024a), DPO (Chen al.) 2025), and RLVR (Wan et al. 2025). We investigate the challenges of applying RLVR in long-context settings and propose a novel framework that substantially enhances its efficacy for alignment.
Long-Context LLM Agent. Recent works (Zhao et al. 2024 Qian al. 2024 Zhang et al. 2024
Zhou al.. 2024) propose utilizing agentic workflows to tackle long-context tasks. Instead of pro- cessing the full context via a single LLM pass, these methods split the text into chunks, processing them sequentially and integrating information through multi-turn collaboration, such as updating states in a chain (Zhang et al. 2024). These approaches circumvent the inherent limitations of long-context capabilities in standard LLMs, making them orthogonal to our contribution. Our work focuses on improving the model's 's native reasoning ability over the full long context. Furthermore, has the potential to enhance agentic frameworks by enabling agents to process larger chunks per step, thereby scaling to even longer contexts.
our approach is complementary:
6 CONCLUSION
In this work, we addressed fundamental limitation of Reinforcement Learning with Verifiable Rewards (RLVR) in long-context scenarios: its inability to effectively learn contextual grounding due to sparse, outcome-only rewards. We formally identified this issue the "vanishing grounding gradient" problem, where the learning signal for retrieving evidence diminishes significantly with the complexity of the task. To overcome this, introduced LongRLVR, a novel training paradigm that augments the standard answer reward with a verifiable context reward. This dense reward signal explicitly teaches the model to first identify and extract relevant evidence before generating an answer. Our extensive experiments demonstrate that LongRLVR substantially outperforms both SFT and naive RLVR baselines across multiple models and benchmarks. Our analyses confirm that this success stems from the synergy between the context and answer rewards for both improved grounding and answer quality. By directly training models to ground their reasoning in provided evidence, LongRLVR provides robust and effective framework for unlocking the long-context reasoning capabilities of LLMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This project was partially supported by the Singapore Ministry of Education Academic Research Fund Tier (Award Number: T1 25 251RES2514) and MiroMind Al Research Intern Program.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. All theoretical claims are formally proven the appendix, with detailed, step-by-step derivations provided for both RE- INFORCE and GRPO estimators in Appx. $A The synthetic data generation pipeline, which is crucial for our method, is described in Alg. [ and further detailed in Appx. $B covering corpus sourcing, preprocessing, and quality control. All implementation details, including model specifies, training hyperparameters, and the learning strategy, are documented in $3.1 The evaluation pro- tocol, including baselines, benchmarks, and metrics, is clearly outlined in $3.2 To facilitate direct replication our results, we will release our source code, the generated dataset, and trained model checkpoints upon publication.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We utilized Large Language Models (LLMs), including Google's Gemini and OpenAl's GPT series, as assistive tools in the preparation this manuscript. Their use was limited to the following tasks:
Generating Python code for the data visualizations in Figures
and
Assisting with the LaTeX formatting of complex elements, particularly Table 1
Proofreading and copy-editing the text for grammatical correctness and clarity.
The core research ideation, theoretical contributions, experimental design, and interpretation of sults are entirely the work of the human authors. LLMS served strictly as productivity and presenta- tion aids.
REFERENCES
Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-fres long-context scaling of large language models. arXiv preprint arXiv:2402. (tti:2402.744454 17463, 2024.
Yushi Bai, Xin Lv, Jiajie Zhang. Yuze He, Ji Qi, Lei Hou, Jie Tang. Yuxiao Dong, and Juanzi Li. Longalign: recipe for long context alignment of large language models. 2024a. URL
https://arxia: www.oogg/aa... / 2401. 18058
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang. Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.1 15004, 2024b.
Guanzheng Chen, Xin Li, Michael Qizhe Shieh, and Lidong Bing. Longpo: Long context self- evolution of large language models through short-to-long preference optimization. arXiv preprint vrXiv:250..13922 2025.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionaliiyy multi cultiigranularity text embeddings through self-knowledge dis- tillation, 2024.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. 2023. URI. httts::/arxxv.orgg org/abs/////. abs/ 2306. 15595
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
11

---

Published as a conference paper at ICLR 2026
Martin Ester, Hans-Peter Kriegel, Jöng Sander, and Xiaowei Xu. A density-basee algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second Inter- national Conference Knowledge Discovery and Data Mining, KDD'96, pp. 226 231. AAAI Press, 1996.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). In ACL, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025, arXiv preprint arXiv:2507.15855, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint vrXiv:2411.15122 2024.
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii. Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dovan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes. Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httpp:///ooeeeevvvwwee/////ooood Reproducibility Certification.
Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089 2025.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. 2023. URL/httss://arxix hww.:/oog///ss///// 2209. 00071
Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, and Zhicheng Dou. Are long-Ilms& a necessity for long-context tasks? arXiv preprint arXiv:2405.15316 2024.
Team Qwen. Qwen3 technical report, 2025. URL
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. arXiv preprint arXiv:2310.20700 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint axXiv:2402.03300, 2024.
12

---

Published as a conference paper at ICLR 2026
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. 2022. URL httpss//arxiv, org/abs/ 1986409864
Fanqi Wan, Weizhou Shen, Shengyi Liao, Yingcheng Shi, Chenliang Li, Ziyi Yang, Ji Zhang, Fei Huang, Jingren Zhou, and Ming Yan. Qwenlong-ll: Towards long-contex( large reasoning models with reinforcement learning. arXiv preprint (xxx:2505.176000 2025.
Suyuchen Wang, Jinlin Wang, Xinyu Wang, Shiqi Li, Xiangru Tang, Sirui Hong, Xiao-Wen Chang, Chenglin Wu, and Bang Liu. Improving context fidelity via native retrieval-augmented reasoning. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing. pp. 11216-21222 2025.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base Ilms. 2025. http::tss:///rxiiii org abs//5// 14245
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen2.5 technical report. arXiv preprint arXiv:2512.15115, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint ar Xiv::22004. 13837, 2025.
Kaiyan Zhang. Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jía, Pengfei Li, al. A survey of reinforcement learning for large reasoning models. arXiv preprint wwwiv:5509.08877 2025.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks. Advances Neural Information Processing Systems, 37: 37:132208-132237 2024.
Jun Zhao, Can Zu, Hao Xu, Yi La, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. Longagent: scaling language models to 128k context through multi-agent collaboration. arXiv preprint 40X5v2240..11550, 2024.
Zihan Zhou, Chong Li, Xinyi Chen, Shuo Wang, Yu Chao, Zhili Li, Haoyu Wang, Rongqiao An, Qi Shi, Zhixing Tan, et al. Llm mapreduce: Simplified long-sequence processing using large language models. arXiv preprint arXiv:2410.09344 2024.
13

---

Published as a conference paper at ICLR 2026
A DETAILED PROOFS FOR PROPOSITIONS AND 2
This appendix provides detailed derivations for the theoretical results presented in Section and Section 2.33 We formally prove that outcome-only rewards lead to vanishing gradients for contextual grounding and show how the proposed context reward resolves this issue. The proofs are provided for both the standard REINFORCE policy gradient estimator and the Group-Relative Policy Opti- mization (GRPO) algorithm.
A. PRELIMINARIES AND NOTATION
We begin by summarizing the formal setup used throughout the proofs. The policy is factorized into a grounding head and an answer head, such that me m X, Q) n (Z X, Q) " "
X, Q, 2). Our analysis focuses the gradients with respect to the parameters of the grounding and head, m
Grounding Head. The long context X is partitioned into set of chunks C. = f...... cN. The grounding head models the selection of each chunk oj via a binary selection vector Z (21...., 2N) E {0. 1) where 2, 1 selected). We parameterize the grounding policy as log-linear distribution over subsets
7(0) exp Σ j=1
8y2ソ + v())),
9
where 8, is the logit associated with chunk Cy. (Z) is an arbitrary potential that can capture depen- dencies between chunks, and (0) is the normalizing constant. This family subsumes the indepen- dent Bernoulli model used the initial version of the paper as the special case (ZZ) = We write gnd Pj = Ee [ = Prę(c, E Z) for the marginal selection probability. Differentiating log тद (Z) with respect to s, yields the score function
Ve, log gud (Z) = 2j Pj. Th
which is the only property of the policy we use in the subsequent analysis.
Ground-Trun Truth and Reward. Let G C C' be the ground-truth set of essential evidence chunks required to answer the question, with |G| = We define the "success" event S as the selection of all essential chunks, i.e., S = (Z 2 G}. The probability of this event is & Pra(S). Under the Sparse Answer Reward (Assumption []. the conditional expected answer reward can be written as B|rans Z = Po + f(Z n G)) for some monotone set function f. 2 + R20 with f(0) = 0. For each Cj E G and subset C G.Y {cz} we define the marginal gain
AT(T) & f(T (c,}) fTT),
and assume it is bounded by A, (T) < 8, for some constant 5, > O. The all-or-nothing reward used in the initial version corresponds to f(T) = 6 1(T 2 G), where (T(T) non-zero only when T
already contains all other evidence in G.
For the proof of Proposition 2
we additionally use an Additive Context Reward of the form
Tex Tex G), =
Σ
ag > 0,
CgEG
so that the total reward is Fiotal Tans + Tetx-
Policy Gradient Estimators. The gradient of an expected reward B[R(Z)] is computed using the REINFORCE identity (the score function estimator):
V., E[R(Z)] E [[( V ay log 5md (Z)] (c, (2,
(10)
Using baseline b that does not depend on zy. this equivalent the covariance between the reward and the action score:
B[R(Z)) E (R(() b) (z3 p3)] Cov (R(2), 2z)-
(11)
14

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF PROPOSITION 1: VANISHING GRADIENTS FOR OUTCOME-ONEE REWARDS
Proposition 1. Under Assumption 7 the gradient of the expected answer reward with respect to the
logit 8j for any essential chunk Cj E G satisfies
v1, amsss - Cov (fov))) 23) - P, - (1)11-)) - (1(0)))/(2n]11111-- and is bounded as
S V B/Tan] S 1y(1-Pj) Pr(E,),
where E A 1 A,((Zn (0t)) 0f is the activation event for cj.
Proof using REINFORCE. Using the covariance form of the policy gradient from Eq. (II) we have
Vay B(rans] Cov((am ; 2j) = Cov + ff 2,) Cov my)
For binary variable 2j E (0, 1), the covariance admits the standard decomposition
())f(Znn),3) 2,) P1 pj) (E( nG) 2,= (2 G) z,=-)
To interpret the difference of conditionals, consider the subset of ground-truth chunks other than Cy that are selected, T()) ~ (Z C foz) When 2j we can write
[T())
+ (T(T)))),
where (T) is the marginal gain defined above. Taking expectations and subtracting the case 2j yields
Bf 12-1 + =(-
By monotonicity, (7( 2 O, and by boundedness, A,(T) S oj. Let &; 12: (T() > 0} be the event that ej has a strictly positive marginal gain given the other selected evidence. We thus obtain
< E(A((T(2)) 2, - 8, Pr(E, 1zz=1) 5, Pr(E,)
0
0)
Substituting back gives the claimed upper bound V., E[rans] < p, - p9) Pro(E,), and the non-negativity of the gradient follows from the monotonicity of f.
Proof using GRPO. GRPO uses a grlup-relative baseline. For a group of K 2 2 iid.. trajectories, the unclipped GRPO surrogate gradient 0 = Osta is proportional the covariance:
K Cov(rams 2z)
K
CORFO (Dola)
B[Tans]
K
K
Therefore, the GRPO gradient inherits the same bound from Proposition I i.e., it is also scaled by the activation probability Pro(E,) and becomes vanishingly small when Pro(E,) is tiny.
A.3 PROOF OF PROPOSITION NON- VANISHING GROUNDING SIGNAL
Proposition 2. For total reward Fioua Tans + Fax where Tex (Z,G) - 0k2k with og 0, the gradient of the expected total reward with respect to the logit $ for any essential chunk Gj E G is
V E (iraaa = Va, El(rms) + aa Var(z;) + Σ as Cov(ZE, 25).
CLEG
In particular, combining this with Proposition 7 shows that the answer-only part is at most p, p3) p5)ō, 55 Pro(E;), while the term oj Var(2,) aj Pÿ) is a dense contribution that does not depend on (I)(E;). the grounding policy exhibits non-negative correlations among related chunks (so that Cov(zk, 23) 2 0 for + j), then
Protal] 2 aj Var(z) =, P, p9)
whenever Pj (0,1).
15

---

Published as a conference paper at ICLR 2026
Proof using REINFORCE. By linearity of expectation, the gradient decomposes: Ve, E E( Pooaal Cov(Fans: 2y) + Cov(Tax: 2,). From Proposition 1, we know (Tansaas. 22) ) = Vay Elrans) for E G.
We compute the contribution from the context reward, Fax >KEG agZki
Cov(Tax 25) = Cov Σ kEG
2k = E an Cov(ze, 23) = =a Var(zz) +
Σ kØj OgE6
or Cov(zR, 2z).
REG
Substituting this expression for Cov(rex: 2n) yields the claimed form for Ve, B| The 0j Var(z;) = O,P,(1 (1-p;) Pú}) is always non-negative and does not depend on the rare activation event E,. so provides dense per-chunk learning signal even when the answer-only component nearly zero. When related chunks tend to co-occur, the croas-covariances Cov(zk. 22) further amplify this signal.
Verification for GRPO and Direct Differentiation. The GRPO gradient is similarly scaled by (K - (K-1)/ yielding
(ORPo(Oola) K (V., E[rans] Cov 24))
so the non non- vanishing term a, Var(z;) appears unchanged. In the special case where chunk selections are independent and all weights are equal (ax a), we have Cov(EE, = 0 for k + and Var(z;) = (1 - Pj), giving
Va [[[rona]] 8. a(1 p2) py(1 py);
which matches the simpler formula reported in the main text of the initial submission. Under the same independence assumptions, direct differentiation the expected total reward, E[riota] = #40 + ōq EEGg Ph: also yields the same result.
DATA CURATION AND GENERATION DETAILS B
This section provides comprehensive overview of the pipeline used to generate the grounded long- context question-ansse answering dataset for training LongRLVR.
B. CORPUS SOURCING AND PREPROCESSING
Our data generation process began with large corpus of long documents from diverse domains, in- spired by Gyo et al. 2025). Book and arXiv documents were sourced from the Long-Date Data-CCollecton dataset, while code documents were sourced from the StarCoder dataset (Li al., 2023). where all files within a repository were concatenated to form a single document. We filtered this raw corpus to retain only documents with token lengths between 8K and 64K tokens, as measured by the Qwen2.5 tokenizer. This step yielded an intermediate corpus of approximately 18K book, 16K arXiv, and 17K code documents.
B.2 DOCUMENT SEGMENTATION AND SEMANTIC CLUSTERING
To prepare documents for evidence identification, each document partitioned into exactly 64 segments. This process was sentence-aware ensuring splits occurred only at natural text boundaries newline) preserve the semantic integrity of each chunk. All segments were
(e.g., after after a period then embedded into a high-dimensional vector space using the BGE-M3 sentence encoder (Chen) al. 2024). We applied the DBSCAN algorithm (Ester al. 1996) the embeddings within each document, grouping semantically related segments into thematic clusters that would form the basis for targeted question generation.
B.3 QA GENERATION AND QUALITY CONTROL
We employed multi-stage generation and filtering process to ensure the final dataset of high quality. For each document, we randomly selected distinct semantic clusters (with a minimum of 4 chunks each) and prompted the Owwn3-235B--A22 model (Qwen 2025) to generate candidate
16

---

Published as a conference paper at ICLR 2026
(Q, G)) tuples per cluster, where G is the set of evidence chunks the model deemed necessary. To maintain high standards, we used the same model as an automated judge to assign a quality rating from to 10 for each generated pair, based clarity, correctness, and evidence relevance. Both generation and judging used chain-of-thought prompting. A two-stage rejection sampling process then selected the single best QA pair for each document: first, selected the top-scoring candidate within each cluster, and second, we selected the best among these four candidates, As final quality filter, we discarded any pair that received final rating below 9. This pipeline resulted in our final dataset of 46K documents, each paired with single, high-quality, and well- well-grounded question- answer pair.
Dataset Example: Long Context QA
Question: What factors contributed to the Mehrikans' eventual disappearance despite their unexpected military victory over the European armada?
Although they achieved a decisive naval victory against the European alliance (a war sparked by their own commercial greed), their civilization ultimately collapsed due to drastic climatic shifts and physiological degeneration (nervous diseases) that reduced their population from 90 million to 12 million.
Answer:
Context (Excerpt): <CHUNK 0> THE LAST AMERICAN By J. A. Mitchell [Illustration: "-In the soft earth was the
imprint of human feet!")... [..]]
<CHUNK
holds the opinion that the Mchrikans were mongrel race wealth, luxury, and gradual decline of the native population: the frightful climatic changes which swept the country like mower's scythe; all this is told by Noz-yt-ahl with force and accuracy.
[...]
CCHUNK29>>
"There were many causes," he answered. the effect of climate upon succeeding generations fatal. They became flat-chested and thin, Nervous diseases unknown us wrought deadly havoc. the population decreased from ninety millions to less than twelve millions. The temperature would skip in single day from burning heat to winter's cold. No constitution could withstand it...
[...]
<CHUNK
have spoken Mehrikans being a greedy race. And their greed, at last, resulted in this war. By means of one-sided laws they secured for themselves a lion's share of all profits until at last the leading powers of Europe combined in self- defence...
[..]]
<CHUNK 61>
lasted just one summer afternoon. But the Mehrikans it was who sent their enemies the bottom. And the sea beneath our feet strewn with iron hulks.
Reference Chunk Dss: [3, 29, 61]
Figure 6: An example instance of training data. The answer requires synthesizing information about the war's outcome (Chunk 61), the war's cause (Chunk 59), and the specific biological and environmental causes of extinction (Chunk 29).
17


Reviews:
[{"review_id": "flLK8IwWkj", "reviewer": "Reviewer_kmU8", "summary": "This paper proposes LongRLVR, a reinforcement learning framework for long-context LLMs that introduces verifiable context rewards to overcome vanishing gradients in grounding. Instead of rewarding only final answers, LongRLVR adds dense rewards for correctly selecting evidence chunks, ensuring effective credit assignment across lengthy inputs. It decomposes the policy into grounding and answering heads, uses F-score–based context rewards, and achieves large gains on long-context QA benchmarks, outperforming outcome-only RL baselines and enabling smaller models to surpass larger ones.", "questions": "Some other gradient vanish work in LLM RL could be discussed in related work. e.g, \"Vanishing Gradients in Reinforcement Finetuning of Language Models\"", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Timely and impactful problem.**\nThe paper tackles a highly relevant and increasingly important issue — how to perform reinforcement learning effectively in long-context settings. As large-context reasoning becomes central to emerging LLM-based agents and search systems, addressing the credit-assignment and gradient-vanishing challenges identified here is both timely and of broad significance.\n2. **Strong motivation, clear formulation, and well-executed methodology.**\nThe study is well motivated and rigorously executed. It formally defines the reward-vanishing problem in long-context RL, provides theoretical analysis to explain why outcome-only rewards fail, and introduces a principled solution through verifiable context rewards. The inclusion of a synthetic yet well-controlled dataset allows precise testing, and the resulting performance gains over baselines are substantial and convincing.\n3. **Clear structure and presentation.**\nThe paper is clearly organized and well written, with intuitive explanations and consistent notation. The conceptual flow, from identifying the issue to formal analysis, method design, and empirical validation, is easy to follow, making the technical contributions accessible and well supported.", "weaknesses": "1. **Strong theoretical assumptions.**\nThe analysis relies on several simplifying assumptions that may not fully hold in practice. In particular, it adopts an all-or-nothing reward assumption, where the answer reward increases only when the entire evidence set G is selected. In reality, LLMs often produce correct answers from partial or alternative evidence, making this assumption less realistic. Similarly, the independent Bernoulli selection assumption overlooks dependencies between evidence chunks—real policies typically select evidence in a correlated or sequential manner, which could alter the theoretical gradient behavior. It would strengthen the paper to discuss under what scenarios these assumptions are likely to hold (e.g., explicit fact-retrieval tasks) and where they may fail (e.g., open-domain reasoning). Such clarification would help readers understand the practical scope of the theoretical results.\n2. **Connection to broader long-context RL not explored.**\nThis is more like a suggestion. The paper could better relate its formulation and method to other long-context settings, such as agent RL, where the context includes environment state and action history (like adding a discussion section to appendix). Discussing which aspects of LongRLVR’s framework may transfer and which may not would improve generality and impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3StNCMbpSt", "reviewer": "Reviewer_fzK1", "summary": "This paper addresses a fundamental limitation of RLVR when applied to long-context reasoning tasks. The authors identify that outcome-only rewards suffer from vanishing gradients for the contextual grounding process. They formally prove this vanishing gradient problem and propose LongRLVR, which augments sparse answer rewards with dense, verifiable context rewards that explicitly supervise evidence selection. The method is validated on RULER-QA, LongBench v2, and LongReason benchmarks, showing consistent improvement against vanilla GRPO and SFT. The approach requires ground-truth evidence annotations, and the authors also propose a data generation pipeline using clustering and rejection sampling.", "questions": "1. Would it be helpful if some existing QA datasets with ground-truth evidence (like HotpotQA) is used partially as the training data or as a seed dataset for the data generation process?\n2. Are the chunk identifiers ([CHUNK_N]) added as new special tokens?\n3. The useful chunks are generated after the thinking process. Does the model actually use or refer to the chunks during the training process?\n4. How does performance degrade with noisy evidence annotations?\n5. Can you evaluate on datasets with evidence annotations (e.g., HotpotQA) for the retrieval accuracy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The formal analysis of why the outcome-only reward is insufficient for the long-context retrieval-based task provides some transferable insights.\n2. The modulated F-score reward, combining unconditional grounding reward and synergistic success reward, is thoughtfully designed.\n3. The paper provides extensive analysis on both synthetic and real-world long-context tasks, and the paper includes thorough ablations examining reward components, data quality, hyperparameters, and chunk number robustness, making the claims more sound.", "weaknesses": "1. The comparison is a bit weak, which hinders the overall soundness of the work. Interleaving reasoning and retrieval is now becoming more popular. I would suggest comparing with some RAG baselines (which do not require RLVR but fit the same scenario), as well as some recent works like [1].\n2. Assumption 1 seems too strong for the analysis. In reality, the reward for retrieved evidence, if applied, should be more continuous than the 0 or 1 sparse reward. Also, the independence assumption for chunk selection might not be true since the evidence should be highly related in a multi-hop QA scenario like the ones in LongBench.\n3. A human evaluation of the validity of the generated data or some examples provided would be very beneficial.\n4. There is a potentially biased evaluation regarding the training data. The vanilla RLVR and SFT baselines are trained on the same generated data with evidence, but they don't have the corresponding training signal, which may introduce bias towards the proposed method.\n\n[1] Wang et al. 2025. Improving Context Fidelity via Native Retrieval-Augmented Reasoning. arXiv:2509.13683.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SfM6WaBT0E", "reviewer": "Reviewer_RAAP", "summary": "This paper focuses on improving long-context reasoning through Rule-based Rewards. They propose a new pipeline to generate data, thereby enabling the use of rule-based rewards to evaluate the evidence used in the LLM output.  Experiments show consistent gains over answer-only baselines, supporting the method’s effectiveness.", "questions": "See weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem is significant, as RLVR may stimulate hallucinations and render the training process unstable, while its sparse rewards make effective exploration challenging in practice. \n\nThe paper addresses the issue of vanishing gradient in RLVR under sparse outcome-reward settings, examining its causes and implications.\n\nThe choice of the F1 score as a reward makes sense to me, since it balances precision and recall rather than encourages the model to cover the evidence as much as possible.\n\nThe experiments appear to support the authors’ claims and show consistent improvements.", "weaknesses": "However, my concerns arose from the data generation pipeline and the usage of the verifier LLM. \n\n1. It seems that the method is only applicable for the Grounded QA, where evidence can be cleanly chunked. However, in such a case, performing rule-based rewards for the evidence suggestion should be straightforward. The usage of the F1-score is also straightforward to me, since recall encourages the policy to cover as many chunks as possible.\n2. A separate verifier LLM is used, which helps identify the evidence and check its alignments with the evidence library; however, it makes the comparison with RLVR unfair. Moreover, an additional LLM can do more (e.g., directly judge whether the final answer matches the reasoning path). Why not use semantic similarity or other similar metrics?\n3. The computational and human costs are nontrivial, both from data collection and the additional LLM verifier. Therefore, I am wondering: do the gains adequately justify the substantial supervision and computation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)