Revisiting Long-context Modeling from Context Denoising Perspective
Authors: Zecheng Tang
Baibei Ji
Juntao Li
Lijun Wu
Haijia Gui
Min Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Language Modeling
Long-context Understanding
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces CDT (Context Denoising Training), a simple yet effective strategy that improves long-context models by reducing the impact of noisy context and enhancing focus on critical information.

Abstract: Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their ability to locate implicit critical information within the context for further prediction. However, recent research reveals that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens, that can mislead model attention. In this paper, we conduct a fine-grained analysis of the context noise and propose an effective metric, the Integrated Gradient (IG) score, to detect and quantify the noise information within the context. Our findings reveal that even simple mitigation of detected context noise can substantially boost the model's attention on critical tokens and benefit subsequent predictions. Building on this insight, we propose Context Denoising Training (CDT), a straightforward yet effective training strategy that improves attention on critical tokens while reinforcing their influence on model predictions. Extensive experiments across four tasks, under both context window scaling and long-context alignment settings, demonstrate the superiority of CDT. Notably, when trained with CDT, an open-source 8B model can achieve performance (50.92) comparable to GPT-4o (51.00).

Published as a conference paper at ICLR 2026
REVISITING LONG-CONTEXI MODELING FROM CON- TEXT DENOISING PERSPECTIVE
Zecheng Tang ?
Baibei Ji Juntao L Lijun Wu', Haijia Gui', Min Zhang'
Soochow University 2 L.M Laboratory (actang, bbji /estu. suda. edu. on
SShangaii Artificial Intelligence Laboratory {ijt, minzhang @suda. edu.
ABSTRACT
Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their ability to locate implicit critical information within the context for further prediction. However, recent research reveals that LCMs of- ten susceptible to contextual noise, i.c., irrelevant tokens, that can mislead model attention. In this paper, conduct a fine-graines analysis of the context noise and propose an effective metric, the Integrated Gradient (IG) score, to detect and quantify the noise information within the context. Our findings reveal that even simple mitigation of detected context noise can substantially boost the model's ss attention critical tokens and benefit subsequent predictions. Building this insight, we propose Context Denoising Training (CDT), a straightforward yet ef- feetive training strategy that improves attention on critical tokens while reinfore- ing their influence on model predictions. Extensive experiments across four tasks, under both context window scaling and long-context alignment settings, demon- strate the superiority of CDT. Notably, when trained with CDT, open-source 8B model can achieve performance (50.92) comparable to GPT-4o (51.00).
INTRODUCTION
The ability to handle long input sequences has become a fundamental requirement for large language models (L.LMs), with cutting-edge models capable of processing context lengths exceeding millions of tokens (Team 2024
LOGO 128K
CDT 128K (ours)
SelF: mel-oooovv
50
LongLoRA-12aK
Mode Performance
Prolong-
Liama3 Base
MiniMax et al., 2025 Metal 2025 Qiu et al.
40
CDT 128K (aurs)
Liama3. Laam3311Instrue
20256). This advancement eliminates the need for complex toolchains and intricate workflows, e.g., RAG (Yu et al.) (2024), and significantly enhances real-world applications, such as LLM agent (Luo et al. 2025; Xi et al. 2025) and project code analysis (Fang et al. 2024a).
LongCE 128K
CE CE-128K
CDT-32K (ours)
LongCE 32K
Lifficient
26
0
Performance Gain per Billion Tokens
Recent studies indicate that LCMs frequently fail when processing long-context tasks (Hsich et al. 2024, Kuratov et al., 2024, Tang et al. 20246 Bai et al.] (024c) and the open-source community mitigates such an issue mainly by using sufficient high-quality synthetic long- context data to post-train the model (Fu et al.
Figure I: Comparative overview model perfor mance
long-content tasks and per- formance gain per billion tokens among different training methods. The bubble size indicates the relative training data volume.
2024a) Chen et al.) 2024b| Gao et al., (024a). However, these approaches are proven be either inefficient or ineffective under limited resources. A controlled study in Appendix A shows that, when trained on 2B tokens using the Llama3-8B backbone, Prolong-64k Baaun-64K-Basss (Gao al., 2024b) improves its average score on 12 real-world tasks from 25.5 to 29, 19.13-qquialem to a gain of 1.8
"Correponndig Author
O Code is available at: htttps://github.comm http:///ggtthh.ccc///o.aaaaaaa// httpa//.hhpppp/............ooooe

---

Published as a conference paper at ICLR 2026
points per additional IB tokens whereas LongCE (Fang et al. 2024b) achieves 32.91 points, cor- responding to a gain of 3.7 points per 1B tokens. As illustrated in Figure [] the training efficiency decreases as the training context length increases to 128K, e.g., Prolong- 128K.
One of the possible reasons is that existing works overlook the fact that LCMs process long input in an implicit retrieval-then serieeallthenn-geeerttinn manner, i.e., first identifying key information within the con-
text and then further generating with the "retrieved-contex" (Liu et al.) 2024b] Wu et al,, 2024] Li al., 2024a, Qiu et al., 2025a). However, the critical tokens in the "retrieved-context" might be overwhelmed by excessive irrelevant tokens (Ye et al.) 2024). Thus, the key to achieving better long-context modeling is effectively detecting the critical tokens, diminishing the effect of irrele- vant tokens (context noise), and strengthening the connection between model prediction and critical tokens. Conventional language modeling training strategy, which relies on uniform token-wise su- pervision through cross-entropy loss, is fundamentally inefficient for long-context modeling because cannot distinguish critical tokens from irrelevant tokens in lengthy inputs.
In this work, we first investigate the impact of context noise on long-context modeling. Specifically, we propose a novel critical token detection metric, the Integrated Gradient (IG) score, based on the concept of information flow (Wang et al., 2023). Our approach achieves remarkable accuracy improvement in the critical token detection task compared to the traditional attention- based method. Then, we leverage the IG score to manually reduce the context noise by subtracting the gradient values associated with irrelevant tokens from the token embeddings. We find that simply suppressing context noise the model input allows LCMs to focus more effectively on critical tokens.
Built upon the above analysis, we further propose a simple yet effective Context Denoising Train- ing (CDT) strategy, which performs denoising at the model input, allowing the model to focus more effectively on critical tokens to better establish the connection between critical tokens and genera- tion. To adapt CDT to senueeeeeece training and further improve training efficiency while reducing peak memory consumption, we theoretically derive method that leverages gradients with respect to -rather token embeddings (Appendix C) rather than directly using IG scores mentioned above-as as identi- fiers to detect noisy tokens. Notably, our CDT approach is analogous to the Signal Denoising in the digital signal processing field (Kopsinis & McLaughlin 2009). where noise reduction in the input sequence can enhance the model's 's attention to essential parts within the context. Experiments two essential long-context training scenarios, i.e., context window scaling and long-context alignments, across 4 different types of long-context tasks (real-world tasks, language modeling task, synthetic tasks, and long-ford long-form reasoning tasks) exhibit the superiority of our method. Our CDT can consis- tently surpass the other methods with an average gain of 2 points on 12 weal-world long-context tasks in LongBench- Bai et al. (2024b) and 13 long synthetic tasks in RULER (Hsieh et al., [024]. Ad- ditionally, with CITT, open-source Llama3. 1.1.....8B... I-8t-unstr model can achieve comparable results with GPT4o on real-world tasks (50.92 points v.s. 51.00 points on LongBench-E testing set).
RELATED WORK
2.1 LONG-CONTENT PRSTIIRAINING
Generally, the purposes of long-context post-training can be categorized into two types: context win- dow scaling and long-context alignment. For context window scaling, prior studies have managed to extend the context length of LLMs with limited computational cost compared to pretraining. It can
be further categorized into two approaches: positional extrapolation et al. 2023a Peng al.,
2023 Ding et al.) 2024 Liu et al. 2024a Zhao et al., 2024a; Zhang et al. 2024c Fu et al. 2024b Lt 2024] Wang et al. 2025b Ge et al. 2025 Xiong et al. [2024] Chen et al. 2024a Liu al.
2024ch and model architecture modification Chevalier et al. [202)] Chen et al.] 2023b; Xiao et al.
2024b. Bertsch et al., 2024) Yuan et al. 2025| [u al. (025). Another line of work focuses on im-
proving models that already support long- contect windows, aiming to enhance the model's ability to capture critical information from lengthy contexts (Liu et al., 2024b; An et al., 2024b] Gao et al. 2024c An et al. 2024a) and to address alignment challenges such hallucination (Zhang et al. 2024b Tang et al. 2024a Li et al., (2024b). However, to date, no existing work has simultancously considered both training efficiency and effectiveness under the two aforementioned settings. Only few preliminary studies (Lin et al. 2024 Fang et al. 2024b] Helm et al. 2025 WWng et al., 2025a) have explored token re-weighting as trivial method to achieve limited trade-off.
2

---

Published as a conference paper at ICLR 2026
Supporting Facts
Interference Facts
Low-frequency Words
Fact 1:
Daniel journeyed to the bathroom.
Pact Mary journeyed to the garden. Fact John went back to the bedroom.
"matata", "safari", "asante", "färōur" ég"" "viă", "betta"
Fact 2:
Daniel picked up the apple.
Fact 3:
Daniel went to the garden.
Fact Daniel took the milk.
random injection among
Irreverent Documents
Fact
Fact "vid"
Pact
Pact
Where was the apple before the ganden?
The apple in
eg
Pact
Fact
the bathroom
Long Context
Question
Prediction
What affects the mudel prediction?
Integrated Gradient (IG) score & Fact Retrieval (FR) score
Figure 2: Task format of our preliminary study, which requires models to predict the final an- swer by reasoning through multi-hop Supporting Facts and distinguishing from the Interference Facts. Simultaneously, the model should also resist the influence of Irreverent Documents and Low- Frequency Words. More details are shown in Appendix B
2.2 RETRIEVAL-TIIE--GEEEERE
MECHANISM OF LONG-CONTEXT MODELS
Existing research has demonstrated that LCMs handle long-conted in a "retrieval -eeeevell--ee---eeeee theenneeneration manner, where LCMs firsi retrieve salient information within the context and utilize this information for further prediction (Wu et al.] 2024; Tang et al., 2024bb Zhao et al.) 2024b) Qiu al.) 2025a). However, Liu et al. (2024b) observes the "lost-in-the tet-tthe-iine thiieeedeemle phenomenon of LCMs, which high- lights that LCMs exhibit a positional bias toward locating key information. Furthermore, Ye et al. 2024) and Fang et al. (2024b) discover that excessive irrelevant long-context can overwhelm criti- cal information, thereby impairing the performance of the model. To mitigate the above issue, some works have explored solutions from various perspectives, including model architecture improve- ments (Ye et al. 2024 Xiao et al., 20244).. enhancements in information extraction mechanisms (Li Zhang et al. 2024a), and optimization of training objective (Fang et al., 2024b Bai
al.
2024a
et al.
2024a In this paper, we revisit critical information location from the context denoising as- pect, helping the model establish better connections between detected salient tokens and predictions.
3 PRELIMINARY STUDY
In this section, we analyze the influence context noise, i.e., irrelevant tokens, on long-context modeling. More concretely, we first design critical token detection metrics in $3. and study the impact of context noise restraint on long-context modeling in $3.2. For evaluation, we construct a synthetic long- form-formaee reasoning task controlled proxy to enable precise assessment, due the lack of real-world testing data with explicitly labeled critical token positions. We conduct experiments with the Llama3. 1-8- (Meta 2024 model, which owns 1 128K context window size.
Synthetic Task Format As shown in Figure 2 there are four types of tokens in the context: sup- porting facts, interference facts, low-frequency words, and irrelevant documents. The model's task to predict the correct answer (e.g.. "bathroom") by reasoning over supporting facts. The interfer- ence facts are seemingly related to the answers and are randomly inserted into the context, aiming to distract the models from providing the correct response. We treat both supporting facts and interfer- ence facts critical tokens, as they are both highly correlated with the answer. The key distinction lies in semantic validity: LCMs must discern which tokens are genuinely supportive are misleading- to predict accurately. Besides, models should also prevent critical tokens from be- ing overwhelmed by irrelevant tokens, including excessive irrelevant documents and low- low-frequency words. The total context length of each sample ranges from OK to 64K.
and which
3.1 CRITICAL TOKENS DETECTION
Given the model input X {a. ) which contains tokens and the ground truth Y {,, mm which contains m tokens, we design two metrics to reflect the influence of context noise: 1=1 Retrieval (FR) score and Integrated Gradient (IG) score.
3

---

Published as a conference paper at ICLR 2026
10
0.8
Normalizze FR score
0.8
Wrong Prediction
Correct Prediction
0.71
Ierelevant Tokens
0.63
0.630:67
Wrong Prediction
Correct Prediction
0.6
IG
0.4
0.4
CHilicl Tokens
0.28g. 25
0.20
0.2
0:13
0.09
0.15
0.00
0.04g 03
Irreleant Tokens
0.050
0.0
0.0 Supporting Interference Irrelvvnt Low- now-frequency
Supporting
Irrelevant Low-trequency
(a) Attention distribution reflected by FR score.
(b) Information flow reflected by average IG score.
Figure 3: Comparison between attention distribution and information flow the critical token loca-
tion task. A significant difference in the distributions of critical and irrelevant contexts is revealed.
1.0
Altention score
Critical Tokens
2.10
29:16 16
25.84
Before
Embead ng Gradient
0.8
84.30%
58.79%
Token-level Overlap
KG score
Embedding Gradient
0.6
0.8
X10 bonnst
X10 boost
After
Value 25
0.4
23.30%
22.38%
0.2
0.2
0.13
0.09
0.0
Supporting Interference
Irrelevant Low-frequency
Supporting Interference irelevantLow-feeuueny
Figure Attention distributions before and after manual context denoising. After context denois- ing, attention scores on critical tokens boost x 10 times, and show reduction on irrelevant tokens.
Figure 5: Relationship between attention IG score and L2-normalized embedding gradients on different types of tokens. shows a propor- tional correlation.
Attention Distribution Metrie: FR score Existing works primarily identify critical tokens based
on the attention distribution (W et al., 2024, Gema et al.) 2024 Xiao et al., 2024a). Similarly, design the Fact Retrieval (FR) score for synthetic task based the attention distribution to quantify the model's attention allocated to different types of tokens. At each step of model prediction yj. if the attention score of 2; ranks within the top-k across the entire sequence, being attended by an attention head h in the I-th model layer. Let 8, be the set of tokens attended by an attention head h at the generation step j. and T, refers the context token of type r {sup, inter, irr, low ). e.g., Toup denotes tokens of the supporting facts. The FR score FR of the h-th attention head in the 1-th model layer can be written as:
define x; as
8 nT T,
F
We average FR scores from all heads to reflect the attention distribution of tokens in T,-
Information Flow Metric: IG score To discover the attention interaction among tokens, i.e., information flow (Simonyan et al.) 2013)), we employ the Integrated Gradient (IG) technique (Wang
al. 2023). We define the IG score of h-th head in model's 1-th layer on segment T, below:
IG o
(Y
1
Σ IGn,(i,5], 2,ET. Wy
Σ
dA
T
where Co (X) is the model's prediction loss on Y, and Ar, denotes the attention matrix of the h-th head in the 1-th layer. The resulting IG score is matrix, where each entry IG.,[i represents the estimated bidirectional information flow between token 2; and token Чý- To assess the overall impact of T. to Y. we compute the total contribution of tokens in T, to the final prediction Y, i.e.,
1 and average across all attention heads and layers the final score, i.., IG(r), higher IG
score (G(r) indicates larger contribution from T, Y. Details shown in Appendix B.2
4

---

Published as a conference paper at ICLR 2026
Critical Token Detection
Emphasizing Training
EM Process within CDT
L(B) logPs & (Y (x)
1.(B)
logP
(X)
Information Flow
back-propagetion
back-propagation
critical token
①
Prediction Head
Prediction Head
detection
back propagation
back propagatinnn
Embedding Grad
benefit from emphasizing training
Transformer Block
Transformer Block
XN
Improving through context denoising
(X)
back-propagation
(x)
proppptoaoottooon
Embedding
bedding
Attention Distribution
X
(X) grad)
Context Denoising
Figure 6: Our proposed CDT (context denoising training) method. It consists of two steps: (1) de- tecting critical tokens within the long context, and (2) utilizing the denoised context for further em- phasizing training. Notably, CDT can be understood as Expeciation Maximization (EM) process, where the model detects noise based on information flow and improves the training by diminishing the noise, thereby enhancing the information flow.
Observation For a clear comparison, we normalize the computed FR and IG scores, and plot them in Figure 3 We find that the IG score detects significantly less noise (irrelevant documents and low-frequency tokens) compared to the FR score on critical token detection. Specifically, as shown in Figure 3al attention-based metrics reflect the distribution of tokens that the model focuses on during the generation process. When the model generates correct responses, its attention focuses more supporting facts; when the model generates wrong responses, its attention focuses more interference tokens. Yet, in both cases, the FR score indicates that the model significantly focuses on irrelevant tokens. As for the IG score shown in Figure 3bb regardless of whether the response is correct or not, the IG score for critical tokens is significantly higher than that for irrelevant tokens.
3.2 EFFECT OF MANUAL CONTEXT NOISE RESTRAINT
Considering that directly suppressing context noise attention is very challenging, we aim re- strain the noise from the input perspective. We first identify irrelevant tokens by computing the IG score on each token and treating the token with the IG score lower than a threshold as the noisy token. Then, we manually suppressed their influence by subtracting the corresponding gradients from their input embeddings. This is motivated by the fact that the model has largely converged on these noisy tokens, resulting in their gradients exhibiting low sensitivity. As shown in Figure 4 observe that after manual context denoising, the attention scores on critical tokens increase nearly 10 times, while the attention scores irrelevant contextual tokens exhibit slight decrease. It is worth noting that this operation can be roughly analogized to denoising in the digital signal pro- cessing field (Kopsinis & Mcl. McLaaugllnn 2009), it reduces noise in the input sequence, allowing the model to focus more effectively on the under-fitting critical tokens.
4 CONTEXT DENOISING TRAINING
Based on the above observation, we propose a simple yet effective Context Denoising Train- ing (CDT) strategy. Building upon the conventional language modeling objective, i.c., cross-entropy loss, CDT explicitly suppresses context noise during training to strengthen the model's attention on critical tokens and help establish better connection between critical tokens and the final prediction. involves two key steps: (1) Critical Token Detection and (2) Emphasizing Training.
4.1 CRITICAL TOKEN DETECTION
Intuitively, we can first apply IG score to detect the critical tokens for the subsequent training. However, computing the IG score in long-context scenario is highly GPU memory-intensive,, as requires storing full attention gradients and weights from every model layer across the entire quence. Even with 8x 92BB GPUs (H20), the maximum computable sequence length for the IG score is limited to 12 12K, making it infeasible generalize to longer sequence. Therefore,
5

---

Published as a conference paper at ICLR 2026
designed simple alternative implementation, which approximates the IG score with token embed- ding gradiente] We derive a proportional relationship between the token embedding gradient and the IG score, and visualize the results in Figure 5 A detailed derivation is provided in Appendix [C
As shown in Figure 6 given the input sequence X = { label Y, and the model fo. we first freeze the model parameters, keeping only the gradients of the input token embeddings (x(X), where & C 0. We then obtain the gradient of each token embedding through the computation of the cross-entropy (CE) loss followed by a loss back-propagation. To identify the critical tokens, we employ an identifier I(.) to detect tokens with large gradients, i.e., critical tokens, in the se- quence. Specifically, we define the calculation of the significance of each token as comparing its L2-normalized embedding gradient against the average of the computed gradients of all tokens, which can be written as:
1. FFoe <t
[xx)
V Eolos)
0. Fe
where T(xx;) = means z; is the irrelevant token (noise); otherwise, is critical token.
4.2 EMPHASIZING TRAINING
To suppress the context noise, we leverage the computed gradients to manipulate the irrelevant token embeddings, leaving critical tokens unchanged. The denoised token embedding can be denoised as:
Ex(x))= = (x, - V(x;)V (m.) x Ir x B,
where Ir is the learning rate and / is the parammer controlling the denoising level. Then, we unfreeze the model and use the denoised token embeddings as the model input for further training, which we refer to as Emphasizing Training. The loss function can be formulated as:
CCDT (xpr(X,X)) (X Los (fo (E,(X)),
(4)
Remark Notably, the above process is conducted online during training rather than pre-computed offline. As shown in Figure 6 although this introduces additional computational overhead, CDT bootstraps the model's long-context capabilities in an model first identifies the critical tokens based information flow and improves the training by diminishing the noise, thereby ultimately enhancing the information flow. In $ 66.3) we will demon- strate that, by training with CDT, the model can continuously enhance its performance compared to conventional training objectives during the post-training stage.
(EM) manner: the
EXPERIMENT
5.1 EXPERIMENTAL SETUPS
Evaluation We evaluate models on 4 different types of long-context tasks, including real- world tasks (LongBench-E (Bai et al., (024b), language modeling task (LongPPL (Fang et al., 2024b).
long-form reasoning task (BABILong (Kuratov et al., 2024)), and synthetic tasks (RULER (Hsieh) et al. 2024)). We compare CDT against existing widely-used methods on two types of models: (1) short-context models (SCMs) that require context window scaling; (2) long-context models (LCMs) select Llama-3-8B-Ban
that require long-context alignment. In our main experiments, as the SCM, of which context window size is scaled x8 times (64K). For LCMs, we select Llama- 3. 3.1-8B-Base and Llama-3.1188 1-8B-Instruct models. We provide
model
evaluation and baseline details in Appendix D and show more evaluation results, such as generalizing CDT to more models,c.g.
Qwen-series (Yang al.) 2024 2025)), in Appendix E We evaluate against current strong LCMs, as well as diverse long-context enhancement methods across training and inference paradigms including token-wisg Miiee- reweighting (LongCE (Fang et al., 2024b)). KV-cache prefilling (Lai et al. 2025). SFT (Chen et al. 2024b), and RL-based optimization (Tang et al. 2024a).
We choose token embeddings for 3 reasons: (1) they easily accessible, (2) the embedding gradients are directly associated with tokens, and (3) they require much less GPU memory compared attention gradients.

---

Published as a conference paper at ICLR 2026
Evaluation results on LongBench-E benchmark. To ensure fairness, we place existing works that do not use the same training data with us in the top group. Our method is implemeted under three settings: context-window scaling (CWS), language modeling (LM), and SFT.
Table 1:
Models
Type
S-Doc QA
M-Doc QA
Summ
Few-shol
Code
Avg.
Prol ong-551K 512K Instruct (Gao et al. 2014bb
SET
40.07
41.24
28.27
64.21
63.08
47.37
NExtLong-511 S12K-Instruct G30
SFT
43.47
43.21
29.88
60.87
44.35
44.35
Llama-3.1-80 SEALONG -
20246
DPO
49.45
44.69
30.96
61.54
57.85
48.90
GPT-4o (version: 2024-11-4 1024-11-20) 11-20)
51.43
60.89
14.78
66.37
61.25
51.00
Results on Short-context Model (all SCMs share the same training data, 8x context window scaling.)
Llama-3-81-- Basee (8K)
25.20
21.52
20.18
32.67
27.92
25.50
+ YaRN (Pegg et ul. 2023
24.37
19.86
24.32
29.99
31.67
26.04
CE
CWS
25.29
21.49
20.36
32.69
27.76
34.62
+ LongCE (Fang et al. 2024b
CWS
17.13
9.59
25,00
59.57
61.83
34.62
+ CDT (ours)
CWS
17.03
24.87
26.61
61.89
66.14
39.31
Results on Long-context Base Model (all LCMs share the same training data.)
1lama-3.1- 188--Bae
18.20
13.19
21.13
63.80
69.32
37.13
+ CE
LM
17.10 19.14
10.82
26.38
62.85
37.55
LongCE. (Fang al. 2024h
LM
10.87
28.63
59.63
66.24
36,90
+ CITT (ours)
LM
19.15
13.01
29.23
63.63
69.44
38.89
Results on Long-context Instruct Model (all LCMs ase same source data with different formats.)
1lama-3.1-88 InStruutttuutt
48.58
45,19
30,30
61.73
57.26
48.61
+ Contriever (Izacard et al.
2021
RAG
42.63
45.55
32.48
62.15
41.85
44,93
FlexPrefill
KV-Prefill
27.37
60.97
47.38
X-Attention
KV-Prefill
48.32
45.60
26.93
61.83
56.39
47.81
+ SFT
SFT
49.23
44.86
30.39
61.96
57.14
48.72
+ LOGO (Tang et 2024a
DPO
49.63
45.39
30.44
62.39
57.19
49.01
CDT (ours)
SFT
50.11
46.04
30.34
62.49
65.64
50.92
Training and Datasets For context window scaling training SCM and post-training LCM- Base, we apply PG-19 (Rae et al.] 2019) as the training data. For each training sample, we organize into 64K tokens and collect 10,000 training samples. For long-context alignment L.C.-Instruuc we utilize data sampled from LongMiT (Chen et al. 2024b) and LongAlpaca (Chen et al. (023c), covering 8,000 samples with context lengths ranging from 16K 128K. Based on the analysis experiment (Section 6.2), we set / = in Equation for Llama-3. and Llama-3 models in the main experiments. More dataset processing and implementation details are shown in Appendix
5.2 RESULTS
Real-world Long-context Understanding Tasks LongBench-E is comprehensive benchmark suite encompassing 12 real-world datasets and various context lengths spread across categories. As shown in table [ we observe that: (1) CDT achieves the best performance among all the sub- tasks. For SCMs, with the same training data, CDT achieved the best performance, outperforming a competitive counterpart (LongCE) by nearly 4.7 points on average. (2) For LCM-Base models, we find when post-training on the base model with language modeling training objective, CDT is the only method that ensures no significant performance drop across all subtasks, and it even achieves some improvements. In contrast, using standard CE or LongCE objective leads to significant per- formance drops on some sub-tasks. For example, LongCE results in a nearly 4-point drop compared to the backbone model on the Few-shot subtask. (3) As for the I.CM-Inttruct models (the bottom group), we find that, due to remarkable performance, existing post-truining methods do not bring significant improvements. For instance, Llama-3, --B- SB-SSALONG (48.90) achieves only around slight 0.3-point average improvement compared to Llama-3, -...-....---BBB -nstttuttttt (49.61). However, our CDT achieves average improvement of more than 2 points compared to that of the backbone model across all tasks. We provide more analysis of results Appendix
Long Synthetic Task and Language Modeling For the long synthetic task, we evaluate the model's performance under 32K, 64K, and 128K context lengths. We choose 13 sub-tasks from the RULER benchmark and report the average results. For the language modeling task, calculate LongPPL (Fang et al.) 2024b) on the GovReport dataset (Huang et al.) 202T). Notably, LongPPL potentially reflect the model's ability to locate salient tokens in the long context. More imple-

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results on long synthetic tasks (RULER), language modeling, and long-form reasoning (BABILong). For RULER, we report the average scores across 13 sampled sub-tasks. To calculate LongPPL, we apply the Llama3-8B- BB-Base model as the evaluator. For BABILong, we report the model reasoning capability from short context (4K) to long context (128K).
Language Modeling
RULER
BABILong
Models
32K
64K
128K
LongPPL
4K
8K
16K
32K
64K
128K
Avg.
ProLong-5 1angg-512K-IInttue
91.68
87.53
80.03
2.97
44.00
45.40
39.20
35.00
35.00
29.80
36.88
NExtLong SeatLoog--5128 onn-5122-Instrruc
90.27
84.62
81.74
3,24
39.60
36.20
35.60
32.00
22,00
38.75
Llama-3. Baama-3.1-0 1------SALLON
91.32
85.97
77.33
3.09
50.20
50.80
42.00
40.80
39.00
31,00
40.72
LJama-3-SB-Bas. -B-Baase
> 100
33.40
26.60
4.80
0.00
0.20
13,00
+ YaRN
39.58
31.46
3.55
35.20
29. 80
24.40
20.20
17.60
25.44
+ CB
36.01
13.82
36.60
26.60
21.60
29.56
LLonCCE
84.02
71.50
3.55
36,00
34,80
34,60
32,60
29.40
33,48
+ CDTT (ours)
84.76
73.40
3.04
38,40
34.60
34,80
31,40
29.60
33.76
Llama-3.1-8B-Bir
89.99
81.96
70.60
3.22
35.00
33.20
27.80
28.00
25.
24.40
28.93
+ CE
86.59
80.87
70.44
3,28
39.20
31.60
31.40
26.60
26.80
19.40
29.17
LLongCE
87.65
81.79
70.79
3.24
37.80
33.40
33.60
32,60
27.60
24.60
31.60
+ CDT (ours)
90.36
82,23
74.12
2,10
38.80
36.60
33.20
29.40
28.20
28,20
32.40
1aama--.1-8B-Inscee
92.49
85,98
76.71
4.05
46.60
49.60
42,40
38.80
37.00
29.60
40.67
+
92.49
86.22
77.33
3,31
47.00
49.40 .4
43.60
41.20
37.40
30.40
41.50
L.OG
92.54
86.93
77.68
4.11
48.20
50.00
42.60
42,20
37.40
31,60
42,00
+ CDT (ours)
93.08
88.01
78.72
2.36
51.40
51,20
41.60
44.00
38.60
33.00
43,30
mentation and calculation details are illustrated in Appendix D.2 We show the evaluation results
in Table 2 where our CDT method achieves the best model performance on the RULER benchmark from 32K to 128K settings. Besides, in the language model task, CDT exhibits the lowest LongPPL, indicating the great potential of CDT to locate salient tokens.
Short-context Long-form Reasoning Tasks We evaluate the model's long-form reasoning ca- pabilities, as well as its short-context capability, on BABILong, synthetic task that requires models to reason through multiple supporting facts hidden in contexts of varying lengths (from 4K to 64K). As shown in Table our CDT achieves the highest overall score in each group. Besides, we observe that our CDT does not compromise the model's performance on short-context tasks. For instance, in the 4K and 8K lengths, CDT achieves either the best comparable results compared to other methods and backbone models.
ABLATION STUDY
In this section, we compare the accuracy of salient token detection of CDT with other detection methods in $6.1 Then, we show the impact of context denoising on the training process $6.2 Finally, we elaborate on the training budget of our CDT method in $6.3 Notably, to help better understand the effectiveness of our CDT method, we also analyze the attention map patterns to reveal how CDT influences the model's attention distribution in Appendix F
6.1 COMPARISON OF CRITICAL TOKEN DETECTION
We compare three different detection methods, including LongPPI., attention-basee detection, and our CDT, on our synthetic task (Figure 2) For attention-based and our CDT methods, we treat the tokens with the top-30 highest attention scores and L2 normalized gradient of embedding as the detected tokens. As shown in Figure 7 we can observe that the attention-basee method can detect a high proportion of supporting tokens and interference tokens, but it also detects a large number of irrelevant tokens. On the other hand, while LongPPL can effectively suppress the detection of irrelevant tokens, struggles to locate supporting tokens. Our CDT method not only identifies the largest number of critical tokens but also effectively suppresses the detection of irrelevant tokens.
8

---

Published as a conference paper at ICLR 2026
1.76
28.20
LongPPL
1e-5
Detection Ratio (%)
18.90
28.20
Attention (ours)
1.75
20-5
Foci more citicall rolonn
Emphasiting Training
10
Value > 10%
1.74
48-5
8e-5
a
6:17
5.01
1.72
Context Denoicing
4.29
4.33
Attention
2:96
1.71
1.70
0.14
Supporting
Irrelevant
Interference
Before CDT
CDT: Step 1
CDT: Step 2
Figure 7: Comparison of critical token detection capability among different methods on our syn- thetic task. CDT achieves best performance.
Figure Impact of context denoising and com- parison of the effect of learning rate on attention scores assigned to critical tokens CDT.
51
SFT
50.92
Information Flow
Performance
CDT
50.38
48
50
49.51
Continual Improvement
Information flow continuously
improves during the training step
19.13
09:04
49.01
49
18.64
48.6)
48,78
Attention Distribution
SFT
DPO
CDT
& 0.
Training
DPO
attention distribution on critical tokens
SFT
20.2
0
50
100
Training Steps
150
200
250
10p2200
stsp133
Vie:200
skxp220
Figure 9: The performance improvement and training duration for every interval of 50 steps. With only a modest cost in training time, CDT significantly boosts the performance LCM.
Figure 10: Illustration of EM process of our CDT method, where both the information flow and attention distribution progressively improve within the training steps.
6.2 IMPACT OF CONTEXT DENOISING STRENGTH
We visualize the changes in attention scores allocated to critical tokens during the CDT training process under different learning rates and the same / = settings. As shown in Figure g we ob- serve that the attention scores critical tokens have already increased significantly after the context denoising step. Furthermore, after the Emphasizing Training stage, there is an additional improve- ment. Additionally, observe that larger learning results in pronounced improvements, further enhancing context denoising. However, a saturation point exists (e.g., at 8e-5), beyond which the benefits plateau. Based this observation, in our main experiments, where ir x B = 5e 5. We also recommend viewing the attention map provided in Appendix(F which shows that CDT enables the model to focus more on key information within long context, without substantially changing the original attention distribution.
adopt learning
Ir of 1e-5 and set /
6.3 TRAINING BUDGETS AND EM PROCESS
Compared to conventional long-context training, which performs one forward and one backward pass to update all parameters, CDT introduces an additional noise detection step. Critically, long- context training, backward passes are typically 2-3x slower than forward passes due to activation recomputation (Shoeybi et al. 2019). Yet CDT adds merely one lightweight backward (where the vast majority of model parameters are frozen) and one extra forward, resulting in minimal wall-clock overhead relative to standard training. We compare CDT with SET (single Forward-Backward and DPO (one batch contains pairwise samples) methods. As shown in Figure g observe that although CDT brings additional cost, i.c., approximately 0.5 hours x A00 GPUs for every 50 steps compared with SFT, it consistently and largely improves the model performance within the 250

---

Published as a conference paper at ICLR 2026
training steps. With the same training steps, DPO only yields marginal improvements, while SFT even demonstrates a decline in performance. We provide the total training duration in Appendix [D] Such great improvement can be largely attributed to the EM process shown in Figure I0. Notably, our approach exhibits a convergence boundary after approximately 250 steps.
CONCLUSION
Prior studies suggest that long-context models typically follow a retrieval-then- thee--enereationn paradigm, where the "retrieval context" may be overwhelmed by excessive irrelevant tokens. To address this issue, we present a fine-grained analysis of contextual noise in long-context inputs. We introduce a novel metric, the IG score, to effectively identify critical tokens, and observe that reduc- ing contextual noise enables models to focus more precisely on critical tokens. Building on these insights, we propose Context Denoising Training (CDT), training strategy designed to both en- hance the model's attention to critical tokens and strengthen the association between salient tokens and the model prediction. Experiments across task types (including both short and long context length) and different models demonstrate the superiority of our method. With CDT, an open-source 8B model can even achieve comparable performance with GPT-4o on real-world long-conten tasks.
ETHICS STATEMENT
We confirm that this work adheres to ethical research practices. All data and LLMs used are publicly available (including API format) and properly cited. No human subjects were involved. The Use of LLM statement is illustrated in Appendix H
ACKNOWLEDGEMENT
We want to thank all the anonymous reviewers for their valuable comments. This work was sup- ported by the National Science Foundation of China (NSFC No. 62576232) and the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001).
REFERENCES
Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of llms fall short? arXiv preprint arXiv:2410. 18745, 2024a.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jia-GGang Lou. Make your Ilm fully utilize the context. arXiv preprint urXiv:2404. 16811, 2024b.
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang. Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1376-1395 Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10. 18253//11/224.
findings-emnlp.P 74. URL
httpss:/aacaaathooogoo org 2224 www.finnnngge igg emn 1p.
Yushi Bai, Xin Ly, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/ v1/2024.acl-long. 172. URL htllhttss// http://ccaaanttooooo wrt./org///e 2024.022422 a00......../022...11-oooggg
Yushi Bai, Shangqing Tu. Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv. Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412. 52044 2024c.
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unlimiformer: Long-range transformers with unlimited length input. Advances in Neural Information Processing Systems, 36, 2024.
10

---

Published as a conference paper at ICLR 2026
Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, and Min Yang. Long context is not long at all: A prospector of long-dependency data for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pp. 8222-8234, 2024a
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023a.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12330 12307, 2023b.
Yukang Chen, Shaozuo Yu, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Long alpaca: Long-context instruction-followie models. htttps://githu com/ 2023c.
dvlab-researcch/Lnng.....
Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Wanxiang Che, Hang Yan, Kai Chen, and Dahua Lin. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. arXiv preprint arXiv:2409.01893, 2024b.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3829-3846, 2023.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang. Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending 1lm context window beyond 2 million tokens. arXiv preprint arXiv: 2024.
Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun, et al. Large language models for code analysis: Do {LLMs} really do their job? In 33rd USENIX Security Symposium (USENIX Security 24), pp- 829- 846, 2024a.
Lizhe Fang, Yifei Wang. Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao. Bolin Ding, and Yisen Wang. What wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771 2024b.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models 128k context. In Forty-first International Con- ference on Machine Learning, 2024a.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Con- ference on Machine Learning, 2024b.
Chaochen Gao, Xing Wu, Qi Fu, and Songlin Hu. Quest: Query-centric data synthesis approach for long-context scaling of large language model. arXiv preprint arXiv:2405. 1944,, 2024a.
Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. Nextlong: Toward effective long-context training without long documents. arXiv preprint arXiv:2501.12766 2025.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint arXiv:2410.026600, 2024b.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-con context language models (effectively). arXiv preprint arXiv:2410.026660 2024c.
Hao Ge, Junda Feng, Qi Huang. Fangcheng Fu, Xiaonan Nie, Lei Zuo, Haibin Lin, Bin Cui, and Xin Liu. Bytescale: Efficient scaling of Ilm training with a 2048k context length on more than 12,000 gpus. arXiv preprint 1rXiv:2502.21231, 2025.

---

Published as a conference paper at ICLR 2026
Aryo Pradipta Gema, Chen Jin, Ahmed Abdulaal, Tom Diethe, Philip Teare, Beatrice Alex. Pasquale Minervini, and Amrutha Saseendran. Decore: decoding by contrasting retrieval heads to mitigate hallucinations. arXiv preprint ar arXiv:2410.18800 2024.
Falko Helm, Nico Daheim, and Iryna Gurevych. Token weighting for long-range language model- ing. arXiv preprins arXiv:2503.092000 2025.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint 4rXiv:2404.06654, 2024.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. 11143, 2024.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.naacl main. 12. URL http:///aaaaaanoaaaa//// org/2021 www.naanclaanan mall-mmain... 112
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021. httpt:ttt///arttt wrg.abs 2112. 09118
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Yannis Kopsinis and Stephen McLaughlin. Development of emd-based denoising methods inspired by wavelet thresholding. IEEE Transactions on signal Processing, 57(4): 1351-1362, 2009.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in prasoning-ii---aayaee hayssakk.. arXiv preprint arXiv:2406.1014 2024.
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Con- ference on Learning Representations, 2025.
Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watan- abe, and Yixuan Su. Alr2: A retrreee-then--reason framework for long-context question answering. arXiv preprint arXiv:2410.03227, 2024a.
Siheng Li, Cheng Yang. Zesen Cheng, Lemao Liu, Mo Yu, Yujuu Yang, and Wai Lam. Large language models can self-improve in long-context reasoning. arXiv preprint 1.X1v22411.08147, 2024b.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, et al. Rho-l: Not all tokens are what you need. arXiv preprint arXiv:2404.00 07965, 2024.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprini arXiv:2310.01889, 2023.
Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang. Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, et al. E. 2-1lm: Efficient and extreme length extension of large language models. arXiv preprint arXiv:240J. 1.6951,,, 2024a.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 11:157-173, 2024b.
12

---

Published as a conference paper at ICLR 2026
Xiaoran Liu, Kai Lv, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. Longwan- juan: Towards systematic measurement for long text quality. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5709-5725, 2024c.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189 2025.
Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush. controlled study on long context extension and generalization in llms. arXiv preprint urXiv:2409. /2181, 2024.
Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460, 2025.
AI Meta. Introducing llama 3.1: Our most capable models to date. Meta Al Blog, 12, 2024.
AI Meta. The Ilama 4 herd: The beginning of new era of natively multimodal ai innovation, april 2025, 2025.
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang. Le Han, Leyang Wang. Lian- fei Yu. Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengeheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu. Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang. Yunji Li, Yumpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying. Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https: / org/ alos//500000 .08313
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2023.
Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. Elic- iting in-context retrieval and reasoning for long-context large language models. arXiv preprint arXiv:2501.08245 2025a.
Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang. Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink
arXiv preprint arXiv:2505.06777 06708, 2025b.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL
05507
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEFE, 2020.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv:1909.0805. 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.660 2013.
13

---

Published as a conference paper at ICLR 2026
Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. Logo- long context align- ment via efficient preference optimization. arXiv preprint arXiv:2410.18533 2024a.
Zecheng Tang, Keyan Zhou, Juntao Li, Baibei Ji, Jianye Hou, and Min Zhang. L-citeeval: Do long-context models truly leverage context for responding? arXiv preprint arXiv:2410.02115, 2024b.
Zecheng Tang, Haitian Wang, Quantong Qiu, Baibei Ji, Ruoxi Sun, Keyan Zhou, Juntao Li, and Min Zhang. Loom-scope: a comprehensive and efficient long-context model evaluation framework. arXiv preprint (rXXx:2507.04723, 2025.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint (rxxix:4433055344 2024.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 9840-9855, 2023.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for I1m reasoning. arXiv preprint (tti::25000019330 2025a.
Zhenghua Wang, Yiran Ding, Changze Lv, Zhibo Xu, Tianlong Li, Tianyuan Shi, Xiaoqing Zheng, and Xuanjing Huang. Layer-specific scaling of positional encodings for superior long-context modeling, arXiv preprint arXiv:2503.04335 2025b.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanisti- cally explains long-context factuality, arXiv preprint arXiv:2404.15577 2024.
Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Jun- jie Ye, Jiazheng Zhang, Wenxiang Chen, al. Agentgym-rl: Training Ilm agents for long-horizon decision making through multi-turn reinforcement learning. arXiv preprint arXiv:2509.088 08755, 2025.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410. 10819, 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2024b.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 4643 4663, 2024.
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprint arXiv:2503. /6428, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15115, 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0923 009388. 2025.
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differrntial transformer. arXiv preprint arX.w:2410.05228, 2024.
14

---

Published as a conference paper at ICLR 2026
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context models effectively and thoroughly. In The Thirteenth International Conference Learning Representations, 2025.
Tan Yu, Anbang Xu, and Rama Akkiraju. In defense of rag in the of long-context language models. arXiv preprint arXiv:2409.0I 01666, 2024.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint urXiv:2502.1008/ 2025.
Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. Longcite: Enabling Ilms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897, 2024a.
Jiajie Zhang. Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longreward: Improving long-context large language models with ai feedback. arXiv preprint arXiv:2410.21252, 2024b.
Yikai Zhang, Junlong Li, and Pengfei Liu. Extending Ilms' context window with 100 samples. arXiv preprint arXiv:2401.0700% 2024c.
Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, et al. Longskywork: A training recipe for efficiently extending context length in large language models. arXiv preprint arXiv:2406.00605, 2024a.
Xinyu Zhao, Fangcong Yin, and Greg Durrett. Understanding synthetic context extension via trieval heads. arXiv preprint arXiv:2410.23316, 2024b.
15

---

Published as a conference paper at ICLR 2026
A ILLUSTRATION OF TRAINING EFFICIENCY OF CURRENT METHODS
Training efficiency comparisons across current long-context methods are inherently challenging: performance gains typically exhibit diminishing returns with increased token budgets under long- context training settings, and reported results often stem from divergent training setups including data composition, optimization hyperparameters. and hardware configurations. These factors render direct ggain-per-tokee comparisons unreliable when conditions unmatched. To fairly compare training efficiency across methods - despite differing dyper-parameter and convergence behaviors adopt controlled proxy: average task gain per 1B tokens, measured under identical data, optimizer, batch size, learning rate, and hardware (8 x A100 GPUs). Specifically, we compare ProLong (Gao al., 2024b) one long-context SFT method, and LongCE (Fang et al. 2024b)
one token-level re- weighting training method, on the Llama3-8B --mma--BB-aa -B--ase model. As shown in Table 3 we evaluate model performance on LongBench-E (12 real-world tasks) per 50 training steps (0.41B tokens per 50 steps), and find that LongCE achieves a 3.7-point gain per 1B tokens versus ProLong's 1.8-point gain per IB tokens.
Performance comparison between ProLong (SFT) and LongCE across training steps, where each step contains the same training setting.
Table 3:
Method
Step 0
Step 50
Step 100
Step 150
Step 200
Prol Long (SFT)
25.50
27.32 28.30
28.15
28.44
29.13
LongCE (same data)
25.50
29.72
31.01
32.91
B PRELIMINARY STUDY DETAILS
B. PRELIMINARY TASK CONSTRUCTION
Task Selection We select 3-hop and 4-hop tasks based qa3 tasks in the BABILong Benchmark to build our datasets, as these tasks generally pose significant challenges for LLMs. However, it is worth noting that the original BABILong qa4 samples do not truly require 4-hop reasoning to pro- duce correct outputs. For example, sample from this subset with Ok context is shown in Figure II In this case, the task only requires attention to a single fact, "The bedroom is west of the bathroom to answer the question, while the first sentence serves as an interference fact. Even in terms of keywords, the model only needs to focus on three keywords: "bathroom", "west", and "bedroom" from the second sentence. Thus, we design our 4-hop dataset based on the BABILong qa3 source data, with one sample shown Figure 12 By carefully arranging the order of facts and reducing the conditions of questions in the long context, we ensure that the model is required to search for all four supporting facts in sequence to produce the correct output.
Table 4: Variable settings, where R. denotes random.
Hops
Samples
Permute
Lengths
2
100
5
8K
3/4
R.
R.
Ok 64k
Controlled Evaluation Data Synthesis We use the 4-hop task with non-zer zon-zero context as an exam- ple here. As shown in Table all variables used for building data include the facts sample, the facts permutation, and the context length. Firstly, we select source samples from the BABILong official file "qa3_ stsssttrcessuppoting--aaae
as our base data. Then, we modify the original BABILong qa3 supporting facts following the pattern shown in Figure 13 Afterward, we add interference to these four original facts while maintaining the relative order of the supporting facts. The process begins by selecting noise context of the appropriate length and inserting the facts into it. Specifically, divide the noise context into 10 eeual- chunks, leaving 10 candidate positions for the insertion of the 4 supporting facts (excluding the tail). Next, we randomly select five permutations from the
16

---

Published as a conference paper at ICLR 2026
Table 5: Performance statisties of using different numbers of attention heads on our preliminary synthetic task. Notably, we find that selecting the top-30 heads yields results that are nearly identical those obtained when using all attention heads.
Supporting
Interference
Irrelevant
Low-frequeney
Head Number
Correct
Wrong
Correct
Wrong
Correct
Wrong
Correct
Wrong
Top-30
0.21
0.11
0.07
0.17
0.72
0.72
0.00
0.00
All
0.20
0.13
0.09
0.15
0.71
0.72
0.00
0.00
full set of Cío candidate position permutations. After injecting noise, we randomly insert interfer- ence facts, i.c., facts that are similar to the supporting facts but irrelevant, among all sentences. We ensure that at least one interference fact is placed after the last supporting fact to test the model's robustness. To ensure the correctness of the samples, we make sure that the objects appearing in the interference facts do not overlap with those in the supporting facts. Additionally, we ensure that the number interference facts is between one and two times the number of supporting facts to avoid making the samples either too easy or too difficult. Finally, for all samples with the same context length, we use the same noise context to maintain consistency. In the end, we randomly insert a few emojis into the constructed context to test the sensitivity of the model to low- www.freuenccy tokens. For the 3-hop task, we directly use the original qa3 task format from BABILong as the base, and the subsequent processing follows similar approach the described above for the 4-hop task.
One BABILong qa4 sample with Ok context
Input:
The bedroom is west the office.
The bathroom is west of the bedroom.
Question:
What is west of the office?
Supporting Facts:
The bedroom is west of the office.
Ground truth: bedroom
Figure 11: A BABILong qu4 sample with context
B.2 DESIGN OF IG SCORE
Prior work (Wu et al. 2024) has shown that not all attention heads behave uniformly, i.e., some are specialized for retrieval like behaviors, while others are not. However, it is important note that these findings are primarily derived from studies focused on copy-oriented tasks, such as NIAH. In contrast, our task involves reasoning and inference, which fundamentally differs from the objectives of retrieval heads. As result, the mechanisms for attending to relevant context in our setting cannot be directly aligned with those used in retrieval-focused tasks. To further analyze the appropriate number of attention heads to select, we conduct experiments where we select the top-k attention heads (k = 30) that retrieve the most relevant information based on the attention scores (Table 55) We find that the performance using only subset of attention heads the results obtained by averaging over all attention heads. Therefore, for simplicity and ease of deployment, adopt the latter approach, i.e., averaging IG scores across all attention heads.
highly consistent with
17

---

Published as a conference paper at ICLR 2026
One of our 4-hop samples with Ok context
Input:
Mary journeyed to the office.
Mike went to the office. Mary got the apple.
Daniel picked up the football. Daniel went back to the bedroom.
Mary journeyed to the bathroom.
Mary dropped the apple. Jonh went to the bathroom.
Question:
Where was the apple's location prior to the place where the apple was discarded, left or dropped?
Supporting Facts:
Mary journeyed to the office.
Mary got the apple.
Mary journeyed to the bathroom.
Mary dropped the apple.
Ground truth:
office
Figure 12: One of our 4-hop samples with Ok context
The pattern of our 4-hop sample
Supporting fact1: [x] {m} the {yl)
Supporting fact2: (x] [p} the {o)
Supporting fact3: (x) (m) the {y2)
Supporting fact4: (x) [d] the {o}
Question:
Where was the (o)'s location prior to the place where the {o} was discarded, left or dropped?
Ground truth:
[yl]
Explanation:
(x) : character name, selected from (Mary, Daniel, Mike, ...]
(m) predicate indicating movement, selected from (went to, journeyed to, travelled to, ...)
[yl), (y2) two
:too different locations, selected from (office, bedroom, bathroom, ...)
{p] predicate indicating picking up, selected from (picked up, took, grabbed, ....
(d] predicae indicating dropping, selected from (dropped, put down, discarded, ...]
(o) an object name, selected from (apple, football, milk ...)
Figure 13: The pattern of our 4-hop sample
C DERIVATION OF RELATION BETWEEN INFORMATION FLOW AND EMBEDDING GRADIENTS
In transformer-based models, the Information Flow in attention is essentially the product of the attention distribution and its corresponding gradient. Therefore, we can transform the derivation into constructing the gradient relationship between the attention score distribution (AA) and the embedding (E(X)). This can be established via the chain rule and implemented through the specific computation steps of the attention mechanism. Notably, in the following derivation, for simplicity, we omit the activation layers in the model. Additionally, considering that transformer- based models are composed of multiple identical network blocks stacked together, one easily
18

---

Published as a conference paper at ICLR 2026
extend the conclusions from single layer to multiple layers. Therefore, focus proving the case with one embedding layer and one attention module.
Given the basic definition of the attention mechanism, have:
QK*
Q B(
A = softmax
vd
K = E(X)WK.
0 A-V,
V E(X)WV,
where We. WK. WV E Idddd are the model parameters, the attention output, E(X) E Raxd the input embedding matrix, and d are sequence length and model dimension, respectively.
Let the loss function be L. By the chain rule, the gradient of the loss with respect to E(X) is:
aL 0L 80 aL 0A eE(X) d0 0E(X) OA O âL Qv
(5)
0V ΠE(X)
Since have ]E(X]
@V W and
A, the gradient relationship between A and ((X) is:
ØT. 0
OL aA DA 0E(X)
O
(6)
To eliminate the influence of the Softmax(.) function, we can further decompose equation into:
S QK Vd
aL
0L
0A as 0E(X)
2
θE(X)
DA
as
04 where 85
is the Jacobian of Softmaxx() function, with elements Aij (dis Aik
Q.K
For each element S,j
Vd E S, the gradient with respect to EXX) can be written as:
a (O)X) Wx))"
as.
Va
0E(X)
AE(X)
(8)
(Wor Kj du + W Qi 0jx)
Vd
Based on equation 7 and equation
we can summary that:
aL
àL
OE(X),
dAi
Sensitivily L to
Ag Ang
Berivation from Softmox
aS.
OE(X)
Linear Transformation
Based on equation we can derive that when Aij increases, indicating higher attention between OL also increases. This results in larger derivatives
token i and token the sensitivity of L to A BAg
18p is the Kronecker delta function. If i equals to kt diz else dsc 0. We can also rewrite this equation into A. Ay).
19

---

Published as a conference paper at ICLR 2026
Table 6: Configuration of context window scal- ing training setting.
Table Configuration of language modeling training setting.
Context Window Sculing Training Setting
Language Modeling Post-training Setting
Backbone
Llama-3-8B-Bal
Backbone
Llama-3. 1..------bss
Training Objective
Language modeling
Training Objective
Language modeling
RoPE base
20,000,000
RoPE base
500,000
Context window size
8K 64K
Context window size
128K
Data seq-length
64.000
Data seq-length
64.000
Deepspeed
Zero2
Deepspeed
Zero2
Global batch size
64
Epoch
2
Epoch
Global batch size
32
Training Steps
160
Training Steps
320
Ring-attention size
Ring-attention size
4
Learning-rate
lc-5
Learning-rate
lc-5
LR-scheduler
cosine_ with min
LR-scheduler
cosine with min
Optimizer
Adam (8) 0.9, .2 0.95)
Optimizer
Adam (8i 0.9, 82 0.95)
GPUs
A100 (80GB) x 8
GPUs
A100 (80GB) 8
Training time
s/8h / epoch
Training time
A=8.5h /epoch
Training data Total consumed tokens
PG19 (Rac et al. 2019
Training data
PG19 (Rae et al. 2019
0.65B
Total consumed tokens
0.65B
Table 8: Configuration of long-context SFT training setting.
Table 9; Testing configuration of RULER
Long-context Alignment Training Setting
Evaluation Configuration of RULER
Backbone
Llama-3.1 ------8----snttecc
Training Objective
Supervised fine-tuning
Question Answering
qa_ qa__ niah pin_single_____ 1,
RoPF: hase
500,000
Single NIAH
Context window size
128K
niah single 2. niah_single niah multikey__ niah multikey 2. niah multikey_ 3 niah multiquery niah multivalue
Data seq-length
4,000-128,000
Deepspeed
Zero2
Multi-keys NIAH
Global butch size
Epach
2
Training Steps
250
Multi-values NIAH
Ring- attentionno size
Multi-queries NIAH
Learning-rare
lc-5
Others
common words extraction (CWE),
LR-scheduler
cosine cosine_with with miinh_min_rrr
frequent words extraction (FWB),
Optimizer
Adam (3: 0.9, 8, 0.95)
variable tracking (VT)
GPUs
A100 (80GB)
Training time
r=6.5h epoch
Length
32K, 64K
LongMIT (Chen al. 2024h
Num samples/task
50
Training data
LongAlpaca (Chen at 0,53B
Total consumed tokens
on the embeddings. Additionally, if A.j becomes excessively large, approaching 1, the value of (1;)) might tend toward O. However, this is often not an issue in long-context scenarios, as the attention scores are unlikely to approach values near 0.5 due to the long context. Even if they exceed 0.5 (possibly for some special tokens), the increase in the first term DA:; effect.
UL helps mitigate this
D IMPLEMENTATION DETAILS
D.1 TRAINING DETAILS
For all experiments, we utilize the open-source training framework OpenRL (uu et al. 2024).
flas-alah-atttntioon (Luu et al., 2022) and DeepSpeed (Rajbhandari et al. 2020). For LongCE training (Fang et al. 2024b), we set the sliding context window size as 8192 and employ the recom-
mended hyper-parameters in the official code
httt:////attbb...ooooo////////ae qit
20

---

Published as a conference paper at ICLR 2026
Table 10: Testing configuration of BABILong.
Metric
QAI
QA2
QA3
QA7
QA8
Num
100
100
100
100
100
Supporting Fact
2
3
1~10
1~88
Interference Fact
1~9
1~66
1~317
1~42
1~42
Table 11: Evaluation results on HELMET Yen et al.
2025
Model
Recall
RAG
ICL
Re-rank
QA
Summ.
Cite
Avg.
Claude-3.5-Sone 355-onnet
94.7
38.1
61.0
7.2
12.6
36.6
18.7
38.4
Mistral-Nemo-122 12B
14.6
40.0
84.0
0.0
22.5
18.5
0.5
25.7
FroLLongS12K-IIsttuu
98.8
63.2
86.5
22.5
43.9
29.2
1.4
49.4
Meta 3aata-Llam--31--3
95.2
59.5
83.9
14.0
43.2
2.9
46.5
+ CDT
97.2
61.8
86.6
18.5
46.7
27.9
9.4
49.7
Context Window Scaling To scale the context window size of the Llama-3-8B-ban model from 8K 64K (8x), we adjust the RoPE base from 500,000 to 20,000,000 and directly train the model. We provide training configurations in Table 6
Data Post- pest-processing Details For the context window scaling experiments, we employ the PG- 19 (Rae et al. 2019) dataset. For long-context SFT and CDT experiments, we construct our data from publicly available long-contens QA datasets, including LongMiT (Chen et al.) (024b) and LongAlpaca (Chen et al.) 2023c). The LongMiT dataset primarily consists multi-hop QA tasks that require reasoning over 2 to 6 evidence passages. To adapt it for our setting, we apply two pre peee-rocessing steps: (i) Length distribution control - we constrain the sampled instances to fall within 16K-128K tokens. This range balances the need for sufficiently long contexts with train- ing efficiency, given our compute resources (8 A800 GPUs). Excessively long sequences were avoided they considerably slow down training. (ii) Evidence balancing across different numbers of supporting passages to obtain a more balanced distribution for multi-hop reasoning. To complement this, we include data from LongAlpaca, which predominantly features single-evidence QA with lengths around 16K tokens (under our model's 's tokenizer). This addition enriches the training distribution by covering shorter single-evidence scenarios, which are underrep- resented in LongMiT. In total, our final training set comprises 7,000 samples from LongMiT and 1,000 samples from Alngaa which are shuffled together before training.
uniformly sample
Language Modeling Post-training and Long-context SFT The language modeling post-training and long-context SFT are directly applied to the Llama3.1-8B-base 1-8B-base and Llama3.1-888 Ins-rtsruut,, re- spectively, which already have 128K context window size. We provide the training configurations in Table 7 and Table respectively.
D.2 EVALUATION DETAILS
We conduct long-context evaluation mainly based the long-context evaluation framework LOCM-Eva Tang et al. 2025).
HELMET HELMET Yen al. 2025) is a comprehensive long-context evaluation benchmark containing different subtasks, including recall, RAG, in-context learning (ICL), re-rank, QA, sum- marization, and citation. The context length of test samples ranges from 0 to 128K tokens. For inputs exceeding I 128K, we truncate from the end to fit within the model's maximum context win- dow. We show the experimental results on HELMET in Table II where our CDT model achieves the best performance.
https://githubs I.Mp http://///ttiiiboooo///oooaaoa/a
21

---

Published as a conference paper at ICLR 2026
Table 12: Evaluation results of two more LLMs on real-world long-context tasks and long-form reasoning tasks.
LongBench-E
BA BBBoonn
Models
Type
S-Doc QA
M-Doc QA
Summ
Few-shot
Code
Avg.
Avg.
Qwen2.5-7B B-Sntutttt
44.54
46.29
28.15
56.03
16.52
38.30
43.32
CDT
SFT
44.93
47.29
28.65
57.33
19.18
39.48
47.56
Qwen3-8B
44.12
48.10
29.30
44.12
29.18
38.85
48.06
CDT
SFT
45.33
49.13
31.89
46.24
32.98
41.11
52.88
Mistral- Lo.3ra--
44.89
40.76
20.52
67.11
47.04
44,06
22.36
CDT
SFT
45.01
41.79
26.08
67.75
57.27
47.58
53.84
LongBench-E LongBench-E variant of LongBench (Bai al. 2024b) designed specifically for long-context real-world tasks. We chose LongBench-E because shares the same test dataset distribution as LongBench while covering wider range of context lengths. For the Llama3-8B- model, we truncate the input to 8K tokens, whereas for other models, we truncate the input to 32K tokens.
Language Modeling For the language modeling task, calculate both LongPPL and PPL met- ries on the GovReport dataset (Huang et al.] (002), which consists ment reports. We sample 50 documents from GovReport, each with context length of up 32K tokens.
of long sequences from govern-
RULER RULER (Hsich al. 2024) is comprehensive synthetic dataset that includes 6 different testing categories to evaluate model's s long-context understanding capabilities. We utilize all test categories, with each category containing 50 test samples covering lengths 32K and 64K. We post the testing configuration of RULER in Table
Long-form Reasoning We evaluate the long-form reasoning capability of models on selected tasks from BABILong (Kuratov et al. 2024). Specifically, select tasks that involve multiple supporting facts, as well as QAI, as the testing dataset. The BABILong testing configurations are shown in Table 10
D.3 BASELINE ILLUSTRATION
We evaluate our method on three foundation models, i.e., LLaMA-3-8B- B-Baase LLaMA-3.1- 8B-Base, and LLaMA-3.1-811 1-8B-Instruct-tt en- sure fair and consistent comparisons across baselines. The baselines include: YaRN, which extends the context window using an improved NTK-based positional scaling method; CE (Cross Entropy), standard language modeling objective without any context-aware weight- ing; LongCE, which builds upon the LongPPL method by identifying key tokens via perplex- ity during training and assigning them higher loss weights; SFT, an instruction tuning setup where input tokens are excluded from the loss calculation; and LOGO, DPO-based training approach designed to mitigate misalignment long-context tasks. Additionally, we com- pare against several strong open-source long- context models: 5112 1agg-512K-Inntuuu IInttrcct and NExtLong-S5 512kong--KK-Innsrruutt which apply long- context scaling techniques on top of LLaMA- In and LLaMA-3.1 S---BB-Istue
Table 13: Model performance language mod-
eling tasks.
Models
LongPPL
PPL
Llama-3-8B-Ban
> 100
100
+ YaRN
3.55
5,60
+ +CE
3,90
6,46
+ LongCE
3.55
5,60
+ CDT (ours)
3.04
5,40
Llama-3.1-88 3.-BBBB.BBBBBBeee
3.22
4.79
+ +CE
3.28
4.86
LLongCE
3.24
5.28
+ CDT (ours)
2.10
5.19
1imma-3.1-BB-Imeee
4.05
5,52
+ SFT
3.31
5.51
+ LOGO
4.11
5.54
+ CDT (ours)
2.36
5.64
22

---

Published as a conference paper at ICLR 2026
Table 14: Statistical significance calculation on LongBench-E data with t-Tst
Models
P-Value
Llama3-80 -B--ase V.S. Llama3-88
3.68e-15
Llama3. 1-8B-Base V.S. Llama3. L----BBs-eCCi
1.53e-2
Llama3. -nSt-lnstruct V.S. Llama3. 1-8-ttstruutt Cettr
2.39e-3
spectively; and S-L.-A-----B-SAALLLO
model specifically optimized for long-
context alignment.
E MORE EVALUATION RESULTS
ANALYSIS OF RESULTS ON REAL-WORLD LONG-CONTEXT TASKS
The strong performance of CDT on code-related tasks, as shown in Table I particularly notable. Code Completion requires models accurately interpret local context and predict missing seg- ments accordingly. CDT especially well-suited for such tasks, as it enhances the model's ability focus local context information during generation, which likely contributes the observed performance improvements. Table 15] offers a more intuitive illustration through a specific Code Completion example. In LongBench-E, this task is evaluated using the Edit Similarity (Edit Sim) metric, which is highly sensitive to the number of tokens senerated-especiallle under the official 64-token generation limit. In the provided example, 3LLMM--.1-8B-Insttii correct outputs, while GPT-4o generates overly lengthy responses that negatively affect the Edit Sim score. In contrast, the CDT-enhanced model generates a concise and accurate response, resulting in significantly higher Edit Sim score. Furthermore, CDT leads to substantial improvements for both -LLMMA-3.--B--M -88-Innsttttt and LLaMA-3-8IB-BA. be attributed to two main factors. First, the training set includes code completion instances (e.g., 263 examples from LongMIT), which enable the model to learn relevant eustruction-followe Second, the baseline model's lower performance in this domain makes the gains from CDT more ap- parent. By contrast, LLaMA-3.J 3.1 --8-BBB- already demonstrates strong performance on code-related tasks likely due to the composition of its pretraining data resulting in smaller relative gains when CDT is applied.
produces entirely
on the Code task. These improvements can
patterns.
E.2 GENERALIZING CDT TO MORE MODELS
We apply our CDT method to more LLMs, including Droo2..57B-Insttree (Yang al. 2024) and Mistral- V0.33--n.trttt (Jiang et al. 2023). We evaluation the model performance on weal-world long- context tasks, long synthetic tasks, and long-form form reasoning tasks. We report the model performance in Table 12] where we can observe that our CDT can significantly improve the model performance on different models. For instance, the Mistral-VO.3-Instueee model obtains more than 30 points on the long-form reasoning task.
E.3 EVALUATION RESULTS ON LANGUAGE MODELING TASKS
Apart from evaluating with LongPPL the language modeling task, also calculate the PPL scores, which are shown in Table 13
E.4 EXPERIMENT STATISTICAL SIGNIFICANCE
We collect the prediction results of the original model and the CDT model the LongBench-E benchmark, and conduct a difference before and after the improvement, shown in Table The results show that method significantly outperforms the baseline model at the 5% significance level, indicating that our method achieves statistically significant improvements.
t-test to assess the statistical significance of the mean
23

---

Published as a conference paper at ICLR 2026
Before CDT, LCM easily misled by context noise, and find no critical token
After CDT, LCM begin to focus critical tokens
(a) Attention map of 24th layer before CDT.
(b) Attention map of 24th layer after CDT.
Figure 14: Comparison between Attention Maps Before and After CDT. In each figure, a deeper (red) color indicates larger model attention to the corresponding context chunk. The final row of each map represents how the question attends to the entire input sequence, including both the context and the question itself. For clearer visualization, we recommend zoom in on this figure.
ANALYSIS OF ATTENTION MAP BEFORE AND AFTER CDT
In this section, we present a visualization of the model's attention patterns before and after applying the CDT training strategy. Given the long input length (12.000 tokens) used in evaluation, evenly partition the input sequence into 46 chunks and calculate the total attention score for each chunk individually. For each chunk, a higher total attention score indicates that the model places greater focus on this chunk. We visualize the attention maps of the 24th layer of the model, this layer provides the clearest representation that, before applying CDT, the model's attention is predominantly concentrated itself (the rightmost portion of the final row in Figure [4a). while key information within the context largely overwhelmed by noise. In contrast, after CDT training, the model not only attends to the question but also shows significantly increased attention to relevant contextual information, as highlighted by the red circles in the final row of Figure [4b] It is noteworthy that the attention map shows no significant changes before and after CDT training, indicating that CDT training does not compromise the original characteristies of the LCM. Instead, it enhances the ability of LCM to capture critical information.
CDT's impact. shown Figure 14 we can observe
the question
G LIMITATION AND FUTURE WORK
Due to the expectation maximization (EM) nature of CDT, includes an additional context noise detection process, which introduces extra computational costs during the training phase. Although we have demonstrated in Section 6.3 that these additional costs are negligible compared to the per- formance gains, theoretically, the noise detection cost will increase as the model size grows since involves complete newaard-backwerd explore simpler method for identifying the context noise or develop more efficient model architectures. For example, designing specific network modules to handle noise, proposed in Ye ct al. (2024), could be a promising direction. Additionally, we observe that the improvement brought by method complex reasoning tasks is not significant that on other tasks, and we are yet to understand the relationship between this and the training data or the training objective function. In the future, aim to further investigate the impact of context noise the model's long-form reasoning abilities, as well as the relationship between the CDT strategy and the enhancement the model's reasoning capabilities.
propagation process. We leave this for future work, aiming
24

---

Published as a conference paper at ICLR 2026
H USE OF LLMS
During the writing of this paper, we leveraged large language models (LLMs) to refine the clarity and fluency of writing, particularly the Abstract and Introduction sections. Specifically, we used the Qwen web interface to access the Qwen series of models (e.g., Qwen-Max), inputting early drafts of these sections and requesting stylistic improvements while preserving technical accuracy and original intent. The model's 's suggestions helped enhance sentence structure, academic tone, and overall readability. All final content was carefully reviewed, validated, and edited by the authors to ensure fidelity to our research and adherence to scholarly standards.
I ERROR ANALYSIS
In this section, we analyze the error pattern of partial model predictions real-world long-context tasks. As shown in Table [5| we use colored text highlight the correct and incorrect parts of the model's predictions.
httpp://ccaat.quen.at
25

---

Published as a conference paper at ICLR 2026
Generation samples of Llama3.1-8B-Instree Llama3.1-BB-Istten (ours), and GPT40 on real-world long-content tasks.
Table 15:
Code Completion Task
Question:
Completing the
Llama3.1-8B-Imn
for (int
code based on the context:
0; s. S.length() () i++) (if
.... Inn self.interface
charAt (i))
wwi.itrfaae.repplaeee.. ") .lower \n self. type
& s. c.arr (i) != ( }
Llama3.1-B--DD
(ours):
if
None in self.if = None An # state \n resulss dict () \n [ommpettion] self.resule def
self.interface == 'all':An
self pellf.para__tpe INTERFACE_AII An
GPT4o: if self. molffinterface. lower ()
dict \n
'all':'n self.param_typp
(self): \n
INTERFACE_AII ALL elif
"""Check all input params""n An if not self. interfaeeee An gelf.module, www.ouue..illssooo..ooooaaao (msg='Erre) Error: Interface name cannot be empty.") [...
' in self.interface or self. s....interface.isiiii. () :nn
self. sela.param_type INTERFACE_FULII_AAN
else: Vn self.if_type
get www.iineefaeeeecc Golden Answer if self.interface ==
\n
'all';\n
Long-context Multí-hop Reasoning Task
Question: What did M. King Hubbert's employer announce it was in the process of doing in April 2010? [.... In April 2010 Shell announced that the corporation is in process of trying to find potential buyer for all of its operations in Finland and is doing similar market research concerning Swedish operations.
Shell announced the sale of some of its assets, including its liquefied petroleum gas (LPG) business, to meet the cost of planned $28
Llama3.1-8B-Insten
Llama3.1-8B-CD (ours): trying to find potential buyer for all of its operations in Finland and is doing similar market research concerning Swedish operations. GPT4o:
MV Miralda's s owner announced it was in the process trying to find potential buyer for all of its operations Finland and doing similar market research concerning.
trying to find potential buyer for all of its operations in Finland
Golden Answer
Single-QA Long-context Task
Question: What metrics are used in evaluation? [...] Figure FIGREF22 shows the results of these experiments wherein we are able to maintain the precision the subset of the training set used for testing while reducing its recall and thus generalising the model. The fact that the model is generalizing on dissimilar dataset is confirmed by the increase in the development set. F1 score. [...
Llama3.1-8B-Instuuct
Macro-averaged F-score.
Llama3.1-8B-CD (ours):
macro-averaged F-score, precision, recall, Fl score. GPT4o:
Macro-averaged F-score.
Golden Answer
precision, recall, Fl score
26


Reviews:
[{"review_id": "CSRMESYnvy", "reviewer": "Reviewer_gbM2", "summary": "the paper address the issue of Long-context language models (LCMs) struggle with contextual noise—when irrelevant information in lengthy contexts overwhelms critical tokens, leading to some issues. The paper propose some measurement.", "questions": "see weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Tackles a crucial challenge in long-context language models: contextual noise overwhelming critical information\n* High practical relevance for real-world applications (RAG, document QA, long-context reasoning)\n* Problem is timely given the trend toward longer context windows in LLMs", "weaknesses": "1. The paper proposes an indirect approach: compute IG scores externally, identify critical tokens, perturb embeddings, then train. However, they provide no compelling explanation for why this is superior to simply training the model to learn token importance directly through standard optimization.\n\n2. The preliminary study (Section 3) relies entirely on synthetic tasks with manually injected noise, which provides no evidence that the observed attention patterns occur in real-world scenarios.\n\n3. The proposed method pre-selects critical tokens for training sequences, which seems to circumvent the model's own ability to learn token importance organically during training. Since intuitive learning suggests independent discovery of salient tokens, the authors should clarify why this indirect training approach outperforms direct training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1PG3ZTnch", "reviewer": "Reviewer_L6Fs", "summary": "This paper identifies a key weakness in long-context models (LCMs): their susceptibility to contextual noise—irrelevant tokens that distract the model from critical information. The authors propose a novel training strategy called Context Denoising Training (CDT), which improves a model’s ability to focus on salient tokens and strengthen their influence on predictions.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The Integrated Gradient (IG) score, which more accurately identifies important tokens compared to attention-based methods.\n\n2. Manual noise suppression experiments: Showing that reducing noise in input embeddings boosts attention on critical tokens by ~10×.\n\n3. The CDT training strategy: A lightweight, online method that detects and suppresses noisy tokens during training, improving model focus without heavy computational overhead.\n\n4.Extensive evaluation: Across 4 task types (real-world, synthetic, language modeling, reasoning) and multiple models, CDT consistently outperforms baselines and even enables an 8B model to nearly match GPT-4o on LongBench-E.", "weaknesses": "1. Is there any relevant literature or experimental evidence supporting the statement in line 759 that \"performance gains typically exhibit diminishing returns with increased token budgets\"?\n\n2. In lines 768–769, it is stated that \"LongCE achieves a 13-point gain per 1B tokens versus ProLong’s 0.3-point gain per 1B tokens.\" How was the 13-point gain calculated?\n\n3. Why are the results for NExtLong-512K-Instruct and ProLong-512K-Instruct shown in Table 1, but not included in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQwn0DieGA", "reviewer": "Reviewer_Rdcm", "summary": "This paper introduces Context Denoising Training (CDT), a straightforward yet effective training strategy that improves attention on critical tokens while reinforcing their influence on model predictions. The authors first propose the Fact Retrieval (FR) score and Integrated Gradient (IG) score to identify and quantify context noise, demonstrating that existing long-context methods fail to distinguish critical from irrelevant information, degrading performance. They show that simple mitigation of detected context noise can substantially boost the model’s attention on critical tokens and benefit subsequent predictions.\n\nBuilding on this insight, CDT subtracts the corresponding gradients to manipulate the irrelevant token embeddings, thus suppressing the context noise. This allows the model to focus on critical tokens and strengthens the causal link between them and the final output. Across LongBench, RULER, BABILong, LongPPL, and other benchmarks, CDT consistently boosts long-context performance.", "questions": "1. The paper introduces the FR score, IG score, and an identifier. However, only the identifier is actually used in CDT. The FR and IG scores serve only as diagnostic tools because they require prior knowledge of the four token types in the context. I would appreciate it if the authors could clarify these two distinct contributions more clearly.\n2. It is unclear why low-frequency words should be singled out. No prior work is cited to justify this taxonomy. A simpler partition into supporting, interference, and irrelevant tokens appears sufficient, and the proposed identifier itself only decides whether a token is irrelevant.\n3. I find it confusing that authors compare CE, LongCE, and CDT with YaRN. YaRN is not a training paradigm. It is merely an interpolation method like NTK. Likewise, comparing CDT at inference time with FlexPrefill and XAttention is also confusing. These techniques aim at enhancing efficiency, not performance.\n4. As a work focused on long-context training, CDT lacks citations to early efforts in this line[1-3]. In addition, the strategy of identifying critical tokens to improve training or inference should be clearly differentiated from similar techniques originally developed in short-input or long-output scenarios[4-5].\n5. The authors note in the appendix that the improvement brought by our method on complex reasoning tasks is not as significant as that on other tasks. Since CDT also identifies critical tokens to enhance reasoning, could the high-entropy token approach[5] be combined to further boost performance?\n6. Typo: irrelevant instead of irrevelant in Figure 3a\n\n[1] Effective Long-Context Scaling of Foundation Models https://arxiv.org/abs/2309.16039\n\n[2] Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models https://arxiv.org/abs/2405.17915\n\n[3] LongWanjuan: Towards Systematic Measurement for Long Text Quality https://arxiv.org/abs/2402.13583\n\n[4] RHO-1: Not All Tokens Are What You Need https://arxiv.org/abs/2404.07965\n\n[5] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning https://arxiv.org/abs/2506.01939", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper conducts extensive experiments across multiple settings and long-context benchmarks, thoroughly demonstrating the effectiveness of CDT over multiple baselines.\n2. The introduction of the FR score, IG score, and identifier $\\mathcal{I}(x_i)$ provides valuable insights into how long-context LLMs comprehend long contexts.\n3. Beyond presenting an effective long-context training paradigm, the authors offer in-depth discussions and visualized analyses that solidify the credibility of their approach.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)