Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
Authors: Haoran Deng
Yingyu Lin
Zhenghao Lin
Xiao Liu
Yizhou Sun
Yian Ma
Yeyun Gong
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Data Curation
Primary area: foundation or frontier models, including LLMs

Abstract: Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across extended spans of text. However, a significant portion of readily available long-text data lacks meaningful long-distance dependencies; most spans can be predicted using only local context. Training on such data is inefficient, making careful data selection crucial. Therefore, we introduce LongFilter, a framework for curating training data tailored to long-context pretraining. LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings, thereby identifying samples where long-range dependencies are essential. Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.

Published as a conference paper at ICLR 2026
BEYOND LENGTH: QUANTIFYING LONG-RANGE IN- FORMATION FOR LONG-CONTEXT LLM PRETRAINING
DATA
Haoran Deng" Yingyu Lin
Zhenghao Lin" Xiao Liu
Yizhou Sun' Yi-An Ma² Yeyun Gong
University of California, Los Angeles, University of California, San Diego,
3 Microsoft Research
[denghaoran, yzsun]ees wclu.cllaaoo edu, lyi1208, Huccsd edu, [xiaoliu2, wwenghaohnhhhnnee yegong @mi crosoft. com
ABSTRACT
Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across tended spans of text. However, a significant portion of readily available long-text data lacks meaningful long-distance dependencies; most spans be predicted using only local context. Training on such data is inefficient, making careful data selection crucial. Therefore. we introduce LongFilter. framework for curating training data tailored to long-context pretraining. LongFilter measures the infor- mation gain provided by extended context by contrasting model predictions un- der long-context versus short-context settings, thereby identifying samples where long-range dependencies are essential. Experiments with LLaMA-3-8B, extend- ing its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.
INTRODUCTION
Modern large language models have shown remarkable capabilities when processing short spans of text, but many real-world tasks, such as reasoning across documents, generating long codebases, or summarizing entire chapters require understanding and integrating information over longer contexts. To enable their long-context capabilities, language models are typically first pre-trained on short- context data and then further continually pre-trained on long-context data, which activates their long-context reasoning abilities. Although recent advances, such as RoPE modifications (Su et al.) 2024) and attention interpolation (Peng 2023 Ding et al.) 2024), have improved efficiency and reduced training costs, the choice training data remains a critical determinant of performance.
While existing methods improve long-context pretraining efficiency, the quality of long-conter. training data remains a critical factor for unlocking a model's long-context abilities. Current data engineering approaches primarily focus on sequence length, for example, by increasing the propor- tion of long sequences in the training set (Fu et al. 2024 Abdin et al. 2024 Yang et al, 2025) or adjusting the ratio between long and short sequences (Gao et al. 2024) However, sequence length alone fails to differentiate genuinely long-context-dependdea data from lengthy sequences domi- nated by repetitions, independent segments, or tokens predictable from short preceding spans. As a result, a considerable portion of long sequences in commonly used corpora does not truly con- tain long-distance dependencies. since their next tokens can already be inferred from short contexts. Even high-quality samples may thus be more appropriate for short-context rather than long-context pretraining.
"Equal contribution.
Work done during his internship at Microsoft Rescarch. Coorrsponning author.

---

Published as a conference paper at ICLR 2026
Long-length data does not mean long-context data. Some long-length texts simply lack contextual information in their extended content. For instance, consider books and poetry collections. Individ- ual poems, even by the same author, often lack inter-poent dependencies, and their relatively short length makes them suitable for short-context models. In contrast, texthooks are more appropriate for long-context pretraining, chapters are tightly interconnected and understanding requires access to preceding chapters. These examples illustrate that not all long sequences convey meaningful long-context information. Including sequences that do not require extended context can dilute the training signal and ultimately lead to performance degradation on long- long-context tasks.
chapter often
Building on this motivation, we explored how to select training data that not only features longer length but also leverages information dependencies over longer distances. Existing long-context pre- training strategies can be viewed "0-to-1" step: by increasing the proportion of long sequences, the model begins to learn long-context dependencies. However, because the training loss is averaged over all tokens, sequences that do not truly depend on long-range context contribute equally to the learning signal, which is suboptimal. In this work, we take a "1-to-2" step by further increasing the proportion of sequences that genuinely require long-context understanding. This approach as- signs more learning signal to tokens that depend extended context, improving the efficiency of long-context pretraining and enabling the model to better leverage long-range dependencies.
To distinguish truly useful long- contect data from merely long-length sequences, we propose Long- Filter, a data selection framework for continued pre-training. Our method is founded on a simple yet powerful principle: data is valuable for long-context training only if the long context actually helps the model make better predictions. We operationalize this insight by developing a scoring function quantify this "information gain." " The score's formulation is derived from the Kullback-Leibler (KL) divergence between a model's next-token prediction distributions conditioned on a long ver- sus short context, which we compute at each token position. These lekenllevel scores are then aggregated to produce final score for the entire sequence. A high score signifies that the extended context provides crucial information, making the sequence high-quality candidate for training.
Contributions
1. This paper suggests that long-context continued pretraining should be conducted on data whose extended context provides additional information for next-token prediction.
2. We propose LongFilter, a data curation method that quantifies the information gain pro- vided by an extended context. Using transformer- based causal language model, LongFil- ter efficiently scores and selects high-quality long-context pre-training data.
3. Extensive experiments show that, without modifying the model or training setup, simply selecting training data with richer long-range information can substantially improve lan- guage model's long-text processing ability during continued pre-training. Models trained LongFilter-seleetee data achieved average gains of over 2 points benchmarks includ- ing HELMET, LongBench, and RULER.
RELATED WORK
2.1 LONG-CONTEXT LANGUAGE MODEL PRETRAINING
Long-context language models have garnered significant attention within the community in recent years due to their high practical value in applications such as code generation and reasoning. A current mainstream approach involves extending the context of an existing language model with short-term context. On top of this, certain techniques have been developed to reduce the amount of training required. For example, some works employ position interpolation (Chen et al. 2023 Peng et al. 2023 Bertsch al.,
2023 Ding et al. 2024 Liu et al. 2024b Zhang et al. 2024 Zhu et al.
2024 on RoPE (Su et al.
2024 to enable the model to better adapt to the positional encoding of
extended context, or manipulating attention module (Xiong et al. 2025 Jin et al. 2024 Bertsch
ct al. 2023). Some of these methods have been applied in certain enterprise-Ieve models (Liu et al.
2024a Yang al. 2025).
2

---

Published as a conference paper at ICLR 2026
2.2
DATA CURATION AND FILTERING FOR LANGUAGE MODEL PRETRAINING
The quality of data exerts direct influence on the performance of language models. This has be- come a standard process for enterprise-level langauge models (Gunasekar al,, 2023 Abdin al,)
2024 Abouelenin et al. 2025). Typically, this complex process involves multiple steps, including
heuristic approaches (Gao ct al. 2020 Laurençon ct al. 2022 Rae et al., 2021), data quality classi- fication (Longpre et al. 2024 Wettig et al. 2024 Xie al. (2023). domain speciifc selection (Feng et al. 2022), deduplication (Borgeaud et al. [022]]Abbar et al. 2023), multilingual filtering (Wen-) zekk et al. 2019), removing toxic content (Penedo al. 2023 Jansen al., [2022). These methods
have achieved tremendous success short-context model pretraining, yet few them are specifi-
cally designed for long-context data.
2.3 DATA ENGINEERING FOR LONG-CONTEXT PRETRAINING
Existing data engineering approaches for long-context pretraining primarily focus on the length of training data, specifically by adjusting data proportions increase the proportion of longer-length training examples within the text corpus (Abdin et al. 2024 Yang et al. (025). Fu et al. (2024) recommends increasing the proportion of data with longer length while maintaining domain balance. Gao et al. (2024) investigated the impact of the ratio of long-to-shoo to-short data mixing and the data source on the performance long-text pretraining. A similar idea to this paper is Long Wanjuan (Liu al, 2024c). which proposes several metrics to measure the quality of long text data. However, most of its metrics are also applicable to short texts, and the context length of the model-based filtering method used in its paper is too short (the longest window in its paper is short as the short windows in this paper). Another related approach, LongAttn (Wu et al.) 2025). uses attention scores to model long-range dependencies, but studies have shown that these attention scores do not reliably capture token importance.
METHODOLOOGY
Our method, LongFilter, is designed to identify and select training data where long-range depen- dencies are semantically meaningful and essential for accurate token prediction. The core insight is to quantify the "information gain" provided by an extended context. We formalize this gain as the (surrogate Kullback-Leible (KL) divergence between the predictive distributions of a language model given a long context versus a short one, which we compute at each token position. These token-level scores then aggregated produce final score for the entire sequence.
Based on this principle, our framework follows a multi-step pipeline:
1. Calculate Token-Level Gain: At each token's 's position, we compute score that quantifies the information gain the extended context provides for predicting that specific token.
2. Aggregate Document Score: We aggregate these token-level scores (e.g., by averaging) to produce single, final score for the entire data instance.
3. Filter and Select: We rank all instances by this aggregate score and select high-scoring subset for continued pre-training.
3.1 PROBLEM FORMULATION AND NOTATION
We operate within the standard causal language modeling framework, where the objective is to predict the next token given preceding context Tet.
Let sequence of tokens be denoted by X (N.,,.) For any given token x, in the sequence, we define two distinct context windows:
Short Context (S): The sequence of Eshort tokens immediately preceding क4. Formally, Stt) (Pe-tshors 11-1).
Longg Context (L): The sequence of fLong tokens immediately preceding z,, where (Long > Eghort- Formally, T(t)) = (Ite Prte-Boogg"-.. Bt-1)
3

---

Published as a conference paper at ICLR 2026
The extended context, denoted E, is the portion of the long context that precedes the short context, ie., E(t = (. 2t-1,)) The long context is therefore the concatenation the extended and
short contexts, L Eo S. See Figure
Short context Long context
hate this
The plot was a mess and the acting was terrible. hate this ?
song
Extended context Next Token
The plot was a mess and the acting was terrible.
movie
thing
An illustration of the token-level long-context information gain. Given only the Short Context (S) "I hate this", the predictive distribution for the next token has high entropy, many words ('song', "thing', 'movie') are plausible. The Extended Context (E), "The plot was a mess..", provides critical information that reduces this entropy, concentrating the probability "movie".
Figure 1:
Given a pre-trained language model M, we can obtain two conditional probability distributions for the next token:
(s(+) = P- S())) and PL(-) PM L(t))
The central question LongFilter addresses is:
How can we quantify the additional information that the extended context E provides for predicting It beyond what is already available the short context S?
3.2 INFORMATION- AHFOMATTIONTTHEORTIIC FORMULATION OF CONTEXTUAL GAIN
The ideal theoretical tool to answer our question is Conditional Mutual Information (CMI). The CMI T(T; 5) measures the reduction in uncertainty about target variable (the next token) after observing extended context E, given that a short context S is already known (Cover & Thomas, 2006).
The CMI can be expressed in two well-known, equivalent forms. The first defines CMI as the reduction conditional entropy:
-
(1)
where H(: is the conditional entropy. A second, equivalent formulation expresses the CMI as the expected Kullback-Leeibee (KI.) divergence between the predictive distributions with and without the extended context:
[[T: E S) Ep(se) [Dxu |S S= = e) p(T S = =s)]]]
(2)
This second form is particularly insightful, as it frames the information gain the expected "dis- tance" between the posterior belief p(1') S, E)) and the prior belief p(T' S). For completeness, we derive the equivalence of these two definitions in Appendix C
For given context instance (e" (e*,s)), to evaluate the effect of the extended context e* on the next token I prediction, we consider the one sample estimate the above CMI:
i(T:E= = SS= DKL (p(T) =" E e²) P(T S S==)))
(3)
3.3 A PRACTICAL SCORING FUNCTION FOR CONTEXTUAL GAIN
Expanding the KI. divergence by its definition in equation 3 we have
p(t 8-
(oT( S ==,=" ll 5= S= ===
s',"""))))) = s" (ee") log p(t 5*)
Ep(t LEV
(4)
This formula has two drawbacks: does not leverage the ground-trui information of t", i.e., the value of DxL (p(T) S s" E = e*) PTT 5 s")) does not depend on t" and it requires costly summation over entire vocabulary V. To create a practical score for a single ground-truth instance ("","", ".s""e"" "e",, we focus on the term corresponding t", which yields surrogate for KL divergence:
P(T = t' E ==e- -..S=
"t"tt",,,,e") p(T = E = = so)
(5)
p(T 1 | (S=8"
4

---

Published as a conference paper at ICLR 2026
This score (Eq. provides fundamental "token-level score". It can be interpreted the gain for predicting the specific target t" that is contributed by the extended context e", given that s* was already observed.
To evaluate an entire data instance X "ei....,,""" ) as outlined in our methodology- aggregate these individual scores. We define the final LongFilter score the average of the per- token scores across the sequence:
Score(X))
score (x) flangit Eshon -1 Eshore 11
(6)
plat 2i (im 1)
p
x Fi-tiomgtt log
p( 26 bshont
For more practical perspective, we can reformulate the LongFilter score in terms of the standard per-token cross-entropy loss, which is equivalent the negative ligelikei Let Clome and çshont be the losses for predicting the ground-truth token 2; given the long and short contexts, respectively:
Clumg He [1 =z ,.).p( (-11--10- x (lome:t-1)
log p(æ;
C He (1 = =.;)pp( 1. (shortt--1)
p(o; 1 (soossoor
where Ho(p,9) Ep log denotes the cross-entropy of the distribution relative to Then we have
Score Score(X*) 1
Σ expl - (Rshhon N
(
L.
This loss-based view offers clear interpretation: the score gives preference examples where the reduction in prediction loss from using longer context (the term Gohor - Cloeg) is large. This loss reduction then weighted by the model's confidence on the token given the full context (exp (-Clong) = p(z² x (1--))) ensuring that the gains are on tokens the model consid- ers plausible.
3.4 LONGFILTER
The framework of LongFilter is shown in Figure LongFilter utilizes pre-trained causal language model to estimate the distribution of the next token across varying context lengths. LongFilter consists of three steps: Long-context Modeling, Short-Context Modeling, and LongFilter Scoring.
Long-Contens Modeling. For an input sequence, we compute the probability distribution obtained by predicting the next token each position based on its prefix context, in a manner analogous to the training stage of causal language model. The process of getting the next token distribution in the long context utilizes the prefix context from all positions.
Short-Context Modeling. For predicting the distribution of the next token in short context, the LongFilter first segments the entire text into shorter chunks. Each short chunk is then fed into pre-trained causal language model, thereby constraining the context of the predicted output the boundaries of the short chunk. To avoid predicting tokens with insufficient context at the beginning of each short chunk, we chose introduce overlap between different chunks during segmentation.
LongFilter Scoring. After obtaining the probabilities for predicting the next token from both short and long contexts, the final score calculated using Equation] / All scores are sorted, and portion of the higher-scoring entries selected as the chosen data.

---

Published as a conference paper at ICLR 2026
Sliding Window
Sliding Window
The
plot
the
acting
temrile
hate
this
movie
Short Context Modeling
Next-laken pradiction distribution
Next-loken prodiction distribution
pl the acting)
al.ala
pC
hate this) .nnl
LongFilter Scoring
Low Contextual
Probabiiity predicting
High Contextual
Information Gain KL div. x
Information Gain KL div. >
predicting
Aggregation Token-Lavel Scoring
The plot acting) al.aia
p(
plot was a... hate this) ls.11
Next-loken pradiction distribution
Next-token prodiction distribution 1
Long Context Modeling
The
plot
and
the
acting
terrible
hate
this
movie
Figure 2: Workflow of LongFilter. The Upper part computes the next-token probability distribution using short-context sliding window (shown as 4 tokens for illustration, though experiments use 4K), while the Lower part computes it using the full long context, LongFilter then scores the information gain (Middle part) by calculating token-level surrogate KL divergence between these two distributions. This gain is low for locally predictable tokens (such as 'was'), but high for tokens that require extended context (such as 'movie"), ). Finally, these token-level scores are aggregated to produce single score for the entire data instance.
EXPERIMENTS
4.1
SETUP
We conduct experiments by continually pre-training LLaMA-3-8B (Dubey al. 2024). which has an initial effective context length of 8K, on different datasets to extend its context length to 64K. For each dataset, we pack the tokenized text into sequences of 64K tokens, score each sample with LongFilter. The model then continually pre-trained on these filtered samples, and its performance evaluated on long-context benchmarks to assess the effectiveness of the data selection process.
Datasets. We SlimPajama-627 (Soboleva al. 2023 as the primary source for long-context pretraining. LongFilter is applied to extract high-quality long- SlimPajama has been widely adopted in recent (e.g.. Gao et al. (2024). Fu et al.] (0244)
text samples from this dataset. long-context data engineering works Specifically, we select three corpora from SlimPajama, that is ArXiv, Books, and CommonCrawl, for our experiments. We similarly categorized each corpus by length, selecting thresholds of 16K, 64K, and 32K for long and short texts in ArXiv, Book, and CommonCrawl, respectively. After applying these thresholds, the vol- ume of data classified as long texts was approximately 19 billion tokens. We con- structed the model training dataset with 80% long texts and 20% short texts. Data selection was applied exclusively the long text portion.
Table I: Statistics of long context tokens.
Name
#Tokens
SlimPajama- Book
19,535,822,844 19,489,295. 5,00
SlimPajama-Arxiv
SlimPajama SlimPajama-Commorte CommonCrawi
19,284,099.072
Model and Training Configuration. For training, we adopt the same configurations as Pro- Long (Gao et al. 2024) when scaling LLaMA-3-8B from an 8K to a 64K context, including op- timizer, learning rate, and RoPE base frequency. The only difference lies in our choice of training data, which is guided by LongFilter-basd basee selection. Apart from increasing the RoPE base frequency from 5 10 to 10% made no further modifications to the model in our experiments. Drawing on configurations from previous studies on long text training (Fu et al. (024). we set the batch size 4M tokens and trained for 1,000 steps, processing total of 4B tokens.

---

Published as a conference paper at ICLR 2026
Baselines. We first compared model with ProLong (Gao et al. 2024). but unified the training data to three corpora from the SlimPajama dataset and adopted the same short-to-long ratio. For a fair comparison, from all training data, while Longrilter's training data was sampled from the selected data. We did not exclude the selected data from ProLong's training set, meaning ProLong and LongFilter share a portion of high-quality long-context training data. We also compared our approach with Lo Wanjuan (Liu et al. (024c), conducting comparative experiments using their best-performing aggregated and holistic data ratio of 1:1 as specified in their paper.
did not use ProLong's ShortMix dataset. ProLong's s training data sampled
Setting of LongFilter. We set the short context window 4K and the long context window to 64K, using the Llama-3.1-88 model (which supports 128K contexts) for scoring. We sorted the scores and selected the top 20% of data as the final training dataset of FingFilter. We run LongFilter on 32 NVIDIA H100 GPUs, enabling each corpus to complete all scoring within a single day.
4.2
EVALUATION ON RECALL (NEEDLE-IN- A -AIINN--HHASTSAE HAYSTACK) TASKS
ProLong
LongWanjuan
LongFilter
SlimPajarma-Arie
SlimPajama-Book
Slimajjama-Commoccraw
100
100
100
95
95
95
Recall (NIAH)
00
80
80
80-
75
75
75
60
60
Trained Tokens
Trained Tokens
Trained Tokens
Figure 3: Performance Recall tasks
w.r.t trained tokens.
We first report the performance of different data strategies series of Recall tasks. This series of tasks has also been referred to as iaalle-in-a-Hasstadee (NIAH) (Kamradi) [002).. This type of tasks directly tests model's ability to utilize information from any position, often serving as one of the most important metrics for evaluating a model's 's performance on long text.
Specifically, reported on the SubEM of Recall task within the HELMET benchmark Yen al. 2025). which encompasses four distinct NIAH tasks: JsonKV, Needle retrieval with multiple keys, UUID retrieval with multiple keys, and value retrieval with multiple keys. Experimental results shown in Figure 3
ProLong
LongWanjuan SlimPajama-Book
LongFilter
SlimPajama-Arxiv
SlimPajama--CommonCraww
58
58
HELMET Score
54
50
54-
54
52
52
44
50
48
Trained Tokens
Trained Tokens
Trained Tokens
Figure 4: Performance HELMET with respect to trained tokens.

---

Published as a conference paper at ICLR 2026
Across the three experimental settings with different type of training data, consistently observe that LongFilter achieves the best overall performance and exhibits the most stable improvement as the training scale increases. For example, in all three groups, LongFilter rapidly surpasses both Prolong and Long Wanjuan at small scales (0.5B-1B) and maintains clear advantage when scaling up 4B, reaching performance ahove 90 in every case. This indicates that filtering with LongFilter can effectively enhance data quality and maximize the benefits of scaling.
4.3 EVALUATION ON LONG-CONTEXT BENCHMARKS
Table 2: Experimental result on LongBench.
Dataset & Model
SingleQA
MultiQA
Summ
ICL
Synthetic
Code
Overall
Arxiv
ProLong
25.26
16.24
24.83
63.04
51.76
68.69
38.58
LongWard Wanjuan
28.43
18.25
25.09
62.85
49.92
69.22
39.36
LongFilter
28.32
17.99
24.48
63.15
51.41
70.00
39.52
Book
ProLong
21.25
15.06
23.64
62.60
52.41
69.76
37.47
Long Wanjuan
22.33
14.84
22.75
63.46
52.62
70.05
37.69
LongFilter
26.58
20.14
24.36
62.69
53.58
70.08
39.81
CC
ProLong
21.95
16.83
25.03
63.87
52.18
69.33
38.37
Long Wanjuan
33.64
18.58
25.18
61.95
47.87
69.79
40.02
LongFilter
30.54
17.21
25.58
62.84
57.02
69.11
40.66
Table 3: Experimental result on RULER.
NIAH Single
NIAH MultiKey
NIAH MultiValue
NIAH MultiQuery
Dataset & Model
Other
Overall
Arxiv
ProLong
77.90
85.40
89.35
85.73
47.12
69.28
LongWanjual
83.8
80.57
83.16
85.04
49.15
69.69
LongFilter
78.68
86.20
89.76
86.92
48.07
70.13
Book
ProLong
93.83
83.80
92.76
95.08
46.56
73.35
Long Wanjuan
90.08
91.03
91.38
82.75
48.23
73.74
LongFilter
95.33
97.87
93.15
80.10
54.71
78.95
CommonCrawl
ProLong 91.31
86.58
94.32
77.87
47.54
72.59
LongWanjuan 90.85
84.65
89.76
80.49
41.25
74.08
LongFilter 92.58
94.50
94.80
77.80
31.71
75.37
To validate whether the data selection strategy of LongFilter is beneficial for training long-text lan- guage models, we evaluate continually pre-trained models on 3 widely used long-context bench- marks: HELMET (Yen et al. 2025). LongBench (Bai et al. (022)), and RULER (Hsieh al.) (024). Since RULER and LongBench require language models to comprehend instructions, we SFT models with IB data using UltraChat dataset (with settings consistent with Gao et al. (2024)).
The final reported score is the average all non-model-based evaluation metries across all tasks in HELMET, encompassing five tasks: Recall, RAG, Re-rank, ICL, and QA. We report the overall performance on HELMET benchmark with respect to the number of trained tokens in Figure g
According to the experimental results, the quality of long-text training data undergoes clear, signifi- cant, and sustained improvement after LongFilter's data selection. LongFilter significantly improves training efficiency. Compared to unfiltered data, length extension training with 1.5B filtered tokens
8

---

Published as a conference paper at ICLR 2026
Vchaptaz (onnlisation))
lechog tereark Cinnol resoos)) Vincclade ande aer untenn)
Viabel i cahap dos cccealasions))
(000666714366111777 ideae (iine vidta-g zae] 1615 2ww.224627724440 24002466774440006 A
(22 5,130 160210776444444 1177737777777 V 12
Viagesibre)
0 Viageti3/4
(12 2415 0112535677334477777733333
01agrt/2/41
ideae [iine vidtan0 zae] 18+5
ACV)
( (12 9607550714415447759
(12.2010 (1334447744477777777777777
S1agrt/3/41
idear [Line vidtbed ben] (13. 2010 000.03626666661111000 (77
Vaagrt
(2/4) 112
(iages(244)
part the neeral ontwork based W.l medel before, such language medel histipplin (000ll00000meilll Wiile globel metrix Exctoriiation beaed latent analyale beltng www.etatiiiiiiaaiiiiii and Irom clustare
Need mibddiings sazede vord to dense vactor rxpresentation, It had sxleted
Uleer [1ine viditbd Zmm]
(12 15.2290.000020 1
Viages (3/4)
5,2200 00002666614444677222222000
Viee Dine vidthed
5,300
salins lossning based appessees like Randris (skip-oras and
5,160 (00066674411447777770000
N (33
11egrt13/10 (24,
stall 2017 scntching) prabes rxpeemnztation lsaming
MLS- seecralned vosi acbendinge becana acantalote stap is the pipalisa desnlpping seural systeme
position
(32 5,a05 www.ES/ASSSSACCEEEEAE.. 0763399377 20199994 Viegric3/8) (3/4)
Cline widkkad
inpas contentunl vord anbeddings,
retwerk, Pretceining!
[13 5,330, www.00400TTEE0006656.. TTT.C/TTTT4TT077777777777777
408825482794 (70022/4440004447011117000001010
Viages(3/4)
Vieget (374)
leazning information itenle instead of from syy particular tasks.
Lis61476i7aan w77.7724791100 stagesck/a)
Vitee Eline widtbad 2ee1 (17 sep 256025
18752941
(a) High-score text segment: Thesis.
(b) Low-score text segment: TikZ codes.
Rank
Rank 2
Rank 3
Rank 1000
Rank 999 LAwgScore:: 0,0113)
Rank 998
LAugScore: 0.5522)
(wggScore: 0.4 0.4616)
LAwgScore: 0.4776)
(wggScore: 0 (1227)
(AvgScore: 0.0097)
StOW
20M
4/c
336
60M
10c Tolson Postion
2/R
6X
25c
206
6/
Tolion
Tokan Poskion
Tobion Position
(c) Context Score each token position for documents of varying ranks.
Figure 5: Token-level context score analysis subset of the processed SlimPajama-Aray dataset. In the top examples, the color of each token is determined by its score: the darker the color, the higher the score. (a) A high-scoring segment of well--ormed academic prose from PhD thesis. (b) A low-scoring segment containing non-prose LaTeX TikZ drawing commands. (c) Context Scores across the full token sequence for documents of the top three ranks and the bottom three ranks.
already achieves performance comparable to training on 3-4B tokens, indicating that only about half the data is required to reach the same level of effectiveness.
On LongBench, the three methods demonstrate different trade-offs across datasets. LongFilter con- sistently achieves the highest overall scores, showing its robustness across diverse domains. In particular, yields notable gains in Synthetic tasks (e.g., 57.02 on CC) and maintains competitive performance in Code, where both categories heavily rely the model's ability to leverage informa- tion from arbitrary positions within the context. This suggests that LongFilter effectively improves the long-rand range information of training data, thereby benefiting tasks needs long-range dependency.
On the RULER benchmark, LongFilter consistently achieves the highest overall scores across all three datasets. Its advantage is particularly pronounced in structured data tasks, such as MultiKey, MultiValue, and MultiQuery, where careful filtering likely enhances the model's ability to capture long-range information.
Overall, these results reinforce the pattern observed in LongBench: data quality and filtering (Long- Filter) provide more consistent and robust improvements than original data and Long Wanjuan.
4.4 CASE STUDY: TOKEN-LEVER ANALYSIS
In this case study, we analyze a subset of the processed SlimPajama- Arxiv dataset containing 1000 samples, each a sequence of 65536 tokens. The analysis is presented in Figure 5 The token-level score is visualized by color intensity in Figure 5a (prose) and Figure 5b (code). where darker text indicates higher score. These results support the intuition that repetitive content like TikZ code, which lacks long-range semantic structure, receives low scores. Figure [5c] shows the token-level scores for the top-three and bottom-three ranked documents. The abrupt score jumps seen in plots

---

Published as a conference paper at ICLR 2026
of Rank I and 3 are artifacts created by concatenating multiple .te files from single arXiv submission during data preprocessing.
CONCLUSION
This paper proposes a data filtering framework tailored for pretraining long-context language mod- els. Unlike short-context language models, long-context models require leveraging semantic in- formation from longer range of positions. Based on intuition, we recommend that long-context language models should be trained on data where this additional length provides information for the next word prediction.
We formalize this process identifying training data where additional context yields higher con- ditional mutual information for predicting the next token. Based on this formulation, we develop scoring function that estimates the informational gain of context using trained language model. To apply this method to practical data filtering, we design model called LongFilter to score the informational value of additional context in long training data, recommending training on data with higher scores.
Sufficient experimental resutls demonstrates the effectiveness of LongFilter. We achieve sustained and significant improvements in long-text capabilities for long-text models solely through data filter- ing. After expanding the Llama-3-8B model from 8K 64K context, experiments benchmarks like HELMET, LongBench, and RULER demonstrate that this simple yet effective method yields up 10% accuracy gain on recall tasks when training IB tokens.
ACKNOWLEDGEMENTS
Haoran Deng and Yizhou Sun were partially supported by NSF 2211557, NSF 2119643, NSF 2303037, NSF 2312501, NSF 2531008, SRC JUMP 2.0 Center, Amazon Research Awards, and Snapchat Gifts.
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning web-scale through semantic deduplication. arXiv preprint 40X5v:2303.09540, 2023.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Séhastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08905, 2024.
Abdelrahman Abouelenin, Atabak Ashfaq. Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary. Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-lora arXiv preprint arXiv:2503.01743. 2025.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3119-3137, 2024.
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unlimiformer: Long-range transformers with unlimited length input. Advances in Neural Information Processing Systems, 36:35522-35543. 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, cean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. International conference on machine learning, pp. 2206-2240. PMLR,, 2022.
10

---

Published as a conference paper at ICLR 2026
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint wrXiv:2306.15595, 2023.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons,
Hoboken, NJ, 2nd edition, 2006.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang. Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending IIm context window beyond 2 million tokens. In Forty-first International Conference on Machine Learning. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2477, 2024.
Yukum Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document selection for efficient encoder pretraining. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9522-9530, 2022.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint urXiv:2402.10171, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:210J.0m 00027, 2020.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:23006 1644, 2023.
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? In First Conference on Language Modeling, 2024.
Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. Perplexed by quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data. arXiv preprint arXiv:2212. 2022.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Selfextend Ilm context window without tuning. In Proceedings of the 41st International Conference on Machine Learning, pp. 22099-22114, 2024.
Greg Kamradt. Needle a haystack pressure testing llms. https:///ithub com/
gkamradt / /LLMTest] 2023. Accessed: 2025-09-25.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31 1809--09--00 1118809-11888 31826, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1924477412.19437 2024a.
Jiaheng Liu, ZhiqiBai ZhiqiBai, Yuanxing Zhang, Chenchen Zhang, YuangZh YuangZh, Ge Zhang, Wang JinkaiWang, Haoran Que, Yukang Chen, Wenbo Su, et al. E2-llm: Efficient and extreme length extension of large language models. In Findings of the Association for Computa- tional Linguistics: ACL 2024, pp. 1243-4255 2024b.
Xiaoran Liu, Kai Lv, Qipeng Guo. Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. Longwan- juan: Towards systematic measurement for long text quality. In EMNLP (Findings). 2024c.
11

---

Published as a conference paper at ICLR 2026
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's ' guide to training data: Measuring the effects of data age, domain coverage. quality, & toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- fies: Human Language Technologies (Volume I: Long Papers), pp- 3245-3276, 2024.
Guilhenme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouci, and Julien Launay. The refined- web dataset for falcon Ilm: Outperforming curated corpora with web data only, Advances in Neural Information Processing Systems, 36:79155-79172 2023.
Bowen Peng. Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1 1446, 2021.
Daria Soboleva, Faisal Al-Khateeb, Joel Hestness, Nolan Dey, Robert My- ers, and Jacob R. Steeves. Slimpajama: A 627b token, cleaned and dedu- plicated version of redpajama.
https: www. atlboaa.abbbaa../.oooog////// and det.eppp redpa jama
s1 sllmmpajjjp jamajama----
June 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputingg, 568:127063 2024.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint
2019.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. In Forty-first International Conference on Machine Learning, 2024.
Longyun WI, Dawei Zhu, Guangxiang Zhao Zhuocheng Yu, Junfeng Ran, Xiangyu Wong, Lin Sun, and Sujian Li. Longattn: Selecting long-context training data via token-level attention. arXiv preprint arXiv:2502. 16860, 2025.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, and Ngai Wong. Parallelcomp: Parallel long- context compressor for length extrapolation. In Forty-second International Conference on Ma- chine Learning, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How evaluate long-context models effectively and thoroughly. In The Thirleenth International Conference on Learning Representationss 2025.
Yikai Zhang, Junlong Li, and Pengfei Liu. Extending llms' context window with 100 samples. arXiv preprint wxXiv:2401.07000 2024.
Dawei Zhu, Nan Yang. Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of Ilms via positional skip-wise training. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2026
A
REPRODUCIBILITY
All code and scripts used in this work are http://gichbbbccoooHooooaaannnn
publicly
available
at:
This repository contains all necessary files for data preprocessing, model training, evaluation, and visualization.
B
USE OF LLM
During the preparation of this paper, large language model (LLM) was used solely for the following purposes:
Sentence-level polishing of grammar and wording.
Translation.
The LLM was not used to generate original content, draft sections of the paper, or make any scientific claims. The authors take full responsibility for all content the submission.
C EQUIVALENCE OF TWO DEFINITIONS OF CONDITIONAL MUTUAL INFORMATION
The equivalence of Eq. equation 1 and Eq. equation 2 can be shown by expanding the definition of entropy: [(T; 5) H(T 5)
Σ pls, t) log -
p(ss, (ppl,,,e s,e)
Σ e,ts,e,t)loggp + Σ
s,e)
see.
s.ett
Σ p(s, e, (logp(t s,e) logp(t s)) s,e.t
p(t s,e) p(l ()
Σ pls, e, log
(8)
Σr(s, e) Σ p(t s, e. log p(t s)
p(t s)
8,6
S s, e) || p(T S == s))
8,6
Ep(se)
[DKI(p(T] S) || p( S))]
13


Reviews:
[{"review_id": "JzBNEXr9wO", "reviewer": "Reviewer_xifA", "summary": "This paper introduces LongFilter, a data curation framework that selects long-context pretraining data based on the information gain provided by extended context, measured via KL divergence between next-token prediction distributions under short (e.g., 4K) and long (e.g., 64K) contexts. The core insight—that sequence length alone is insufficient to ensure meaningful long-range dependencies—is well-motivated and validated through experiments on LLaMA-3-8B extended from 8K to 64K context. Results on HELMET, LongBench, and RULER show consistent gains (e.g., +2+ average points), with particularly strong improvements in recall (Needle-in-a-Haystack) tasks.", "questions": "* How sensitive is LongFilter to the choice of base model for scoring? Would a smaller or differently trained model yield comparable rankings?\n\n* Could the scoring be approximated more cheaply (e.g., via attention patterns or n-gram statistics) without sacrificing effectiveness?\n\n* Does LongFilter improve reasoning depth (e.g., multi-hop QA) or primarily retrieval fidelity? The current benchmarks emphasize the latter.\n\n* How does performance scale with the selection ratio (e.g., top 10% vs. 20%)? Is there a point of diminishing returns?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Novel and principled metric: The use of conditional mutual information (via KL divergence) to quantify long-range dependency is theoretically grounded and practically effective.\n\n* Strong empirical validation: Experiments across multiple benchmarks and data domains (ArXiv, Books, CommonCrawl) demonstrate robustness. LongFilter outperforms length-based baselines (ProLong) and a recent alternative (LongWanjuan).\n\n* High practical impact: Achieves comparable performance with half the training tokens, significantly improving data efficiency.\n\n* Transparent methodology: The scoring function (Eq. 6–7) is interpretable as a confidence-weighted loss reduction, and token-level visualizations (Figure 5) provide intuitive validation (e.g., low scores for repetitive TikZ code).", "weaknesses": "* Computational cost: Scoring requires forward passes with both short and long contexts using a large model (Llama-3.1-8B), which may be prohibitive for very large corpora despite optimizations.\n\n* Limited model scope: Evaluation is restricted to LLaMA-3-8B; generalizability to other architectures (e.g., Mamba, RWKV) or smaller models is unverified.\n\n* Task coverage: Benchmarks focus on retrieval and structured reasoning; performance on narrative coherence or open-ended generation is not assessed.\n\n* Baseline fairness: ProLong and LongFilter share high-quality data (since ProLong uses unfiltered data that includes LongFilter’s top samples), potentially underestimating LongFilter’s true advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ugtftuXUGw", "reviewer": "Reviewer_hWVa", "summary": "Much readily available long-text data does not genuinely require extended context, as most spans can be predicted using only short-range context. Not all long sequences provide meaningful long-context information, and some long sequences can dilute the training signal. To solve this, the authors propose LongFilter, a framework for curating training data tailored to long-context pretraining. The core idea of LongFilter is to quantify the \"information gain\" provided by an extended context. The method uses a scoring function derived from the Kullback-Leibler (KL) divergence between the next-token prediction distribution given the long context versus the short context. The score is formulated as the reduction in prediction loss when using the long context compared to the short context, weighted by the model’s confidence on the token given the full context.A high score signifies that the extended context is crucial for making accurate predictions.\nThe authors conducted experiments by extending the Llama-3.1-8B model from an 8K context length to 64K. They filtered data from the SlimPajama dataset, selecting the top 20% of high-scoring data for training. Models trained on LongFilter-selected data showed \"substantial improvements\" on benchmarks like HELMET, LongBench, and RULER, achieving average gains of over 2 points. Meanwhile, LongFilter significantly improves training efficiency. Experiments showed that training with 1.5B filtered tokens achieved performance comparable to training on 3-4B unfiltered tokens.", "questions": "The Llama-3.1-8B model was used for scoring. Have you experimented with using other models for this step? How does the choice of the scoring model impact the quality of the selected data and the performance of the final trained model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper show the fact that sequence length alone is an insufficient proxy for data quality. Much long-text data does not genuinely require long-range dependencies, which can dilute the training signal. \n\nThe proposed method, LongFilter, quantifies the informational value of extended context, which is a significant contribution for LLM pretraining.\n\nThe experiments are well-founded. The experiments use Llama-3.1-8B, extend it to a significant context length (8K to 64K) , and train on a large-scale, standard dataset SlimPajama. The method is also compared against relevant baselines like ProLong and LongWanjuan. The results show substantial improvements across a suite of demanding long-context benchmarks, including HELMET, LongBench, and RULER, and demonstrate that training on 1.5B filtered tokens can achieve performance comparable to training on 3-4B unfiltered tokens.", "weaknesses": "The paper highlights the training efficiency gains, but the data curation step itself appears to have a high computational cost. The authors report that scoring each corpus required 32 NVIDIA H100 GPUs for a single day.\n\nThe paper uses the Llama-3.1-8B model as the scoring model to conduct experiments and achieve favorable results. However, the paper does not test other models for generating the scores, which suggests a lack of generalizability regarding the choice of the scoring model in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m5Z8bSDa1F", "reviewer": "Reviewer_bFEo", "summary": "This work focuses a key challenge in training long-context language models (LLMs): much of the available long-text data does not actually require extended context to be modeled effectively.\nTo tackle this, the authors propose LongFilter, a framework that quantifies the information gain from using longer contexts by comparing model predictions under short and long context settings.\nThis allows them to curate pretraining datasets where genuine long-range dependencies are present.\nThey demonstrate that applying LongFilter during pretraining, specifically when extending LLaMA-3-8B’s context length from 8K to 64K, leads to substantial improvements on several benchmarks (HELMET, LongBench, RULER).\nThe paper also provides analyses showing which types of text segments most benefit from long-context modeling.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work addresses an important but often overlooked issue in scaling up LLMs’ context windows, i.e., not all “long” data is useful for learning true long-range dependencies.\n2. The authors propose a clear and interpretable metric (information gain) for identifying valuable training samples.\n3. The paper provides an additional analysis on what kinds of text benefit most from extended contexts, informing future research and practical data selection.", "weaknesses": "1. All the results and analysis are made on one model (i.e., LLaMA-3-8B). It could be better to show the effectiveness on more backbone models.\n2. Beyond the long-context understanding tasks, to further demonstrate the improvements on processing long-range information , it could be better to show the performance on long generation tasks, such as long reasoning and long writing tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xF1wkQ8zGo", "reviewer": "Reviewer_GcpK", "summary": "This paper challenges the prevailing data engineering practice for long-context LLMs, which primarily relies on sequence length as a proxy for quality. The authors argue that many long sequences do not genuinely require long-range understanding, and training on them is inefficient. They introduce LongFilter, a data curation framework designed to select training data based on its \"long-range information\" content.\n\nThe core of LongFilter is a scoring function derived from information theory. It quantifies the information gain provided by an extended context by measuring the KL divergence between a model's next-token predictions under a long context versus a short context. A practical and efficient score is developed based on this principle, which prioritizes sequences where access to a longer context significantly reduces the model's prediction loss. The authors conduct experiments by continually pre-training LLaMA-3-8B to extend its context from 8K to 64K. The results show that training on data selected by LongFilter leads to substantial performance improvements on a suite of long-context benchmarks compared to baseline data selection strategies.", "questions": "The final scoring function (Eq. 7) weights the loss difference (L_short - L_long) by exp(-L_long). This means that tokens on which the model is very uncertain even with long context (i.e., high L_long) will receive a very low score, regardless of the information gain. Did you consider alternative formulations, such as using the raw loss difference L_short - L_long? What is the justification for this specific weighting scheme?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The central thesis—that data for long-context training should be selected based on its actual need for long-range information—is both intuitive and powerful.\n2. LongFilter is elegantly derived from the information-theoretic concept of Conditional Mutual Information, lending it strong theoretical credibility.\n3. The experimental results are compelling, showing consistent and significant improvements across multiple models and benchmarks. The ability to achieve the performance of 4B tokens of training with only 1.5B tokens of filtered data is a strong testament to the method's efficiency.\n4. The proposed filtering method, while computationally intensive to run, is conceptually simple and does not require modifying the model or training process. It can be applied as a preprocessing step to any long-text dataset.\n5. The token-level case study provides insightful qualitative analysis, visually demonstrating that the filter correctly identifies semantically coherent prose as high-value and repetitive or unstructured content as low-value.", "weaknesses": "1. The primary drawback of LongFilter is its high computational cost. The paper notes that scoring a large corpus required 32 H100 GPUs for a day. This is a significant barrier for researchers with limited resources. A discussion on more efficient approximations of the score would make the work more accessible.\n2. The quality of the selected data is contingent on the capabilities of the model used for scoring. The paper uses a very strong model (LLaMA-3.1-8B). It is unclear how performance would be affected if a smaller or less capable model were used for scoring.\n3. The choice of l_short (4K) and l_long (64K) seems somewhat arbitrary. The effectiveness of the filter might be sensitive to these hyperparameters, but this sensitivity is not explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)