EntropyLong: Effective Long-Context Training via Predictive Uncertainty
Authors: Junlong Jia
Ziyang Chen
Xing W
Chaochen Gao
Zijia Lin
Songlin Hu
GuoBinghui
Venue: ICLR 2026 Poster (2026)
Keywords: Longcontext
Primary area: foundation or frontier models, including LLMs

Abstract: Training long-context language models to capture long-range dependencies requires specialized data construction. Current approaches, such as generic text concatenation or heuristic-based variants, frequently fail to guarantee genuine long-range dependencies. We propose \textbf{EntropyLong}, a novel data construction method that leverages predictive uncertainty to verify dependency quality. Our approach identifies high-entropy positions in documents, retrieves semantically relevant contexts from large corpora, and verifies their utility by assessing whether they reduce prediction entropy. This \textit{model-in-the-loop verification} ensures each dependency represents measurable information gain rather than spurious correlation. We construct training samples with long-range dependencies by combining original documents with these verified contextual supplements. Using FineWeb-Edu and Cosmopedia, we generate a dataset of 128K-length sequences with verified dependencies. Models trained on this data demonstrate significant improvements on RULER benchmarks, particularly in tasks requiring distant information. Following instruction fine-tuning, our models also achieve substantial gains on LongBench-v2, demonstrating enhanced long-context understanding. Extensive ablation studies further validate the necessity and effectiveness of entropy-based verification for long-context training.

Published as a conference paper at ICLR 2026
ENTROPYLONG: EFFECTIVE LONG-CONTEX. TRAIN- ING VIA PREDICTIVE UNCERTAINTY
Junlong Jiat-s Ziyang Chen", Xing Wu Chaochen Gao", Zijia Lin', Songlin Hu² Binghui 2,4,5
School of Artificial Intelligence, Beihang University
Institute of Information Engineering, Chinese Academy of Sciences
TTsinghua University
*Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beijing 'LMIB, NLSDE, Beihang University, Beijing
[jiajunlong guobinghut Jêbuaa edu.cn
(gacchaochen, wuxing, husongl wtw.iiattttninguau..mm org.rr.g.ooo..... cn
1 in, chenzi @ chenzlyangJ
ABSTRACT
Training long-context language models capture long-range dependencies quires specialized data construction. Current approaches, such as generic text concatenation heuristic-based variants, frequently fail to guarantee genuine long-range dependencies. We propose EntropyLong, novel data construction method that leverages predictive uncertainty verify dependency quality. Our approach identifies high-entrope positions in documents, retrieves semantically relevant contexts from large corpora, and verifies their utility by assessing whether verification ensures each
they reduce prediction entropy. This model-in-the-lod dependency represents measurable information gain rather than spurious correla- tion. We construct training samples with long-range dependencies by combining original documents with these verified contextual supplements. Using Fine Edu and Cosmopedia, generate dataset of 128K-length sequences with ver- ified dependencies. Models trained on this data demonstrate significant improve- ments on RULER benchmarks, particularly in tasks requiring distant information. Following instruction fine-tuning, our models also achieve substantial gains on LongBench-v2, demonstrating enhanced long-context understanding. Extensive ablation studies further validate the necessity and effectiveness of entropy- based verification for long-context training.
INTRODUCTION
Processing and reasoning over extensive information spans represents a fundamental challenge for
Large Language Models (LLMs) (Huo et al. 2025 Liu al.) 2024a Yang et al. 2025). While
architectural innovations such as Longformer (Beltagy et al. (0220), Big Bird (Zaheer et al. 2020).
and rotary position embeddings (Su et al. 2024 Peng al. 2023 [ al. 2024) have expanded
theoretical context windows to millions of tokens, a fundamental bottleneck persists: the scarcity of training data that genuinely enables effective utilization of this extended context (Liu et al., 2023 An et al., 2024). The predominant practice (Chen et al. 2023) of concatenating short, unrelated documents frequently fails to establish meaningful long-range dependencies. resulting in models with extended attention spans but limited capability to leverage them effectively.
To address this data scarcity, recent research (Gao et al. 2024 2025) has focused on synthesizing long-context training examples through heuristic principles. Query-centric methods like Quest (Gao et al. 2024) construct sequences by retrieving semantically related documents for topical coherence. discrimination-drivee approaches like NExtLong (Gao et al. 2025) interleave relevant documents with distractors to enhance dong-distance information discrimination. Taskedriven synthesis (Wei ctt al. 2024; Nadāy et al. 2025) generates data for specific long-contens skills, while self -elf-ggeeratee synthesis (Li et al. 2024) leverages LLMs create their training data. Despite significant

---

Published as a conference paper at ICLR 2026
Step1 Adoptive Threshold-Bases High-Entropy
Query
Quary 2
Query
Position Selection
{-x) Root Document
Threshold
N-11
Eneray
Low Entropy Token
High Entropy Token
Step2 Iffrmaation-Therrete Context Retrieval
Chunks
Chunis
Quary- Buary-Bused Search
Chunks
Step3 Entropy Reduction Verification
Concarenation
Encray
Abondon
Eneray
Shuffle
Threshold
Entroy
Keep High
Step4 Strategic Concatenation
Root Document
Figure 1: Overview of the EntropyLong framework. Step 1. Adaptive Threshold- High- Entropy Position Selection: Identify high-entropy tokens (red) exceeding the adaptive threshold and generate queries for uncertain positions. Step 2. Information- Theoretic Context Retrieval: Retrieve relevant document chunks from large corpora using query-based search. Step 3. Entropy Reduction Verification: Verify whether retrieved chunks reduce entropy retain successful chunks (green checkmark) and discard ineffective ones. Step 4. Strategic Concatenation: Shuffle verified chunks and concatenate with the root document to create training sequences with validated depen- dencies.
progress, these strategies share a critical limitation: they presuppose what constitutes a "good" long-context example without directly verifying its utility from the model's 's perspective.
This paper introduces LongopyLong, new paradigm for long-context data construction that re- places heuristic-basse synthesis with principled, model-in-the-lon verification. Our key insight is that a model's 's predictive uncertainty (entropy) directly signals an information deficit. High-entropy positions-vv whhre the model struggles to predict the next laken-mate locations where additional context may resolve uncertainty. When introducing relevant information from far earlier in the se- quence reduces such entropy, this creates genuine long-range dependency: accurate prediction requires integrating distant context. This establishes principled connection between information deficits, verified entropy reduction, and effective long-range dependencies, drawing inspiration from active learning (Settles 2009) and uncertainty -uudded data curation (Pang et al.) 2024). EntropyLong operationalizes this insight through four-stage pipeline: (1) identifying high-entropy positions us- ing adaptive thresholding. (2) retrieving semantically relevant contexts for these positions, (3) em- pirically verifying that retrieved contexts reduce prediction entropy, and (4) concatenating verified contexts with original documents to create training sequences with long-range dependencies.
We validate approach by applying this methodology FineWeb-Edu (Lozhkov et al.) and Cosmopedia (Allal et al.). constructing a comprehensive 128K-length training corpus with verified long-range dependencies. Our experimental evaluation demonstrates that models trained with En- tropyLong substantially outperform baseline approaches on RULER (Hsich et al.) (2024) benchmarks across 8K-K context lengths. Following instruction tuning, our approach achieves substantial performance gains on LongBench-v2 (Bai et al. 2024). We further conduct comprehensive ablation studies validate the necessity and effectiveness of entropy-basee based verification.
Our primary contributions are:
1. Entropy Long Framework: We propose novel data construction paradigm using model- in in-the-loop verification to identify high-entropy positions and empirically verify that dis- tant context reduces prediction uncertainty, establishing genuine range dependencies through information-theoreti validation.
2. High-Quality Long-Context Dataset:
We construct and plan to open-source 128K- length training corpus from Fine FineWeb-BB Edu and Cosmopedia, containing samples with veri- fied long-range dependencies that demonstrate measurable entropy reduction.
2

---

Published as a conference paper at ICLR 2026
3. Comprehensive Validation: We achieve substantial performance improvements RULER and LongBench-v2, with ablation studies confirming the necessity and effective- ness of entropy-beet basedd verification for long-context training.
RELATED WORK
Our work integrates two primary research areas: creating data for long-context model training and using predictive uncertainty for data curation.
2.1 DATA CONSTRUCTION FOR CONG-CONTEXT TRAINING
Early long-context training methods (Xiong et al. 2023 Glm et al., 2024) often use curriculum learning, training models progressively longer sequences, for instance by simple document con- catenation. However, this may not create genuine long-range dependencies, leading to more ad- vanced synthetic data generation techniques. Task-Driven Synthesis: This method (Gao et al.) 2025) generates data for specific long-contedd skills. prominent example adapts the "Needle- in-a-Haystacc" (NIAH) (Kamradt] (2023) evaluation create training data that teaches models to find facts in distractor text (Liu et al. (023). Other work synthesizes long-form question-answering datasets to enhance multi-step reasoning (Nadãs et al. (2025). Corerence-Driven Synthesis: This approach (Jiang et al. 2023] Shi et al. 2024; Gao et al. 2024 2025) focuses on creating seman- tically fluent long documents by merging topically related texts. Approaches like Quest exemplify this by generating interconnected texts that demand long-range understanding, assuming topical rel- evance fosters useful dependencies (Gao et al., (024). Self-Generated Synthesis: An emerging method involves an LLM generating its own long-context training data (Li et al. [024). This offers a scalable solution that bypasses data scarcity and licensing issues associated with external teacher models. Perification-Driveee Synthesis: A recent method is REASYN (Zhang al. 2025 which retrieves similar documents and infers dependencies by reordering candidates using perplexity from small proxy model. This doc-level, proxy criterion is misaligned with the target LM's parametric knowledge [ and coarse in granularity. In contrast, Entropyl Long keeps the target model in the loop: it locates token-level uncertainty via predictive entropy and retains only contexts that empirically duce it. This position-level, model-aligned test guarantees measurable information gain, suppresses pseudo-dependencies and redundancy, and scales without permutation search.
2.2 PREDICTIVE UNCERTAINTY AS A GUIDING SIGNAL
A model's predictive uncertainty is foundational signal in machine learning, most notably used in active learning to select the most informative data for human annotation, thereby improving effi- ciency (Settles 2009). The principle is that models learn most from data they uncertain about. In the LLM era, model feedback is vital for data curation, especially
reacher-studen setups for align- ment and instruction tuning (Pang al. | 20244. Uncertainty also guides active preference learning to request human feedback efficiently. While entropy improves reliability in retrieval- -aumeetl- gen- eration, its application to the unsupervised construction of pre-training data is less explored. Some research uses predictive difficulty diagnostically, for instance, to design better evaluation metrics like LongPPL (Fang et al. 2024). In contrast, EntropyLong uses this signal constructively. It creates closed loop where the model's learning state directs the generation of data it needs, grounding cura- tion information- theorreti principles. This eupervuee approach differs from teacher-student models by allowing model to autonomously refine its pre-training data, paving the way for more automated data curation pipelines.
3 THEORETICAL MOTIVATION FOR ENTROPY-DRIVEN DATA CONSTRUCTION
Before detailing our methodology, we establish the theoretical framework that motivates Entropy- Long. We argue that model's own predictive uncertainty is the most direct and reliable signal for identifying information deficits. By leveraging this signal, we can move beyond heuristic-based
As transfer the target I.M.
3

---

Published as a conference paper at ICLR 2026
data construction towards more principled, information coormation-thoortiie approach. Our framework is built upon a core premise, from which we derive a set of guiding principles and testable hypotheses.
3.1 THE PREMISE: PREDICTIVE ENTROPY AS SIGNAL FOR INFORMATION DEFICIT
Our central premise is that high predictive uncertainty a specific position may reflect an informa- tion deficit. The model lacks sufficient context to make a confident prediction. This uncertainty can be precisely quantified using Shannon Entropy (Shannon 1948). Given a language model M with parameters 0, the entropy at position t is:
Hz (F<a)
Σ
Po(vl log Po(u) NEV
where V denotes the vocabulary set of all possible tokens. A position with high entropy, which we term Hlih-UUncetaaie Position, thus serves as a natural anchor point-a location where external context could be maximally beneficial.
This premise shifts the focus from curating data based human-defined heuristics (e.g., semantic similarity) to directly addressing the model's expressed needs. The challenge, then, is to find and verify the context that most effectively resolves this uncertainty.
3.2 THE PRINCIPLE: MAXIMIZING INFORMATION GAIN THROUGH EMPIRICAL VERIFICATION
From our core premise, we derive a guiding principle: effective long-context data should be con- structed by introducing distant information that demonstrably reduces the model's 's predictive uncer- tainty at points of information deficit. We formalize this principle using the concept of Contextual Information Gain.
Given document D with high-entropy position t and candidate context C, the Contextual Information Gain (C, D) measures the relative reduction in entropy after prepending C:
(C, Ha (z, (x))) - Ho( IC:D| <1+C)))
(2)
Ho(x)(=2.)
where x
<ct+|C denotes the context from the concatenated sequence [C:
[C:D]
This principle has two practical implications for data construction:
1. Empirical Verification: We must empirically confirm that candidate context provides a meaningful information gain (i.e., A > <) before accepting as a valid long-range dependency. This verification step is the cornerstone of our approach, ensuring that every constructed dependency is genuinely useful to the model.
2. nnformation-Theoree
Selection: The optimal context C* is not merely one that is topi- cally related, but one that maximizes the information gain: C = arg maXceR (C, D).
This moves beyond similarity metries more direct measure of utility.
3.3 TESTABLE HYPOTHESES
This principled framework leads to set of clear, empirically testable hypotheses that guide experimental validation:
Hypothesis 1 (Necessity of Verification): We hypothesize that the empirical verification of entropy reduction is crucial. Training data built with verified contexts will significantly outperform data constructed using semantic retrieval alone, especially on tasks requiring fineggrineed long-range dependencies.
Hypothesis 2 (Optimality of Thresholds): We hypothesize that the thresholds for iden- tifying high-uncertainty positions and for verifying information gain are critical. Optimal thresholds exist that balance the trade-off between the quality of dependencies and the quantity of available training signals.
These hypotheses are systematically tested in our analysis (Section 6). where we validate the impor- tance of verification (Section 6.1) and the impact of thresholding (Section 6.2)
4

---

Published as a conference paper at ICLR 2026
4 METHODOLOGY
We present the detailed methodology of EntropyLong. Our approach operationalizes the theoretical framework from Section 3 through four main stages: (1) adaptive threshold-based high-entropy position selection, (2) information-theore
context retrieval, (3) entropy reduction verification, and (4) strategic concatenation with root documents. The complete algorithm is provided in Appendix
4.1 ADAPTIVE THRESHOLD-BASSE HIGH-ENTROPY POSITION SELECTION
We use document-specifie entropy statistics to identify high-uncertainty positions. Given doeu- ment D {I1, ..2......n]...... Jn. tive entropy at each position using Equation
pass it through a base language model Me to compute the predic-
We identify high-entropy positions using an adaptive threshold based on the document's entropy distribution:
TH PH 000
where μA and σи are the mean and standard deviation of entropy values within the document, and a is a selectivity parameter (we use a = 2.0 in practice).
Positions where Hg (zy xx))) are marked as high-entropt positions U = {t1, .2,....,.............. )
4.2 PPORMATION-THEORRE CONTEXT RETRIEVAL
For each identified high-entropy position content from the corpus. The surrounding context is composed of w words before and after the high-entropy position:
the surrounding context to retrieve relevant
qi It wst,+
where is the context window size (we use 16 in practice).
Using this surrounding context as query, we retrieve potentially relevant documents from large pretraining corpus R. using dense retrieval. We employ a pre-trained sentence transformer (Sturua al. 2024) encode both the query and candidate documents, then retrieve the top-K most similar documents based on cosine similarity:
embed(4:) embed (dj)
sim dd) lembed (g:)) |embed (dy))
(5)
4.3 ENTROPY REDUCTION VERIFICATION FOR RETRIEVED CONTENT
We empirically verify that retrieved contexts actually reduce model uncertainty. For each retrieved candidate context Cj. we prepend it to the original document and re-evaluate the model's 's entropy at the high-entropy position:
(C,:D ch+lc,
Σ
[C,:D] Po log, Po(u|=
CC,DD
(6)
We only retain context Oj if satisfies the entropy reduction threshold:
(C, (e( (1 | [C,:D] (=24+
Ho Ho 122))
where we use = 0.4.
4.4
STRATEGIC CONCATENATION WITH ROOT DOCUMENTS
For each original document D with verified contexts {Ci, Cz,..., Cm). we construct training quences through strategic concatenation. We employ two main strategies:
Shuffle Strategy: We randomly shuffle the retrieved contexts (which correspond to different high- entropy positions in the original token order) before concatenating them with the root document.
5

---

Published as a conference paper at ICLR 2026
Table 1: Main results on the RULER benchmark. Best results are in bold.
RULER
8k
16k
32k
64k
128k
avg
Quest
91.39
89.72
84.37
77.07
60.11
80.53
NExtLong 89.99
88.58
86.04
85.22 83.52
77.99
Long
91.50
90.11
88.95
85.04
81.26
87.37
Sequence Strategy: We organize the retrieved contexts according to the original token order of their corresponding high-entropy positions in the document.
The final training sample constructed as:
$ = [C>(1)i "i DJ
Cr(m)ii
where т a random permutation {1, 2,... m) for the shuffle strategy, or the identity permutation for the sequence strategy. We use the random shuffle strategy it demonstrates better performance in our experiments.
5 EXPERIMENTS
We conduct comprehensive set of experiments to evaluate the effectiveness of EntropyLong. Our evaluation aims to answer two key questions: (1) Does EntropyLong improve performance on long- context benchmarks compared to existing data construction strategies? (2) Do these improvements during pretraining translate to better performance on downstream tasks after instruction fine-tuning?
5.1 EXPERIMENTAL SETUP
Base Model and Training. We use Meta-LLama-3-- (Dubey et al. 2024) base model, extending the context window to 128K tokens. Following NExtLong (Gao et al. (025). we only modify the RoPE (Rotary Position Embedding) base value 200,000,000 achieve this context extension. The model is trained for 1000 iterations with a global batch size 4M tokens. Complete training hyperparameters
provided in Appendix
Dataset Construction. We construct the EntropyLong dataset using FineWeb-Edu (Lozhkov et al.) and Cosmopedia (Allal et al.). We sample 100K documents source texts and use the full corpora (totaling over IB documents) for retrieval. Our methodology generates 4B tokens of 128K-length sequences, where each sequence contains verified long-range dependencies with average infor- mation gain of A 0.68 per dependency. Implementation details are provided Appendix D
We compare against two leading methods as baselines, using identical Llama-3-8B train- ing configurations and 4B tokens. Quest (Gao el al. 2024): Coherence-driven retrieval and catenation of semantically similar documents. NExtLong (Gao et al. 2025 Discrimination-drieen document interleaving with distracting negative examples.
Baselines.
Evaluation Benchmarks. We evaluate on two main benchmarks: (1) RULER (Hsieh et al. 2024). which includes needle-in-a-haysttcc. multi-hop reasoning, variable tracking, and pattern extraction tasks at 8K, 16K, 32K, 64K, and 128K context lengths; (2) LongBench-v2 (Bai et al. 2024) ter instruction fine-tuning, which tests real-world long-context capabilities across diverse domains. Complete benchmark descriptions are in Appendix
5.2
MAIN RESULTS
5.2.
RULER BENCHMARK RESULTS
As shown in Table I EntropyLong achieves superior performance on the RULER benchmark across different context lengths. Our model consistently outperforms baselines, with particularly strong results at longer context. EntropyLong achieves average score of 87.37 across all context lengths, obviously outperforming Quest (80.53) and NExtLong (85.22). On 128k context length, Entropy- Long achieves of 81.24 1.26, significantly outperforming Quest (60.11) and NExtLongg (77.99).

---

Published as a conference paper at ICLR 2026
Table 2: Results on LongBench-v2 after instruction fine-tuning (UltraChat Ding et al.
2023
Model
Easy
Hard
Short
Medium
Long
Overall
Quest
17.70
25.10
25.60
20.00
21.30
22.30
NExtLong
21.40
25.70
27.20
21.90
23.10
24.10
Long
25.50
28.90
30.00
23.70
31.50
27.60
Table 3: The verification step is critical for performance. We report RULER scores across different context lengths and the average.
RULER
8k
16k
32k
64k
128k
avg
91.44
88.76
86.29
83.85
79.47
85.82
EntropyLong (Full Method)
91.50
90.11
88.95
85.04
81.26
87.37
5.2.2 INSTRUCTION FINE- FINE-TUNING RESULTS
The benefits of our pretraining strategy carry over strongly to downstream tasks after supervised fine- tuning (SFT). As shown in Table 2 our model achieves significant improvements on LongBench-v2 across different task categories. EntropyLong demonstrates particularly strong performance on Long tasks (31.50) compared to Quest (21.30) and NExtLong (23.10) 13), showing significant improvement of 8.40 points over the best baseline. This indicates that the model has learned to effectively locate and utilize information from extensive contexts. These results confirm that building pretraining dataset with empirically verified dependencies leads to more capable long-contes models.
The superior performance can be attributed to the quality of our constructed dependencies. Entropy- Long dataset achieves average information gain of A 0.68 per verified dependency, indicating substantial uncertainty reduction compared to unverified retrieval. Examining the performance gains more closely. we observe that EntropyLong's advantage over NExtLong scales progressively with context length on RULER. This non-uniform scaling pattern provides empirical evidence that our information-theonee theoretic approach data construction translates to performance improvements.
ANALYSIS
This section provides a comprehensive empirical validation of our theoretical framework from Sec- 3ion We examine each of our two core hypotheses through targeted experiments: (1) the necessity of empirical verification for effective data construction (Hypothesis 1), and (2) the existence of opti- mal threshold parameters (Hypothesis 2). Additionally, we conduct detailed ablation studies on key hyperparameters and provide qualitative analysis of the learned attention patterns.
6.1 VERIFICATION NECESSITY (HYPOTHESIS 1)
We test Hypothesis 1 by comparing our full method against "EntropyLongg- Verrif. which skips empirical verification and always accepts the top-retrieved document based on semantic similarity alone. Table 3 strongly supports Hypothesis 1. The full EntropyLong method achieves 87.37 vs. 85.82 for the non-verified version (+1.55 points), with consistent improvements across all context lengths: 16k (+1.35), 32k (+2.66), 64k (+1.19), and 128k (+1.79). The performance gaps demon- strate that empirical verification is essential for effective long-conted training data, as it filters out spurious correlations that may degrade model performance.
6.2 THRESHOLD OPTIMALITY (HYPOTHESIS 2)
We validate Hypothesis 2 by examining how two key threshold parameters affect the balance be- tween data quality and data quantity: the high-entropy position selection threshold (Tн) and the entropy reduction verification threshold (e). Our hypothesis predicts that optimal threshold values exist to achieve the best balance between these competing factors.
High-Entropy Position Selection Threshold (тн) The parameter controls how many high- entropy positions we select: lower values select more positions (more data) while higher values select fewer but more selective positions (higher quality). Table confirms hypothesis. When
7

---

Published as a conference paper at ICLR 2026
Table 4: Performance on RULER benchmark with varying adaptive threshold a values. #Tokens indicates the number high-entropy tokens selected for context retrieval. Higher a values result in fewer but more selective high-entropy positions.
RULER
#Tokens
8k
16k
32k
64k
128k
avg
Long (a = 1.5)
913
90.25
87.16
82.07
79.49
73.47
82.49
EntropyLong (a: 2.0)
292
91.50
90.11
88.95
85.04
81.26
87.37
EntropyL.ong (a = 2.5)
83
90.96
88.29
85.52
82.83
80.02
85.52
Table 5: Performance on RULER benchmark with varying entropy reduction thresholds. #Veri- fied indicates the number of verified dependencies retained after entropy reduction validation. The threshold of 0.4 provides the best empirical performance.
RULER
#Verified
8k
16k
32k
64k
128k
avg
Long (c = 0.2)
62
91.44
88.44 44
84.97
83.53
78.84
85.45
EntropyLong (c = 0.4)
46
91.50
90.11
88.95
85.04
81.26
87.37
Entropyl (e 0.6)
29
90.31
88.64
86.19
84.09
81.46
86.14
EntropyL.ng (e 0.8)
91.39
90.29
87.60
83.90
79.19
86.47
a = 1.5, we get many positions (913 tokens) but performance drops (82.49 vs 87.37) due to noisy selections. When 2.5, get very few positions (83 tokens) with high quality but insufficient training data. The optimal a = 2.0 (292 tokens) strikes the right balance between having enough high-quality training signals.
Entropy Reduction Verification Threshold (c) The threshold e determines how strict we are when verifying that retrieved contexts actually help reduce uncertainty. Table strongly supports Hypothesis 2. With a low threshold (c = 0.2), we keep many contexts (62 verified dependencies) but accept weak ones that barely help, hurting performance (85.45 87.37). With high thresholds (e = 0.6, 0.8), we only keep very helpful contexts but get too few for effective training (29, 13 dependencies). The optimal quality verified contexts for effective training.
0.4 (46 dependencies) gives us the best balance: enough high-
Finding clear optimal values for both (2.0) and (0.4) strongly validates Hypothesis 2. These results show that our method works best when we carefully tune these thresholds to balance data quality and quantity. This confirms that our information-theoreiic approach is both theoretically sound and practically effective.
6.3 IMPACT OF DIFFERENT WINDOW SIZES w
The window size w determines how much surrounding context is used to retrieve relevant documents from the corpus. We tested different values of w to understand its impact on model performance. The window size analysis reveals an interesting non-monotonic relationship between context richness and retrieval effectiveness. While larger windows (w = 16) generally provide richer contextual information leading better retrieval quality, the relationship not strictly monotonic-ar shows superior performance at shorter contexts (8k, 16k) but degrades at longer contexts. This suggests that the optimal window size may depend on the target context length. Our setting of w = 16 achieves the best overall balance across the full range of context lengths.
6.4 IMPACT OF DIFFERENT CONCATENATION STRATEGIES
We compare two different strategies for concatenating retrieved contexts with root documents: sequence-based and random chuffle-based concatenation. The random shuffle strategy slightly out- performs the sequence-based approach. Random shuffling helps prevent the model from learning a positional bias tied to the relative ordering of supplementary documents. As result, the model is forced to search the entire context to locate relevant evidence rather than relying on shortcut posi- tional cues. This reliance global retrieval becomes especially beneficial at long context lengths (e.g.. 32K and 128K), which likely explains why the random shuffle variant yields noticeably larger gains in these settings.
8

---

Published as a conference paper at ICLR 2026
Table 6: Performance on RULER benchmark with varying window sizes w. Larger windows provide more context for retrieval but may introduce noise.
8k
16k
32k
64k
128k
avg
Entropyyong (w 2)
91.12
89.55
88.16
84.49
79.75
86.61
Enng (w = 4)
90.91
88.91
86.96
84.39
79.03
86.04
EntropyLoong (w
92.34
91.01
86.97
83.86
80.06
86.85
EntropyLong (w 16) (current setting)
91.50
90.11
88.95
85.04
81.26
87.37
Table 7: Performance comparison between different concatenation strategies. shows better performance than sequence-based approach.
Random strategy
RULER
8k
16k
32k
64k
128k
avg
EntropyLong (Sequence)
91.55
90.91
87.47
84.91
80.43
87.06
EntropyLong (Random) (current setting)
91.50
90.11
88.95
85.04
81.26
87.37
Table 8: Performance on short text benchmarks. LntropyLong maintains competitive performance on short texts while excelling long contexts.
arc.c
arc.e
hellaswag
piqa
logiqa
winogrande
avg
Llama3-8b-bbe 50.34
80.18
60.13
79.60
27.50 26.27
72.85 72.22
61.77
+EntropyLong
51.45
80.98
59.61
80.47
61.83
6.5 SHORT TEXT PERFORMANCE
To demonstrate that our method does not negatively impact short text performance, we evaluate several short text benchmarks. EntropyLong maintains very similar performance short text tasks (61.83 vs. 61.77 average), demonstrating that our long-context training approach does not compromise overall short capabilities.
6.6 ENTROPY REDUCTION FROM SUPPLEMENTARY DOCUMENTS
10
Entropy Value
8-
Entropy Value
6
4.
2-
w/o Context
w/ Context
w/o Context
w/ Context
Figure 2: Distribution of entropy before and after adding the supplementary document that maxi- mally reduces entropy for each high-entropy token. (a) shows the entropy distribution for tokens that pass verification before and after adding the retrieved document, (b) shows the corresponding tropy distribution for high-entropy tokens that fail verification before and after adding the retrieved document.
We analyze the entropy distribution of high-entropy tokens before and after adding. for each token, the retrieved document that yields the largest entropy reduction, as shown in Figure 2aland Figure
For verified high-entropy tokens, the average entropy before adding any supplementary document is 5.62. After incorporating the document that maximally reduces entropy for each token, the average entropy drops to 1.70, indicating that verified supplementary documents substantially resolve local uncertainty at these positions.

---

Published as a conference paper at ICLR 2026
0.025
X125
Entrepylong NEel.ong
50
Relative Attention Change (%%
Average Attenton Values
0.020
40-
97.0%
0%
2017
0.015
30
8:122
5013
0.010
1.99
8.00
20
0.038
to-
0.000
10
15
20
25
30
0
25
50
75
100
Context Number
Distance to Answer (%)
(a)
(b)
Figure 3: EntropyLong'ss attention patterns analysis. (a) Attention to correct answers vs NExtLong across different context chunks with answers at front; (b) Relative attention vs NExtLong with answers at different positions, where AA represents the attention difference.
In contrast, for high-entropy tokens that fail verification, the average entropy before adding any supplementary document is 4.40. After incorporating the document that yields the largest entropy reduction among the retrieved candidates, the average entropy decreases only slightly to 3.84, in- dicating that supplementary documents provide limited information gain for these positions and do meaningfully resolve the model's uncertainty.
6.7 ANALYSIS OF ATTENTION PATTERNS
To validate the effectiveness of our uncertainty-driven training approach, we further construct ex- perimental sequences by concatenating answer, question, and irrelevant text chunks, then analyze attention patterns correct answers for evaluating long-range dependency capabilities. Figure reveals the differences in how EntropyLong and NExtLong process long-range dependencies.
Context Length Scaling (Figure 3a): We analyze attention allocation patterns by fixing correct answers at the sequence front and comparing attention scores between EntropyLong and NExtLong context length increases. Entropy consistently maintains higher attention correct answers than NExtL.ong, with the attention advantage remaining stable across increasing context lengths, demonstrating superior capability to foeus relevant information regardless of sequence length.
Lost-ietthe-Middle Mitigation (Figure 3bb: We evaluate positional bias by comparing attention scores to correct answers positioned at various locations within sequences. EntropyLong alleviates the "lost-in-the-middle" phenomenon, achieving substantial relative improvements over NExtLong: 12.5% (0%), 37.6% (25%), 44.4% (50%), 99.0% (75%), and 35.2% (100% distance). The peak im- provements at middle positions (25%-75%) 75%) directly validate mitigation of this pervasive limitation in standard long-conters models.
CONCLUSION
We introduce Entropyl.ong, novel data synthesis method that leverages predictive uncertainty to construct high-quality long-context training data. Our key insight is that model's uncertainty sig- nals where additional context is most beneficial, enabling construction of training datasets with verified functional dependencies rather than heuristic similarity. Experiments show superior perfor- mance on RULER and LongBench-v2, with ablations confirming empirical verification is essential. Long shifts from heuristic-based to evidence-based data construction, offering a more reli- able path toward robust long-context understanding.
ACKNOWLEDGMENTS
This work is supported by the Brain Science and Brain- like Intelligence Aaaanology-Nationas Sci- ence and Technology Major Project (Grant No. 20212D0201302.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your 11m fully utilize the context. Advances in Neural Information Processing Systems, 37:62160- 62188, 2024.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang. Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning realistic long-context multitasks. arXiv preprint arXiv:2412.1520 15204, 2024.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint www.:xi004.0555550 2020.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. 12307, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprini arXiv:2507.0626. 2025.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint 2xX.::2402..3733.. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Lizhe Fang, Yifei Wang. Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What wrong with perplexity for long-context language modeling? arXiv preprint arXhv:2410.33711,00
Chaochen Gao, Xing Wu, Qi Fu, and Songlin Hu. QUEST: Query-centric data synthesis approach for long-context scaling of large language model. arXiv preprint arXiv:2405. 19846, 2024.
Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. NExtLong: Toward effective long-context training without long documents. arXiv preprint arXiv:2501.12766 2025.
Glm, Zeng, Xu, Wang, Zhang, D Yin, D Zhang, D Rojas, Feng, Zhao, el. Chatglm: A family of large language models from glm-1300 130b to glm-4 all tools. arxiv 2024. arXiv preprint arXiv:2406. 12793, 2024.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint 44Xiv:2404.06654 2024.
Bi Huo, Bin Tu, Cheng Qin, Da Zheng, Debing Zhang, Dongjic Zhang, En Li, Fu Guo, Jian Yao, Jie Lou, et al. dots. Ilml technical report. arXiv preprint 2XXv:2506600566, 2025.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt com- pression. arXiv preprint arXiv:2310.08839, 2023.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion scale similarity search with gpus. JEEE Transactions Big Data, 733):535-547, 2019.

---

Published as a conference paper at ICLR 2026
Gregory Kamradt. Limtest Limtestnnedlennnhhasstte.. GitHub repository, 2023.
Urvashi Khandelwal, Omer Levy, Dan Jurafšky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint 1rXv:1911.00072 2019.
Jerry Li, Subhro Das, Aude Oliva, Dmitry Krotov, Leonid Karlinsky, and Rogerio Feris. Long context understanding using self- elf-generated synthetic data. In First Workshop on Long- Cong-Context Foundation Models@@ ICML 2024, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024a.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling unbounded n-gram language models trillion tokens. arXiv preprint arXiv:2401. 173777 2024b.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni. and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03112, 2023.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024.
Mihai Nadãs, Laura Diosan, and Andreea Tomescu. Synthetic data generation using large language models: Advances in text and code. IEEE Access, 2025.
Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. Improving data efficiency via curating Ilm-driven rating systems. arXiv preprint arXiv:2410.1087 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007/, 2023.
Burr Settles. Active learning literature survey. 2009.
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representations, 2024. heLLhhts: :tttps://ope
n FozumP1G--LVVssIIHHO
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang. Markus Krim- mel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina-embeddings- v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409. 10173, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputingg 568: 127063, 2024.
Junfeng Tian, Da Zheng, Yang Chen, Rui Wang, Colin Zhang, and Debing Zhang. Untie the knots:
An efficient data augmentation strategy for long-context pre-training in language models. In Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1223-1242, 2025.
Jerry Wei, Chengrun Yang, Xinying Song. Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, et Long-form factuality in large language models. Advances in Neural Information Processing Systems, 37:80756-80827, 2024.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 16039, 2023.
12

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 00Xiv:2500500388, 2025.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences, Advances in neural information processing systems, 1517283--7297, 2020.
Zhiyang Zhang, Ziqiang Liu, Huiming Wang. Renke Shan, Li Kuang. Lu Wang, and De Wen Soh. Re3syn: A dependency-based data synthesis framework for long-context post-training. In Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31468-31480) 2025.
13

---

Published as a conference paper at ICLR 2026
Lotn Acc:
10
90.0
80.0
8
70.0
Depth Percent
60.0
50.0
Score
40.0
4
30.0
20.0
2
10.0
0.0
0
90k
131k
Token Limit
Figure 4: EntropyLong's performance on the geedle-i--a-hasstae task within 1 128K context win- dow. The heatmap shows accuracy across different text lengths and needle positions, with darker colors indicating higher accuracy. EntropyLong achieves perfect accuracy across all configurations, successfully locating the target information regardless of position within the context.
A SHRDLE-IN-A-HAYSTTEE
ANALYSIS
To further validate EntropyLong' effectiveness in inn-ersstanttudddesndnnnnttt imental validation on the classic -Needle-in-a-Hasstee task (Kamradt) 2023). This task requires the model to accurately locate and extract specific information within large amounts of irrelevant text, serving as an important benchmark for evaluating long-context model performance.
conducted exper-
As shown in Figure Entropy demonstrates exceptional performance the needle-in-a- haystack task. The experimental results reveal:
Perfect Accuracy Across All Positions: EntropyLong achieves 100% accuracy in locating the target information regardless of where it appears within the 128K context window, from the very beginning to the end of the sequence.
Length Robustness:
The model maintains perfect performance across different context lengths within the 128K window, demonstrating that our uncertainty-driven training approach effectively handles varying sequence lengths without degradation.
Position Independence: Unlike traditional long-con long-context models that often suffer from the "lost- in-the-middle" problem, EntropyLong shows no performance drop when the target information is positioned in the middle of long sequences, indicating robust attention mechanisms across all posi- tions.
These results confirm that EntropyLong's incertainty-driver data construction method enables the model learn effective long-range dependencies, resulting in superior performance on tasks requir- ing precise information retrieval from extended contexts.
B ALGORITHM DETAILS
This section provides the complete implementation details for reproducing our EntropyLong frame- work.
After selecting dependency positions under different thresholds, we obtain set of verified sup- plementary context segments {C,} along with their corresponding root document D. To strictly maintain total sample length of Luagett 128K tokens, we allocate and fill context budget as follows:
(1) Compute the remaining budget for supplementary context. For each root document, compute the maximum budget available for its supplementary contexts:
Baup
Tearget
|D],
14

---

Published as a conference paper at ICLR 2026
where [D] denotes the token length of the root document and Bupp is the total length available for all supplementary context segments.
(2) Sort supplementary segments by entropy reduction. We rank all candidate segments in de- scending order according to their entropy-reduction value the target high-entropy positions, so that the most informative segments are added first.
(3) Fill the budget segment by segment. We iterate through the sorted list and add segments one by one, updating the remaining budget after each addition.
adding the next segment would exceed Bxup, we stop and discard the remaining segments, en- suring that the final sample does exceed Learget 128K tokens. Finally, if the total length of "all available supplementary context + root document" is still below Lurget (i.e., not long enough to form sufficiently long training instance), discard the sample entirely and do not include it in the training set.
Al Algorithm D provides the complete pseudocode for our Entropy data construction framework. The algorithm implements our four-stage pipeline: identifying high-entropy positions through adap- tive thresholding, retrieving semantically relevant contexts, empirically verifying entropy reduction, and constructing training sequences through strategic concatenation. The key innovation lies in the verification step, where ensure that each retrieved context demonstrably reduces prediction uncertainty before inclusion in the final training data. This principled approach distinguishes En- tropyLong from heuristic-based methods by grounding data construction decisions in measurable information-theoretie criteria.
Algorithm EntropyLong Data Construction
Require:
Training corpus DD (D1, Dz...., Dx), retrieval corpus R. base model Mo. target
length Luaget
Ensure: Synthesized long documents S
1: Initialize +-0
2: for each document D D do
3: H. + Conmputentropy((D M4) {Compute entropy for all positions)
4: TH u1 Ndw (Aduptive threshold}
5: (1:
Th} {Identify uncertainty points]
6:
Cverified 0
7:
for each uncertainty point t U do g+ -r-wrt-w {Extract query context)
8:
9:
Ceandidates + Retrieve(q, R, K) {Retrieve top-K candidates] for each candidate if C' & Cverified then (Check for duplicates) AT CerifylnfommaiinGGaanCCCCC D, Mo)
Ceandidates do
11:
12:
13:
if A > e then
14:
Everified t Cverifed U {C}
15:
break (Use first verified context for this position)
16:
end if
17:
end
18:
end for
19:
end for
if [Creritee then
21: S S ConstrotSSquenncCCCaantt D, Lerger)
22:
S
23:
end if
24: end for
25:
26: return
15

---

Published as a conference paper at ICLR 2026
Table 9: Complete training hyperparameterss for 128K model.
Parameter
Value
Initial Model
Meta-Llamaa3-81 (Dubey et 2024
rotary-emb-base
200,000,0000
31
0.9
(8
0.95
Learning Rate
4 x 10
Precision
6
Gradient Clipping
1.0
Weight Decay
0.1
LR Decay Style
cosine
Training Iterations
1000
Warmup Iterations
200
Sequence Length
131,072
Global Batch Size
4M tokens
C TRAINING CONFIGURATION
Table 19 presents the complete training hyperparameters used for our 128K context length mod- els. extend the Meta-Llama-3-SB base model by modifying only the RoPE base parameter to 200,000,000 to accommodate the extended context window. Our training uses global batch size of 4M tokens with cosine learning rate decay and gradient clipping to ensure stable training on long sequences. These hyperparameters ensure fair comparison with baseline methods by following NExtl configuration.
D IMPLEMENTATION DETAILS
This section provides the complete implementation details for training with data generated by En- tropyLong.
We detail all hyperparameters used our entropy-driven data construction pipeline, including the adaptive thresholding mechanism, context retrieval system, and verification process. Additionally, we specify the configuration used for instruction fine-tuning to ensure complete reproducibility of our experimental results.
EntropyLong Hyperparameters:
Query window size: w 16 words
Retrieval candidates:
Entropy reduction threshold: 0.4
Adaptive threshold parameter: 2.0
Sentence transformer: jina-embeddings-v3 Sturua al. 2024
Dense retrieval index: Faiss (Johnson et al.
2019) with 768-dimensiom embeddings
Instruction Tuning Configuration:
Dataset: UltraChat (Ding al. 2023
Learning rate: 2 x 10-*
Batch size: 2M tokens
Training Iterations: 250
COMPUTATIONAL COST ANALYSIS
Table 10 summarizes the cost breakdown for both methods under the full 4B-token training setup.
We decompose the cost of long-context pretraining into two stages: (1) data construction and (2) model training, and compare EntropyLong against NExtLong, the strongest baseline in our exper- iments. For fair comparison, fully reproduce NExtLong's data construction pipeline. In the
16

---

Published as a conference paper at ICLR 2026
Table 10: Computational cost breakdown for NExtLong and EntropyLong. "GPUh" denotes GPU hours; both models are trained on 4B tokens.
Method
Embed
High-Ent.
Verify
Train
Total
NExtLong
928 GPUh
768 GPUh
1696 GPUh
EntropyLong
928 GPUh
3 GPUh
352 GPUh
768 GPUh
2051 GPUh
Table 11: RULER performance when NExtl ong and Entropyl are matched under similar total GPU cost, NExtLong is trained on 6B tokens equalize total GPU hours with EntropyL.ong (trained on 4B tokens).
Method
8k
32k
64k
128k
avg
Total
NExtL (6B tokens)
91.97
90.52
85.81
81.88
78.60
85.75
2080
EntropyLong (4B tokens)
91.50
90.11
88.95
85.04
81.26
87.37
2051
data construction stage, EntropyLong incurs approximately 355 additional GPU hours relative to NExtLong, mainly due to per-candidate entropy-reduction verification.
However, reach the same RULER performance as NExtLong, Entropy requires substantially fewer training tokens (2.4B vs. 4B), reducing the training cost by 307 GPU hours. This shows that EntropyLong more eefeneefficient than NExtLong at matched performance.
We also examine a setting where both methods use a similar total computational budget. When creasing NExtLong'ss 's training tokens 6B so that its total GPU cost reaches 2080 GPU hours, com- parable to EntropyLong's 2051 GPU hours, its RULER performance reaches 85.75, which remains lower than EntropyLong's 87.37 under similar cost. As shown in Table IT] EntropyLong achieves higher performance at longer context lengths (32k to 128k). The results show that, when aiming for better performance- typically needs more computational resources resources-Entropyy EntropyLong demon- strates higher computational efficiency compared to previous methods.
BENCHMARK DETAILS
RULER Benchmark Tasks:
(NIAH): Four retrieval-based tasks including Single NIAH (vanilla needle retrieval), Multi-keys NIAH (retrieving one needle among multiple distractors), Multi-values NIAH (retrieving all values with the same key), and Multi-queries NIAH (retricving multiple distinct needles)
Variable Tracking (VT): Multi-hop tracing task that evaluates coreference resolution by tracking variable assignments through linear chains (e.g., X2 X1, X3 = X2) and returning variables pointing to the same initial value
Common Words Extraction (CWE) and Frequent Words Extraction (FWE): Aggre- gation tasks requiring models to identify the most frequent words from long sequences, serving as proxies for summarization capabilities where relevant information spans large portions of context
Question Answering (QA): Real-world adaptation of NIAH where golden paragraphs con- taining answers are inserted among distracting paragraphs from the same dataset, with questions serving retrieval queries
LongBench-v2 Tasks:
Single-Document QA: Question answering across six domains (Academic, Literary, Le- gal, Financial, Governmental, Detective) plus event ordering tasks, testing comprehension of individual long documents
QA: Cross-document reasoning tasks requiring information synthesis from multiple sources across Academic, Legal, Financial, Governmental, and Multi-news domains
17

---

Published as a conference paper at ICLR 2026
Table 12: Comparison between UTK and EntropyLong on the RULER benchmark.
Method
8k
16k
32k
64k
128k
avg
UTK 89.51
84.66
80.23
77.84
71.98
80.84
Eong
91.50
90.11
88.95
85.04
81.26
87.37
Table 13: RULER results for different verification thresholds a under similar verified token num- bers.
Method
8k
16k
32k
64k
128k
avg
0.4
90.73
88.08
85.63
82.16
77.93
84.91
0.6
89.48
88.16
84.91
83.05
80.82
85.18
Long In-context Learning:
Three challenging tasks including user guide QA, new lan- guage translation (Zhuang and Kalamang), and many-shot learning with anonymized labels for classification tasks
Long-dialogue History Understanding: Analysis of extended conversation histories in- cluding agent interaction games (GAMA-Benchh and multi-turn chat sessions (Long-
MemEval) requiring memory of distant context
Code Repository Understanding: Deep comprehension of large codebases requiring inte- gration of multiple code components and understanding of complex software architectures Long Structured Data Understanding: Reasoning over extensive tabular data (financial reports) and large-scale knowledge graphs requiring multi-hop logical inference across in- terconnected entities
G ADDITIONAL RULER RESULTS WITH UTK BASELINE
For completeness, we also compare EntropyLong against the UTK Tian et 2025
baseline on the RULER benchmark. Table 12 reports scores at different context lengths. Entropy Long consis- tently outperforms UTK across all lengths, with especially large gains 16k- 128k, highlighting the advantage of our intropy-verifice dependency construction for very long contexts.
H EFFECT OF HIGHER VERIFICATION THRESHOLD a UNDER SIMILAR VERIFIED TOKEN NUMBERS
To verify our claim that the optimal context C is the one that maximizes information gain while controlling for the effect of data quantity, we maintain an approximately constant number of verified tokens.
We then increase the verification threshold obtain more strictly filtered subset, while keeping the number of verified dependency relations roughly similar to those identified under the original configuration. This results in dataset of approximately 2.4B tokens, on which train models with 0.4 and a 0.6 and evaluate them on RULER (Table
We observe that, under a comparable token budget, using higher verification threshold (a = 0.6) yields larger gains longer context lengths (64k and 128k), confirming that more strictly verified dependencies can improve very-long-contexx performance.
THE Q-E TRADE-OFF AND ITS IMPACT ON MODEL PERFORMANCE
To further analyze the joint effect of a and e, we track RULER performance at different training steps for three configurations: high-a, low-e setting (a 2.5, 0.2), our final balanced setting (a = 2.0, e = 0.4), and a low-a, high-c setting (a = 1.5, e = 0.8). The high-a, low-c setting yields large set of weakly verified positions. This results in fast initial improvements but also leads to early saturation. In contrast, the other two settings produce similar improvement curves over time, with the balanced setting achieving consistently better final performance.
18

---

Published as a conference paper at ICLR 2026
Table 14: Training dynamics of different (ar, e) configurations on RULER at steps 200, 600, and 1000 (corresponding to 0.8B. 2.4B, and 4B tokens, respectively).
Step / Setting
8k
16k
32k
64k
128k
avg
Step 200 (0.8B tokens)
a = 2.5, c 0.2
90.58
88.95
85,43
80.94
78.74
84.93
2.0, 0.4
90.90
87.03
83.13
77.82
72.99
82.37
1.5, 0188
91.33
89.57
82.75
79.22
70.13
82.60
Step 600 (2.4B tokens)
2.5, E 0..2
91.20
89.71
87.13
84.46
79.80
86.46
a 2.0, c
90.73
88.08
85.63
82.16
77.93
84.91
a = 1.5, E 0.8
88.87
87.63
85.57
81.97
79.33
84.68
Step 1000 (4B tokens)
0 2.5, c 0.2
91.75
89.26
87.09
83.65
78.89
86.13
2.0, 0.4
91.50
90.11
88.95
85.04
81.26
87.37
1.5, 0.8
91.27
89.63
87.02
83,99
80.55
86.49
Table 15: Performance comparison between sequence-based and front-loaded supplementary con- text insertion across different context lengths (RULER scores).
Method
8k
16k
32k
64k
128k
Sequence
92.00
91.17
85.10
83,26
78.25
Front
91.69
88.33
86.42
84.14
79.75
This comparison highlights the trade-off between and both extremely loose and extremely strict thresholds (high a with low c, or low a with high c) eventually converge to similar effectiveness, while the balanced setting (a 2.0, 0.4) achieves the best overall performance under compa- rable token budget.
EFFECT OF SUPPLEMENTARY CONTEXT INSERTION POSITION
To examine how the insertion position of supplementary documents affects performance, duct controlled experiments on synthetic long-context sequences. For each root document, we first construct a 64K-token sequence by keeping most the top two entropy-reducing supplementary documents for each dependency and concatenating them with the root document. We then con- catenate such 64K sequences to form 128K context containing root documents and their corresponding supplementary contexts.
We compare two concatenation strategies:
(1) Sequence: for each root document, build two 64K samples via our shuffle strategy and concatenate them sequentially:
'supp context shuffle strategy] [root dac1] [supp context 2 shuffle strategy] [root doc2).
(2) Front: we shufile all supplementary contexts corresponding to the two root documents and place them at the beginning, followed by the two root documents.:
[supp context shuffle strattgy [oc] [root doc2].
As shown in Table 15 the trend suggests a shift in what helps the model depending on context length. In shorter ranges (8K-16K), keeping supplementary context close to its corresponding root document (sequence) provides clearer local cues and yields stronger performance. In longer ranges (32K-128K), front-loading the supplementary information (front) allows the model to better ag- gregate globally relevant signals over long spans, alleviating the difficulty of attending to distant supporting content.
19

---

Published as a conference paper at ICLR 2026
Table 16: LLM-based categorization of relationships between verified contexts and high-entropy tokens (1,036 items).
Category
#Items
Proportion
Positive signal (overall)
762
73.55%
Semantic claboration
268
25.87%
Topical consisteney
256
24.71 %
Contextual background
146
14.09%
Factual support
84
8.11%
Narrative continuation
6
0.58%
Syntactic linking
0.19%
Negative signal (overall)
274
26.45%
Unrelated
265
25.58%
Contradictory
0.87%
Table 17: Annotation of the primary level at which entropy is reduced for verified high-entropy tokens.
Reduction Level
#ltems
Proportion
Semantic- reduction
717
71.70%
Lexical-level reduction
257
25.70%
Syntactic- reduction
26
2.60%
K LLM-BASED ANALYSIS OF DEPENDENCY RELATIONSHIPS
To further understand the nature of the dependencies constructed by EntropyLong, perform LLM-based qualitative analysis of the relationship between verified contexts and their associated high-entropy tokens. We sample 1,036 high-entropy positions and their verified contexts and use Gemini 2.5 Flash (Comanici et al. 2025) to annotate which type of relationship expressed. The specific prompt used for annotation is shown in Figure 5
Table [6 summarizes the annotation statistics. Overall, 73.55% of cases are judged as positive signals, indicating that the verified contexts meaningfully support prediction at the high-entropy po- sitions. Within these, the most common patterns are Semantic Elaboration (25.87%), where the context expands
refines the local meaning, Topical Consistency (24.71%), where the context maintains the same topic and reinforces coherence, and Contextual Background (14.09%), where broader background or legal/historical information is supplied. The remaining positive cases include Factual Support (8)1)%) Narrative Continuation (0.58%), and Syntactic Linking (9.19.) Negative (26.45%) mostly labeled as Unrelated (25.58%), with small fraction Con- tradictory (0.87%), illustrating that the entropy- verification is not perfect but strongly biases toward useful dependencies.
For verified high-entropy tokens, we further ask the LLM to determine the level which entropy reduction primarily occurs. As shown in Table 17 71.70% of cases are labeled as Semantic-level Reduction, where the context clarifies meaning, fills in implicit premises, or disambiguates legal factual references. 25.70% of cases Lexical-level Reduction, where the context narrows down the plausible next words without substantially changing the underlying semantics, and 2.60% Syntactic-level Reduction, where the context mainly constrains grammatical structure. These results suggest that Entropyl predominantly constructs semantic rather than purely lexical or syntactic dependencies.
We also analyze high-entropy tokens for which verification failed, meaning no retrieved context passed the entropy-reduction threshold. For these cases, Gemini 2.5 Flash assigns qualitative causes such as Inherent Diversity (Open Choice) 10%%, where multiple continuations remain equally plausible even with additional information, Transitional or Structural Boundary (36.15%) where high entropy reflects discourse or section transitions rather than missing knowledge, and Informa- tion Gap (12.75%). where the retrieved supplementary context did not pass the entropy-reducctin threshold. These result show that many unverified positions arise from genuine ambiguity or struc- tural factors rather than retrieval failure.
20

---

Published as a conference paper at ICLR 2026
LLM Prompt for Relationship Classification
You expert linguistic and semantic analyzer.
Classify the relationship between:
(1) The original document segment
(2) The retrieved supplementary information
Choose ONE category from:
1. Factual Support
The retrieved content provides factual information (entities, events, numbers, definitions) that directly fills or clarifies the missing information at the high-entropy position.
Narrative Continuation
The retrieved content continues supports the same narrative flow, event sequence, sto- ryline as the high-entropy context.
3. Syntactic Linking
The retrieved content provides structural or grammatical patterns that clarify the syntactic role needed at the high-entropy position.
4. Semantic Elaboration
The retrieved content offers definitional, descriptive, or semantic expansion related the concept near the high-entropy position.
Contextual Background
The retrieved content adds broader background or contextual information (historical,
domain-specific, situational) relevant to understanding the high-entropy position.
6. Topical Consistency
The retrieved content is topically related to the high-entropy context but does not provide direct semantic, narrative, or factual support.
Contradictory
The retrieved content conflicts with contradicts the meaning factual information implied by the high-entropy context.
Unrelated
The retrieved content is irrelevant to the high-entropy context with no topical, semantic, narrative, syntactic connection.
Original document: ( text}
High-entropy word: [word)
Sentence context: (sentence)
Retrieved supplementary information: (retrieved_texi)
Only output the relationship type.
Figure 5: Prompt used for LLM-based classification of relationships between verified contexts and high-entropy tokens.
21

---

Published as a conference paper at ICLR 2026
Original Document (D):
...Statting from a suggestion of Stephen Toulmin and through an interpretation of Neurath's (one of the founders of the Vienna Circle) criticism of Descartes' views science, the paper attempts to outline pattern of modernity opposed to the Cartesian one, that has been obtaining over the last four centuries...
Model State "criticism""
High Entropy: The model is unsure of what specific aspect of Descartes' views Neurath criticized. could be "methodology." "epistmoooo"""" "metaployy"""mmmmtsssiiii "metaphysies." "dualism," "duliss" etc.
Retrieved & Verified Context (D'):
...Neurath disputed the existence of universal truths natural laws. Decisions would never cease toentail a measure of uncertainty and men would always err in the forest of Descartes, without anyhope of ever exiting it. Rationality limited the pain of deciding ex- actly as other methods did andcould not claim to be the way out of the forest. Rationality, though, even if its results did not standthe test of truth...
Model State with Prepended Context ([D'; D)):
Low Entropy at "criticism": The model can now confidently predict "criticism" based on the retrieved context showing Neurath's 's opposition Cartesian rationalism and universal truths.
Top Prediction: ('criticism') } + completing the phrase "Neurath's criticism of Descartes' views science"
Figure 6: An example of contextual uncertainty resolution. Retrieved context about Neurath's 's philo- sophical stance (disputing universal truths and Cartesian rationality) enables the model to confidently predict "criticism" by connecting this background knowledge with the original document's 's reference to Neurath's 's position toward Descartes' views on science.
L ADDITIONAL DISCUSSION: RELATION TO RETRIEVAL-BASED METHODS
Nearest-neighhor language models (Khandelwal et al. 2019) and Infinigram (Liu et al. 2024b) both operate at inference time: they retrieve text fragments that semantically statistically lly similar to the current context order to reduce predictive entropy and directly improve next-token prediction. proxy signal during pretraining construct
In contrast, EntropyLong uses entropy reduction and filter training samples, thereby enabling the model to effectively learn long-range dependencies over extremely long contexts in the course of training.
M QUALITATIVE ANALYSIS: LONG-RANGE DEPENDENCY CONSTRUCTION
To illustrate how EntropyLong constructs meaningful long-range dependencies, we present quali- tative example showing how retrieved contextual information helps the model establish connections across extended contexts. As shown in Figure 6 our method identifies uncertainty positions and provides relevant background knowledge that enables better understanding of distant relationships in the text.
Contextual Knowledge Integration Analysis. This example demonstrates three critical aspects of Entropyl approach: (1) Uncertainty Detection: The model correctly identifies "criticism" as high-entropy position requiring external context, (2) Relevant Context Retrieval: The system successfully retrieves Neurath's philosophical stance on Cartesian rationalism, which provides the necessary conceptual framework, and (3) Synthesis Verification: The retrieved context demonstra- bly reduces entropy by enabling the model to connect Neurath's 's opposition to universal truths and Cartesian rationality with the specific critique mentioned in the original document.
This example illustrates how EntropyLong constructs training instances that require genuine multi- step reasoning rather than simple pattern matching. The model must perform conceptual reasoning (understanding Neurath's philosophical position against Cartesian views) and contextual inference (connecting this philosophical stance to the specific criticism mentioned) resolve the uncertainty.
22

---

Published as a conference paper at ICLR 2026
Such complex dependencies precisely what enable effective long-context understanding in real- world scenarios.
N AI ASSISTANCE DISCLOSURE
We acknowledge the use of large language models (LLMs) to assist with language polishing and grammar checking during the writing process. All core ideas, methodology, experimental design, and scientific contributions are entirely the work of the human authors. The LLM assistance was limited to improving readability and correcting grammatical errors, without contributing to the con- ceptual or technical content of this work.
23


Reviews:
[{"review_id": "xV0aMLYpi4", "reviewer": "Reviewer_dtra", "summary": "This paper proposes EntropyLong, a model-in-the-loop data construction framework for long-context language model training. It identifies high-entropy positions where the model is uncertain, retrieves semantically related contexts, and retains only those that empirically reduce prediction entropy, ensuring genuine long-range dependencies. Using Llama-3-8B with a 128K context window trained on FineWeb-Edu and Cosmopedia, EntropyLong achieves higher scores than Quest and NExtLong on RULER (87.4 avg) and LongBench-v2 (27.6 overall) while preserving short-text performance. Ablations confirm that entropy-based verification is essential and that optimal thresholds balance data quality and quantity, leading to improved long-context reasoning and mitigation of the “lost-in-the-middle” problem.", "questions": "1. How do you ensure that high-entropy tokens truly correspond to informational gaps rather than discourse transitions or stylistic boundaries? Have you examined any qualitative examples of entropy peaks to confirm this assumption?\n\n\n2. The paper treats entropy reduction as evidence of successful dependency capture. Could you clarify whether the reduction tends to occur at the lexical, syntactic, or semantic level? Have you compared this with other dependency metrics (e.g., attention shifts or mutual information)?\n\n3. Does the entropy-based selection bias the resulting corpus toward more homogeneous or easier-to-predict contexts? Have you evaluated the lexical or topical diversity of the constructed dataset versus the raw corpus?\n\n4. Given that each sample requires per-candidate forward verification, how does the total cost compare to standard long-context pretraining? Are there any approximations (e.g., smaller proxy models, cached logits) that could make the process scalable?\n\n5. The experiments focus on a single model (Llama-3-8B). Do you expect the same entropy distribution patterns and improvements to hold for larger or smaller architectures, or for instruction-tuned models?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents a novel application for long-context data construction, offering a model-driven perspective on how to identify and verify informative dependencies across distant text segments. \n\nIts originality lies in connecting information-theoretic uncertainty with practical dataset synthesis, moving beyond heuristic concatenation methods. \n\nThe paper is well-organized and clear, making a complex pipeline easy to follow.", "weaknesses": "1. The central assumption is that high predictive entropy indicates missing long-range information, which lacks strong theoretical grounding. Empirically, many high-entropy tokens correspond to discourse openings, transitions, or rare words rather than genuine dependency points. Thus, the “verified” concatenations may enhance topical smoothness rather than causal linkage.\n\n2. The paper does not analyze what types of dependencies are actually captured after verification (e.g., factual consistency, narrative continuity, or syntactic linking). Without qualitative inspection or causal tracing, the claim of “true long-range reasoning” remains speculative.\n\n3. The framework requires per-sample forward passes for entropy computation and re-verification, which substantially increases computational cost. The paper does not discuss efficiency trade-offs or scaling to larger corpora.\n\n4. Although the paper frames its contribution as constructing long-context dependencies, the retrieval query for each high-entropy token is limited to a 16-token local window. This narrow context restricts semantic scope to near-sentence continuations rather than true cross-document or distant dependencies, effectively reducing the retrieval process to localized topical or lexical matching. As a result, the constructed samples may improve surface fluency but do not necessarily capture genuine long-range information flow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fg8x2PFgnl", "reviewer": "Reviewer_2y86", "summary": "This paper proposes EntropyLong, a novel data-construction recipe for the “long-context continual pre-training” stage of LLM development. Given base model M, training corpus D, and retrieval corpus R, the authors present a procedure consisting of 4 steps:\n1. For each document from D, define high uncertainty tokens (the tokens the autoregressive model M is unsure about)\n2. For each uncertain token: \n\n2.a. Form a query by taking the surrounding context\n\n2.b. Embed the query (jina-embeddings-v3)\n\n2.c. Use this query to retrieve 32 documents from R using cosine similarity\n\n3. Filter documents that reduce the uncertainty of the token (only keep C if it did not appear for this doc before)\n4. Shuffle and merge filtered documents into a context\n\nThen, starting with the M checkpoint (LLaMa3-8B) as the initialization, they continue its training with increased sequence length (8K to 128K) as per standard practice with modified RoPE hyperparameter.\n\nThe authors compare their method to baseline methods: Quest (just retrieve related documents) and NExtLong (retrieve related documents, but also unrelated documents for distraction)", "questions": "Here are several questions for the authors:\n- How are 100K root docs sampled? Uniformly or using some other random strategy?\n- Are the root documents included in the retrieval corpus?\n- How is underflow or overflow in ConstructSequence handled?\n- How is context deduplicated across root documents (sequences)? How do you know there is no retrieved document that is a part of every - single sequence you train on?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The authors clearly explain their strategy, formulate and test relevant hypotheses.", "weaknesses": "The authors claim that dataset release is a key contribution, but no comparison with other datasets is presented in the experiments section. They only compared their method against other published methods.\nOne comparison that could have been done is the comparison with the dataset that was used for training LLaMa-3.1-8B, which has the same context length as the resulting model presented by this work. Another comparison could be made with LongMIT-128K.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3EXQNWBTA", "reviewer": "Reviewer_PDbx", "summary": "In this paper, the authors introduce a methodology for improving the long-context performance of language models by modifying training data based on high-entropy regions. The approach begins by identifying positions in a training document where the model shows high predictive uncertainty, measured through token-level entropy. For each such high-entropy region, semantically similar documents are retrieved using embedding similarity. Among the retrieved candidates, only those that, when prepended to the original document, lead to a measurable reduction in entropy for that region are kept. These verified documents are then added to the beginning of the training document, producing an extended version that contains additional context proven to lower uncertainty.\n\nThe modified corpus is then used to fine-tune a pretrained model with RoPE-scaled positional embeddings to support longer context lengths. Experimental results show clear gains on long-context benchmarks such as RULER and LongBench-v2, confirming that entropy-guided data construction improves a model’s ability to utilize distant context. The paper also includes ablations on the entropy threshold, the necessity of the verification step, and the effect of the retrieval window size.", "questions": "- **Q1**: I am curious about the placement and ordering of the retrieved, entropy-reducing documents. Table 7 suggests that random or sequential shuffling sometimes yields noticeable gains—especially at 32 K and 128 K context lengths—which might indicate sensitivity to document ordering. Do the authors have an explanation for this behavior?\nAdditionally, connecting to works such as Lost in the Middle (Liu et al., 2023) and LDAM (Chen et al 2025), which study how the position of relevant information affects model performance, it would be interesting to see experiments that vary where these documents are inserted. My intuition is that placing them at the beginning helps the model learn to attend to earlier parts of the context, potentially explaining the stronger performance at longer lengths.\n\n- **Q2**: It is somewhat unclear why the authors choose to append multiple retrieved documents rather than focusing on a single or very limited number that produce the largest entropy reduction. Would using only the top document (K = 1) that most strongly reduces entropy yield similar improvements? Exploring smaller K values could help disentangle the effects of information quantity from information placement, and—when combined with varying document position—might offer clearer insights into how LLMs actually utilize extended context.\n\n- **Q3**: I am having a hard time understanding Figure 2 from description and text. Could the authors please provide an explanation of the setup, formulation for the attention pattern analysis and how it relates to the long entropy method ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The experimental setup in this paper is mostly clear, and the idea of using uncertainty in text and explicitly targeting those regions is interesting. The core motivation—to improve model performance by providing additional, helpful information in the training text—is well-founded. By doing so, the approach encourages the model to make better use of context and, directly or indirectly, to rely on information that is placed farther away from the main document, thereby improving its ability to utilize long-range context. Overall, the idea opens the door to several interesting research directions that explore uncertainty-guided data modification and its role in enhancing context understanding in language models.", "weaknesses": "- **W1)** One weakness is the lack of clarity in how high-entropy regions are identified and represented. The paper mentions detecting areas of high predictive entropy, but it is unclear how these local and often volatile entropy values are aggregated into meaningful regions that actually require additional context. In practice, entropy can fluctuate sharply even within short spans—for instance, the first part of a word or phrase often has higher entropy than later parts due to language-level artifacts. Without clear illustrations or examples of how such regions behave during next-token prediction, it is difficult to assess whether they truly capture model uncertainty rather than incidental variation.\n\n- **W2)** Conceptually, the approach still resembles retrieval augmentation, where related text is added to provide more context, and it is not entirely clear whether entropy is the main factor behind the gains. Although the authors compare verified versus unverified document additions, the difference could stem from how the top-k similar documents are chosen or from context-overload effects. It would help to clarify whether documents that reduce entropy tend to rank higher in similarity, and how many are included when verification is skipped. While EntropyLong operates at a finer granularity than RE³SYN, a direct comparison would strengthen the argument that entropy reduction—rather than dependency linking alone—is what drives improvement.\n\n- **W3)**  The paper largely treats high entropy as a sign of poor model performance or uncertainty, but this link is not always straightforward. This claim is somewhat reflected by the language of this paper, specially cases where the authors claim this entropy increase as a theory for poor model performance. Entropy in language can also reflect diversity or ambiguity rather than failure, and its interpretation varies across models and data domains. It would help if the authors discussed whether lower entropy consistently aligns with better predictions or cited supporting evidence for that assumption. From my own experience with large language models, entropy behavior can differ substantially across model families and scales, suggesting that the same thresholds for $\\alpha$ and $\\epsilon$ may not generalize. Including experiments that vary these parameters across different models would clarify how model-dependent the method is. I understand that the selection of such values is document dependant, but I believe the model itself could play a role in this method which explicitly uses entropy  of a language model for targetting points of improvement. \n\n- **W4)** The paper does not provide information or comparisons regarding the computational cost of the entropy-based data curation process. I assume that the lookup and top-k extraction steps could be computationally intensive, and it would be helpful if the authors included at least an approximate cost analysis or comparison with other data construction methods, even if only in the appendix.\n\n- **W5)**  (Minor) The paper does not discuss several closely related approaches that also reuse similar regions of text to improve next-token prediction. Khandelwal et al. (2020, Nearest Neighbor Language Models) retrieve semantically similar examples from the training set using embedding similarity between hidden states, improving predictions by grounding generation in past contexts. In contrast, Liu et al. (2025, Infinigram) rely on n-gram frequency statistics to reuse highly similar text fragments during inference. While these methods apply retrieval at inference rather than training, both demonstrate that referencing similar parts of text can enhance next-token accuracy. Highlighting this connection would make it clearer how EntropyLong extends these retrieval-style ideas into a training-time, entropy-guided framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HcCh3lPn2b", "reviewer": "Reviewer_L2EC", "summary": "This paper tackles the lack of genuine long-range dependencies in long-context training data. It proposes EntropyLong, a novel data pipeline that uses a \"model-in-the-loop\" to verify data quality. The method identifies high-uncertainty (high-entropy) token positions in a document, retrieves relevant context, and verifies that this context reduces the model's prediction entropy at that position. Only verified context is used to construct final training samples. Training a Llama-3-8B model on this data shows significant performance gains on RULER and LongBench-v2 benchmarks over strong baselines, with ablations confirming the necessity of the entropy verification step.", "questions": "1. For each document, you will find prepended contexts by changing two thresholds including high-entropy tokens and effective contexts to reduce entropy. How did you maintain the size of total length to be 128K in your case? If the contexts + original document > 128K, will you just separate as different samples like <context1><doc> and <context2><doc>? If so, does this mean we may train more on the tokens in original document <doc> than other documents?\n2. The proposed threshold will affect the quality and quantity of the data. I think this work can be stronger if we analyze the trade-off between $\\alpha$ and $\\epsilon$ to have similar total token budget but one is high $\\alpha$ and low $\\epsilon$ and the other is low $\\alpha$ and high $\\epsilon$.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The authors proposed a novel idea to create true long-context dependencies in long-context pre-training data. Their method using entropy to select tokens and relevant contexts shows effective results to answer the premise that a good dependency is one that measurably reduces model uncertainty.\n2. This paper demonstrates clear and consistent performance improvements over two baselines (Quest and NExtLong) on RULER and LongBenchv2 benchmarks.\n3. The analysis in section 6 is solid and convincing, providing detailed hyper-parameter selection and validating the paper's core hypothesis.", "weaknesses": "1. The proposed method needs a series of ablation studies to find effective thresholds. If we want to apply to a new model, we need to carefully find high entropy tokens and select dependent contexts. I'd like to see if this dataset can be generalized to other models besides the one (Llama3-8B) used to construct the dataset. \n2. The baselines and the proposed method focus on finding extra documents for each short chunk to create long dependencies. We may need additional baselines by creating long dependencies based on current chunks like CIP[1] or UTK[2]. \n3. In section 3.2 item 2, the authors claim the optimal context C is the one that maximizes the information gain. However, in table 5, we saw the performance doesn't improve if we pick highest information gain contexts. The authors believe the problem is because of the data quantity. Then can we ensure the total tokens are the same with higher threshold to validate this claim?\n\n[1] CIP: LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models \n[2] UTK: Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)