NExtLong: Toward Effective Long-Context Training without Long Documents
Authors: Chaochen Gao
Xing W
Zijia Lin
Debing Zhang
Songlin Hu
Venue: ICML 2025 poster (2025)
Keywords: Long-Context Model
Synthetic Data
Primary area: deep_learning->large_language_models

Abstract: Large language models (LLMs) with extended context windows have made significant strides yet remain a challenge due to the scarcity of long documents. Existing methods tend to synthesize long-context data but lack a clear mechanism to reinforce the long-range dependency modeling. To address this limitation, we propose NExtLong, a novel framework for synthesizing long-context data through Negative document Extension. NExtLong decomposes a document into multiple meta-chunks and extends the context by interleaving hard negative distractors retrieved from pretraining corpora. This approach compels the model to discriminate long-range dependent context from distracting content, enhancing its ability to model long-range dependencies. Extensive experiments demonstrate that NExtLong achieves significant performance improvements on the HELMET and RULER benchmarks compared to existing long-context synthesis approaches and leading models, which are trained on non-synthetic long documents. These findings highlight NExtLong's ability to reduce reliance on non-synthetic long documents, making it an effective framework for developing advanced long-context LLMs.

NExtLong: Toward Effective Long-Context Training without Long Documents
Chaochen Gao Xing Wu
Zijia Lin 3 Debing Zhang
Songlin Hu !2
Abstract
Ng (RAG)
Helpalga (RAG)
(Recall)
Niah mik 3 (Rocal))
Large language models (LLMs) with extended context windows have made significant strides yet remain challenge due to the scarcity of long documents. Existing methods tend syn- thesize long-context data but lack clear mech- anism to reinforce the long-range dependency modeling. To address this limitation, we pro- pose NExtLongg, a novel framework for synthesiz- ing long-context data through Negative document Extension. NExtLong decomposes a document into multiple meta-chunks and extends the con- text by interleaving hard negative distractors re- trieved from pretraining corpora. This approach compels the model to discriminate long-range dependent context from distracting content, en- hancing its ability to model long-range depen- dencies. Extensive experiments demonstrate that NExtLong achieves significant performance im- provements on the HELMET and RULER bench- marks compared to existing long-context synthe- sis approaches and leading models, which are trained on non-synthetic long documents. These findings highlight NExtLong's ability to reduce reliance on non-synthetic long documents, mak- ing it an effective framework for developing ad- vanced long-context LLMs. Our code avail- able in https://github. com/ wtt.://sw...ccc//a...a......./// Dongoontext/trree/maii////aa
Popqa (RAG)
Niah, mn
(Hecall)
Triviaga
(RAG)
Json, kky (Recal))
RULER (RULER)
(ICL)
Tree_ fine (ICL)
Infbench choice
(LongQ
Bunking (ICL)
lafbench
(LongQA)
Click
Nureativequ (LangQA)
(Re-rank)
V2-LLie
Qwen2.5 5 7B
Liuma 3.1 Llama 3 MExtLLnn 512K, Base (ours)
Figure 1. Comparison existing remarkable models and NExt- Long on the METTYYn et al., 2024b) and RULERR(ssich al., 2024) benchmarks. We evaluate various task types classified by HELMET. All results averaged over sequence lengths of 8K. 16K, 32K, 64K, and 128K.
Li et al., 2025). For example, the Llama series models in- crease the context length from 4k in Llama 2 (Touvron et al., 2023b) to 128K in Llama 3.1 (Meta, 2024). The increased context window enables LLM to unlock more challenging tasks, such as Document Summary (Wu et al., 2023b), Long- book QA (Caciularu et al., 2023) and Code Planning (Bairi et al., 2023). To model long-range dependencies, main- stream methods (Fu et al., 2024; Gao al., 2024b) typically continue training existing L.MMs pre-trained on a 4K or 8K context length with long documents that reach the target length, e.g., 128K. However, the scarcity of high-quality long documents in most domains remains a significant challenge, particularly as the target context length continues to increase (Gao et al., 2024a).
1. Introduction
I Large language models (LLMs) have garnered significant attention due to their powerful and versatile capabilities. Recently, the context length of LLMs has been rapidly ex- tended (Peng et al., 2023; AI et al., 2024; Yang et al., 2024;
'nntittte of Information Engineering, Chinese Academy of Sci- ences "School of Cyber Security. University of Chinese Academy of Sciences TTinnguua University "Xiaohongshu Inc. Correspon- dence to: Xing Wu <wuxing Sieixinn@iie.ac.cn... Songlin Hu <hu- songlin acc.ac.cn>.
To address the challenge of scarcity of long documents, isting approaches synthesize long-context data by concate- nating shorter documents. Similarity- based methods, such
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we define "long-context data" as synthetic long- form datasets, and "long documents" non-synthetic long docu- ments that meet the target training length.

---

NExtLong: Toward Effective Long-Context Training without Long Documents
as KNN (Guu et al., 2020; Levine et al., 2021), aggregate the top-k semantically similar short documents into longer document. Other studies (Roziere et al., 2023; Ouyang et al.. 2022; Touvron et al., 2023a) randomly sample and concatenate short documents, often compromising coher- ence and relevance. Recently, Quest (Gao et al., 2024a) aims to balance semantic correlation and contextual diversity by retrieving documents relevant to specific keywords How- ever, those methods typically concatenate short documents based on random or similarity-basel rankings, lacking clear mechanism for capturing long-range dependencies.
Quest (Gao et al., 2024a). Moreover, compared to existing remarkable models trained by long documents, NExtLong achieves extraordinary results, as highlighted in Figure 1. These results demonstrate that NExtLong is a highly ef- fective method for synthesizing long-context data. The synthesized data significantly alleviates the dependence on training large long-context models on long documents and holds the potential train ultra-long context models that are not constrained by the scarcity of long documents.
Our main contributions can be summarized as follows:
An intuitive approach to building documents with long- range dependencies is to insert additional text between de- pendent segments (Tian et al., 2024; Zhao et al., 2024), thereby transforming short dependencies into long-range ones. However, recent studies show that large language models can be easily distracted by irrelevant context (Shi et al., 2023a), and this issue is exacerbated as the context length innn et al., 2023). This raises a critical chal- lenge: how can we enhance a model's 's ability to discriminate long-range dependent information from distracting content within extended contexts?
We propose NExtLong, a simple and effective method that extends the document to strengthen the model's ability to model long-range dependencies.
We provide an in-depth analysis of the key components that contribute to the effectiveness of NExtLong.
Our experiments show that NExtLong achieves signifi- cant improvement across multiple long-context evalua- tion tasks, demonstrating the effectiveness of negative document extension in training long-conted LLMs.
Inspired by the hard negative technique (Robinson et al., 2020; Kalantidis et al., 2020; Zhan et al., 2021) from con- trastive learning, which introduces hard negatives to en- hance a model's ability to discriminate relevant samples from distracting ones, we adapt this concept to create hard negative distractors that reinforce long-range dependency modeling. The key idea to generate negative-extended documents by inserting semantically similar yet distracting texts between dependent fragments. These distractions in- crease the model's learning difficulty, thereby enhancing its capacity to model long-range dependencies. Specifi- cally, NExtLong works by first chunking a document into multiple chunks, termed meta-chunks. We retrieve hard negatives from a pretraining corpus for meta-chunks and interleave them between dependent meta-chunks. Since the pretraining corpus undergoes extensive deduplication, these hard negatives share partial semantic similarities with the meta-chunks but do not replicate their content. By inserting these distractors between originally dependent meta-chunks, NExtL.ong not only increases the distance between depen- dent chunks effecse-ef transforming the dependencies into long-range ones-but also introduces distracting noise, which compels the model to reinforce its ability to discrimi- nate long-range dependent context from distracting content.
2. Related Work
Unlocking LLMs' ability to process long-context tasks. Train-free methods bypass parameter updates for long- context handling. LM-Infinite (Han et al., 2023) employs a A-shaped attention mask with a distance limit for length gen- eralization. StreamingL (Xiao et al., 2023) mitigates the "attention sink" phenomenon by balancing attention scores. Self-Extend (Jin et al., 2024b) introduces group- wise atten- tion to map unseen relative positions, while DCA (An et al., 2024) uses token-wise attention and memory-effciint mech- anisms for effective context extension. Train-based meth- ods enhance performance through continued training. Chen et al. (2023b) extend RoPE-based (Su et al., 2021) LLMs via positional interpolation, and PoSE (Zhu et al., 2023) applies positional skip-wise training to decouple training and target lengths. Recently, upsampling long documents across diverse domains has emerged as critical factor in advancing long-context modeling (Fu et al., 2024; Gao et al., 2024b; Xiong et al., 2023).
However, those train-based methods are dependent on the availability of high-quality long documents, which are scarce in many domains and become increasingly harder to obtain as context lengths grow. In contrast, NExtL overcomes this challenge by extending documents with hard negatives, alleviating the reliance on naturally occurring long documents.
Extensive experiments on the HELMET(Yen et al., 2024b) and RULER(Hsich et al., 2024) benchmarks demonstrate that NExtLong significantly improves the model's 's ability to capture and utilize long-range dependencies. Overall, NExtLong delivers an average performance improvement of 7.33% over the previous long-context synthesis method
Synthesizing long-context texts by concatenating short documents. Past approaches to synthesizing long-context data primarily rely on concatenating short documents.
2

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Stagel: Nogative Document Extension
Hard Negative
Meta- Chunk1
Mining
Neg- text I(U)
Neg- text ((K)
Meta- Chink1
Hard Negative Mining
Chank
Meta- Chunk2
Docurment Extension
Meta- Chunk2
Neg. 2(1)
Neg text 2(K)
documnent
Meta-
Hard Negative Mining
ChunkN
Meta- Chunky
Neg-lext N(1)
Neg- Beegllext N(K)
Stag22: Long-Range Dependence Modeling
Model Long-Range Dependence
Distinguish Hard Negative
Meta-
Neg-texl I(1)
Nog-loxt I(K)
Meta-
Neg- text 2(1)
Neg- text 2(K)
Next Token Prediction
Mota-
Cbunk1
ChurkN-1
Distinguish Hard Negative
Model Long-Range Dependence
Figure 2. The NExtLong method consists of two stages. In the first stage. document is chunked into multiple meta-chunks, and each meta-chunk is mined numerous hard negatives. These hard negatives are then concatenated with the meta-chunks create long document. In the second stage, the model is trained using this synthesized long document, focusing modeling long-range dependencies by identifying meta-chunks across a wide range of hard negatives.
Those methods often lack a mechanism to ensure that the concatenated documents maintain explicit long-range depen- dencies. Some methods randomly sample and concatenate short documents (Roziere et al., 2023; Chen et al., 2023c), while others attempt to preserve semantic coherence by clus- tering similar documents using KNN (Guu et al., 2020). Recent works like Quest (Gao et al., 2024a) try to balance semantic relevance and diversity by retrieving keyword- related documents. However, those methods typically fail hold mechanism effectively model long-range depen- dencies across distant documents. In contrast, NExtLong not only synthesizes long-context data but also explicitly introduces hard negatives between document chunks, which reinforces the model's ability to learn and utilize long-range dependencies.
handle longer contexts. In this work, we adapt hard neg- ative mining for document synthesis by interleaving hard negatives between meta-chunks in NExtL.ong, which helps the model better focus long-range dependent context and improves its ability to process long-context tasks.
3. Method
This section introduces our proposed method, NExtLong, which comprises two stages: Negative Document Exten- sion and Long-Range Dependence Modeling. NExtLong aims enhance long-context modeling by synthesizing extended-length documents. An overview of the approach is shown in Figure 2, and the corresponding pseudocode is presented in Appendix D.2.
3.1. Negative Document Extension
Hard negative technique. Hard negative mining is well- established technique in contrastive learning and dense re- trieval, aimed at improving model discrimination by intro- ducing samples that are semantically similar yet incorrect (Robinson et al., 2020; Xiong et al., 2020). dense retrieval, enables the model to effectively distinguish between rele- vant and irrelevant information by selecting challenging neg- ative samples (Zhan al., 2021; Wu et al., 2023a). While traditionally applied to retrieval tasks, recent studies explore its potential in LLMs, such as Jin et al. (2024a), which in- vestigates the role of hard negatives in Retrieval-Augmented Generation (RAG). However, those approaches have not yet been applied to document synthesis for adapting LLMSs to
The Negative Document Extension stage consists of two steps: document chunking and hard negative mining.
3.1.1. 1. DOCUMENT CHUNKING
We sample document from the training dataset as a meta- document and divide it into sequential meta-chunks. We define the documents to be expanded meta-documents. The meta-dooummntts is divided into several chunks accord- ing to certain chunking granularity. These chunks are defined as meta-chunks. To ensure sentence integrity, we define a maximum length s as the chunking granularity. The
3

---

NExtLong: Toward Effective Long-Context Training without Long Documents
chunking process follows two-step approach:
context length. Details for calculating k are provided in Appendix D.I.
1. Splitted by newline: The meta-document is first split- ted into paragraphs based on newline characters (n), preserving the coherence of each paragraph.
Finally, we synthesize a long document t by concatenating the extended chunks:
t= 11.12.....
(5)
2. Form chunks:
These paragraphs are concatenated se- quentially to form meta-chunks my until the cumulative length reaches the maximum length If adding another paragraph exceeds this threshold, the current group is finalized as a complete meta-chunk, and the process con- tinues with the remaining text. The effect of chunking granularity s is analyzed in Section 5.3.
3.2. Long-Range Dependence Modeling
In alignment with the pretraining stage, we employ next token prediction (NTP) loss (Radford, 2018) to extend the
context length of the base model. The loss function is de- fined as:
In this way, the meta-document is divided into p meta- chunks:
Loss Σ log (22,4122,,22,.....
(6)
chank {m,, M2........ .... mp)
(1)
The key distinction of NExtLong lies in the differentiation of tokens during training. The tokens in the synthesized long document t are classified into two categories: meta- chunks m, and hard negatives niy, Together, they form an extended chunk 4. For simplicity, we use my, ni,, and 1 to denote the encoded tokens of meta-chunks, hard negatives, and extended chunks, respectively. The loss function can thus be reformulated as:
The number of meta-chunks y depends on the length of the meta-document r and the chunking granularity s.
3.1.2. HARD NEGATIVE MINING
To obtain distracting texts as hard negatives for each meta- chunk, we build a FAISS index from the pretraining dataset, which undergoes extensive deduplication. Unlike methods that treat entire documents indivisible units, we also divide each document in the pretraining dataset into smaller chunks based on the same granularity s. This chunking enables more fine-grained and efficient content retrieval during the extension process. Formally, each document d; in the pretraining corpus is divided into q chunks:
Loss
log P( [mg R1, migs....
log P( l2,
de chank
Cigy
The NTP loss encourages the model distinguish relevant meta-chunks from surrounding hard negatives and model long-range dependencies effectively. In Section 4 and Sec- tion 5, we empirically demonstrate that incorporating hard negatives in the loss function improves the model's 's ability to model long-ran range dependencies across extensive contexts.
Each chunk is indexed individually for precise and efficient retrieval. We compute the embedding vector e, for each chunk and insert into the FAISS index:
{Ci,s Cia..,,..... ) project
{ei, Cly........ Ci,) index FAISS
(3)
After building the FAISS index, we retrieve the top-k most similar chunks as hard negatives ni, for each meta-chunk m. These hard negatives are then concatenated with the meta-chunk to form an extended chunk Z;:
4. Experiments
In this section, evaluate the effectiveness of NExtLong by comparing it with other data synthesis methods (Section 4.2) and state-of-the-art (SOTA) models (Section 4.3),
[, Blg Bigs. Mig
(4)
4.1. Experimental Setups
We conduct ablation experiments on the position of meta- chunks (Appendix B.),, confirming that placing the meta- chunk before the hard negatives yields better performance. The number of hard negatives, i.c., kt depends on the length of the neca-doc,m,nt, chunking granularity s, and target
Datasets We select two commonly used pretraining datasets composed entirely of short documents (Refer to Appendix B.2 for document length distribution): Cosmope- dia v2 (Ben Allal et al., 2024) and FineWeb-EEde (Lozhkov et al., 2024). Both datasets are used for the main experi-

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Table Comparing NExtl onng with other data synthesis methods on HELMET and RULER benchmark. All results averaged over
sequence lengths of 8K,16K.32K,6o 64K, and 128K. O results from Yen et al. (2024b): 4 results evaluated by ourselves.
Model
Max Len
Avg.
Recall
RAG
ICL
Re-rank
LongQA
RULER
Meta-L Sott--Llama---S--bbs &
8K
13.37
18.00
12.68
13.60
7.74
10.38
17.80
Standard +
128K
52.85
62.33
58.67
71.24
19.18
28.99
76.68
KNN +
128K
50.97
64.24
56.00
60.28
18.77
32.27
74.30
ICL.M +
128K
50.37
64.04
54.48
72.36
14.04
28.17
69.14
Quest
128K
55.25
69.13
57.47
72.08
22.35
33.82
76.63
+ NExtLonng (ours) +
128K
62.58
82.56
60.91
81.76
31.47
37.30
81.50
NExtLong
Standard
ICLM
2024; Yen et al., 2024b), leading to the creation of various evaluation datasets. Accordingly, we the target context length 128K for comprehensive evaluation. We evalu- ate the models using the HELMET (Yen et al., 2024b) and RULER (Hsieh et al., 2024) benchmarks. The evaluation spans five task types from the HELMET benchmark: syn- thetic recall, retrieval- -uugeete-d generation (RAG), many- shot in-context learning (ICL), passage re-ranking, and long- document QA, covering total of 17 sub-tasks. Detailed descriptions of the HELMET benchmarks can be found in Appendix C.3. Additionally, the RULER benchmark includes 13 synthesis sub-tasks.
Quest
Knn
Recall
RAG
90
verformance
60
75
60
45
52
48
8k
16k
32k
64k 128k
8k
16k 32k
64k 128k
ICL
Re-Rank
88
40
Performance
80
30
72
64
10
4.2. Comparison with Other Data Synthesis Methods
56
16k
32k 64k 128k
8k
16k
64k
128k
We first compare NExtLong with previous long-context data synthesis methods on the 128K context length setting.
LongQA
RULER
90
Verformance
80
35
Experimental Settings for Extending Context Length to 128K. We fine-tune the Meta-Llama-3-SB-ber (Meta, 2024) model using a batch size of 4M tokens for 1000 steps with the open-source source framework GPT-NeoX2. The RoPE frequency base is increased from 500,000 Meta-Llama-3- 8B-base 200,000,000. The same training configuration is applied to all methods for a fair comparison. Further details are available in Appendix C.I.
70
225
60
25
50
8k 16k 32k 64k 128k
8k
16k 32k 64k 128k
Context Length
Context Length
Figure 3. Comparison of NExtl with other data synthesis meth- ods on HELMET and RULER henchmarks across different context lengths. NExtL Long shows significant performance improvements across various tasks.
Baseline Methods
We compare NExtLong with several methods that synthesized 32,000 128K-length samples (ap- proximately billion training tokens) from short documents:
ments, and we also provide ablation studies on their selec- tion in Appendix B.3. Various methods, including NExt- Long and baseline approaches, are employed synthesize target-length samples concatenated from these short docu- ments. The datasets described as follows:
Standard Method: Randomly samples and concate- nates short documents (Ouyang et al., 2022; Le Scao al., 2023; Touvron et al., 2023a).
KNN (Guu al., 2020; Levine et al., 2021): Pairs each document with the top k most similar retrieved documents.
An advanced version of the largest synthetic dataset for pretraining, comprising over 39 million generated samples from textbooks, blog posts, and stories.
Cosmopedia v2:
ICLM (Shi et al., 2023b): Uses traveling salesman algorithm to reduce redundancy and improve diversity.
Fine Web-Edu: Consists of 1.3 trillion tokens of educa- tional web pages filtered from the Fine Web dataset.
Quest (Gao et al., 2024a): Balances semantic corre- lation and diversity by clustering documents based on predicted queries.
Evaluation Recent long-conted evaluations have focused on a 128K context length (Zhang et al., 2024c; Hsieh al.,
Phtpps: // tttp///tttttccooce com/ wwt.c/////eeeeerrlggttt----oo.a.
5

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Table Comparing NExtl. .ong with other open-souree hase models the HELMET and RULER benchmarks. All results averaged
over sequence lengths of 8K. 16K, 32K. 64K, and 128K. O: results from Yen et al. (2024b); A results evaluated by ourselves.
Model
Max Len
Avg.
Recall
RAG
ICL
Re-rank
LongQA
RULER
Open-source base models
Yarn- Yarn-Llama-2-77 2-7---2------88
128K
35.61
18.58
43.47
71.32
13.27
25.91
41.14
DeepSeek- -a2-sekk-V2-Liee V2-Lite
160K
42.62
37.00
46.93
72.36
14.31
29.97
55.17
Yi-9H-200K +
200K
53.91
65.88
57.31
62.36
22.86
39.47
75.61
Qwen2.5-7B
128K
49.53
59.04
49.44
73.84
18.37
28.62
67.84
Distral-Nemo-BRe +
128K
50.34
57.04
54.73
74.68
18.98
35.53
61.08
Llama-3-8BB --8---rooooo-------- 512K-Base
512K
60.34
86.95
60.93
79.20
31.66
24.68
78.60
Llama-3,1-88 +
128K
61.07
83.18
61.22
71.40
29.10
37.35
84.20
3-----8-8--N-Long--- Saaggg-1KK-BBBB (ours)
512K
65.76
91.58
63.68
84.08
31.27
38.42
85.52
Closed-source models
GPT-4o-mi mini <
128K
70,98
94.90
69.64
78.12
52.30
43.13
87.82
GPT-40
128K
77.43
98.22
72.30
85.76
64.70
47.61
95,96
Gemini- (imini--,,5-PPe 1,5-Pro +
2M
71.71
84.60
72.06
78.74
69.04
45.99
79.84
Sosonne 0
200K
51.19
93.30
41.10
59.80
9.10
10.83
93.00
GPT-4o
Llama-3-88 18B-NExtonng-511 --aoongg----bbbeeeeeee
GPT-4o-mini
TREC Coarse
TREC. Fine
BANKING77
CLINIC150
NLU
ICL Performante
21
49
79
74
400
3300
6600
400
3200
6400
360
2000
5900
3525
7050
510
3080
0296
Number of Shots
Figure 4. Comparison of NExtLong with GPT-4o on five In- Context Learning (ICL) tasks from the HELMET benehmark. Each polyline
represents the model's performance across context lengths of 8K, 16K, 32K, 64K, and 128K.
NExtLong Outperforms Existing Data Synthesis Meth- ods. Table and Figure 3 compare NExtl Long with other data synthesis methods across different context lengths (8K, 16K, 32K, 64K, and 128K) the HELMET and RULER benchmarks. Table 1 presents the averaged results, indi- cating that NExtL.ong surpasses all baseline methods with an average improvement of at least +7.33%. Notably, it achieves 13.43% gain in Recall and a 9.12% improve- ment in Re-Rank over the Quest method, demonstrating effectiveness in enhancing long-context performance.
a two-stage training strategy: first training on shorter con- texts, then extending to longer ones (e.g., 512K). ProLong evaluates models using "train-long, test-short" approach, testing shorter contexts (e.g., 128K). For fairness, adopt the same strategy, training up to 512K and evaluating on 128K benchmarks.
Experimental Settings for Extending Context Length to 512K. Unlike other models such as ProLong (Gao et al., 2024b), which are trained on naturally occurring long doc- uments, we utilize NExtl NExtLong-sse ong-synthesized data for train- ing. Specifically, we synthesize two long-context datasets, NExtLong-64K and NExtLong-512K, both derived from the Fine Web-Edu and Cosmopedia v2 corpora. The detailed training prperpparaeeters are provided in Appendix C.2.
Figure 3 further illustrates that NExtLong outperforms other methods across varying context lengths, with the gap widen- ing as context length increases. The results highlight NExt- Long's 's superior capability to model long-range dependen- cies and maintain robust performance even at 128K context length, demonstrating the versatility and reliability of NExt- Long across diverse tasks and its effectiveness handling ultra-long contexts.
Baseline Models We select open-source base models with comparable parameter sizes for evaluation, including Llama- 3.1-8B and Llama-3-8B-PP Sio--B-ProLLo--1111 -12K- 5--2--assee Additionally, we compare against current SOTA sourde models, such as GPT-4o, Gemini, and Claude.
4.3. Comparison with SOTA Models
We compare NExtLong-tr models with state-of-the-art models, including ProLong (Gao al., 2024b), which uses
6

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Table 3. Comparing NExtLLong with ProLong models on the LongBench v2 henchmark.
Model
Overall
Easy
Hard
Short
Medium
Long
Booo---o---ooo-n----- Insttruct
27.2
31.8
24.4
31.7
29.3
15.7
30.4
33.3
28.6
32.2
30.7
26.9
Without Using Long Documents, NExtLong Outper- forms Other Open-Source Models. Table 2 shows that Baain-3----NEExLLoo 512K- 512K-Base model surpasses other open-source models, outperforming 512K-Base by +5.42% and Llama-3.1 by +4.69% on average. These results demonstrate that synthesized data can match even surpass von-synthesized long documents in enhancing long-context capabilities, positioning NExtLong for ultralong context extensions.
0.115
LanggA Performance (Normalizen) Long-Dependence Attention Weight
0.110
Attention Weight
0.105
0.100
37.30
0.095
0.090
32.27
0.085
0.080
28.99
28.17
Standard
ICLM
KNN
Quest
NExtLongg
In ICL Tasks, NExtLong Matches or Surpasses GPT- 4o as the Number of Shots Increases. Recently, Long- context models' ICL performance has garnered significant attention (Bertsch et al., 2024; Agarwal et al., 2024; Anil al., 2024). Agarwal al. (2024) highlight ICL perfor- mance valuable metric for evaluating long- long-context mod- els. Figure 4 shows that the number of shots increases, NExtLong matches GPT-4o in the Banking77 task and out- performs it in four other tasks. Its strong performance and moderate computational cost make NExtLong suitable for future ICL applications.
Methods
Figure 5. NExtLong enhances long-range dependency modeling. The bars represent the model's 's ability to capture long-range de- pendencies, measured by the attention weights assigned to the first third of the context. The dotted line indicates the model's perfor- mance, demonstrating a positive correlation between improved long-range dependency modeling and better performance the LongQA task.
5.2. NExtLong Performs Strongly After Supervised Finetuning.
5. Analysis
To evaluate how NExtLong performs after supervised fine- tuning, we follow the approach in ProLong (Gao et al., 2024b) and fine-tune our base model using the UltraChat (Ding et al., 2023) short-context SFT dataset. We test the model on the recently proposed LongBench v2 benchmark (Bai al., 2024). As shown in Table 3, NExtLong outper- forms ProLong overall, especially on the Long metric. The results demonstrate that
This section provides an in-depth analysis of the NExtLong method. Due to the high computational cost of experiments, ablation studies are conducted with a 128K context length.
5.1. NExtLong Enhances Long-Range Dependency Modeling
To assess the improvement in long-range dependency mod- cling achieved by NExtLong's 's negative document extension, we conduct a probing experiment using the Longbook QA dataset (Zhang et al., 2024b), which features long-range de- pendencies up 128K in length. In this experiment, use the normalized attention weights assigned to the first third of the context, when predicting the last token, as metric for evaluating the model's long-dependency modeling ability.
122-Bae performs strongly as base model. With the same SFT dataset, the improved long-context base model enables the training of a superior fine-tuned model.
5.3. The Impact of Chunking Granularity
We perform an ablation study on chunking granularity s using values of 512, 1024, 2048, 8192, and 32768. The results, shown in Figure 6, indicate that the model performs best with a granularity of 2048. While a granularity of 1024 yields optimal performance for 128K context length, it underperforms in the 8k and 16k ranges compared to 2048. We conclude that too small a granularity disrupts semantic integrity, while too large introduces redundant information, negatively impacting the hard negative mining stage. A moderate granularity offers the best balance for performance.
As shown in Figure 5, we observe a positive correlation between this denendeeyy metric and the model's ' per- formance on LongQA. Complementarily,
discussed in Appendix B.4, NExtLong reduces the model's dependence on proximal text (the last third context). These findings demonstrate that models trained with NExtLong's nega- tive document extension exhibit enhanced long-dependency modeling capabilities, resulting in significantly improved long-context performance.
7

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Table Comparison of short text performance across methods. Overall, NExt Long shows minor performance Aluctuation in short
benchmark as the method improves the long-context ability of Meta- Meeta-Llaaa- 3- 3-BB-B-BBBBBBBB 80- base.
Model
Avg.
Hel.
Lam.
AR-C.
AR-E.
PIQA
Win.
Log.
1ecta-llama-3-8B---
63.75
60.13
75.66
50.34
80.18
79.60
72.85
27.50
+ Standard
63.66
59.57
72.87
49.83
81.73
80.36
73.64
27.65
KNNN
63.23
59.74
72.25
49.15
80.85
80.30
73.56
26.73
ICLM
63.89
61.24
71.67
52.47
81.73
80.14
73.56
26.42
+ Quest
63.96
59.72
73.20
50.68
81.14
80.41
74.74
27.80
+ NExtLong
63.83
60.32
72.06
51.45
82.03
79.87
73.09
27.96
62
52
61.36
Avg.All
1.0
Avg. Performance (Normalized) Cosine Similarity
58,07
56,39
58.55
57,53
56,79
Avg. 128k
50
Similariy
Performance
60.35
62.02
61.00
Avg. Avg,64k
56,72
62,58
63,69
63,06
62.02
Aug 32k
60.51
60
69
28.27
02
65.07
52
Avg_16k
50,22
61.33
61,38
62
03.03
65,25
63.79
Avg_8k
52
29.48
RandomD
Tbil-k
RandomR Top-
Relf-Repeat
512
1024
2048
8102
32768
Chunking Length
Strategles of Negative Selection
Figure 7. The impact of negative selection long-context per- formance. The bars represent the cosine similarity of documents concatenated by different strategies. The dotted line indicates the average performance on the MEL and RULER benchmarks, with all results normalized align within the specified similarity range.
Figure The impact different chunking granularities S on performance of NExtLong. The six curves, from bottom to top, correspond to the average performance across six task types at document lengths of 8K, 16K, 32K, 64K, and 128K, as well as the overall average across all these lengths.
5.4. The Importance of Hard Negatives for Achieving Better Results
Mcanwhile, using repeated meta-chunks brings false neg- atives, further degrades model performance, which is con- sistent with the phenomenon observed contrastive learn- ing (Chen al., 2021).
To evaluate the impact hard negatives on performance, we design 5 document retrieval strategies. For each meta-chunk, retrieve 512 documents from the Faiss index and select k from these documents using the following strategies:
5.5. NExtLong Shows No Significant Performance Degradation on Short Text.
1. Self-Repeat: Repeat meta-chunks without including retrieved documents.
To verify how well NExtLong maintains model performance on short tasks, following Quest (Gao al., 2024a), we select 7 widely-used short-text datasets: HellaSwag (Hel.) (Zellers et al., 2019), Lambada. OpeeAA (Lam.) (Paperno et al., 2016), ARC-Challenge (AR-C.) (Clark et al., 2018), ARC-Fasy (AR-E.), PIQA (Bisk et al., 2020), WinoGrande (Win.) (Sakaguchi et al., 2021), and Logiqa (Log.) (Liu et al., 2020). As shown Table 4, compared to the Meta- Llama-3-4 Hamma-3-BB- 3-B---as-B-eee model, the performance on short text eval- uations shows significant degradation after continued training with long-context data synthesized by NExtLong.
2. Top-k (hard negatives): Concatenate documents in descending order of similarity until the target length is reached.
3. RandomR: Shuffle retrieved documents randomly and select from them.
4. Tail-k:
Concatenate documents in ascending order of similarity.
5. RandomD: Randomly select documents from the train- ing dataset, ignoring retrieval documents, which share similar idea to (Tian et al., 2024).
6. Conclusion and Future Works
Figure shows that the choice of hard negatives (the top-k setting) plays a crucial role in NExtLong. Low-similarity negatives reduce training difficulty, weakening performance.
This paper introduces NExtLong, a framework that im- proves long-range dependency modeling LLMSs through
8

---

NExtLong: Toward Effective Long-Context Training without Long Documents
negative document extension. By dividing document into meta-chunks and inserting hard negative distractors, NExt- Long increases learning difficulty, encouraging the LLMs to better model long-range dependencies over extended con- texts. Experimental results show that NExtLong outper- forms existing methods on HELMET and RULER bench- marks, achieving notable performance gains. In the future, we plan to explore more effective negative chunk mining strategies, such generative approaches to creating more diverse and harder distractors, further enhancing the model's ability to learn fine-grained long-range dependencies.
Author, N. N. Suppressed for anonymity, 2021.
Babenko, A. and Lempitsky, V. The inverted multi-index. IEEE transactions on pattern analysis and machine intel- ligence, 37(6): 2014.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L.., al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308. S.ti::230...408.. 2023.
Bai, Y., Tu, S., Zhang. J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204 2024.
Acknowledgement
This work supported by the National Natural Science Foundation of China (No. U24A20335).
Bairi, R., Sonwane, A., Kanade, A., C, V., Iyer, A., Parthasarathy, S., Rajamami, S., Ashok, B., and Shet, S. Codeplan: Repository-level coding using Ilms and planning. Sep 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. Smollm-corpus, 2024. URL https://huggingge
co datasets/ HuggingFFaeeTB/ smo Im-corpus.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. I.., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 08X77 2023.
Bengio, Y. and LeCun, Y. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007.
Bertsch, A., Ivgi, M., Alon, U., Berant, J., Gormley, M. R., and Neubig, G. In-context learning with long- context models: An in-depth exploration. arXiv preprint arXiv:2405.000 002000, 2024.
Agarwal, R., Singh, A., Zhang, M., Bohnet, B., Rosias, L., Chan, S., Zhang. B., Anand, A., Abbas, Z., Nova, A., al. Many-shot in-context learning. arXiv preprint vrXiv:2404.11018, 2024.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401. 029544 2024.
i, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q. Yue, S., Yang. S.. Yang. S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P. Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by ai, 2024.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Alabdulmohsin, M., Neyshabur, B., and Zhai, X. Revisit- ing neural scaling laws language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022.
An, C., Huang, F., Zhang, J., Gong, S., Qiu, X., Zhou, C., and Kong, Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463, 2024.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Anil, C., Durmus, E., Sharma, M., Benton, J., Kundu, S., Batson, J., Rimsky, N., Tong, M., Mu. J., Ford, D., et al. Many-shot jailbreaking. Anthropic, April, 2024.
Caciularu, A., Peters, M., Goldberger, J., Dagan, I., and mod-
Cohan, A. Peek across: Improving eling via cross-document question-answering May 2023.

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Cai, X., Xiao, M., Ning. Z., and Zhou, Y. Resolving the im- balance issue in hierarchical disciplinary topic inference via Ilm-based data augmentation. In 2023 IEEE Interna- tional Conference on Data Mining Workshops (ICDMW), pp. 1424-1429. IEEE, 2023.
Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim,
Y., and Peng, H. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.001111 2024.
Gao, C., Wu, X., Fu, Q.. and Hu, S. Quest: Query- data synthesis approach for long-context scaling of large language model. arXiv preprint arXiv:2405. 19846 2024a.
Chen, G., Li, X., Meng. Z., Liang. S., and Bing, L. Clex: Continuous length extrapolation for large language mod- els. arXiv preprint arXiv:2310. 16450, 2023a.
Chen, S., Wong, S., Chen, L.,, and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595,, 2023b.
Gao, L., Biderman, S., Black, S., Golding, L.,, Hoppe, T. Foster, C., Phang, He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Chen, T.S.., Hung, W.-C., Tseng, H.-Y., Chien, S.-Y., and Yang. M.-H. Incremental false negative detection for contrastive learning. arXiv preprint arXiv:2106.03719, 2021.
Gao, T. Wettig, A., Yen, H., and Chen, D. How to train long-context language models (effectively). 2024b. URL https: http::/appp..aaaaaaaae org/Corpus ( 273098476.
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. 12307, 2023c.
Goodfellow, Bengio, Y., Courville, A., and Bengio, Y. Deep learning, volume 1. MIT Press, 2016.
Chevalier, A., Geng, J., Wettig, A., Chen, H., Mizera, S., An- nala, T., Aragon, M. Fanlo, A. R., Frieder, S., Machado, S., al. Language models as science tutors. arXiv preprint arXiv:2402.111111 2024.
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929- 3938. PMLR, 2020.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 1oX3:v/80030555.. 2018.
Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308. 2023.
Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03320 2020.
DeepSeek-AL. v2 A strong, economical, and efficient mixture-of-experts language model, 2024.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Scaling laws for autoregressive generative modeling. arXiv preprint (:XXi:2010..47000 2020.
Ding, H., Wang, Z., Paolini, G., Kumar, V., Deoras, A., Roth, D., and Soatto, S. Fewer truncations improve lan- guage modeling. In Proceedings of the 4|st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Hinton, G. E., Osindero, S., and Teh, Y. W. A fast learning algorithm for deep belief nets. Neural Computation, 18: 1527-1554. 2006.
Ding, N., Chen, Y. Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 1..00,,.4233. 14233, 2023.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint urXiv:2203. 15556, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et The llmaa herd of models. arXiv preprint arXiv:2407.2178. 2024.
Hsich, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg. B. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.0665 2024.
Duda, R. O., Hart, P. E., and Stork, G. Pattern Classifi- cation. John Wiley and Sons, 2nd edition, 2000.
10

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Ivison, H., Wang. Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv22311.17002, 2023.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H.,, Lewis, M., Yih, W.-., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459 2020.
Jin, B., Yoon, J., Han, J., and Arik, O. Long-contexi Ilms meet rag: Overeoming challenges for long inputs in rag. arXiv preprini arXiv:2410.05983, 2024a.
Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bho- janapalli, S. Functional interpolation for relative posi- tions improves long context transformers. arXiv preprint arXiv:2310004418, 2023.
Jin, H., Han, X., Yang. J., Jiang, Z., Liu, Z., Chang, C.-Y., Chen, H., and Hu, X. Llm maybe longlm: Self-extend Ilm context window without tuning, 2024b.
Li, Z.-Z.. Zhang. D., Zhang, M.-L.,, Zhang, J., Liu, Z., Yao, Y. Xu, H., Zheng. J., Wang, P.-J., Chen, X., Zhang, Y.. Yin, F, Dong, J., Li, Z., Bi, B.-L., Mei, L.-R., Fang, J., Guo, Z., Song, L., and Liu, C.-L. From system I to system
Jung, V. and van der Plas, L. Understanding the effects of language-specifer class imbalance in multilingual fine- tuning. arXiv preprint arXiv:2402.13016 2024.
2: A survey of reasoning large language models, 2025. URL https: / perrxiv.orge abs wtt.ooraaabbb..11111/11 221.2222222...1..1
Kalantidis, Y., Sariyildiz, M. B., Pion, N., Weinzaepfel, P., and Larlus, D. Hard negative mixing for contrastive learning. Advances in neural information processing systems, 33:21798-21809 2020.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08266 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Liu, J., Cui, L.,, Liu, H., Huang. D., Wang, Y., and Zhang, Y. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.
Kaushik, R., Krishnamurthy. R., Naughton, J. F., and Ra- makrishnan, R. On the integration of structure indexes and inverted lists. In Proceedings of the 2004 ACM SIG- MOD international conference on Management of data, pp. 779-790, 2004.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. arXiv preprint 171 2017.
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu, May 2024. URL petps://hugginee
Kearns, M. J. Computational Complexity of Machine Learn- ing. PhD thesis, Department of Computer Science, Har- vard University, 1989.
co /ataaattsess/////o HuggingFaceeW/ Fineweb-edu.
Mallia, A., Khattab, O., Suel, T., and Tonellotto, N. Learn- ing passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pp. 1723-1727, 2021.
Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al. The stack: tb of permissively licensed source code. arXiv preprint arXiv:2211 1555333, 2022.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Mazumder, S. and Liu, B. Lifelong and continual learning dialogue systems. Nov 2022.
Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL sttps:///lant meta com/ / 1lama3/.....
Le Seao, T., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open- multilingual language model. 2023.
Michalski, R. S., Carbonell, J. G., and Mitchell, T. M. (eds.). Machine Learning: An Artificial Intelligence Approach, Vol. Tioga, Palo Alto, CA, 1983.
Levine, Y., Wies, N., Jannai, D., Navon, D., Hoshen, Y., and Shashua, A. The inductive bias of in-context learning: Rethinking pretraining example design. arXiv preprint arXiv:2/10.045. 2021.
Mitchell, T. M. The need for biases in learning generaliza- tions. Technical report, Computer Science Department, Rutgers University, New Brunswick, MA, 1980.

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Newell, A. and Rosenbloom, P. S. Mechanisms of skill acquisition and the law of practice. In Anderson, R. (ed.), Cognitive Skills and Their Acquisition, chapter 1, pp. 1-51. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, 1981.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case gener- alization. arXiv preprint
2019.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99 106, 2021.
Nogueira, R., Yang, W., Lin, J., and Cho, K. Docu- ment expansion by query prediction. arXiv preprint arXiv:1904.08375. 2019.
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Samuel, A. L. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):211-229, 1959.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, H., Schärli, N., and Zhou, Large language models can be easily distracted by irrelevant context. In Inter- national Conference Machine Learning, pp. 31210 31227. PMLR, 2023a.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bemardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint orXiv:1606.06031, 2016.
Shi, W., Min, S.. Lomeli, M., Zhou, C., Li, M., Lin, V., Smith, N. A., Zettlemoyer, I.,, Yih, S., and Lewis, M. In-context pretraining: Language modeling beyond doc- ument boundaries. arXiv preprint arXiv:2310.10638 2023b.
Paster, Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2330.06740 2023.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama, 2023. URI. http::///////ggggaaaaea/s tetpssets//erebbaa/siinnie
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007. 2023.
627B.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 2024.
Radford, A. Improving language understanding by genera- tive pre-training 2018.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 67, 2020.
Su, H., Tian, Z., Shen, X., and Cai, X. Unraveling the mystery of scaling laws: Part i. arXiv preprint arXiv:2403.066 0000000663,,.. 2024.
Su, J. Rectified rotary position embeddings. https//giinubboom/oojoneeeeerr 2023.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2021.
Robinson, J., Chuang, C.-Y., Sra, S., and Jegelka, S. Con- trastive learning with hard negative samples. arXiv preprint XXXi:2000000592 2020.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https: http///wwmmmmmmmmmmm/..././mmmmm git.i.iiihhii github. blog/ /w/p//./n..////mm./
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code Ilama: Open foundation models for code. arXiv preprint a008 12950, 2023.
Tian, J., Zheng, D., Cheng, Y., Wang, R., Zhang, C., and Zhang, D. Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. arXiv preprint arXiv:2409.0 04774, 2024.
12

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023a.
Xiong, Y., Chen, X., Ye, X., Chen, H., Lin, Z., Lian, H., Niu, J., and Ding, G. Temporal scaling law for large language models. arXiv preprint arXiv:2404.17788 2024.
Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. arXiv preprint 2eXiv:2406.08445 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint N.0i:22307.09288, 2023b.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C.. Li, C.,, Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang. J., Wang, J., Yang. J., Tu, J., Zhang. J., Ma, J., Xu. J., Zhou, J., Bai. J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang. P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W. Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y. Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang. Z., and Fan, Z. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., and Mitoš, P. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
van der Maaten, L. and Hinton, G. Visualizing data using sne. Journal of Machine Learning Research, 9(86):2579- 2605, 2008. URL http:: / jml httgpppprs///// vandermaaten08a.F1 html.
Wang, L., Ma, C., Feng. X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. A survey large language model based autonomous agents. Frontiers of Computer Science, 18 18(6):1-26, 2024.
Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang. P., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Retrieval- multimodal language modeling. arXiv preprint urXiv:2211.225 2022.
Wu, X., Ma. G., Qian, W., Lin, Z., and Hu, S. Query-as- context pre-training for dense passage retrieval. arXiv preprint ar arXiv:2212.09500 2022.
Yen, H., Gao, T., and Chen, D. Long-context language modeling with parallel context encoding. arXiv preprint arXiv:2402. 2661:22002.6617, 2024a.
Wu, X., Ma, G., Lin, M., Lin, Z., Wang, Z., and Hu, S. Con- textual masked auto- entoder for dense passage retrieval. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 37, pp. 4738 4746, 2023a.
Yen, H.,, Gao, T., Hou, M., Ding, K.,, Fleischer, D., Izasak, P., Wasserblat, M., and Chen, D. Helmet: How to evaluate long-contedt language models effectively and thoroughly. arXiv preprint arXiv:2410.02694, 2024b.
Wu, Y., Iso, H., Pezeshkpour, P., Bhutani, N., and Hruschka, E. Less more for long document summary evaluation by lmss. arXiv preprint 2xXiv:2399003882 2023b.
Yu, H., Zhang, Y., Bi, W., et al. Trams: Training-free mem- ory selection for long-range language modeling. arXiv preprint arXiv:2310.15494 2023.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
Zellers, R., Holtzman, A., Bisk. Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? 2019.
arXiv preprint
Xie, S. M., Pham, H., Dong, X., Du N., Liu, H., Lu, Y., Liang, P. S., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training. Advances in Neural Information Processing Systems, 36, 2024.
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., and Ma, S. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, pp. 1503-1512, 2021
Xiong, L., Xiong, Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., and Overwijk, A. Approximate nearest neigh- bor negative contrastive learning for dense text retrieval. arXiv preprint 0tt::2200.0000000 2020.
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang. Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language to vision. arXiv preprint (tt::2406.168552 2024a. URL https: https://arxit //arxiv.. org/ ala/bbb//22 w.bp/2406..... 16852.
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of founda- tion models. arXiv preprint arXvv22309 16039, 2023.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., and Sun, M. Bench Extending long context evaluation beyond 100K tokens.
13

---

NExtLong: Toward Effective Long-Context Training without Long Documents
In Ku, L.-W., Martins, A., and Srikumar, V. (eds.). Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 15262-15277, Bangkok, Thailand, August 2024b. As- sociation for Computational Linguistics. URL ht : //aclanthology org/ 2024. acl long. 814.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L.,, Wang, S., Liu, Z., et al. ocbench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024c.
Zhao, L., Wei, T., Zeng, L., Cheng, C.,, Yang. L., Cheng, P., Wang, L.,, Li, C., Wu, X., Zhu, B., Gan, Y., Hu, R., Yan, S., Fang, H., and Zhou, Y. Longskywork: A training recipe for efficiently extending context length in large language models. arXiv preprint arXiv:2406. 00605, 2024.
Zhu, D., Yang, N., Wang, I.,, Song, Y., Wu, W., Wei, F. and Li, S. Pose: Efficient context window extension of Ilms via positional skip-wise training. arXiv preprint wxXi:22309.10400 2023.
14

---

NExtLong: Toward Effective Long-Context Training without Long Documents
A. The Result on Needle-in-a-Haystack Benchmark
Baam--3-B--EExxLnng--- Acc: 1.0
90.0
70.0
Depth Percent
60.0
50.0
40.0
30.0
20.0
2
10.0
25k
150k
275k Token Limit
400k 500k
Figure 8. The Basllein--a-Haystan task assesses a model'ss capability to extract specific information (the needle) from a large corpus of documents (the haystack). The y-axis indicates the position of the "needle" within the document, spanning from 25K to 500K tokens.
Following previous works (Gao al., 2024a; Zhang et al., 2024a; Liu et al., 2024), evaluate the Llama-3-8BNNixLLorr 512K-Bat Base model on the widely used Needle-in-a- Haysttaa task. As shown in Figure 8, ----------NEELLnng 512K- 5-sseBse achieves a 100% accuracy on the Boddl-in-a-Haystann task.
B. More Ablations
B.1. Placing Meta-Chunk Different Positions
We explore three different strategies for combining meta-chunk and hard negatives, which represented by the following descriptions:
1. Head:
Placing the meta-chunk the beginning of the retrieved hard negatives.
2. Tail:
Placing the meta-chunk the end of the retrieved hard negatives.
3. Random:
Randomly inserting the meta-chunk within the retrieved hard negatives.
Table 5 shows that placing the meta-chunk the beginning (Head) yields better performance. We believe this method helps establish longer dependencies, resulting in enhanced effectiveness.
Table Performance Comparison of Different Insertion Strategies. All results are averaged over sequence lengths
and 128K.
Model
Avg.
Recall
RAG
ICL
Re-rank
LongQA
RULER
Head
62.58
82.56
60.91
81.76
31.47
37.30
81.50
Tail
60.01
72.66
63.67
80.68
30.73
34.09
78.26
Random
58.95
74.51
63.05
71.64
32.78
33.00
78.73
B.2. Document Length Distribution of Cosmopedia V2 and FineWebEEd
We analyze the document length distribution of two datasets, Cosmopedia V2 and FineWebEdu, by sampling million documents from each dataset and encoding them using the Meta-Lllama-3-88 tokenizer. Document lengths categorized into two ranges: [0, 8192] and > 8192. Table shows that the majority of documents in both datasets are relatively short (under 8K). We apply the NExtLong algorithm to extend the document length to 128K and 512K, achieving approximately 64-fold increase compared to the original.
Table 6. Document Length Distribution of Cosmopedia V2 and FineWebEdu.
Dataset
0 < Length < 8192
Length > 8192
Cosmopedia V2
100.00%
0.00%
WebEdu
99.19%
0.81%
15

---

NExtLong: Toward Effective Long-Context Training without Long Documents
B.3. Dataset Ablation Study
We compared three different dataset selection strategies: (1) using FineWeb-Edu alone for long-context data synthesis, (2) using Cosmopedia v2 alone for long-context data synthesis, and (3) combining both datasets for long-context data synthesis. The results shown in Table 7. The findings indicate that the combined strategy achieved the best performance, highlighting that diverse dataset significantly enhances data synthesis.
Table Dataset Comparison on HELMET and RULER Benchmark.
Dataset
Avg.
Recall
RAG
ICL
Re-rank
LongQA
RULER
Cosmopedia
59.26
79.94
59.58
81.00
27.07
29.13
78.84
Fine WenbWu
62.04
83.56
61.53
83.76
28.31
34.46
80.60
Cosmopedia FineWebEdu
62.58
82.56
60.91
81.76
31.47
37.30
81.50
B.4. NExtLong Reduces Dependence on Proximal Text
Complementary with Section 5.1, we investigated the dependence of different models on proximal text (the last third the text), As shown Figure 9, NExtLong demonstrates lower degree dependence on proximal text. This shift in attention toward long-range text contributes improving the model's performance.
0.90
Short- Dependence Attention Weight
0.88
Attention Weight
0.86
0.84
Standard
ICLM
KNN
Quest
NExtLong
Methods
Figure NExtLong reduces the model's dependence proximal text. We calculated the degree of dependence proximal text (the last third of the text) for different methods when completing the LongQA task. It can be observed that NExtLong significantly reduces the model's dependence on proximal text.
C. Experiment Details
C.1. Training Laama---B--NExLLoog--2228 detailed setup
We use the parameters listed in Table 8 train the 128K model. For other data synthesis methods, we only modify the training dataset while keeping all other training parameters unchanged.
C.2. Training 3-0
detailed setup
Given that mixing documents of different lengths during training widely adopted in prior work (Gao et al., 2024b; Meta, 2024), we follow the ProLong (Gao al., 2024b) approach by synthesizing both 5 512K-length and 64K-length documents for training the 512K NExtLong model. For synthetic 512K-length documents, apply the full attention mechanism, as each document constitutes an entire training sample. For synthetic 64K-length documents, we concatenate them into 512K -1eKthentth training samples (each sample contains eight 64K-length documents) and employ intra-document attention (Ding et al., 2024) to restrict information flow within each 64K-length document. We use the parameters listed in Table to train the 512K model. The training samples are sourced from the NExtLong-511 512K and NExtLong-66l 66k datasets in ratio of 1:2.
16

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Table 128K model training configuration.
128K training setting
Initial Model
Meta-I.lama-3 3-88 (base model)
rotary-emb-base
200,000,000
31
0.9
B2
0.95
lr
4e-
precision
66
gradient-clipping
1.0
weight-decay
0.1
It-eecay-tyyle
cosine
train-iters
1000
warmup-iters
200
seq seq-length
131072
GPU-type
H100
GPU-numbers
64
training-time
1.5
Table 9. 512K model training configuration.
512K training setting
Initial Model
Llama-3-8B- Panoooo---- -64k Basc
rotary-emb-beet
128,000,000
81
0.9
82
0.95
lr
le
precision
bfloat16
gradient-clipping
1.0
weight-decay
I--dccy-style
cosine
train-iters
500
warmup-iters
50
seq-length
524288
GPU-type
H100
GPU-numbers
128
training-ting -tme
20h
C.3. Evaluation Metric and Task Category of the HELMET Benchmark.
The task category and evaluation metric of the HELMET benchmark (Yen et al., 2024b), which we use this work, are shown in Table 10.
D. More Details in NExtLong
D.1. The Calculation Method for the Number of Hard Negatives k
In the Negative Document Extension stage, the meta- meta-document targeted for extension is chunked into meta-chunks. Each meta-chunk retrieves the top-k similar texts as hard negatives from the Faiss index, with the value of k adaptively adjusted based on the target length T. Let E represent the encoding rate of the model tokenizer. The total number of characters Q needed for retrieval is calculated as follows:
Q x Ex
(8)
Here, the adjustment factor w accounts for variability and ensures that a sufficient number of hard negatives are recalled for each meta-chunk. In our experiments, we set w = 1.5. Then, we compute the remaining characters L necessary for synthesis beyond the total character length S of the meta-document. This calculated by subtracting S from Q:
= Q
(9)
17

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Table 10. Summary of datasets and metrics in HELMET benchmark.
Category
Dataset
Metrics
Description
Natural Questions
SuhEM
Factoid question answering
TriviaQA
SubEM
Trivia question answering
Retrieval-augmented generation
PopQA
SubEM
Long-tai tail entity question answering
HotpotQA
SubEM
Multi-hop question ans answering
Passage re-ranking
MS MARCO
NDCC@ @ I0
Rerank passage for query
NarrativoQAA
ROUGE
Book movic script QA
Long-document QA
<ENCH QA
ROUGE
Novel QA with entity replacement
BBENCH MC
Accuracy
Novel multiplc-choice QA with entity replacement
TREC Coarse
Accuracy
Question type classification, 6 labels
TREC Fine
Accuracy
Question type classification, 50 labels
Many-shot in-context learning
NLU
Accuracy
Task intent classification, 68 labels
BANKING77
Accuracy
Banking intent classification, 77 labels
CLINC150
Accuracy
Intent classification, 151 labels
JSON KV
SubEM
Retrieve key JSON dictionary
RULER MK Needle
SubEM
Retrieve the needle (a number) within noisy needles
Synthetic recall
RULER MK UUID
SubEM
Retrieve the needle (a UUID) within noisy needles
RULER MV
SubEM
Retrieve multiple values for one needle (key)
The meta-document is divided into p meta-chunks. The number of characters P required for retrieval from each meta-chunk distributed evenly across all p meta-chunks:
p=
(10)
The number of hard negatives that need to be retrieved for each meta-chunk can be calculated follows:
P
k
(11)
Substituting P into this equation gives:
L pxs
Q Q pxs
TxEx
k
(12)
pxs
This formulation ensures that the number of hard negatives is proportional to the chunking granularity s and the target length T. Additionally, to enhance content diversity, we ensure that the same hard negative is not repeatedly used across different meta-chunks.
D.2. Pseudocode of NExtLong
We present the complete process of constructing the NExtLong dataset using pseudocode in Algorithm 1.
18

---

NExtLong: Toward Effective Long-Context Training without Long Documents
Algorithm Negative Document Extension (NExtLong)
Input:
Training corpus D (d1, d2,,,22.... dy) chunking granularity s, number of retrieved hard negatives k. target long length Luaget.
Faiss index Faiss
Output: Synthesized long documents for long- ong-dependence modeling
I:
Initialize an empty list T storing synthesized long documents
2: function DOCUMENT CHUNKING(r, 8)
3: Split into paragraphs (ri, ....... rP) by newline characters
4: Initialize empty list chunks
Initialize an empty buffer buffer = with length counter é for each paragraph r; in do if + (r. S then buffer buffer r.: f L + Length(rs)
6:
7:
8:
10:
11:
chunks + chunks U [huffer)
12:
buffer + { Length(r:))
13:
15:
end for
16: if buffer * Ø then
17:
chunks + chunks (buffer)
18: end if
19:
return chunks
20:
end function
21:
function NEGATIVE MINING(m., k. Faiss)
22:
niy Rig....... ..., Big Top-k similar chunks from Faiss m, return [M,,, Re....,,
23:
24:
end function
25:
procedure
8, Luuget: Faiss)
26: Build Faiss index by segmenting each d, E into chunks
27: Insert the embeddings of all chunks into Faiss
28:
for each document E D do
29:
{ m....... .... M.ppp
DOCUMENTT.HUNKKIN((;
30:
for + 1 to do + NEGATIVE MINGNIIV k, Faiss)
31:
32:
end for t < [11,12.....4]
33:
Concatenate into single long document
34:
Length)()) Lunga then T UUTUU {t}
35:
36:
end
37:
end for
38: return
Synthesized long documents for training
39:
end procedure
19


Reviews:
[{"review_id": "paPu3LZaE6", "reviewer": "Reviewer_syXt", "summary": "Most LLMs require long documents to handle long-context processing, but in practice, high-quality long documents are scarce. Existing methods typically concatenate short documents either randomly or based on similarity, which is not effective for learning long-range dependencies. This paper proposes splitting documents into multiple meta-chunks and inserting hard negatives—text segments that are semantically similar but unrelated to the actual context—between them. This encourages the model to distinguish between true context and misleading context, thereby enhancing its ability to learn long-range dependencies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.\n\n---\n\n**Q1: \"It would be better to explain why improving the long-context ability of LLMs can decrease the performance on short text benchmarks.\"**\n\n**A1:** \nPrevious long-context extension approaches [1] often degrade short-context performance because they rely exclusively on non-synthetic long documents, which are scarce across most domains (as noted in lines 35–37). This view also aligns with [2], which states that \"*a data mixture that keeps the domain mixture ratio the same as the pretraining mixture...this is the primary reason our solution improves long context tasks while maintaining short context performance.*\"\n\nQuest [3] demonstrates that using a long-text dataset synthesized from short texts with good domain diversity does not decline short-text performance. Following Quest, we also use short-text data sources with sufficient diversity, such as Cosmopedia V2, which covers over 34,000 topics. As a result, Table 5 demonstrates that NExtLong does not degrade short-text performance on average.\n\nThanks for your valuable suggestion! We will elaborate on this point in the revised version.\n\n**Q2: \"Compared to prior methods, the novelty of this approach feels somewhat limited. It would be helpful to more clearly highlight which aspects of the previous methods made them suboptimal.\"**\n\n**A2:** As mentioned in lines 51-53: \"*those methods typically concatenate short documents based on random or similarity-based rankings, lacking a clear mechanism for capturing long-range dependencies.*\" Previous studies (e.g., Quest [3]) synthesize long-context data by concatenating similar documents sequentially, allowing models to perform the next token prediction without relying on preceding documents. That concatenation strategy weakens the learning of long-range dependencies, and Table 1 shows that those methods achieve only marginal improvements compared to the Standard approach.\n\nWhile approaches like [4][5] increase dependency length using interleaved chunk data, they do not capture the real-world challenge of extracting long-range dependencies amid extensive distracting information. To verify this, we replace hard negatives with random chunks to implement a baseline (RandomD) similar to [4][5], and Figure 7 shows that it underperforms NExtLong.\n\nIn contrast, as mentioned in lines 83-88: \"*By inserting these distractors between originally dependent meta-chunks, NExtLong not only increases the distance between dependent chunks—effectively transforming the dependencies into long-range ones—but also introduces distracting noise.*\" **This design better reflects real-world scenarios where models must extract relevant information despite interference, thereby fostering more robust long-range dependency learning.**\n\nWe will provide a more detailed comparative analysis in the revised version. Thanks for your constructive suggestion!\n\n---\n\n[1] Chen Y, Qian S, Tang H, et al. Longlora: Efficient fine-tuning of long-context large language models. The Twelfth International Conference on Learning Representations (ICLR'24).\n\n[2] Fu Y, Panda R, Niu X, et al. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference on Machine Learning (ICML'24).\n\n[3] Gao C, Wu X, Fu Q, et al. Quest: Query-centric data synthesis approach for long-context scaling of large language model. The Thirteenth International Conference on Learning Representations (ICLR'25).\n\n[4] Tian, Junfeng, et al. Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. arXiv preprint arXiv:2409.04774 (2024).\n\n[5] Zhao, L., Wei, T., et al. Longskywork: A training recipe for efficiently extending context length in large language models. arXiv preprint arXiv:2406.00605 (2024).", "claims_and_evidence": "Yes, it's supported by the experiments section.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria seem make sense for the problem.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I checked the validity of the experimental designs. It would be better to explain why improving the long-context ability of LLMs can decrease the performance on short text benchmarks.", "relation_to_broader_scientific_literature": "Improving the ability of LLMs to handle long context is important, and this paper addresses how to effectively construct synthesized long documents.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n1. Writing is clear. \n2. The motivation behind the method is clear.\n3. The efficacy of the method is demonstrated through the experiments on HELMET and RULER benchmark.\n\n**Weaknesses**\n1. Compared to prior methods, the novelty of this approach feels somewhat limited. \n2. It would be helpful to more clearly highlight which aspects of the previous methods made them suboptimal.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQVu21unAC", "reviewer": "Reviewer_3FUJ", "summary": "The paper introduces NExtLong, a new framework designed to address the challenge of training LLMs with extended context windows, particularly in the face of limited availability of long documents. The key contributions of the paper are the usage of hard negative documents mining for the construction of long documents. Experimental results on RULER and HELMET show the strong performance of this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.\n\n---\n\n**Q1: Dependency on FAISS retrieval quality**  \n\n**A1:** The quality of FAISS retrieval depends significantly on the effectiveness of the embedding model. As embedding models advance, retrieval performance is expected to improve accordingly. Given ongoing advancements in this area [1][2], we anticipate these advancements will further enhance the performance of NExtLong and plan to explore the impact of more advanced embedding models in future work.\n\n---\n\n[1] Choi, Chanyeol, et al. Linq-Embed-Mistral Technical Report. arXiv preprint arXiv:2412.03223 (2024).\n\n[2] Wang, Liang, et al. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024).\n\n\n**Q2: \"computational cost for indexing\"**\n\n**A2:** The computational cost of indexing is only incurred during the generation of synthetic training data and does not impact the inference phase of the model. Since this is a one-time cost, the resulting performance improvements provide lasting benefits for downstream tasks. Given these long-term advantages, the computational expenditure is justified and contributes to overall performance.\n\n**Q3: \"not comparing against models using realistic long documents.\"**\n\n**A3:** We would like to clarify that **some of the models compared in Table 2 are indeed trained with realistic long documents**. For example, ProLong [1] uses realistic long documents (as mentioned in lines 323-325) and extensively explores how to utilize them effectively.\n\nTo ensure a fair comparison, Table 2 directly compares NExtLong with ProLong. It shows that Llama-3-8B-NExtLong-512K-Base outperforms Llama-3-8B-ProLong-512K-Base, further validating the effectiveness of NExtLong (lines 319-322).\n\n---\n\n[1] Gao, Tianyu, et al. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660 (2024).\n\n**Q4: \"Although I understand it is not the main point of this paper, but I am curious about if the proposed method can be applied to long documents to further extend the effective context length of the model? ...\"**\n\n**A4:** Based on the hypothesis that shorter original documents pose more significant challenges for a fixed target length and given that training a long-context model is resource-intensive, we prioritized creating a more challenging experimental setting within our limited training resources. Our current experiments deliberately use relatively short documents, creating an effective 64x increase in sequence length compared to the original documents (as mentioned in lines 651–652).\n\nDespite this challenging setting, NExtLong still achieved strong performance, **leading us to believe that when the original documents are longer, they can be merged into even longer sequences while maintaining promising results**. We are currently collecting longer documents and will systematically investigate the impact of documents exceeding 8K tokens in future work.\n\nThank you for your insightful question!", "claims_and_evidence": "The main claim is that the proposed method enhances long-context modeling by synthesizing data with hard negatives, addressing the scarcity of long documents.\n\nEvidence: It outperforms all baselines by a large margin (e.g., +7% over Quest) on HELMET and RULER (Table 1). The superior performance on long-context ICL is especially impressive.", "methods_and_evaluation_criteria": "Method: chunk documents into meta-chunks, retrieve hard negatives via FAISS, interleave them, and train with next-token prediction loss.\nEvaluation: Two commonly used long-context benchmark, including HELMET and RULER (several subtasks), measuring recall, RAG, ICL, re-ranking, LongQA, and synthetic tasks. Metrics include Accuracy and ROUGE F1.", "theoretical_claims": "No, no theoretical claim in this paper.", "experimental_designs_or_analyses": "Comparisons: Tested against KNN, Quest, and SOTA models (Llama-3 and other open models, with GPT-4o, Gemini-1.5-Pro and Claude).\nAblations: Analyzed chunking granularity (Figure 6), negative selection strategies (Figure 7), and dataset combinations (Table 7).", "relation_to_broader_scientific_literature": "Builds on contrastive learning (hard negatives) and long-context methods (e.g., ProLong, Quest). Addresses document scarcity, a key challenge in long-context model training.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: neat and good use of hard negatives, comprehensive evaluations, minimal performance drop on short-text tasks.\nWeaknesses: Dependency on FAISS retrieval quality, computational cost for indexing, not comparing against models using realistic long documents.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Although I understand it is not the main point of this paper, but I am curious about if the proposed method can be applied to long documents to further extend the effective context length of the model? For example, using a dataset which contains more documents > 8K length, and apply the proposed method on it to get a much better long-context performance on the evaluation benchmark.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9eeOmrYtK6", "reviewer": "Reviewer_hZK6", "summary": "The paper introduces the NExtLong framework, which aims to alleviate issues arising from the scarcity of high-quality long documents in long-context training. Traditional methods that concatenate shorter documents do not effectively capture necessary long-range dependencies, leading to problems with coherence and relevance. NExtLong addresses these challenges by decomposing long documents into meta-chunks and incorporating hard negative distractors from pre-training corpora. This method increases the difficulty of the training process, encouraging the model to differentiate between relevant long-range dependencies and distracting information, thereby improving its overall modeling capabilities. Experimental results indicate that NExtLong outperforms existing long-context synthesis methods and state-of-the-art models, yielding an average performance improvement of 7.33% on key benchmarks such as HELMET and RULER.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.\n\n---\n\n**Q1: \"I would like to suggest to include the above work in the related section. It is the first work try to use interleaved chunk data to improve long-context performance.\"**\n\n**A1:** We sincerely appreciate your valuable suggestion. We acknowledge the significance of the mentioned work as the first to explore the use of interleaved chunk data for improving long-context performance. **We will incorporate this important reference in the revised version** to ensure a more comprehensive discussion in the related work section. Thank you for bringing this to our attention.\n\n**Q2: \"Limited Novelty: The novelty is limited given the existence of previous works [1] and [2].\"**\n\n**A2:** We would like to emphasize that **our contribution is not using interleaved chunks**. As we explicitly state in lines 55–57:  *\"An intuitive approach to building documents with long-range dependencies is to insert additional text between dependent segments.[1]\"*  Additionally, we discuss the limitations of those methods in lines 62–65, noting that **they do not fully capture the real-world challenge of extracting long-range dependencies amid extensive distracting information.** \n\nWe replace hard negatives with random chunks to implement a baseline (RandomD) similar to [1][2], and Figure 7 shows that it underperforms NExtLong. The results indicate that our method better reflects real-world scenarios where models must extract relevant information despite interference, thereby fostering more robust long-range dependency learning (As mentioned in lines 83-88).\n\n**Q3: \"Compatibility with Modern Techniques: The proposed data preparation method is not compatible with the commonly used intra-document attention approach for long-context training.\"**\n\n**A3:** We would like to clarify that **our method is indeed compatible with intra-document attention**. In fact, **Llama-3-8B-NExtLong-512K-Base is trained using the intra-document attention approach and achieves strong results (Table 2)**. Specifically, we construct 512K and 64K datasets (lines 326–329) and apply intra-document attention to the 64K subsets, enabling their concatenation into 512K context length. This strategy aligns with ProLong[4] (lines 318–320), which uses a 64K dataset to train 512K models. Our results in Table 2 demonstrate that NExtLong performs better under this setup.  \n\n**Q4: \"For Table 1, I suggest that the authors provide the results averaged over 32K, 64K, and 128K in the appendix, following the same evaluation protocol as in ProLong.\"**\n\n**A4:** Thank you for the suggestion. We will incorporate the results averaged over 32K, 64K, and 128K in the appendix. We report the averaged results in the table below, and NExtLong maintains a performance advantage across sequence lengths beyond 16K, further supporting the effectiveness of our approach.\n\n| Model                              | Avg.  | Recall | RAG   | ICL   | Re-rank | LongQA | RULER |\n|------------------------------------|-------|--------|-------|-------|---------|--------|-------|\n| Llama-3-8B-ProLong-512K-Base       | 60.34 | 82.38  | 60.10 | 84.07 | 24.77   | 33.44  | 77.26 |\n| Llama-3-8B-NExtLong-512K-Base      | **64.71** | **87.96**  | **62.42** | **88.67** | **25.81**   | **41.27**  | **82.14** |\n\n**Q5: \"In Table 2, the results are averaged over 8K, 16K, 32K, 64K, and 128K. Does Table 5 follow the same procedure? I suggest that the authors provide more detailed explanations of the experimental setup.\"**\n\n**A5:** Yes, **Table 5 follows the same reporting procedure**. Specifically, the \"Head\" strategy in Table 5 corresponds to the NExtLong results in Table 1.  \n\nWe sincerely apologize for any confusion caused by the insufficient explanation of the experimental setup. We will include more details in the revised version. Please let us know if any further information would be helpful.\n\n---\n\n[1] Tian, J., Zheng, D., Cheng, Y., Wang, R., Zhang, C., & Zhang, D. (2024). Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. *arXiv preprint arXiv:2409.04774.\n\n[2] Zhao, L., Wei, T., Zeng, L., Cheng, C., Yang, L., Cheng, P., ... & Zhou, Y. (2024). Longskywork: A training recipe for efficiently extending context length in large language models. *arXiv preprint arXiv:2406.00605.\n\n[3] Wang, H., Liu, Q., Du, C., Zhu, T., Du, C., Kawaguchi, K., & Pang, T. (2024). When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training. *arXiv preprint arXiv:2411.13476.\n\n[4] Gao, T., Wettig, A., Yen, H., & Chen, D. (2024). How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660.", "claims_and_evidence": "This work claims that the proposed data cooking method is effective for long-context training. This main claim is supported by clear and convincing evidence. In the experimental section, the authors compare the proposed method with various alternatives and demonstrate the effectiveness of hard negatives through ablation experiments (section 5.4).", "methods_and_evaluation_criteria": "The proposed method is evaluated on the two most important long-context benchmarks: RULER and HELMET.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is standard and sound.", "relation_to_broader_scientific_literature": "The key findings regarding the data composition strategy have the potential to enhance the long-context training methods used in current LLMs.", "essential_references_not_discussed": "Zhao, L., Wei, T., Zeng, L., Cheng, C., Yang, L., Cheng, P., ... & Zhou, Y. (2024). Longskywork: A training recipe for efficiently extending context length in large language models. arXiv preprint arXiv:2406.00605.\n\nI would like to suggest to include the above work in the related section. It is the first work try to use interleaved chunk data to improve long-context performance.", "other_strengths_and_weaknesses": "In general, this work is solid, with well-presented experimental results.\n\nWeaknesses:\n1. Limited Novelty:\n   The novelty is limited given the existence of previous works [1] and [2]. Compared with [1], this work omits training strategies like Knot Tokens and instead introduces a negative sample data preparation strategy. Moreover, [2] was actually the first to use interleaved chunks to achieve improved performance. The authors should, at a minimum, include [2] in the related work.\n\n2. Compatibility with Modern Techniques:\n   The proposed data preparation method is not compatible with the commonly used intra-document attention approach for long-context training. After Llama3 demonstrated the effectiveness of intra-document attention, in terms of both performance improvement and reduced memory requirement, the Qwen series and recent research works [3, 4] have adopted this approach. While the proposed method can improve the older full attention mechanism, it may not compare favorably with emerging methods.\n\n----\n\n[1] Tian, J., Zheng, D., Cheng, Y., Wang, R., Zhang, C., & Zhang, D. (2024). Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. *arXiv preprint arXiv:2409.04774.\n\n[2] Zhao, L., Wei, T., Zeng, L., Cheng, C., Yang, L., Cheng, P., ... & Zhou, Y. (2024). Longskywork: A training recipe for efficiently extending context length in large language models. *arXiv preprint arXiv:2406.00605.\n\n[3] Wang, H., Liu, Q., Du, C., Zhu, T., Du, C., Kawaguchi, K., & Pang, T. (2024). When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training. *arXiv preprint arXiv:2411.13476.\n\n[4] Gao, T., Wettig, A., Yen, H., & Chen, D. (2024). How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660.", "other_comments_or_suggestions": "1. For Table 1, I suggest that the authors provide the results averaged over 32K, 64K, and 128K in the appendix, following the same evaluation protocol as in ProLong.", "questions_for_authors": "1. In Table 2, the results are averaged over 8K, 16K, 32K, 64K, and 128K. Does Table 5 follow the same procedure? I suggest that the authors provide more detailed explanations of the experimental setup.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZDfW5qpNde", "reviewer": "Reviewer_HH7d", "summary": "This paper introduces NExtLong, an effective framework that improves long-context modeling in LLMs through negative document extension. It first divides a document into meta-chunks and then inserts hard negative distractors to force LLMs learn the dependency between long documents. Experimental results have illustrated the efficiency of NExtLong.\n\n## update after rebuttal\nThe author's rebuttal addressed my previous concerns and thus I have updated my score correspondingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.\n\n---\n\n**Q1: \"Regarding Table 1, it seems that NExtLong is using extra negatives when pre-training on the two datasets compared to other baselines, meaning that it may use more training examples, which makes the comparison not fair.\"**\n\n\n**A1:** Thanks for your question. We appreciate the opportunity to clarify this point.\n\nWe would like to clarify that **NExtLong does NOT use any additional training example (lines 251-256)**. The negatives utilized by NExtLong are also drawn from the two datasets (lines 208-212). Specifically, we construct a FAISS index from these datasets, and NExtLong retrieves hard negatives from the FAISS index to synthesize long-context data. The FAISS index used by the baseline methods is also built from the same datasets; **the only difference between those methods and ours is how they rearrange documents into long-context data.**\n\n\n**Importantly, the total number of training examples and the experimental settings are identical across all methods, ensuring a fair comparison.** This is explicitly stated in the following sections of our paper:  \n\n- Lines 208-212: \"*Various methods, including NExtLong and baseline approaches, are employed to synthesize target-length samples concatenated from these short documents.*\"  \n- Lines 251-256: \"*...The same training configuration is applied to all methods for a fair comparison.*\"  \n- Lines 263-272 provide further details on how baseline methods synthesize long-context data.  \n\nWe apologize for any confusion and hope this clarification addresses your concern. Please let us know if any further details would be helpful.", "claims_and_evidence": "The effectiveness of the proposed method NExtLong is proved by the experimental results.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No issues.", "experimental_designs_or_analyses": "Experimental settings are good except Table1 (please see the weakness section)", "relation_to_broader_scientific_literature": "The proposed method is novel and effective.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n\n1. Regarding Table 1, it seems that NExtLong is using extra negatives when pre-training on the two datasets compared to other baselines, meaning that it may use more training examples, which makes the comparison not fair.\n\nI am happy to raise my score if the authors could answer my question clearly (or point out my misunderstanding)", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles a critical bottleneck in long‑context LLM pretraining—namely, the scarcity of high‑quality, naturally long documents—by proposing NExtLong, a synthetic data framework that interleaves “hard negative” distractors between contiguous meta‑chunks. By forcing the model to distinguish true long‑range dependencies from semantically similar but irrelevant text, NExtLong significantly boosts long‑context understanding without degrading short‑text performance. Empirical results on HELMET and RULER benchmarks demonstrate consistent gains (≈7 % average improvement) over strong baselines—including Quest, ProLong, and Llama‑3 variants—across multiple context lengths (8 K–512 K).

All reviewers elevated or maintained their scores after rebuttal. The paper addresses an important problem, presents a clear, well‑motivated method, and achieves substantial empirical gains. While the core idea bears similarity to previous work (e.g., LongSkywork), NExtLong’s use of retrieval‑based hard negatives and its systematic study across extremely long contexts (up to 512 K) represent a meaningful advance. The requested clarifications and added comparisons can be accommodated in the camera‑ready.
Decision: Accept (poster)