Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation
Authors: Linda He
Jue WANG
Maurice Weber
Shang Zhu
Ben Athiwaratkun
Ce Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Long Context
Instruction-Tuning Data
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts surpassing 100K tokens. To bridge this gap, we introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data, which effectively addresses the scarcity of raw long-context data. 
Through a step-by-step rotary position embedding (RoPE) scaling training strategy, we demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench and maintains robust performance on general language tasks.

Published as a conference paper at ICLR 2025
SCALING INSTRUCTION-TUNED LLMs TO MILLION- TOKEN CONTEXTS VIA HIERARCHICAL SYNTHETIC DATA GENERATION
Linda He Jue Wang² Mauriee Weber2 Shang Zhuz Ben Athiwaratkun Ce Zhang
'Harvard University TToeeherr AI *University of Chicago
Lindaheecollege hww.haaaaaadd... {jue, maurice, shang, ben/etoogtther.a.. cez@uchicago..cr
ABSTRACT
Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts sur- passing 100K tokens. To bridge this gap, we introduce novel post-training syn- thetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of avail- able real-world data, which effectively addresses the scarcity of raw long-context data. Through step-by-step rotary position embedding (RoPE) scaling training strategy, demonstrate that our model, with context length of up IM tokens, performs well on the RULER benchmark and InfiniteBench and maintains robust performance general language tasks.
INTRODUCTION
The capabilities of Large Language Models (LLMs) have significantly advanced, enabling impres- sive performance across a wide range of natural language processing tasks (Wu et al., 2023 Jiang ct al. [2023 Wei et al. 2022). However, managing long contexts remains major challenge, which limits the practical utility of LLMS in tasks such as document comprehension and summarization. code generation, lifelong conversations, and complex agent scenarios (Liu et al.) [003) Meng al.) 2023). Extending context lengths to IM tokens marks a critical breakthrough for applications re- quiring processing beyond 128K token limit. For instance, company-wide document retrieval benefits from efficiently analyzing extensive organizational histories stored in unstructured formats, while interconnected project timelines and legal documents gain from enhanced reasoning across multi-document datasets.
To extend the context length of LLMs, current approaches focus on either architectural innova- tions like efficient attention mechanisms (Katharopoulos et al. 2020 Gu & Dao 2024) or scaling positional embeddings (Chen et al. 2023 Peng al.] (023) and continual pretraining on natural long-form data, such books and web data. However, the RULER benchmark (Hsieh et al.) (024)) shows that many models struggle to maintain consistent performance as context length increases, when claiming to support longer contexts. This highlights the need for high-quality instruction data to fully utilize the nuances long-fom content. Acquiring such data is challenging and costly, as open-source datasets often fall short in document length, relevance, and tasks requiring genuine long-range understanding. To date, no open-source instruction-tuning datasets exceed 100K tokens, creating significant gap between theoretical and practical long-context capabilities of LLMs (Li et al. 2024 Zhao et al. 2024)..
To address limitations in extending LLM context length, propose an effective long-context in- struction data generation pipeline, as illustrated in Figure Our pipeline leverages short-context
Work done during an intersship Together AL.

---

Published as a conference paper at ICLR 2025
Depth Broodth Sesoration
Long Context becument
Hararchy-owere QA
Molti-op Q4
Sonn Bellomnrn pnd Butt
Foumsdey 2.2
? Pool of Locel Summaries
loch camaryy satanued
hartine uies orter From
Paclleene Bosug
stocal
Specific
closs:
Complet Diverse Q4
Niva postett astromation
Hicrochical Summory Sampling
LLM Generator
booelorer, grscrete neterr boord
Sloesl
Dac sae Touaes (eatasdas, Daste
Nowes cacked Lediosation
Q
Phr netoos prociee This apneice
Conenire boaed foo Thenter
infernation.
compuras
Figure 1: High-level overview over our approach automatically generate QA pairs for long context documents. (1) In the first step, we split a document into small and medium chunks which are then (2) summarized by an off-the-shelf L.LM requiring only smaller context windows. In (3) we sample summaries at different localities in hierarchical manner, balancing local and global views of the original document. In (4) we generate questions based on the sampled summaries. In the right panel, show subset of prompts used to generate diverse and complex questions, given the sampled summaries.
models to create long-context instruction data using three key methods: (a) Hierarchical question ordering: structuring questions in a logical sequence to ensure coherent reasoning across contexts; (b) Diverse question type pool: maintaining wide range of question types, including hierarchical- aware, multi-hop, local-specific, and other complex types to handle varied tasks; and (c) Multi- document integration: incorporating multiple documents to generate data with arbitrary context lengths. The contributions of this paper are threefold:
Extensive and scalable long-context data generation strategy: We present, to the best of our knowledge, the first extensive strategy for synthetically generating long-context data with com- prehensive ablation tests and evaluations. Our highly scalable approach is unconstrained by the length of available real-world data, effectively combining multiple documents with diverse, com- plex questions. This hierarchical method ensures logical coherence and sequence integrity.
2. Extensive evaluation of core strategies: We conduct extensive evaluations shorter context lengths (100K and I80K) to demonstrate the effectiveness of our hicrarchical strategy, multi- document combinations, and diverse question-answee pair generation. These evaluations validate that our core strategies work well across various tasks and context lengths.
3. Scaling to 1M context length: We successfully extend LLaMA-3.1--0 1.-AA--.--B--IISSS I-8B-Instruct a context length of million tokens. Our model significantly outperforms the LLaMA-3.1-BB-Instee model in zero-shot RoPE scaling to a 1M context window on the RULER benchmark and sur- passes the Pradientai/Llamm-33 3-BB 3-8B--Inee ditionally, our model outcompetes strong performance on LongBench and MMLUU
1048k model trained by Gradient AL. Ad- 1---A--.----B-A.A
on InfiniteBench while maintaining
The remainder of this work is organized as follows. In Section we place our work in the land- scape of existing literature around methods to address long context capabilities of LLMs. Section 3 presents our method for generating long-context instruction tuning data. Our approach is then val- idated in Section 4 with a series of extensive and representative experiments. Finally, we conclude
in Section 5
2 RELATED WORK
Adapting transformers to enable longer context capabilities is a critical area of research in natural language processing. This effort primarily focuses three key directions: (1) architectural modifi- cations to the transformer model itself, (2) improvements in positional embedding techniques, and (3) the development and utilization of more extensive long-context datasets.
Efficient Attention Mechanisms. To address the quadratic computational and memory demands of standard transformer self-attention, researchers have developed various architectural modifica-
2

---

Published as a conference paper at ICLR 2025
tions to improve efficiency and extend context lengths. Notable examples include Longformer (Beltagy et al. 2020), which combines sliding window and global attention, and Bransforafrrrmr (Ho et al. 2 which employs hierarchical global-to-loall modeling, Linear Attention methods Katharopoulos et al., 2020) reformulate self-attention for linear complexity, while Infinite'Trans former (Munkhdalai et al., 2024) incorporates unbounded long-term memory through continuous- space attention. State space models like Mamba (Gu & Dao (202) capture long-range dependencies efficiently without explicit attention mechanisms. Despite these advancements, bridging the gap with high-quality data remains critical challenge and is the focus of this work.
Position Embedding Extension. Advances in positional encoding methods have enabled language models to handle longer sequences effectively. Techniques like RoPE (Su et al.) (023). ALiBi (Press al.] 2022), and xPos (Sun al., 2022) have emerged as prominent solutions. RoPE has gained widespread adoption in LLaMA (Touvron et al., 2023), b) and PaLM (Anil et al., 2023), due to its ability to represent relative positions and its theoretical grounding the complex plane. A breakthrough showed that RoPE's embeddings could extend to longer contexts with minimal or no fine-tuning (ee et al. 2024), leading to two key approaches: Positional Interpolation (PI)
(Chen et al. 2023) which linearly scales positional indices to extend context length, and NTK- Scaling RoPE (Peng et al., (023) which combines high-frequency extrapolation with low-frequency interpolation. While these developments improve model performance with longer inputs, they rely heavily on limited long-context data for fine-tuning.
Long Context Data. Recent work, such as LongT55 (Guo al. 2022 and LongAlpaca (Chen it al. 2024), has shown the benefits of additional pretraining on long sequences, enabling models to better capture extended context. Methods like combining multiple short-context sequences (Xiong et al. 2023) have also emerged as promising ways to efficiently extend context lengths. However, significant gap remains in generating high-quality instruction-tunm lengths. Few open-source efforts address this need. Our work introduces a scalable pipeline for generating long-context instruction-tuning data by systematically combining multiple documents, diverse questions, and hierarchical strategy to ensure coherence and structure.
data exceeding 100K context
Synthetic Data Generation. Synthetic data generation offers a promising path for scaling language models across diverse tasks and complex instructions. AutoEvol-Instruet (Zeng al. 2024). au- tomates the evolution of instruction datasets using large language models, reducing the need for extensive human intervention. WizardLM (Xu et al. (023) employs Evol-Instruct to iteratively evolve and scale instruction complexity, achieving strong results on benchmarks like MT-Bench and Vicuna's 's evaluation set. Auto Evol-Instruct (Zeng et al., 2024) further refines this process with an iterative evolution strategy, while Self-Instruct (Wang et al. 2023) enhances instruction-followii performance through data synthesis. Our work extends this research by generating long-context data tailored for instruction tuning.
3 METHOD
In this section, we describe our methodology for generating coherent instructions from a single document and scaling it to multiple documents to curate long-context datasets beyond the context length of available raw data. Section(3.Toutlinee our strategy for ensuring (1) quality and complexity and (2) coherent ordering of generated question-answer pairs. Section 3.2 expands on scaling to longer context lengths using multiple documents. Figure provides an overview of our long-context synthetic data generation pipeline.
3.1 COHERENT INSTRUCTIONS FROM SINGLE DOCUMENT
The quality of long-context instruction -utning datasets is driven by two key factors: (1) the com- plexity and diversity of the generated instructions, and (2) the structured ordering of questions and instructions. To address these, we devised a bifurcated strategy targeting each component.
Quality, Diversity, and Complexity of Instructions. As illustrated in Figure 1 our methodology for generating rich, diverse, and complex instructions leverages the key insight that short-context models can be used to generate long-context instruction-tuning data. The core approach involves dividing the input document into smaller chunks, typically 4K tokens, enabling models optimized for shorter contexts to process these segments with greater precision and clarity. We curate an initial set of prompts covering multiple dimensions of instruction complexity, such as temporal reasoning, thematic inquiry, and character-based scenarios (full set in Appendix(B) During question-
3

---

Published as a conference paper at ICLR 2025
Long Confext Decumant
QM Gkbbi Summoriattinn
④
Atteratt butwan Eapla
Deep Medium
Hecarciial and Summuriation
OR
Small Chuik
Figure 2: High-level overview over our approach to generate order-following QAs. (1) Input a raw long context document. (2) Split the document into small, medium, and global chunks, and generate summaries each level. (3) The first QA based on the global summary. (4) We randomly select medium chunk to generate QA, (5) then delve deeper by selecting a small chunk within it for another QA. (6) To continue, the process alternates between exploiting the same small chunk or exploring new medium small chunks to generate further QAs.
Genenate diverse and Nenarchcca Q
Extrrct hierarchical and diverse QAs from each decument
4
Concatenate QAA from different documents
approach to curate long context data using multiple doc- uments. (1) Diverse and hierarchical QAs are generated at different levels of granularity for each document. (2) N hierarchical and diverse QAs are sampled and extracted from each document. (3) QAs from different documents are combined, maintaining a balance of hierarchical and diverse questions across the entire set. N in our algorithm, and when we revisit previous documents in step (3), we sample 3 hierarchial questions for each document with 60 % probability as well as 9 total diverse questions from all previous documents.
Figure 3: High-level overview
answer pair generation, a small chunk and one question are randomly selected to generate a pair. To ensure broader contextual understanding, we incorporate multi-ho hop questions spanning 2-4 chunks, enabling cross-chunk question-anse -nnswer pairs.
hicrarchical strategy to split, summarize, and generate questions from long documents (see Figure 2 balancing exploration and exploitation. The document is first divided into large sections of 12K tokens, then into smaller 4K-token chunks linked hierarchically to connect broader and granular segments. The first QA is based on the global summary give high-level overview of the document. Then, we randomly select medium chunk to generate a QA, and then delve deeper by selecting small chunk within it for another QA. To continue, the process alternates between exploiting the same small chunk or exploring new medium or small chunks to generate further QAs. This iterative process ensures a balance between specificity and diversity, covering both localized details and broader document sections. The hierarchical structure ensures logical progression from broad QAs to detailed ones. The detailed algorithm and pseudocode are provided in Appendix A
Ensuring Coherent Order. To ensure logical and coherent QA generation,
3.2 EXTENDING TO LONGER CONTEXT LENGTHS USING MULTIPLE DOCUMENTS
Here extend methodology handle longer contexts by concatenating multiple documents and generating coherent hierarchical and diverse QA pairs across them. The workflow is visualized in Figure and the detailed algorithm is provided in Appendix A Below, we clearly define the pa- rameters Ni. N2, and N3. which govern the selection of hierarchical and diverse QA pairs, ensuring logical continuity and broad reasoning across documents. For each document, the process proceeds as follows:
1. N1 hierarchical QA pairs and N1 diverse QA pairs: After processing each document, Ni 5 hierarchical follow-up questions are added. These questions are designed to capture contextually related information within the document. creating logical order of reasoning and flow across sections. Moreover, another N = diverse QA pairs for this document is added as well, designed to capture specific details of the document.
4

---

Published as a conference paper at ICLR 2025
2. N2 diverse QA pairs: Next, N2 9 diverse QA pairs added. These questions sampled from all previously visited documents where diverse QA pairs have not already been sampled. This approach ensures cross-referencing between documents.
3. N3 revisiting hierarchical QA pairs: For every previously visited document, there is a 60% probability of sampling N3- = 3 hierarchical follow-up questions. These are added to revisit earlier contexts, fostering richer and interconnected understanding of the content.
This process is repeated iteratively for all K documents in the dataset to create a comprehensive instruction-tuning dataset that balances within-document reasoning, cross-document relationships. and revisiting earlier content for contextual continuity. We also present an example of a concatenated data example in Appendix C
4 EXPERIMENTS
In this section, we validate our long-context data generation approach through a series of exper- iments. In Section 4.2 we extend LLaMA-3.1 1-88-88-nstttet a 1M context-length model using stepwise RoPE scaling and hierarchical, diverse QA data generated by Qwen-2-72B. Our IM model delivers excellent results on ultra-long contexts while maintaining strong performance on short and medium-length contexts. In Section 4.3 we evaluate robustness using smaller and same-sized gen- erator models (Qwen-2.5-7B and L-MM--.1----I--S-
confirming our models achieve strong performance across ultra-Iong, short, and medium contexts. These findings highlight the scalability and effectiveness of our approach across generator model sizes. In Section 4.4] we present abla- tion studies showing how our hierarchical strategy and diversified questions significantly improve long-context instruction tuning, focusing on 180K with two documents.
4.1 SETUP
Models. We use 110---3.--8B-Intttrue as the base model for instruction-tuning given its capa- bility as a leading open-source LLM. To validate robustness, we employ various generator models for synthetic data: Qwen-2-72B- Instruct (large, high-quality data), Diven-2.5-7B-Inntrru (smaller), and
(same size). This demonstrates that our improvements are not reliant on very large models and that smaller models can achieve similar gains. We also benchmark against the 1aama-3--B-IIssiii
Gradient Al model trained on 1.4 billion tokens, showing that our method outperforms existing baselines.
1048k). a 1M context-length model
Hardware. We fine-tuned our models on a SLURM cluster using 8 to 32 H100 GPUs across up to 4 nodes, connected via InfiniBand for efficient multinode training. We used FSDP to shard the model across GPUs and implemented DeepSpeed Ulysses sequence parallelism for long-context training.
Datasets. Our primary dataset is the Together long books datase processed into approximately 1.4 billion tokens, distributed across these stages: 2000 samples of 180K tokens, 1280 samples of 350K tokens, 600 samples of 650K tokens, and 200 samples 1M tokens. We generated 582,900 QA pairs with hierarchical and diverse questions for robust instruction-tuning using the Together AI inference API 2 By sending 32 simultancous API requests, took about two days to create our full long-context instruction dataset, comprising 7,772 books. For each book, generated 25 hicrar- chical and 50 diverse questions, resulting in 582,900 QA pairs alongside global summaries. During training, we calculate loss solely answers, masking out questions and context ensure the model focuses on reasoning and generating accurate answers without being penalized for reproducing input content.
Evaluation Protocol. We evaluated our models using: 1) InfiniteBench (Zhang et al. 2024 De- signed for LLMs on extended contexts, it includes tasks like key-value retrieval, summarization, and QA on data exceeding 100K tokens. We evaluated the first 150 samples per task, excluding coding tasks as our data lacks code. 2) LongBench (Bai al. 2024): Focused on medium-context tasks (10K tokens), it assesses summarization, QA, and fact-checking across multiple domains, of- fering a measure of general capabilities. We excluded coding tasks. 3) RULER (Hsieh al.) (024)) RULER is a synthetic benchmark designed to evaluate how well models handle complex, woald tasks in long contexts. Unlike traditional retrieval-based tasks like aaalle-in---Hasstad (NIAH), which focus extracting specific pieces of information from distractor texts, RULER tests models'
https://api. xyz/
5

---

Published as a conference paper at ICLR 2025
100
110
50-
50
63X
L31K
200K
20
1304
20C
524e
2255. 2828
5246
Seguance Longth
Sequuerion berggh
Sequerce Lengen
(a) Context length 350K
(b) Context length 650K
(c) Context length 1M
Figure Effective context length up 1M tokens using Qwen-2-724 72B- 12s-ruutt as generator on RULER.
Table I: Model performance on InfiniteBench (100K tokens) using Qwen-2-72B-Insttne generator.
LLaMA-3.1-
gradient- ai-model
Metric
180K
350K
650K
1M
8B-Instruct
Cetrieve.PassKe
100.00
100.00
100.00
100.00
100.00
100.00
Retrieve.N Numbber
95.33
99.83
99.33
100.00
100.00
100.00
Retrieve.KY
42.66
15.60
88.66
92.00
63.33
57.33
En.Sum
27.63
17.02
24.01
23.51
23.68
23.06
En.QA
24.83
14.31
34.26
33.23
31.72
31.97
En.MC
68.00
57.20
74.00
72.00
75.33
74.00
En.
16.66
5,00
18.00
18.00
22.00
16.00
Math.Find
35.33
19.42
37.33
35.33
36.00
36.00
Average
51.31
41.04
59.45
59.26
56.51
$4.80
ability to comprehend deeper relationships and manage long-range dependencies. Given specified context length, RULER generates synthetic tasks across multiple categories, including multi-hop reasoning and document tracing, and measures the model's accuracy. In our evaluation, we sampled 130 tasks for each context length across 13 categories, totaling over 150 million tokens. 4) MMLU (Hendrycks et al.] 2021): This benchmark evaluates general model performance across multiple domains, assessing both breadth and depth understanding. It includes tasks spanning STEM, hu- manities, and social sciences, with varying difficulty levels. MMLU ensures that improvements in handling long-context tasks do not cause regression in overall model capabilities.
4.2 MAIN RESULTS: SCALING UP TO LONGER CONTEXT LENGTHS (350K, 650K, 1M)
To extend Llama-3.1 B.ttna--.--BB-nie Instruut to 1M context model, applied stepwise rope scaling. Training started with 180K tokens and progressed through checkpoints at 350K, 650K, and 1M tokens, con- catenating 4, 8, and 12 documents as per the algorithm in Section We compiled 2000 samples 180K, 1280 at 350K, 600 at 650K, and 200 at IM tokens. Data was generated using Qwen-2-72B. 6e-5 learning for 1 epoch. Training
fine-tuned on Llama-3.1 1.ama-3.1-8B-Imme -nsttrruttttt with rope scaling the 650K model took 30 hours, and the 1M model took an additional 52.5 hours.
An earlier ablation test combining two documents (Section 4.4) showed that combining hierarchical and diverse questions with a fixed number of QAs and global summarization is optimal for handling long contexts. We extended this setup for ultra-long context data, with each document followed by N1 5 hierarchical and Nj = 5 diverse questions. When revisiting previous documents, there is 60% chance of extracting N₂ 3 hierarchical question from each document and N3 diverse questions sampled from all prior documents.
Figure shows the effective context lengths of the 350K, 650K, and 1M models on the RULER benchmark. For comparison, we performed zero-shot rope scaling on the LLaMA-3. 1-8-IIntetttt model and included results using input truncation for context lengths above 128K as an additional baseline. On contexts shorter than 128K, our models performed comparably to LLaMA-3. 1-BB- Instruct and surpassed it with zero-shot rope scaling. This demonstrates the robustness of our mod- els short and medium contexts. For contexts longer than I 128K, models significantly outper- formed both baselines, with their strengths becoming more evident as context length increased. Raw evaluation results in Appendix D

---

Published as a conference paper at ICLR 2025
Table Model performance on LongBench (10K tokens) using Qwen-2-72B-Insee as generator.
LLaMA-3.1- 8B-Instruct
Gradient- AI-Model
180K
350K
650K
1M
Single Document
46.91
30.71
45.83
45.88
45.24
45.15
Multi-Document
41.45
12.45
41.71
41.75
41.13
41.29
Summarization
26.10
21.72
25.14
24.97
24.26
24.98
Few-shot Learning
63.48
59.69
62.22
61.66
60.00
59.27
Synthetic Tasks
67.48
55.50
68.17
67.50
65.00
66.42
All
48.11
35.89
47.58
47.34
46.18
46.42
Table 3: Model performance on MMLU using Sowen22-7BBInntrruue as the generator.
LLaMA-3.I- 8B-Instruct
gradient- ai-model
Category
350K-model
650K-model
1M-model
mmlu
68.21 + 0.37
60.48 0.39
66.29
0.38
65.80
0.38
65.08 1: 0.38
humanitics
64.23 + 0.67
55.75 = 0.69
61.51 0.68
61.02 + 0.68
61.02 + 0.68
other
73.03 + 0.77
67.04 = 0.82
72.84 + 0.77
71.84 + 0.78
71.84 + 0.78
social sciences
77.48 + 0.74
70.46 = 0.80
76.81 0.74
75.27 = 0.76
75.27 = 0.76
stem
60.36 + 0.83
51.32 = 0.86
59.44 0.84
57.72 0.84
57.72 + 0.84
To further validate our approach, we compared it to the Gradient Al model Batiiientiiillmaa-----133 Instruct- Stradient-10488 1M context model, on InfiniteBench, LongRench, and MMLU bench- marks. Table | compares models across context lengths InfiniteBench, while Table 2 focuses on LongBench. All our models (180K, 350K, 650K, IM) consistently outperforms the Gradient Al model on InfiniteBench, showcasing the effectiveness of our hierarchical, diversified QA-based nata-generation strategy. The 180K and 350K models scored 59.45 and 99.26, significantly exceed- ing the LLaMA-3.1-888 1-..-...-----nnttreee bascline of 5..31. The 650K model scored 56.51, and the 1M model achieved a strong 54.80.
Notably, while the Retrieve.KV task shows the most significant improvement, tasks like Re- trimbe,Number, En.MC, and Math.Find also display meaningful gains. The improvement on Re- trieve.KV stems from our data-generation methodology, which uses a structured mix of hierarchical and diverse questions while revisiting prior documents. This encourages the model to associate relevant sections, aligning with the demands of key-value retrieval and RAG techniques. where ac- curate context memorization is critical. Beyond key-value retrieval, our model excels on other tasks: on En.MC, the 650K model scored 75.33, surpassing the bascline (68.00) and Gradient Al model (57.20). On Find,Find, it scored 36.00 at 650K, outperforming the Gradient AI model (19.42), showcasing improved reasoning capabilities.
As shown in Table our models maintain robust short-context performance on LongBench, despite being trained for significantly longer contexts (up to IM tokens). For example, our 1M context- length model achieves an average score of 46.42, comparable to the baseline LLaMA-3.1- Instruct model 148. This demonstrates that while optimized for ultra-long contexts, the model generalizes effectively to shorter contexts, such those on LongBench. Minor regressions in tasks like summarization are due to trade-offs when training for extended contexts. As the model adapts to handle extremely long contexts, small task-specific adjustments may impact short-context perfor- mance. However, these regressions are minimal and expected, given the differences between short- and long-context tasks, Despite these trade-offs, our model consistently outperforms the Gradient AI model (35.89) on all LongBench tasks, demonstrating the effectiveness of our hierarchical and diversified instruction tuunonnttttnnnnnnnn approach.
As detailed in Table 3 our model demonstrated minimal regression in general task performance despite significant improvements in ultra-long-conte tasks. For instance, our model retained com-
The results dropped likely due to multi-node training, we helieve 650K and 1M models are under- trained because of the extended time required to train and the communication overhead from NCCI.

---

Published as a conference paper at ICLR 2025
100
90
90
Accuracy
80
Accuracy
80
70
70-
60
60
50
50
40
40
30
30
0OK
131K
202K
524K
Sequence Length
Sequence Length
(a) 350K model using Owen-22.577B I--nsrrruct as generalor.
(b) 650K model using Qwen-2.5-78 Instruct as generator.
90
Accuracy
Accuracy
80
70-
60
50
40
30
8K
65K
131K
262K
131K
262K
521K
Sequence Length
Sequence Length
(c) 350K model using 1lama-31188--Inttie generator.
(d) 650K model using Llama-3. 1-St-nnstrett generator,
Figure 5: Effective context length using Llama-3. 3.1-8B-Innnrrue Qwen-2.5-7B-Instee as generators on RULER.
Table 4: InfiniteBench performance with Llama-3 3.1 1-8B-Innsreen and Qwen-2.5-77 2.cc--225-B-Insttrucc as generators.
LLaMA-31-
gradient- ai-model
180K-
350K-
650K-
180K-
350K-
650K-
SB--Inttrutt
lamm-ggen
Bamm-gee
llama-gen
qwen-gen
qwen-gen
qwen-gen
Retrieve. Pass Key
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Retrieve. Number
99.33
99.76
Retrieve.KV
42.66
13.33
85.47
89.33
42.14
89.52
85.33
52.66
En.Sum
27.63
17.02
25.68
26.85
26.64
26.97
27.70
26.74
FA.QA
24.83
15.84
33.39
35.67
33.37
32.30
29.55
29.67
Er.MC
68.00
61.33
58.00
60.66
66.00
63.33
61.33
64.66
16.66
4,00
20.00
Math.Find
35.33
26.66
36.66
32.66
35.33
30.00
34.66
38.00
Average
51.31
42.19
57.22
57.48
52.94
58.65
57.49
54.38
petitive MMLU scores (e.g., 68.21 0.37 for the baseline and 65.08 + 0.38 for the 1M model), whereas the Gradient AI model showed marked degradation both MMLU and LongBench. This reinforces the robustness of our method, ensuring that gains in ultra-long-contet models excel ultra-long-context tasks
performance do
not compromise broader capabilities. In conclusion, on RULER and InfiniteBench, outperforming the base LLaMA-3. 1---A--..-8--Inaae 1-8B-Instruct and Gradient AI models while maintaining strong performance general tasks like MMLU and LongBench.
4.3 VALIDATING ROBUSTNESS ACROSS GENERATOR MODELS
To validate that observed improvements are not solely due to using large generator model (e.g.. Qwen-2-72B), we trained and evaluated models with Qwen-2.5-7B and 1..MA--..--B-Instraet as generators. By employing smaller or similarly sized models, we demonstrated the robustness and generalizability of our hierarchical QA data-generation strategy. Additionally, we benchmarked against the Gradient AI model ggradientai/Llama- ---llaa--------seee----- Instrruct-rradeen-- 1048k), a 1M context --a-aaaa-------snnn--nnaaaaddd
model trained on 1.4 billion tokens. While our models were trained only up 650K tokens to validate the approach, the same method can seamlessly scale to 1M tokens. Our models outper- formed the Gradient AI baseline across all long-context benchmarks, achieving higher accuracy on InfiniteBench and RULER, while preserving general task performance MMLU and LongBench.
Figure 5) highlights effective context length using Llama-3. 1-B--....-----stteeee and Qwen-2.5-7B as gen- erators on RULER. On all settings (350K, 650K), our hierarchical approach outperformed the Gra- dient Al model and the zero-shot baselines across context lengths. Table summarizes results on InfiniteBench (100K context length). Our approach again consistently outperformed both the base LLaMA-3.1 1--MA-3,--BB-Instruue IB-Inttruct model and the Gradient Al model. This demonstrates that even smaller generator models produce high-quality data for instruction-tuning.
8

---

Published as a conference paper at ICLR 2025
Table 5: LongBench performance Llama-3.J 1.1ma-3.1181 3.-suuctttt and Qwen-2.5-7B Insstrrcctt as generators.
LLaMA-3.1- 8B-Instruct
gradient- ai-model
180K-
350K-
650K-
180K-
350K-
650K-
Task
Haamagge
Hamaa-gen
Haama-gen
qwen-gen
qwen-gen
gwen-gen
single-document
30.75
46.48
46.53
46.20
46,70
46.28
multi-document
41.45
12.45
38.69
38.75
37.54
40.76
41.90
39.31
summarization
26.10
21.72
25.28
25.10
24.68
25.05
24,83
24.90
shot learning
63,48
59.70
61.56
62.79
60.50
61.92
61.56
60.69
synthetic tasks
67.48
55,50
66.17
67,75
66.00
67.11
67.60
67.10
Average
48.11
35.89
47.23
47.72
46.20
47.95
47.97
47.00
Table MMLU performance with Llama-3. 1--ma-3.-8B-IIntrucc Qwen-2.5-7B-Instmmn. as generators.
Category
LLaMA-3.- uu-Instruct
gradient- al-model
180K-
350K-
650K-
180K-
350K-
650K-
Ilama- gen
Bama-gen
Ilamn-
qwen-gen
mmlu
68.21 0:37
60.48 = 0.39
66.90 0.38
66.74 2 0:38
65.93
0.38
67.33
0.38
65.78 0.33
64.60
0.38
hamanities
64.23 + 0.67
55.75
0.69
62.52 + 0.67
61.38 +: 0.68
00.57
0.68
62.81
+ 0.67
59.68 + 0.68
59.45
0.68
0:77
0.76
social sciences
77.48
0.74
0.74
75.53
0.74
75.66
71.87
0.77
stens
60.36 + 0:83
51.32 = 0.86
58.67 3 0.84
58.61 + 0.84
57.72
0.84
58.77 + 0.84
58.14 0.84
56.49
0.85
Table model performance LongBench (10K context length). Despite being optimized for ultra-long contexts, our approach retains strong performance on shorter contexts, comparable the generator, our model scored 47.00 at 650K, closely matching LLaMA-3. 1--M---.---B---- 1--8B--nsstuccts 48.11. Our model also outper- forms Gradient AI (35.89) across all LongBench tasks. Table shows our models' minimal regres- sion in MMLU performance. The 650K trained using LLaMA-3. 1---MA-..1-BBImmaa -B-Inrttt as generator scored 65.93 0.38, close L.LaMA331 1-8--nsttttntttrutt (68.21 + 0.37). In contrast, Gradient Al showed notable regression. This underscores our hierarchical approach's 's ability to support long-context learning while maintaining general task performance.
LLaMA-3.1 1-8B-Instruct. For example, with Qween225-7B-Inntrruct
4.4 ABLATION STUDIES
Our 100K context length single-document ablation studies, detailed in Appendix demonstrate that hierarchical ordering significantly boosts performance, particularly when combined with diverse question sets. Configurations with hierarchical ordering consistently outperformed those without, highlighting its importance for structuring instruction-tuning data. These findings provide solid foundation for extending our experiments to larger context lengths and exploring the interaction of hierarchical and diverse question compositions. Building on these results, we expanded our experimentation to a 180K context length combining two documents, aiming to determine whether the patterns observed at 100K scale effectively with rope scaling. We also explore which question types (hierarchical or diverse and complex) perform best for questions directly following documents referencing previous ones.
For each experiment, we generated 300-600 training samples of 180K tokens (concatenating two documents) using Qwen-2-72B and fine-tuned the data L.LMA-331 188---1-----n-ttuu with learning rate of 6e-5 for epoch. As the 180K context length exceeds LLaMA-3. ---.M--..--------- 1-81-----Bttttc native 128K window, we applied rope scaling. The following compositions were tested: a) Random
fixed number of questions: Follow-up questions were either randomized (2-10) or fixed (6 main and 4 follow-up) to maintain consistency, b) Hierarchical tested hierarchical ordering questions (h) against questions targeting specific, diverse, and complex reasoning (s). Each experiment is labeled as x-y-z, where x refers questions following the first document, the second, and to questions referencing the first document after the second is pro- cessed. For instance, h-h-s-fixed includes hierarchical questions for each document and diverse follow-ups referencing the first document after the second. c) Summarization: Some experiments excluded global summarization at the start to assess its impact on model comprehension.
diverse and complex questions: We
Table shows the ablation results InfiniteBench. Notably: 1) All experiments outperformed the baseline LLaMA-3.1 1.1----..-BBB.. In-Irutt model by a significant margin, demonstrating the effective- ness of our strategy with rope scaling. 2) Fixed questions outperform randomized ones: hs-hs-hs- fixed scored 59.45, surpassing hs-hs-hs ---bs-hs-randdmized (58.51). 3) Hierarchical questions paired with diverse questions achieve the best performance: hs-hs-hs-fixed yielded the highest score (59.45),

---

Published as a conference paper at ICLR 2025
Table 7: Ablation study on InfiniteBench with 180K context length. Each experiment labeled as x-y-z, where x is the type of question after the first document, y is the type of question after the second document, and the type of question referencing after the second document is processed. For example, h-hhs-fixed is the dataset with 6 hierarchical questions following the first document, 6 hierarchical questions following the second document, and follow-up diverse questions referencing the first document after the second document is processed. Randomized signifies that the number of questions sampled is randomized, and no-sum signifies that the global summary is removed.
LLaMA-3.I-
hs-hs-hs- randomized
hs-hs-hs- fixed
h-h-s-
Task
IB-Instruct
randomized
Cetrieve.PassKa
100.00
100.00
100.00
100.00
Aetrieve.Number
95.33
100.00
99.33
100.00
Retrieve.KV
42.66
82.66
88,66
84.66
En.Sum
27.63
23.42
24.01
24.33
En.QA
24.83
33.32
34.26
31.84
Fn.MC
68.00
71.33
74.00
73.33
En.
16.66
18.00
18.00
14.00
Matd.Find
35.33
39.33
37.33
36.66
Average
51.31
58.51
59.45
58.10
h-h-s-fixed-
h-h-s-
h-h-
h-h-h-
Task
no-sum
fixed
randomized
randomized
Retrieve. Payseyy
100.00
100.00
100.00
100.00
Extrieve.Number
99.33
99.33
98.66
99.33
Retrieve.
84.00
83.33
76.66
84,66
En.Sum
24.11
24.74
24.33
23.86
En.QA
32.81
33.88
30.69
31.97
Mr.MC
70.66
73.33
72.00
72.00
Dia
16.66
14.66
15.33
18.00
Math.Find
36.66
39.33
35.33
35.33
Average
58.03
58.58
56.63
58.14
highlighting the benefits of structuring and diverse, complex questions. 4) Summarization improves performance: ns-hs--fieednno-um scored 58.03, slightly below hs-hs-hs-fixed (58.58). Based on these findings, for longer context lengths (Section 4.2, we retain summarization, fix the number of questions/answers, and ensure both hierarchical and diverse questions are generated after direct documents and for those referencing previous ones.
CONCLUSION
This paper presents a novel strategy to generate high-quality, long-context iustruction-tunigg datasets that exceed the typical data context length. incorporates hierarchical ordering sure logical coherence while maintaining diversity and complexity in questions. Systematic ablation studies show that combining diverse questions with hierarchical ordering enhances performance, particularly in long-context scenarios. Our IM model demonstrates strong capabilities, outper- forming LLaMA-3.1 1-BB-Instrucc InfiniteBench and significantly surpassing on RULER, while maintaining robust performance on chorter-conten. tasks, as shown by LongBench and MMLU. Our data curation strategy highly scalable, enabling efficient creation of instruction -uning datasets ex- ceeding I million tokens and scaling up to 10 million or more. With sufficient resources and a strong training stack, our method supports increasingly longer context lengths, potentially unlimited.
While our approach has significantly improved instruction tuning for long-context scenarios, promising direction for future work developing a self-evolutionary strategy that diversifies and adapts prompts. A short-context model could autonomously generate long-context instruction data using our methodology and evolve independently, creating diverse and adaptable prompts for various scenarios. This could enable models to progressively evolve into longer-contex. models. Addition- ally, combining our data-centric approach with architectural optimizations offers another promising avenue for future research.
10

---

Published as a conference paper at ICLR 2025
Ethics Statement. In conducting this research, ensured adherence to the highest ethical stan- dards in the development and testing of our models. No human subjects were involved in data col- lection, ensuring that there privacy concerns or risks associated with the handling of personal information.
Reproducibility. We included the code generate bunch of hierarchical questions and di- verse questions for single document (see Section in supplementary material (see generating- data.py). We also included the code concatenate multiple documents (see Section 3.2 in supple- mentary material (see concatenate-350K.yy py)) . To enable long context training. we described detailed hardware setup in Section 4.1 Details about evaluations are also mentioned in in Section 4.1
REFERENCES
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry. Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxi- aoyu Feng. Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Itycheriah,, Matthew Jagielski, Wenhao Jia, Kathleen Ke- nealy. Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hycontack Lim, Hanzhao Lin, Zhongtao Liu, Freder- ick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Mous- salem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Mar- tin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli. Xuezhi Wang. Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023. URL wttpt:tt::// http://awwvvvv..aaaa//// abs 2305. htt.aab///3300000
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, mul- titask benchmark for long context understanding, 2024. URL attps://arxiv.oot
2308.14500
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL (ttthttps: //apxiv...ivvv///a org. wtagg/bb//// 2abbs/2004.0555 05150
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL httts::/arxiv..om
abs/ /20s/2336... 15595
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning long-context large language models, 2024. URL pictps://arii: erg/abs/ 2309. 12307
Albert Gu and Tri Dao. Mamba: Linear-tima sequence modeling with selective state spaces, 2024.
URL (RRLshtts://axxtt org/ abs//bss/11 00752
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences, 2022. URL https:
//axii.orrgaae abtp://b//22/ /tt.//////l... 07916
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja-
cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
w/wrxii..rg//baaaae... wttp://.wwggaba////0//// 03300
11

---

Published as a conference paper at ICLR 2025
Namgyu Ho, Sangmin Bae, Tachyeon Kim, Hyunjik Jo. Yireun Kim, Tal Schuster, Adam Fisch, James Thome, and Se- Young Yun. Block transformer: Global-to-lowe language modeling for fast inference, 2024.
hbtp://s/22440 http://aww.....////////////6/ 02657)
Cheng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's s the real context size of your long-context language models?, 2024. URL htRp::tts://axxxs org/abs 2tt.//4444006666
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv.org/abs 06825
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention, 2020. URL https://arxiv. org/ 2006 16236
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context languug
models understand long contexts?, 2024. URL
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023, URL
https::/aaxiis org/ hwag//as//a// abbp//07
Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length, 2024. URL http::://axxiiv.orraabb/////////
1459I
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
www.orgg/abb///ooo 05262
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave context behind: Efficient
infinite context transformers with infini-attention, 2024. URL https://axxiv.org//
2404.07143
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023. WRRLtttps://aaat wttp:o/ggabbb.....//// 00071
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
2208. 12409
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En hanced transformer with rotary position embedding, 2023. URL
2104. 09864
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaud- hary, Xia Song, and Furu Wei. A length-extrapolatable transformer, 2022. URL https:
7/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models, 2023. URL pettps://arxta org/ abs///ab//aat 2302.13971
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. URL URL ptttltttt///axiiit wrg. abs 2212 10560
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022. URL htt.::ttp/////
.rg. abs /200 07682
12

---

Published as a conference paper at ICLR 2025
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023. URL JRLpps://aaxxx org/ abs//303...................... 7564
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023. URL https:://axxiy wtwp://///ooooooooooooooooo///o 2309 16039
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023, URL
fitps://aaxiv: org/ als///b///// /ww.a/////.3. 2304 12244
Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. Automatic instruction evolving for large language models, 2024. URL htt.:tts//aaaivv.rr///////220 00770
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. cobench: Extending long context evaluation beyond 100k tokens, 2024. URL
Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang. Peng Cheng, Lijic Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. Longskywork: training recipe for efficiently extending context length in large language models, 2024. URL httph:tt//// //arxiv org/ abs 2406. 00605
13

---

Published as a conference paper at ICLR 2025
A APPENDIX: ADDITIONAL DETAILS ON DATA GENERATION ALGORITHMS
In this section, present the pscudocode for the hierarchical QA generation strategy described in Section 3.1. along with the algorithm for combining multiple documents, as outlined in Section 3.2
Algorithm Hierarchical Question Generation Strategy (Single Document)
1: procedure GENERRATEEXTTSPPRROOREEEECCCCEE
N. Questions To Generate)
2: chunks < HicearchicclSppiiccccume.....
3: summaries, fulmasu Summmariiefeererchhhccchhhhi
4: conversations + InnitalSummaarydocumeut... summarry]
5: for i = 1
CoteesttionnToGGGnerrre do
6: context, summary < SelectContext((huntee summaries, last medium, dastssmall, i)
7:
qa-pair < CeeeeeareeeeeAAAPaaarrcreeeettte summary)
8:
AmpeantTDCConessaiooocccvvrrrrot ga_pair)
UpdateLasiCCunnsssasst mediumm, last_small) 10: end for
9:
11: return conversations
12: end procedure
13: procedure POLLETTCONTEXT((cunk, summaries, last medium, last small, iteration index)
14: if first iteration then
15:
return random medium chunk
else if no small chunk selected then
17:
return first small chunk of current medium
else
19:
random choice + RandomChoice((O, 1, 2])
>> Equal 1/3 probability for each
random choice 0 then return deeper content of current small chunk else random_choice then
21:
22:
23:
return next small chunk in current medium else
24:
25:
return next medium chunk
26:
end end if
27:
28:
end procedure
procedure GENERATTEOAAAIRRComiitt,, summary)
30:
if' Context(sSpeciic((cmeeex then
return GenerateSpecffiiAAAAPerreeeee))
32:
else
return GenertteGeneaal//PPaiiieeeee,,,e summary)
34:
end if
35:
end procedure
B ADDITIONAL INFORMATION ON DATA GENERATION PROMPTS
Here list all prompts used in the different stages of our synthetic data generation pipeline.
Document Summarization
"Suumarizee the following text concisely in more than [word_limit] words:
(chunk;
14

---

Published as a conference paper at ICLR 2025
Algorithm 2 Concatenating Multiple Documents
Input: Set of K' documents, each with hierarchical and diverse questions Initialize: conversation list C 0
for each document D; where i 1, K do
S < Goneratefiicarcccca@QQutisissss D;)
H₁ f
C t C U antiaalHHeerrrccical@eeestssssss (D)
C C' U Sandom/ySamppeeiivseeeeeesttttt (
Store remaining unselected diverse questions from S
end for
for each document D; where 2.3.... K do
C TextHierarrhiialluusttooo H;-1)
Cnspleeesseeeeeedee Update hierarchical index for document D; end for for each document D; where = ,, do
C'
if Randomoondiiiion( then
+ CU Follow Uppiienacchiccluuuuttissss///
end if end for Process remaining specific and diverse questions:
Lengih(S;)
if xr ThiresoolaforSppciiieuuuttttttti then
Select and append follow-up specific questions to
Remove selected follow-up specific questions from pool
end
Output: Final conversation list C
Diverse Questions
"""Context information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based on the below quezy. You are Teacher/Professor. Your task is to set up diverse temporal question about the The question
context for an upcoming quiz/examination should cover different time periods and events described in the context. Restrict the question to the context information provided. You must return the result in JSON: (' l'question': <question>, 'answer' <answer>]""!
15

---

Published as a conference paper at ICLR 2025
Diverse Questions
Context information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based on the below query. You are a Teacher/Professor. Your task to create character-based question from the context for an upcoming quiz/examination. The question should explore different aspects of the characters, such as their mot motivations, actions, and relationships. Restrict the question to the context information provided. You must return the result in JSON:
l'question': <question>, 'answer': <answer>) ""*
"""Context information below.
$ (context)
Given the context information and not prior knowledge. Generate content based the below query. Formulate complex question that requires analysis of multiple aspects from the context for an upcoming quiz/examination. The question should encourage critical thinking and synthesis of different pieces of information within the context. Restrict the question the context information provided. You must return the result in JSON: (' q'quettio'; <question>, 'answer': <answer>) nn"
"""Context information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based the below query. You are a Teacher/Professor. Ask 1 question about the main themes or messages of the text for an upcoming quiz/examination. The question should cover different aspects of the themes and how they are developed in the context. Restrict the question to the context information provided. You must return the result in JSON: (' l'question': <question>, ansser'; <answer> ""*
16

---

Published as a conference paper at ICLR 2025
Diverse Questions
Context information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based on the below query. You are a Teacher/Prooessorr. Create question that compare different elements within the context for an upcoming quiz/examination. The question should highlight similarities and differences between various elements such as characters, events, and themes. Restrict
the question to the context information provided.
You must return the result in JSON:
l'question': <question>, 'answer': <answer>)"" ""*
"""Context information below.
$ (context)
Given the context information and not prior knowledge.
Generate content based the below query.
You are a Teacher/Professor. Develop question that explore the cause and effect relationships within the context for an upcoming quiz/examination. The question should focus on understanding the
reasons behind events and their outcomes. Restrict the question to the context information provided.
You must return the result in JSON: (' l'question': <question>, 'answer': <answer>)
"""Context information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based the below query.
You are a Teacher/Professor. Create hypothetical question based on the context for an upcoming quiz/examination. The question should explore what-if scenarios and possible alternate comes. Restrict the question to the context information provided. You must return the result in JSON: ('question': <question>, 'answer'; <answer>) nen
17

---

Published as a conference paper at ICLR 2025
Diverse Questions
*Cnttex information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based on the below query.
You are a Teacher/Professor. Formulate that require interpretation of the context for an upcoming quiz/examination. The question should encourage students to provide their own insights and interpretations based on the information given. Restrict
question
the question to the context information provided.
You must return the result in JSON: l'question': <question>, 'answer': <answer>)"" ""*
"""Context information below.
$ (context)
Given the context information and not prior knowledge. Generate content based the below query.
You are a Teacchr/Profesor.. Ask 1 detail-oriented question about the context for an upcoming quiz/examination. These question should focus on specific details, facts, and figures mentioned in the context. Restrict the question to the context information provided. You must return the result in JSON: {' quqestionn: <question>, 'answer'; <answer>) "** """
"""Context information is below.
$ (context)
Given the context information and not prior knowledge. Generate content based on the below query.
You Teacher/Professor. Create question that explore different perspectives or viewpoints within the context for an upcoming quiz/examination. The question should examine how different characters or groups might view events themes differently. Restrict the questions to the context information provided. You must return the result in JSON: [' ('question': <question>, 'answer'; <answer>) """
18

---

Published as a conference paper at ICLR 2025
Mupti-Hop Questions
" " "Context information is below.
[selectee_chunk_]]]
[seleccte__chunk]]]
{[selected_churk)
You are a Professor designing a final exam for an advanced interdisciplinary course. Create 1 complex question requiring deep analysis and synthesis of information from all three chunks. Do not mention that there are three chunks/your questions. Do not mention excerpts either. For example, instead of question that says "Analyze the theme of justice and its various forms as portrayed in the three provided literary excerpts. How do the characters' actions and the outcomes of their situations reflect challenge traditional notions of justice? Consider the legal, personal, and societal implication of justice in each excerpt and discuss the role of power
dynamics in shaping justice." You should say: "Analyze
the theme of justice and its various forms
as portrayed. How do the characters' actions and
the outcomes of their situations reflect challenge
traditional notions of justice? Consider the legal, personal, and societal implications of justice and discuss the role of power dynamics in shaping justice." Question Guidelines:
1. The question must integrate and require reasoning across all three chunks.
2. The question should be milti-layered, promoting analysis, synthesis, and evaluation.
Answer Guidelines:
1. Provide a comprehensive answer addressing all question aspects.
2. Reference and interconnect information from each chunk. Return 1 question-answer pair in JSON format: "question": <question>, "answer": <answer> mun
19

---

Published as a conference paper at ICLR 2025
Specific Detail Question
e""Context information is below.
$ (context)
Given the context information and not prior knowledge, generate content based the below query. You are a Teachet/Professor. Create 1 specific question about the details, events, characters, and settings from the context provided. This question should have an exact, unambiguout answer that can be directly found in the given information. The question should be similar in style to the following examples: "Where does Character A meet Character B for the first time?"
"What is Character C's religion?"
"Where does Character D live for the majority of the story?"
"Which of the following is NOT one of Character g' responsibilities?* "Which among [list of names] is not [Character Fl's child?"
"Who among [list of names] is the final perish?"
"What's s the name of [Family namel's summer home?" "Who accompanied [Character G] [specific activityl at last?" Ensure that the question and answer are strictly based on the context information provided. The question may include miltiple-choice options when appropriate. You must return the result in JSON: (' ('question's <question>, 'answer': <answer>)""F ""*
C COMPLETE CONVERSATION DATA
Here present an example of our instruction tuning data that is developed by concatenating four documents (350K context length). The data consists of three types of content: long documents and their summarizations (yellow) (constituting QA pairs), QA pairs extracted from each document (blue), and QA pairs that reference past documents (green). These are color-coded in the following example for clarity.
"conversations".
"role": "user", "content"; "Produced by Charles Bowen from page scans provided the Web summary the book"
Notes: ... Please give
"role": "assistant",
"content"; "Charles Bowen created digital book from. Web Archive scans, ....
"role": "user", "content"; "What are main reasons for the tensions and confllets among the characters in 'The Patricians'?
"role": "assistant.",
20

---

Published as a conference paper at ICLR 2025
"content": "The tensions and conflicts in 'The Patricians' arise from issues pride, ..."
"role": "user",
"content": "What class in ancient Rome did the 'Patricians' typically refer to?"
"roie": "assistant",
"content"; "The 'Patricians' typically referred the aristocratic class in ancient Rome, who were affluent, powerful, and held considerable political sway in the Roman Republic."
"role": "user",
"content"; "Who among the following did not attend Christophor's banquet? A) Tausdorf B) Althea с) Schindel D) Francis"
"role": "assistant",
"content"; "D) Francis"
"role": "user",
content"; "Analyze the theme of justice and its various forms as portrayed. How do the characters' actions and the outcomes of their situations reflect or challenge traditional notions of justice? Consider the legal, personal, and societal implications justice and discuss the role of power dynamics shaping justice."
"role": "assistant", content"; "The theme of justice explored through the complex interplay of legai, personal, and societal implications reflecting and chall chalinnging traditional notions of justice.
"role": "user"
"content": "Produced by D Alexander and the Online Distributed Proofreading Tean at http://wwwpgdp..rt + Please give me summary of the book"
"role": "assistant",
"content"; "The text acknowledges Alexander ..."
"role": "user", "content"; "who does the professor and his party meet at the sulphur spring?"
"role": "assistant",
"content"; "Jranvin, the chief of the islanders"
"role": "user",
21

---

Published as a conference paper at ICLR 2025
"content": "What does Jim recognize that lifts his spirits while confined on the Marjorie?"
"role": "assistant", "content"; "Jim recognizes his own ship, which lifts his spirits while he is contined on the Marjorie."
"role": "user", "content" "How did the professer and his party find out about the location
of the Marjorie and what did they discover upon investigating the harbor?"
}
"role"; "assistant",
"content": "The professor and his party, including Tom, Jo, and Berwick, hiked to a cliff and climbed to the top where they spotted the Marjorie and ancther ship in harbor five miles away. Upen investigating the harbor, they found the sea Eagle leaving and chaos on the Marjorie. They also captured
the steward who revealed he helped Jim escape."
"role": "user",
"content"; "Analyze the theme of justice and its various forms portrayed.
How do the characters' actions and the outcomes of their situations reflect
or challenge traditional notions of justice? Consider the legal, personal, and sccictal implications justice and discuss the role of power dynamics in shaping justice."
,,
"role": "assistant", "content"; "The theme of justice is intricately woven through the narrative,
"role": "user",
"content"; MnnuntnnnPoouuuee by Marius Masi, Juliet Cutherland and the Cnline InDistributed Proofreading Team Please give me a summary of the book"
"role": "assistant", "content"; "The text describes a collection illustrated books about Japan, ....
22

---

Published as a conference paper at ICLR 2025
"role": "user", "content"; "How does the text compare Japanese and western art in terms
of their integration into daily life and appreciation by the public?"
"role": "assistant",
"content"; "The text emphasizes that Japanese art, unlike Western art
is deeply integrated into daily life and universally appreciated, similar to the vitality of art in ancient Greece..."
"role": "user", "content"; "How does the book contrast Japanese and Western art in terms of
their integration into daily life and societal appreciation?"
"role": "assistant", "content": "The book highlights that Japanese art is more integrated into daily
life and universally appreciated, ...."
b
"role": "user",
"content": "Analyze the character development of Tausdorf throughout
the passage. How does his perception of Bona change, and what factors contribute
to this change? Additionally, discuss the role of the setting and the use of imagery in shaping Tausdorf's actions and emotions."
"role": "assistant", "content"; "Tausdorf's character development complex and multifaceted,
"role": "user",
"content"; "What is the name the yacht that Jim and Berwick consider hiring?"
"role": "assistant",
"content"; "Storm King"
("role": "user", "content":
by David Rdwards. and the Online Distributed ... Please give summary the book"), ("role:: "assistant", "content"; "{"The Curlytops Snowed In by Howard 3. Garis follows siblings Ted and Janet Martin, known as the Curlytops, .).
("role": "user", "content"; "What challenge did Nicknack, the goat, face during the winter adventures, and how did the chi 1dren address it?"),
23

---

Published as a conference paper at ICLR 2025
1M model
650K model
350K model
Context Length Accuracy (%)
Context Length Accuracy (%)
Context Length
Accuracy (%)
000 000 62.95%
650 000 57.24
350 000 71.15 %
524 288 76.88%
524 288 74.88 %
262 144 78.26 %
262 144 76.24 %
262 144 78.26 %
131 072 82.72 %
131 072 82.05 %
131 072 82.29
65 536 82.63
536 83.73 %
65 536 81.81
32 768 87.26
32 768 86.19 %
32 768 87.96 %
16 384 91.45 %
16 384 90.06 %
16 384 92.86 %
8 192 92.62 %
192 92.96 %
8 192 91.92
096 96.13 %
096 95.59 %
95.49
Table 8: Context length accuracy of RULER for different models
"rolee: "assistant", "content": "Nicknack had difficulty standing ice due to the unsuitability his hooves..."
("role": "user", "content": "What causes the snow house to collapse in Chapter VI, and what do the children decide to do after the incident?"),
("role": "assistant", "content"; "The house collapses when Nicknack the goat...."!.
("role"; "user", "content"; "Compare and contrast the portrayal of art and its integration into daily life in Japan and the West, described the text. How does the Japanese approach to art, particularly in the context of theater, reflect broader cultural values and phi Ahilosophiess Discuss the role of realism, balance, and the environment in shaping the artistic experience in both cultures .",
"role": "assistant", "content"; "The text illustrates stark contrast between the Japanese and Western approaches to art, particularly in theater, which reflects deeper cultural values and philosophies, In Japan, art is not merely conscious effort achieve beauty but an instinctive expression of inherited taste that permeates all aspects of life...";,
"role": "user", "content": "Analyze the theme of survival and its varicus forms portrayed in the narratives. How do the characters actions and the outcomes of their situations reflect or challenge traditional notions of survival? Consider the natural, social, and personal implication of survival and discuss the role of rescurcefulness in shaping survival strategies.") ("role": "assistant", "content": "The theme of survival is intricately woven through the narratives, reflecting the characters' resilience and adaptability in the ...",,
1)
D RULER NUMERICAL RESULTS
E 100K CONTEXT LENGTH ABLATION STUDIES
The 100K ablation studies aim to assess whether hierarchical ordering and diverse question types improve results on single-document instruction tuning data. We also aim to identify which these factors most significantly influences overall performance. In particular, whether hierarchical ordering enhances outcomes, (2) whether diverse question sets contribute pos- itively, and (3) whether the use of multi-hop questions further boosts results.
want to explore (1)
24

---

Published as a conference paper at ICLR 2025
Table Context length of RULER for LLaMA-3.--BB-nsttuue models
LLaMA-3JJ-B-IIntruucc
Zero-shot Rope Scaling 1M
Context Length Percentage ())
Context Length Percentage (%)
28.93
000 000 48.81 %
262 144 30.34
524 288 62.53 %
131 072 83.06 %
262 144 66.44 %
65 536 82.26 %
131 072 72.68 %
32 768 88.44
65 536 77.81 %
384 93.13
32 768 84.01
8 192 92.08 %
16 384 87.36 %
4 096 95.49 %
192 90.73 %
4 096 95.94 %
(a) Context length of RULER of LLaMA-3.1-8B-- 1-8--Instrt
(b) Context length RULER with
zero-shot rope scaling to 1M context length
Each experiment uses 300-600 data samples, each with 100K tokens, fine-tuned on LLaMA-3.1- 8B-Instruct for 1 epoch at a 6e-5 learning rate. The specific ablation tests we included are 1) hierarchies: from single document, we generated hierarchical ordering data using the algorithm specified in Section 3.1 2) 4 hierarchies with multi-hop reasoning: In addition to the 4 hierachies set up in Section every time we generate a new QA pair, there is 20 % chance that multi-hop question-an answer pair will follow. 3) 4 hierarchies without order: hierarchical questions were gen- erated without enforcing the order from Section 3.1] testing if strict hierarchy enforcement improves outcomes. 4) Diverse questions: this setup generated various question types to test if diversity improves performance, as outlined in Section
The results of these ablation studies on InfiniteBench are summarized in Table 11 The key find- ings include: 1) Multi-Hop Reasoning Improves Performance: Among all configurations, multi-hop reasoning achieved the highest average score of 54.70, demonstrating the importance of captur- ing cross-document relationships and broader reasoning capabilities. 2) Diverse Questions Pro- vide Broad Improvements: The diverse questions setup achieved the second siiond-hghhettt score of 52.41, highlighting the value of introducing variety in QA generation for instruction-tume -uning data. 3) Hier- archical Ordering Boosts Performance: Both the striet hierarchical model (52.08) and the random hierarchical model (50.69) outperformed the base LLaMA-3. 1AMMAA-.1-8----A. 1-8B-Instrrut (51.31), validating the effectiveness of hierarchical structuring, even when strictly ordered.
The LongBench results (presented in Table 10 provide additional insights, though the differences between configurations are relatively minor. This is likely because LongBench evaluates models on short contexts (up to 10K tokens), which do not fully leverage the strengths of hierarchical or multi- hop structures designed for longer contexts. summary, the ablation tests show that hierarchical ordering, multi-hop reasoning, and diverse questions are key to optimizing performance on long- context tasks.
25

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study on LongBench with 100K context length.
L.AMA-3.1-8B-
hierarchies multi-hop
4 hierarchies random
diverse
Task
Instruct
hierarchies
questions
NarrativeQA
25.48
25.89
25.10
25.04
27.91
Qasper
45.33
47.02
44.79
46.00
46.25
MuliFicldQAAcn
54.98
54.86
53.96
54.86
53.75
MultiFicldQA-zh
61.83
55.75
54.87
59.89
56.14
Single Document
46.91
45.88
44.68
46.45
46.01
HotpotQA
55.00
56.67
56.91
55.83
58.34
2WikiMQA
44.95
52.19
52.96
48.74
52.71
Musique
31.76
29.15
28.55
29.85
28.10
DuReader
34.10
36.83
36.32
35.57
36.74
Multi-Docunnn Document
41.45
43.71
43.69
42.50
43.97
GovReport
35.07
34.39
33.72
35.31
35.33
QMSum
25.13
25.15
25.27
25,52
25.38
MultiNews
27.08
27.34
27.48
27.29
27.46
VCSUM
17.10
16.12
16.75
16.13
16.40
Summarization
26.10
25.75
25.81
26.06
26.14
TREC
72.50
73.00
73.00
73.00
72.00
TriviaQA
91.65
92.28
92.25
91.87
91.83
SAMSum
43.77
43.81
43.98
44.49
45,48
LSHT
46.00
46.00
47.00
47.00
48.00
Few-shot Learning
63.48
63.77
64.06
64.09
64.33
Passage Count
6,55
4.00
3.00
7.56
5.00
P
99.50
99.00
99.00
98.50
98.50
PassageRetrieval-z
96.38
98.50
100.00
94.63
99.50
Synthetic Tasks
67.48
67.17
67.33
66.90
67.67
All
48.1 1
48.31
48.15
48.27
48.67
Table 11: Ablation study InfiniteBench with 100K context length.
L.IMAA3.1-SB-
diverse questions
hierarchies random
hierarchies
hierarchies
Instruct
multi-hop
Retrieve. PassKey
100,00
86.66
86.66
86.66
100.00
Cetrieve.Number
95.33
86.66
86.00
85.33
96,66
Retrieve.KV
42.66
60.00
58,00
58.66
57.33
En.Sum
27.63
23.02
24.11
22.77
22.67
En.QA
24.83
29.66
32.50
25.40
30.25
En.MC
68.00
70.66
72.00
70.00
70.66
En.
16,66
24.66
23.33
20.66
26.00
Math.Find
35.33
35.33
36.66
36.00
34,00
Average
51.31
52.08
52.41
50.69
54.70
26


Reviews:
[{"review_id": "lpmxAU32uK", "reviewer": "Reviewer_L4p6", "summary": "This paper introduces a novel post training synthetic data generation strategy for long context.  The approach works by splitting a long context document into smaller documents and generating question-answer pairs from separate smaller documents as well as combinations of the documents.  The generated data is used in combination with a step-by-step rotary position embedding scaling strategy to scale the model context.  The proposed approach is tested in several benchmarks for longer context and overall results appear higher for the proposed finetuning.", "questions": "- What is the Retrieve.KV task and are there hypotheses about why this task in particular has large improvements?\n- What are $N_1$, $N_2$ and $N_3$? Can the authors include some ablations if needed on these values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper makes contributions to improving long context reasoning of LLMs. The exploration of algorithms and fine-tuning strategies for increasing the context length is an important problem for language models and finetuning on synthetic data appears to be a promising direction for this work\n- The proposed method appears simple to implement and should be reproducible as many of the proposed prompts are provided in the appendix, and the training strategy is similar given the generated data.", "weaknesses": "- A main concern with the proposed work is that for many of the results (e.g. Table 1, 5) the improvements appear to primarily come from a single task Retrieve.KV.  Improvements on the other datasets are smaller.  While this leads to an overall increase, it's important to understand the importance of this subtask rather than simply the reported overall average increase. \n- The scale of the datasets in experiments is rather small and there are no uncertainty estimates provided.  This is important particularly as for some of the tasks, there is only 100-200 samples, where a few correct answers can increase.  I would encourage the authors where possible to increase the amount of evaluation data.\n- The authors only test a larger model for generating the synthetic data.  The method would further be justified by including comparisons for models of the same size that are used to generate the data as it is unclear the dependency on model size.  Further the experiments are only done for the Llama-8B model, but experiments on other models even smaller would be beneficial to see whether smaller or larger models benefit more from increasing context length.\n- there is some confusing wording in the paper that the authors should clarify particularly around Section 3.2 Authors could do a better job clarifying what are $N_1$, $N_2$ and $N_3$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "duNB6WBiM3", "reviewer": "Reviewer_DSHN", "summary": "The paper introduces a novel post-training method for generating synthetic data designed to efficiently extend the context window of LLMs, all while preserving their task performance. This approach leverages a summarizer to reduce the context length, enabling a hierarchical pipeline for data generation.", "questions": "Would it be possible to up scale a single document? Although multiple document concatenation was considered to scale to larger context lengths, the proposed methods do up-scaling to a single document that might be 1M. For example, for a document that might only be 128k, increasing the total context to 256k. This would make the entire data synthetic pipeline a more holistic data synthetic pipeline for all long context documents/instructions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an important need for long context instruction data.\n- The proposed methods allows for diverse instruction tasks\n- Finetuning on this data retrains performance on medium context lengths (~10k)", "weaknesses": "- The impact of the summarizer’s quality on the data generation pipeline is unclear. The paper uses Qwen-72b, but further discussion and experimentation are needed to understand how different summarizer models could influence the method’s effectiveness. For example, different model sizes and a different model family like Llama-2 or Llama-3.\n- RULER evaluations at extended context lengths (like 1M): It would be insightful to include evaluations with context lengths of 1M tokens (or 500k), as most evaluations in the paper are currently under 200k. This would help clarify whether the proposed method maintains performance at significantly larger context lengths.\n- Can the author provide performance in small contexts like MMLU or hellaswag after FT'ing? This would show that the model retains performance on shorter contexts while improving long-context capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s6ku8vcFJ4", "reviewer": "Reviewer_LvUc", "summary": "This paper introduces a novel approach to generate a high-quality, long-context instruction-tuning dataset that significantly surpasses the context length of typical raw data. It incorporates a unique hierarchical ordering strategy to ensure logical coherence while preserving the diversity and complexity of the questions. The experimental results on RULER and InfiniteBench demonstrate that the proposed method significantly enhances the performance of llama3.1 in longer contexts.", "questions": "Reference to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a scalable long-context data generation strategy to significantly improve the long-context capacity of LLama3.1 and extend its context length to 1M.\n2. Comprehensive ablation tests. Analyze the impact of data construction strategies from the perspectives of data complexity, diversity of questions, etc.", "weaknesses": "1. The paper uses the Qwen-2-72B model  to generate the QA pairs,  which may result in strong models having a distillation effect on small models. Can you provide experimental results using the Qwen2 7B or Llama3.1 8b as the generator model?\n2. Lack of baseline models to validate data generalization . The paper only used llama 3.1 as a baseline. Can you provide more experimental results on models such as Qwen2 7B and deepSeek-V2-Lite?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MD4NiZjC8c", "reviewer": "Reviewer_ZqBD", "summary": "The paper \"Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation\" aims to address the challenges of scaling large language models (LLMs) to handle extended context lengths, particularly up to one million tokens, without compromising general task performance.", "questions": "1. **Significance of Extending to 1M Context Length**: The authors should provide more justification for the significance of extending to a 1M context length. A 128K context is sufficient for most real-world long-context tasks, and demonstrating practical use cases that necessitate 1M tokens would help strengthen the motivation for this work.\n\n2. **Figure Improvement**: Consider changing the layout of figures 2 to horizontal format for better readability and comparison.\n\n3. **Loss Calculation During Training**: Clarify whether only the answer part of the generated question-answer pairs calculate to the loss during training or if both the question and answer are involved.\n\n4. **Potential Explanation for Decreased LongBench Performance**: Consider discussing whether Section 3.2 of \"How to Train Long-Context Language Models (Effectively)\" could provide insights into why LongBench performance decreases as training progresses with long-context data, addressing the concerns raised in Weakness 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a synthetic data generation pipeline for instruction tuning, which allows LLMs to effectively extend their context lengths without the need for vast, annotated long-context datasets.\n\n2. Hierarchical Approach: The use of hierarchical splitting and question generation is a significant advancement. By logically structuring questions across multiple levels—hierarchical-aware, local-specific, and multi-hop—the authors ensure coherent instruction generation and improve the model's ability to reason over long contexts.\n\n3. Comprehensive Evaluation: The authors extensively evaluate the proposed approach on multiple benchmarks such as RULER, InfiniteBench, and LongBench. The detailed ablation studies show the importance of different components in their data generation strategy, highlighting that hierarchical order and diverse question generation are critical to achieving better long-context performance.", "weaknesses": "1. **Lack of Strong Baseline Comparisons**: The paper lacks comparisons with suitable, strong baselines that have also achieved extensions to 1M tokens. For example, EasyContext (https://github.com/jzhang38/EasyContext) has similarly attempted to extend context lengths to 1M using long-text data. Especially since the comparisons are made against the original LLaMA-3.1-8B after additional fine-tuning on top of it.\n\n2. **Decreasing Performance on LongBench**: The results on LongBench suggest that the model’s performance decreases as context length increases. This is in contrast to what one would expect after training on long-context data, raising questions about the effectiveness of the proposed methods in maintaining or improving performance across all types of tasks. An explanation should be provided as to why LongBench performance degrades with extended context length training.\n\n3. **Limited Value of Ablation Study in Tables 3 and 4**: The results of the ablation studies in Tables 3 and 4 align closely with expectations, as instruction-tuning data types (e.g., Diverse Questions) are more closely aligned with the evaluation benchmark used (e.g., LongBench). Therefore, these tables have limited value in demonstrating broader insights. A deeper analysis of Table 5 would be more insightful, focusing on the types of questions and the effect of different question strategies during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)