LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Authors: Chaochen Gao
Xing W
Zijia Lin
Debing Zhang
Songlin Hu
Venue: NeurIPS 2025 poster ()
Keywords: Long-context Model
Synthesis Data
Primary area: deep_learning

Abstract: High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Chaochen ?2 Xing ; Zijia Lin', Debing Zhang', Songlin Hu
IIntiittee of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences
3 Xiaohongshu Inc, *Tsinghua University (gaochaochen, www.ii..uuooggiiii @iie.ac.cn dengyang @ xiaohongshu.com linzijia@tsiin tsinghua.edu.cn
Abstract
High-qualijg long-come congeconext instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-contex: instruction data remains proprictary. Human annotation is costly and challenging, while template-bas based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, self-synthesis framework that automatically generates large-scatl scale long-context instruction data. Our key insight is that aligned congecontext LLMs, when presented with document followed by special tokens preceding user turn, auto-regressively generate contextually rele- vant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-conten tasks while maintaining competitive perfor- mance short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.
Introduction
Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range
of tasks, with recent advancements significantly extending their context lengths 22 I 18] The ability to process long documents is essential for complex applications such as Longbook QA [7].
document summarization [49). and code planning (5). However, fine-tuning LLMs to leverage long contexts requires access to high-quality long-context instruction data [8] (2). While the model weights of several open-source LLMs, such as Qwen [54] and Llama [99, have been made publicly available, the corresponding instruction datasets for long-contes training remain proprietary. This closed-data paradigm poses substantial barrier to the advancement of open-source long-context models.
Existing methods for creating open-soure instruction data face substantial limitations when extended to long contexts. (1) Human labor costs are prohibitively high for creating diverse, high-quality long-context instruction data. The annotation difficulty substantially greater than for short-context data, requiring individuals to read documents spanning thousands of tokens before formulating instructions- a demonstrably challenging task. (2) Existing synthetic approaches, often relying on predefined templates [39] seed questions [47] do not guarantee the diversity needed for effective
long cong-coneext instruction. While existing projects (26 52 2] attempt to broaden seed data diversity, creating large-scale long-context instructions with high quality and diversity remains an expensive and time-comeen consuming process.
Cooresspodiigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Stagel: Query and Answer Gencration
System Prompt
Query
startt-rystmm
<iim <jim_stan(iwue start
historical significance pond lowered by one shaku...
Document
Yamada Pond was the largest est reservoir Shiodadairn until...
+
Query
The historical signifseance Yamada Pond being constructed...
<jmm endi>
Query Gonocation
Query Appending
Answer Generation
Query
Filteing
Stage2: Multi- Document Extension
Query
Mulli-Doe
historical significance of the pond lowered shaku...
Negutivate Mining
Neg-Doel
Documents
Document
LongMapie
Dataset
Answer
The historical significance Yamada Pond being
Neg-DocN
constructed...
Figure LangMagpie pipeline overview. Stage one: document serves as a system prompt, a special user token triggers query generation, followed by the model response. Stage two: combines the query-response pair with the source document and sampled documents from the corpus to create challenging multi-document long-instruction data.
recently proposed self-synthesis method, Magpie [53] has gained widespread attention for elim- inating the need for seed instructions and prompt engineering required by previous approaches 52 2]. creates alignment data by prompting aligned LLMs with only special tokens preceding a user turn, leveraging their auto-regressive nature. Inspired by Magpie, we introduce LongMagpie, self-synthesis method for generating large-scale long-context instruction data without human annotation or complex prompting. A key observation is that long-context understanding often involves document- based question answering, such as RAG or long document QA. Thus instruction- tuned LLMs such as Qwen [54] and Llama [19 internalize patterns of document-query relationships during their long-context instruction training. Thus, when aligned models are presented with only document, followed by the special tokens that typically precede user query, they auto-regressively generate contextually relevant queries about that document. By leveraging this behavior, we can automatically create high-quality instruction-respone datasets for long-context training without explicit prompting or manual intervention.
47 26
This approach offers advantages: it scales efficiently to generate diverse, high-quality long-context instructions without labor costs complex prompt engineering; produces naturally varied queries that probe different aspects of documents: and eliminates complex pipeline components required by previous methods. Furthermore, we extend LongMagpie beyond single documents multi-document contexts, creating more challenging scenarios that require distinguishing relevant information such as RAG 117. This multi-document extension enhances the model's 's ability to handle complex real- world applications that frequently involve reasoning across multiple information sources while providing natural way to increase context length and task difficulty without additional computational overhead.
To further balance the long-context and short-context capabilities, we introduce the p-Mix strategy, which addresses the performance degradation on short-context tasks when models are predominantly trained on long-context instructions. This strategy employs a probabilistic mixing approach that begins by prepending a short-context instruction each training sequence, followed by dynamic sequence constructed through probabilistic sampling. Specifically, with probability PL. a long- context instruction (generated by LongMagpie) is appended; otherwise, with probability 1 Pt. another short-context instruction is selected. This process continues iteratively until approaching maximum sequence length T.mar- p-Mix effectively prevents the model from overfitting to long-context patterns while maintaining strong performance across diverse task scenarios.

---

and Longbench v2 [ benchmarks, we demonstrate that models trained on LongMagpie generated data achieve leading performance. When incorporated with p-Mix, our approach maintains competitive performance on short-context tasks. We conduct detailed analytical experiments on the MongMagpie method explain its effectiveness. The positive experimental results demonstrate that LangMagpie represents a meaningful step toward democratizing long-context capabilities for LLMs, making high-quality long-context instruction accessible to the broader research community.
Through extensive evaluation on HELMET 55 RULER 222
Our main contributions are:
We introduce novel self-synthesis approach for generating high-quality long-context instruction data that leverages the auto-regressiv nature of aligned LLMs, eliminating the need for human annotation or predefined examples.
We propose the p-Mix technique, a probabilistic mixing strategy that effectively balances model'ss performance on both long-context and short-context tasks.
We conduct extensive evaluations demonstrating that models trained LongMagpie-generatee data achieve leading results on long-context benchmarks compared to existing methods.
We provide in-depth analyses revealing the key factors contributing to LongMagpic' 's effectiveness, including query diversity and quality.
Method
This section introduces LongMaappi, method for synthesizing long-context instruction data. We first describe the key insight of our approach, followed by the detailed pipeline of LongMagpie and p-Mix strategy for balancing long-context and short-context capabilities.
2.1
Key Insight: Auto-Regressive Ducument-Quer, Generation
The foundation of LongMagpic a key observation about aligned long-context LLMs: with document followed by tokens that typically precede a user query (without the query itself), these models generate contextually relevant queries about that document. This behavior stems from the fact that long-contex, understanding often involves document-basse question answering tasks such as RAG and long document QA. During instruction tuning, models like Qwen and Llama internalize document-query relationship patterns, enabling them regressegsys predict meaningful questions when presented with document-only contexts. This capability allows us to synthesize diverse, high-quality instruction data without human annotation, predefined templates, seed questions.
when provided
Formally, for aligned LLM M with vocabulary V, we define the document-query generation process as follows: given document = [dl, d2, ..., J V and pre-query template T Tpre {41, (2..., km n (containing tokens indicating user query role, e.g., < im_start >user), we provide input = D Tyre, where @ denotes sequence concatenation. The model then generates a sequence Q = {qi, 42. .... 9k} E D representing a query related to document D. This process can be described as:
PM(Q D. D,Tme)
II pM1(: Dppee 9<i)
This approach differs fundamentally from traditional prompt engineering or instruction-foll foollociionfoolooingg. explicitly instructing the model generate query about the document. Instead, leverage the model's learned patterns of document-query relationships that emerge from its instruction training.
Langia Pipeline
The MangMagpie pipeline consists of two main steps: (1) query and answer generation, and (2) extension to multi-document setting.

---

2.2.1 Query and Answer Generation
Document Preparation We collect diverse documents from various domains and lengths, primarily using curated resources like Fineweb. These documents span domains including science, history, literature, and technical topics, with an average length of approximately 1 1.6k tokens in our primary dataset. This provides a range of context lengths while focusing on truly long-context scenarios.
Query Generation For each document D, we construct an imput sequence = D E Tree where Tpre contains tokens preceding user query in the model's instruction template. For example, >user and for Qwen-22.5IInttrut <|im_start/ start| >user. We pass X to the aligned LLM and sample completion Q until an end-of-template token is generated or a maximum length is reached. This completion represents a contextually relevant query. By generating multiple queries per document with different sampling parameters, we create diverse document-quer pairs that naturally vary in complexity.
the tokens for Llama-3-Instruct model are < aistar_heaade,
Response Generation For each document-query pair (D, Q), we construct standard instruction prompt by combining the document, query, and tokens that precede assistant response (e.g.. < <leot_id >< settar__header. then generate response R, forming complete instruction triplet (D, Q, R) for long-context training. If the same model is used for both query and response generation, these steps can be consolidated without manual intervention.
>assistant lenc<<eenddaaaa _eadderedr___ _idd_ Ldl> for Llama-3-Instructt We
Query Filtering In query generation, we observed that LLMs occasionally continue the input document rather than generate queries, particularly when the model size is small. To ensure the quality of the generated queries, we upplied two filtering strategies: (1) Rule-based filtering: we retain queries that end with a question mark as simple heuristic to identify interrogative sentences; (2) Length-based filtering: we discard generated texts longer than 1.5k characters, as they are typically descriptive passages rather than valid queries.
2.2.2 Multi-Document Extension
To enhance task diversity and real-world applicability, we extend LongMagpie to multi-document settings. Many tasks require reasoning over several related documents rather than a single one. Our approach involves:
Obtaining x documents {Di, } negative documents via random sampling, where is
drawn uniformly from 0 to n (with n = O reducing to the standard single-document QA setting)..
Concatenating documents using a special separator token (e.g., (edoc_sep>>) to form Dmmutii =
Dj @ < <dcc_sepp>> ① D..
Generating queries and responses as in the single-document pipeline, producing triples (Duulti Q, Rt) requiring cross-document reasoning.
2.3 p-Mix: Balancing Long-Context and Short-Context Capabilities
Fine-tuning predominantly on long-context data degrades performance on short-instruction tasks 2. (2). To balance these capabilities, we introduce p-Mix, novel instruction data hybridization strategy. The core idea is twofold. First, to emulate the typical non-contextual start of general tasks, we sample short-context instruction at the beginning of each training sequence. Second, we append subsequent data segments probabilistically to construct a mixed-conted sequence up to length Lmaaz With probability PL, long-context context instruction (generated by LongMagpie) is chosen; otherwise, with probability 1 PL, another short-context sample is chosen. This process repeats until approaching the target sequence length, ensuring each instance starts with short, context- instruction followed by dynamically mixed sequence of long and short segments. This prepares the model for diverse real-world scenarios. The procedure formalized in Algorithm T and we conduct an ablation study of the parameters related to p-Mix in Appendix A.9

---

Experiments
In this section, we describe our experimental setup, present our main results, and analyze the factors that contribute to LongMegpie's performance.
3.1
Experimental Setup
Dataset Generation
Using the LangMa pipeline described Section 1 we generate long- context instruction dataset using Sween.5-70B-Insteeee with documents sampled from Fine Web-Edu [4]. FineWeb-Edu is subset of the FineWeb dataset, comprising 1.3 trillion tokens extracted from educational web content.
Compared Datasets We compare LongMagpic LongMagpie-eeneratte generated data against several widely used instruc- tion datasets. These include datasets specifically designed for long contexts and standard short-context datasets adapted for long-context fine-tuning based on ProLong [6]]
Long Instruction Datasets We compare with two long-context datasets: ChatQA 52 combines multiple data sources, including LongAlpaca122 12k (8) and GPT-4 samples from Open Orca [28]. containing 1.5 million synthetic instructions. In this work, we refer to ChatQA2 as ChatQA by default; LongAlign
generates questions and answers for long documents by prompting LLMs. Short Instruction Datasets Following findings that concatenated short instructions benefit long- context capabilities [16], we include: Tulu (24). open- collection based on Llama 3.1; Magpie [53], a self-synthesis method using template prefixes; and UltraChat (IT]. comprising 1.5 million multi-turn dialogues. We concatenate samples from these datasets to reach the target context length during fine-tuning.
2
3.1.1 Model Training
We select Llama-3 ---aa----------- 3--B-NExtLong-555 511KK-aaseee [5] as our base model, which has undergone exten- sive long-conteds continued pre-training. The batch size is 4M tokens for 250 steps, total of 1B tokens for baseline datasets and LongMagpie. The same training configuration is applied across all datasets to ensure fair comparison. Further details are provided in Appendix A.1
3.1.2 Evaluation Benchmarks
Long-context Evaluation We evaluate our models on three comprehensive long-context bench- marks. These benchmarks provide holistic assessment of models' abilities to utilize long contexts effectively across different tasks and complexity levels.
evaluates long-context models across diverse pnpiication-centera tasks with context lengths up to 128k tokens, using model-based evaluation that prioritizes complex tasks for better real-world performance prediction.
HELMET 55
RULER 12 provides fine-grained evaluation of long-context reasoning with synthetic tasks that offer flexible control over sequence length and complexity identify performance bottlenecks beyond simple retrieval.
LongBench-Ve v2 [4) upgrade to LongBench (3), assesses extremely long- long-context understanding (8k to 2M words) through 503 expert- validated questions across six categories, revealing a need for improved ultra-long reasoning capabilities.
Short-context Evaluation To further evaluate the model's ability to follow short instructions, we select widely-used short-context datasets: HellaSwag (Hel.) [57], Lambada OpenAI (Lam.) [35].
ARC-Challenge (AR-C.) [9], ARC-Easy (AR-E.), PIQA [6), WinoGrande (Win.) (38), and Logiga
(Log.) [30].
3.2 Main Results
As shown in Table I models trained solely on LongMagpie data already set leading performance on long-contens evaluation, topping HELMET (62.10), RULER (91.17), LongBench-v2 (34.4) and the LongAVG score (62.56) within the Long Instruction Data group. The performance gains are
5

---

Table 1: Main experimental results comparing LangMa with other methods long-context and short-context benchmarks. Best scores in each column are bolded. LongAVG is the average of HELMET, RULER, and Longbench v2, ShortAVG is the average of different short-context tasks.
Long Evaluation
Short Evaluation
Datasct
HELMET
RULER
Longbench v2
LongAVG
ShortAVG
Short Instruction Data
Tulu
61.93
87.92
28.4
59,42
63.90
Magpie
60.18
87.06
31.4
59.55
63.32
UltraChat
60.55
83.85
30.4
58.27
64.43
Long Instruction Data
ChatQA
60.23
89.82
30.8
60.28
63.58
LongAlign
57.79
86.08
24.5
56.12
60.97
LongMagpie
62.10
91.17
34.4
62.56
62.37
p-Mix: Long + Short Instruction Data
ChatQA + UltraChat
60,80
87.42
31.4
59.87
64.38
LongAlign UltraChat
60.98
89.49
30.6
60.36
64.17
LongMagpic + UlrrChaa
62.11
89.70
33
61.60
64.10
substantial compared to existing long- ong-context instruction datasets: LongMagpie outperforms ChatQA by +1.87 on HELMET, +1.35 on RULER, and +3.6 LongBench-v2. yielding a +2.28 improvement on AVn. The gap is even more pronounced when compared with LongAlign, where LongMagpie delivers gains of +4.31 on HELMET, +5.09 on RULER, and +9.9 on LongRench-v2. resulting in remarkable +6.44 improvement on LVngAVG. The strong performance of Mangi on long- context tasks demonstrates the effectiveness of our self-synthesis approach for generating high-quality long-context instruction data without human annotation seed examples.
Among the models trained with p-Mix strategy, which mixes LongMagpie with other short-instruction datasets, LongMagpie + UltraChat achieves the best or tied-bes bi--best scores on HELMET (62.11). RULER (89.70) and LongAVG (61.60) among all mixed datasets. It also retains a competitive Short- AVG accuracy (64.10). only 0.33 below the overall best, confirming that 1) The long-context signals produced by our self-synthesis method are highly complementary to existing short-instruction data, and 2) The probabilistic mixing schedule effectively balances these two instruction regimes, yielding models that are robust across both ultra-long reasoning and everyday short-instruction scenarios. These results highlight the practical value of p-Mix: it preserves the strength of Long long-context tasks while simultaneously mitigating the typical performance drop on short-context benchmarks. We provide further analysis to demonstrate the advantages of p-Mix compared to alternative mixing strategies in Section
4 Ablation Studies
This section first analyzes the key configurations that influence LongMagpic's performance, then evaluates the quality and diversity of its generated queries, and finally assesses the its resource efficiency.
4.1 Impact of Different Multi-Document Settings
To increase instruction difficulty and further enhance the model's ability to capture long-range dependencies, we introduce a multi-document setting. With certain probability, the document associated with a generated query-answe pair is mixed with randomly sampled documents from the corpus, where x is drawn uniformly from 0 to n (with n reducing to the standard single- document QA setting). Table[]provides the detailed performance scores for different values of n the multi-document setting, corresponding to the trends shown Appendix AA.]. We observe that the multi-document strategy significantly improves performance long-context tasks (from 60.19 19 to 62.56). As the value of n increases, the performance on long-context tasks improves and degrades, with the best performance observed when n 10. We hypothesize that this trend is due to

---

Table 2: Detailed results for the impact of the maximum number of documents (n) in user prompt.
HELMET
RULER
Longbench v2
ShortAVG LongAVG
0
60.13
89.04
31.4
60.19
63,20
5
61.42
89,91
31.4
60.91
61.98
10
62.10
91.17
34.4
62.56
62.37
20
61.75
91.08
32.8
61.88
62.04
40
62.08
90.77
31.0
61.28
62.37
80
61.15
90.65
31.0
60.93
62.13
Table p-Mix better balances the performance long- Long-context and short-context than other mixing
strategics.
Strategy
HELMET
RULER
Longbench
LongAVG
ShortAVG
No Mix
62.10
91.17
34.4
62.56
62.37
Sequential Mix
61.60
88.85
31.8
60.75
61.89
Simple Mix
61.84
89.65
31.2
60.90
64.04
p-Mix (Ours)
62.11
89.70
33.0
61.60
64.10
an excessive number of documents increasing the task difficulty beyond the model's learning capacity, thereby leading to drop in performance.
4.2 Impact of Different Mixing Strategy
To investigate the effectiveness of the p-Mix Strategy, we compare p-Mix with three alternative mixing approaches: (1) No Mix: training solely on LongMagpie data without short-context SFT datasets; (2) Sequential Mix: first training short-context data (UltraChat) then fine-tuning long-context data (Long Magpie), similar to [I]] (3) Simple Mix: directly combining and shuffling long and short data in single training stage, similar approaches used with LongAlign [2]; and (4) p-Mix (Ours): our proposed strategy from Algorithm I that pre-pends short instructions and probabilistically mixes segments. As demootddesssssrttess alternative strategies struggle balance long-context and short-context performance compared to our p-Mix approach. In contrast, our p p-Mix strategy demonstrates superior balance: it achieves competitive AVngAVG of 61.60 (notably better than sequential and simple mixing, and only slight trade-off compared to no mixing) while attaining the best ShortAVG score of 64.10. This highlights the efficacy of the p-Mix approach in maintaining strong long-contex reasoning abilities while significantly bolstering performance on short, non-contextual tasks. More details can be found in Appendix A.9
4.3 Impact of Difffrrnt Data Size
To investigate the impact of data volume on model performance, we train our models using two different sizes of LongMagpie-generatte data: 190k and 450k samples. As shown in Table scaling up the training data from 190k to 450k samples leads to consistent improvements across all long- context evaluation benchmarks. Specifically, we observe gains of +0.81 on HELMET, +0.52 on RULER, and +1.8 on Longbench v2, resulting +1.05 improvement the overall LongAVG metric. This demonstrates that increasing the volume of high- long- long-context instruction data significantly enhances the model's 's ability to comprehend and reason over extended contexts.
4.4 Impact of Different Source Model Size
To assess the impact of different models on data synthesis, we use LongMagpie to generate two 450k long-context instructions respectively by the Qwen-2.5-7B model and the Owen-2.5-70BB model. As shown in Table using the larger 70B model improves LongAVG performance (59.61 - 62.56), and shows similar performance on ShortAVG. This superior performance likely stems from larger models' enhanced ability to model long-context capabilities [50), which translates better results when applied to the LongMagpic method.

---

Table Increasing the volume of training data improves performance on long-contexs benchmarks.
Source Model
Data Volume
HELMET
RULER
Longbench v2
LongAVG
ShortAVG
Qwen- 2.B-70B Qwen-2.5-70B
190k 450k
61.29
90.65
32.6
61.51
62.30
62.10
91.17
34.4
62.56
62.37
Table 5: Using the larger source model improves performance on long-contens benchmarks..
Source Model
Data Volume
HELMET
RULER
Longhench v2
LongAVG
ShortAVG
Qwen-2.5-7B Qwen-2.5-70B
450k 450k
59.28 62.10
86,95
32.6
59.61
62.18
91.17
34.4
62.56
62.37
Longhtagple (sun)
Longlhagge (oun)
ChatpA
ChatCh
Largelign
Longhign
Percatage P0
(a) Reward model scores for different datasets.
(b) Query similarities within different datasets.
Figure 2: Analysis SengMaggie-geneaaae data quality and diversity. (a) higher reward model scores indicates higher quality. (b) lower pairwise query similarity indicates better diversity.
Longttague tours)
ChaiDA
Lung- Context rerfrmencc
Longdign
MungMagolesourg
LongAlign
80
ChalQA
Hagple
39
Ultracnat
LongMaggie tourst
Chel
5000 Context Length
15000
10600
(a)
(b)
(c)
Figure 3: Visualizations of LongMagpic characteristics: (a.b) t-SNE visualizations of query embed-
dings from different datasets showing LongMagpie's 's dispersed distribution indicating diversity; (c)
Long-context performance vs. token consumption showing LongMagpie's 's superior performance.
4.5 Analysis of LongMagpie Queries
Higher Quality of LongMagpie Queries We use the Reward Model FsfairX-LLama--RM - [2] to score three long-context fine-tuning datasets. As shown in Figure 22h, the x-axis represents the scores given by the reward model, and the y-axis represents the proportion of data within each dataset corresponding to that score. The overall data quality LongMagpie significantly higher than that of ChatQA and LongAlign.
Better Diversity of Mangia Queries To investigate the diversity of different datasets, we sampled 300 queries from each dataset, inferred their embeddings using the jinaeemeeddings-33 [2] model, and visualized their distribution using t-SNE [55] as shown in Figure 3 It can be observed that LongMagpie's 's distribution is more dispersed, reflecting its better diversity.
Furthermore, we repeated the following experiment 30 times: sampled queries from each dataset, calculated the pairwise similarity between the sampled queries within each dataset, and aggregated the distributions of similarities, as shown in Figure 2h It can seen that LangMagpie queries generally exhibit lower similarity among themselves, which also reflects their good diversity.

---

4.6 Sample Efficiency of LongMagpie
We analyze the sample efficiency of various long-context instruction synthesis methods by quantifying the average token processing requirements during instruction synthesis. As illustrated in Figure Bp. LongMagpie exhibits exceptional sample efficiency, achieving superior long-context performance while processing substantially fewer tokens per instruction (averaging 1. 1.6K tokens D This efficiency stands in stark contrast to methods like ChatQA and LongAlign, which consume 10- 13x more tokens per instruction during synthesis yet produce inferior performance outcomes. LongMagpie' remarkable sample efficiency facilitates greater scalability and diversity.
4.7 Sample-Count-Controll Comparison
To ensure sample-count-controllee comparison, train 190k samples across different methods. For ChatQA, we use its original 190k dataset; for LongAlign, we follow its original construction strategy generate a 190k version. Results shown Table
Table 6: Comparison under equal data size (190k samples).
Method
Data Size
RULER HELMET
LongBenchV2
ChatQA
190k
60.23
89.82
30.8
LongAlign
190k
60.63
87.36
33.0
LongMagpic
190k
61.29
90.65
32.6
We further scale up the LongAlign dataset to 450k samples compare its scalability. Results shown in 7
Table 7: Scalability comparison with increased data size (450k samples).
Method
Data Size
HELMET
RULER
LongBenchV2
LongAlign
450k 450k
60.62
88.77
33.2
LongMagpie
62.10
91.17
34.4
As shown in Table 6 and Table 7 LongMagpic consistently outperforms LongAlign on average. especially the data scale increases. We attribute this to its ability generate more diverse and higher-quality questions (as illustrated in Figure 2) through adaptive query generation, rather than relying on fixed prompt templates seed questions.
Moreover, prior methods often depend on Nomain-specifhi long-context data long- caaeee LLMs, which hinders their scalability. For example, ChatQA synthesizes data using NarrativeQA and needs be combined with LongAlpaca requires long documents and long-context models for data synthesis, and also needs to be mixed with short-text instruction data. In contrast, Magpie uses only general short-document datasets (around 1.6k tokens average, shown in Figure 3c) and simple, scalable method, enabling efficient synthesis at scale without external instruction data.
and OpenOrca reach 190k samples. LongAlign
5
Related Work
5.1 Long-Context Data Synthesis
Existing approaches synthesizing long-context data can be divided into two categories.
Continuation-Orienter Methods Approaches this category generate long-context data by concatenating shorter documents. Early methods [37] used random sampling and concatenation, but failed to maintain meaningful long-range dependencies, Later approaches preserved semantic
coherence through document clustering (20) nearest-neighbor retrieval [40). Quest [14] balances
"Our multi document extension approach enables arbitrary context length extension without incurring additional computational overhead.

---

relevance and diversity using keyword matching. NExtLong [5] decomposes document into multiple meta-chunks and extends the context by interleaving hard negative distractors retrieved from pretraining corpora. However, these methods focus on pre-training rather than instruction tuning. In contrast, MaggMe directly generates instruction-foll foolooiinn data with the model's auto-regressive capabilities.
Instruction-Oriented Methods There exist many approaches to generate long-context instruction data [59] 46 23 43 Representative works include WildLong [26] uses templates and seed questions, LongAlign [2] employs Self-Instruct with packing strategies but requires curated examples, ChatQA 31 blends QA datasets with conversational QA. ChatQA 2 [52) packs documents into 32- 12-122K token contexts, LOGO 44 adapts self-synthesis for long-context alignment, and GATEAU [4] focuses valuable instruction selection. These methods obtain high-quality data through complex pipelines. In contrast, LongMagpie eliminates seed questions, and complex pipelines by leveraging aligned LLMs' ability generate contextually relevant queries when provided only with documents.
5.2 Synthesis Methods for Short-Context Instruction Data
Recent studies scale synthesis across various dimensions: Unnatural Instructions 21 yields diverse instructions through paraphrasing: WizardLM [51] uses evolutionary strategies for challenging vari- ants; GLAN (25 eliminates templates by generating tasks from taxonomies; BARE (58) improves factual correctness; and Humpback 277 performs instruction back-translation. Domain-specific approaches like MetaMath [56] generate specialized content. Magpie [53 demonstrates aligned
LLMs autoregressively generate diverse instructions without human annotation seed examples. Motivated by Magpie, Maggie extends this paradigm to long-context settings by leveraging document-query relationship patterns from instruction tuning, enabling diverse long-context instruc- tion data without specialized prompting.
6 Conclusion
This paper introduces LongMagpie, a self-synthesis method that automatically generates large-scale long-context instruction data without human annotation seed examples. Extensive experiments HELMET, RULER, and Longbench v2 demonstrate that models trained on LongMagpic data achieve leading performance long-context tasks while maintaining competitive short-context capabilities when combined with our proposed p-Mix strategy. This work establishes LongMagpie as an effective approach for democratizing long-contess capabilities.
7 Limitations
First, LongMagpie unavoidably inherits biases from the source instruction-tunet LLMs, which future work should detect and mitigate. Second, the current implementation of LongMagpie inadequately covers tasks requiring long-form outputs, as it primarily focuses on document-query relationships rather than extended reasoning or generation. Future research should expand support for diverse output formats and complex analytical tasks.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (No. U24A20335).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: recipe long context alignment large language models. arXiv preprint 24X1v:2401.18058 2024.
10

---

[3] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
[4] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper under- standing and reasoning on realistic congcconextt multitasks, arXiv preprint 24X2:24412.12044, 2024.
[5] Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, VageeshD C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. Codeplan: Repository-leve coding using iims and planning. Sep 2023.
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[7] Avi Caciularu, MatthewE. Peters, Jacob Goldherger, Ido Dagan, and Arman Cohan. Peek across: Improving multi-document modeling via cross-document questtion-answering May 2023.
[8] Yukang Chen, Shengju Qian, Haotian Tang. Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint ar09v:2309. /2307, 2023.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 180300/800305557 2018.
[10] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint atXiv22307.08691 2023.
[11] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Sheng Si, Yun Liu, Zhiyuan Zhang, Yu Wu, Chao Li, et al. Ultrachat: A large-scale gento-eeeeetedd data for diverse conversations with large language models. arXiv preprint arXiv:2305.14233, 2023.
[12] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.00 247507766, 2024.
[13] Yury Fu, Peter Levin, Nikos Casas, Orhan Firat, and Rohan Anil. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402. 10171, 2024.
[14] Chaochen Gao, Dongfu Li, Liantao Si, Yuanhang Zhao, Xing Wu, Debing Zhang, and Songlin Hu. Quest for long context with 12 norm enhanced position embeddings. arXiv preprint 1rXiv:2402.17320. 2024.
[15] Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. Nextlong: Toward effective long-context training without long documents. arXiv preprint arXiv:2501. /2766, 2025.
[16] Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). 2024.
[17] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval aammentd generation for large language models: A survey, arXiv preprint 102.v923/2.1099., 2:1, 2023.
[18] Google DeepMind. Gemini model updates: March 2025, March 2025. Accessed on May 8, 2025.
[19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.21183,, 2024.
[20] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Retrieval augmented language model pre-training arXiv preprint arXiv:2002.08909 2020.
11

---

[21] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689 2022.
[22] Tianyu Hsieh, Gupta Otkrist, Jeff Wu, Devamanyu Lin, Yuntian Li, Yue Tian, Yann LeCun, and Wenhan Xiong. Ruler: Discrimination-aven long-context benchmarking. arXiv preprini arXiv:2405. 17781, 2024.
[23] Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. Longform: Optimizing instruction uning for long text generation with corpus extraction. In Findings of EMNLP 2024, 2024.
[24] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers in open language model post-tratt training. arXiv preprint arXiv:2411.15124 2024.
[25] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang. Shaohan Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong. Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402. 13064, 2024.
[26] Jiaxi Li, Xingxing Zhang, Xun Wang, Xiaolong Huang, Li Dong, Liang Wang, Si-Qing Chen, Wei Lu, and Furu Wei. Wildlong: Synthesizing realistic long-contex instruction data at scale. arXiv preprint MXXi::2502.166444 2025.
[27] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason We- ston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.0622. 2024.
[28] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://
huggingface co/ ht///daaass///// wwwpaaasttsteepenrrraaaaaaaaaaee / 2023.
[29] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 2IX1:24122.1437,, 2024.
[30] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: challenge dataset for machine reading comprehension with logical reasoning, arXiv preprint arXiv:2007.08124 2020.
[31] Zihan Liu, Wei Ping. Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Building gpt-4 level conversational qa models. CoRR, 2024.
[32] AI @ Meta Llama Team. The llama herd of models, 2024.
[33] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[34] Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf, Fineweb-edu, May 2024.
[35] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi. Sandro Pezzelle. Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv:1606. 06031, 2016.
[36] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEF, 2020.
[37] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Liu, Denis Lebowitz, Piero Molino Ferrer, Tom Cochrane, et al. Code llama: Open foundation models for code. arXiv preprint 230.::::30..990000 2023.
12

---

[38] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106. 2021.
[39] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M.Saiful Bari, Can- wen Xu, Urmish Thakker, Shanya Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, NihalV. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, ZhengXin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and AlexanderM. Rush. Multitask prompted training enables zero-shot task generalization. arXiv preprint 2021.
[40] Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, et al. In-context pretraining: Language modeling beyond document boundaries. arXiv preprint arXiv:2310. 10638, 2023.
[41] Shuzheng Si, Haozhe Zhao, Gang Chen, Yunshui Li, Kangyang Luo, Chuancheng Lv, Kaikai An, Fanchao Qi, Baobao Chang, and Maosong Sun. Gateau: Selecting influential sample for long context alignment. arXiv preprint 1..iv:2400.16611 2024.
[42] Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. jin-eembeddings-33. Multilingual embeddings with task lora, 2024.
[43] Rui Sun, Zhiwei Sun, Yang Li, Yi Ren, and Wei Bi. Efficient training of ultra-long context large language models. arXiv preprint arXiv:2504.06214 2025.
[44] Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. Logo-long context alignment via efficient preference optimization. arXiv preprint arXiv:2410. 1...2333,,8833,. 2024.
[45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911), 2008
[46] Liang Wang, Nan Yang. Xingxing Zhang, Xiaolong Huang, and Furu Wei. Bootstrap your own context length. arXiv preprint arXiv:2412.18866 2024.
[47] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. ielf-instruut: Aligning language models with self-generated instruc- tions. arXiv preprint arXiv:2212. 2022.
[48] Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mecha- nistically explains long-context factuality. arXiv preprint arXiv:2404. 15574, 2024.
[49] Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. Less is more for long document summary evaluation by llms. arXiv preprint arXiv:2309.07388, 2023.
[50] Wenhan Xiong, Jingyu Liu, Igor Molyhog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. /6039, 2023.
[51] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
[52) Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa 2: Bridging the gap proprietary lmm in long context and rag capabilities. arXiv preprint arXiv:2407. /4482, 2024.
[53] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, arXiv preprint arXiv:2406.088 08464, 2024.
13

---

[54] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[55] Tse- Yu Yen, Tsu-Jui Cheng, Paul Pu Liang, Xiang Dong, Tianyu Zhao, Wenhan Liu, Wenhao Wang, Min Peng. Oleksiy Shliazhko, Li Zhang, et al. Helmet: A hierarchical efficient benchmark for long context evaluation with modular units and comprchensive taxonomy. arXiv preprint arXiv:2405. 18696, 2024.
[56] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR 2024, 2024.
[57] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.000 07830, 2019.
[58] Alan Zhu, Parth Asawa, Jared Q Davis, Lingjiao Chen, Boris Hanin, Ion Stoica, Joseph Gonzalez, and Matei Zaharia. Bare: Combining base and instruction-tunet language models for better synthetic data generation. arXiv preprint arXiv:2502.01697, 2025.
[59] Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexan- dra Birch. Effective data synthesis for long-context instruction tuning. arXiv preprint arXiv:2502. 15592, 2025.
14

---

A Detailed Experimental Results
A.1 Training Contig
We employ the AdamW [33] optimizer with parameters 31 = 0.9 and 82 0.95. Following
ProLong [16), we concatenate samples up to 64K sequence length and apply the document mask- ing technique to prevent interactions between independent sequences. Additionally, we utilize 2 [0] and ZeRO [36] optimize memory usage and accelerate training. The detailed
training config is shown in Table 8
Table 8: Model Training Configuration.
training setting
Initial Model
Llama-3- Baama-3--B---- -on---5og
rotary-emb-bass
128,000,000
31
0.9
92
0.95
Ir
2e-5
precision
6
gradient-clipping
1.0
weight-decay
0.1
It-decay-style
cosine
train-iters
250
seq-length
65536
GPU-type
H100
GPU-numbers
8
training-time
10h
A.2 Detailed Results of HELMET
We present results across a comprehensive suite of HELMET tasks, including Recall, RAG, ICL, Re-rank, LongQA, Cite, Summ, and RULER. The complete evaluation results shown Table In Section 3 we report the average performance excluding the Cite and Summ tasks, as these two are newly included and evaluated in the latest version of our experiments.
Table 9: Evaluation results across HELMET tasks.
Method
Recall
RAG
ICL
Re-rank
LongQA
Cite
Summ.
RULER
ChatQA
93.34
66.47
80.36
23.74
37.25
15.18
20.61
89.82
LongAlign
92.43
59.05
81.20
27.12
29.14
17.93
24.32
86.08
LongMagpie
97.53
63.37
85.84
28.60
35.16
19.99
26.36
91.17
A.3 Detailed Results of LongBench v2
We further evaluate our approach the LongBench V2 benchmark, which measures multi-domain long-context understanding across variety of tasks, including multi-document QA (Multi-Doc QA), long in-context learning (ICL), single-document QA (Single-Doc QA). code repo understanding (Code), long-dialogue history understanding (Long-dial.), and long structured data understanding (Long Stru.). The detailed results are shown in Table 10 Our proposed method (LongMagpie) consistently outperforms prior approaches across most categories, showing powerful performance on long dialogue history understanding and multi nulti-document question answering.
Table 10: Evaluation results across LongBench v2 tasks.
Method
Multi-Doc QA
ICL
Single-Doc QA
Code
Long-dial.
Long Stru.
ChatQA
25.6
34.57
36.0
24.0
25.64
30.30
LongAlign
20.0
27.16
28.57
24.0
17.95
21.21
LongMagpie
28.8
35.8
37.14
28.0
46.15
33.33
15

---

A.4 Distribution of Generated Query Types
We categorize the generated QA pairs into various task types. As shown in Table [1 our frame- work already generates a substantial number of instances beyond traditional ducument-query pairs, including tasks related summarization and complex structured extraction.
Table 11: Distribution of generated QA pair types.
Category
Count
Precise Retrieval
201.306
Summarization
91,118
Advice Seeking
51,609
Planning or Reasoning (Multi-step Analysis)
38,526
Comparative Choice-Basee Based Task
25,342
Math Data Analysis
8,679
Complex Structured Extraction
5,725
Creative Task
3,339
Coding & Debugging
2,999
In addition, the generated task types in LongMagpie often align closely with document content e.g.. code documents yield code-related queries, and structured texts lead to extraction tasks. We will expand to more diverse domains and formats broaden task coverage, with concrete examples to be included in the future.
A.5 Effect of Retrieval-Focused Training Data
Based on our experimental results, we find that serreval-focused training data do limit the model's generalization to other cong-context skills. On the contrary, the improved retrieval capability facilitates performance across various tasks. Intuitively, effective retrieval a foundational skill for handling long-context inputs, as models must first identify relevant information before generating accurate responses. The importance of retrieval in long-context models has also been widely recognized in prior work [48 [3]
To investigate this more directly, we conduct additional experiments on the 190k-sample dataset. Specifically, we vary the proportion of Precise Retrieval data while adjusting the other data distribu- tions accordingly. One setting reduces the Precise Retrieval portion from 50% 30%, and the other increases it to 70%.
Table 12: Performance comparison under different proportions Precise Retrieval data.
Precise Retrieval (%)
Recall
RAG
ICL
Re-rank
LongQA
Cite
Summ.
RULER
30% 50%
97.63
62.99
80.92
25.44
34.72
19.30
26.12
90.71
97.29
62.72
85.12
26.26
35.05
20.39
24.32
90.65
70%
98.85
63.38
84.16
26.85
36.26
22.02
24.86
90.93
As shown in Table 12 increasing the proportion of Precise Retrieval data improves the model's Recall performance, which also leads to consistent gains in downstream tasks such as RAG, Re-rank, LongQA, and Cite, confirming that retrieval-centric training benefits general long-context capabilities.
A.6 Replacing Ultrachat with Magpie
We conduct experiments using the combination of Maggie and Magpie. The results were roughly comparable to the LongMagpic Ultrachat setting. As shown in Table [3] observe a slight improvement in long-context performance, while the performance on short-contexl tasks decreased slightly.
A.7 Safety Analysis
We performed a safety analysis using the Hama-Guard-3-3 32 model to classify the generated content. As shown in Table [4] the resulting dataset is overwhelmingly safe, with 99.86% of samples
16

---

Table 13: Comparison between LongMagpie Ultrachat and LongMagpie + Magpie.
Method
HLLMET
RUELE
LongBenchV2
LongAVG
ShortAVG
LangAVG ShortAVG
LongMagpic + Utrachat
62.11
89.70
33.00
61 60
64.10
62.85
LongMagpic + Magpie
61.95
90.47
33.40
61.94
63.17
62.56
Table 14: Safety classification results of the Mangia dataset.
Category
Percentage (%)
Safe
99.8603
Specialized Advice
0.1147
Intellectual Property
0,0072
Non- Violent Crimes
0.0063
Hate
0.0033
Indiscriminate Weapons
0.0028
Child Sexual Exploitation
0.0009
Violent Crimes
0.0009
Defamation
0.0009
Elections
0.0002
Sexual Content
0,0007
Code Interpreter Abuse
0,0007
Privacy
0.0005
Sex-Related Crimes
0.0002
Suicide & Self-Harm
0.0002
categorized as safe. This suggests that our pipeline can produce high- figh-quality instructional data with minimal safety concerns.
A.8 Impact of mutti-Document Setting
Figure illustrates the performance variation under different multi-document configurations.
1.0
NET
BILER
Longbercch
Normalted Nerformencg
0.8
Shart Ph famfmmanc
0.4
0.2
0.0
10
90
Maximum Number of Documents it User Prompt
setting model performance. As the number of documents increases, the performance on long-cone cong-context tasks improves and then decreases.
Figure 4: Impact of the
A.9 Ablation Study on p-Mix Strategy Parameters
To further understand the behavior of the p-Mix strategy, we conducted an ablation study on key parameters: the number of initial short-context samples pre-pended (Ns), and the probability (Pr) of selecting long-context sample during the probabilistic mixing phase (see Algorithm [T] The results, presented in Table 15] showcase how different configurations impact overall performance on both long and short tasks evaluation benchmarks. These experiments were conducted with n = 10 for the multi-document context length parameter.
17

---

Table 15: Detailed ablation results for different parameter settings of the p-Mix strategy. Ng is the number pre-pended short tasks. PL is the long-context selection probability.
Pp
HELMET
RULER
Longhench
LongAVG
ShomAVG
0.2
61.38
$8.52
59.83
0.4
61.84
89.65
31.20
60.90
64.04
0.6
61.64
90.51
31.00
61.05
63.92
0.8
61.48
90.54
30.40
60.81
63.41
0.4
62.11
89,70
33,00
61.60
64,10
0.6
61.74
90.58
29,80
60.71
63.71
0.8
61.45
90.66
28.80
60.30
63.33
0.4
61.70
88.67
31.20
60.52
64,13
0.6
61.90
90.07
30,00
60.66
63.97
0.8
61.34
90.53
31,00
60.96
63.68
85.67
0.4
60.77
85,30
30.00
58.69
64,25
0.6
60.67
86.09
30.80
59.19
64.39
0.8
60.60
84.42
30.00
58.34
64.21
Algorithm Hybrid SFT Data Construction with short-context Pre-pending and Probabilistic Mixing
1: procedure CONSTRLCTHYMRISSAANNE
PL. Lmmax: sep)
2: Initialize Sconcat +- empty sequence b Ds: set of short-context SFT samples, Dz: set of long-context SFT samples Pe:
probability selecting long-context sample, Lmax: sequence length
sep: separator loken/sequence between samples
3: #p +- RundomSanplee Du)
4: Sconeut <- FormatSample(55)
5: perrent_length +- Length(Seonnaat)
6:
while caurenttJengle < Limaz
7: rand RandomReal(O,
8:
raaddd < Pt. then
D. Select long-context sample with probubility Pt.
9:
lnest (andonSSample((D.))
10:
formatted (neet FormatSample(Incal)
11:
current, Jenghh Length (sep) Length( formoffed Limax
12:
Sconent Seancat sep formetted
13:
current length 4- Length(Sconeat)
14:
break else
15:
D Next sample exceeds Lmax
16:
17:
else
D. Select short-context sample with probahility Pz.
18:
Snezt RandomSample(Ds)
19:
formatted Enese FormatSampll(enesr if current, Jenghh + Length + Length( formatted Anout) S Lmmaa then
20:
21:
Sconcat Scancae eep formetted Sneze current_length +- Length(S.oncat) Sconeat)
22:
23:
24:
break end end if
Next sample exceeds Timax
25:
26:
27:
end while
28:
Sconcut
29:
end procedure
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
18

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
19

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section and Appendix A.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way access this model for reproducing the results way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case closed-source models, it may that access to the model limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release our data and model in the future version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines lhttps://nips.. cc/ gublic/guidess httpl////uiddes/ddddbubbmisssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https phiips.cc/ubllie httpa////uuueee/ooooesssssssssao
for more details.
20

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 3.1.1 and
and Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: error bars are not reported because it would be too computationally expensive. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eитоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Appendix A.I
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp/////neppppppppp/p////////e
Answer:
[Yes]
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there is a special considera- tion due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We only open-source data and models, and our research focuses improving long-context model performance.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, Ict alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Question:
22

---

Justification:
We currently do not release any model or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers that we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, (ppperswithheeeec thoode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors encouraged reach the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We currently do not release any model or data.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
23

---

Our paper does not involve crowdsoureing nor research with human subjects. Guidelines:
Justification:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our paper describe the usage of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components. ctt.pppsscccceooooeceeeeeeeeeeee 20ticceee//22///1/ /LMM) for
Please refer LLM policy what should or should not be described.
24


Reviews:
[{"review_id": "wctfCUiUOt", "reviewer": "Reviewer_iG7h", "summary": "This paper addresses the need for high-quality, large-scale long-context instruction data for LLMs. \nBuilding on the Magpie framework, which generates queries from domain-specific prompts using pretrained LLMs, LongMagpie replaces these prompts with long context documents. This allows aligned LLMs to self-synthesize relevant document-query-response triplets by simply being presented with a documen, an empty user prompt and special tokens indicating start of users' prompt.\n\nThe method extends to multi-document contexts for more complex reasoning and introduces the \"p-Mix\" strategy to blend long and short-context instructions during training, preventing performance degradation on short-context tasks.\n\nExperiments show models trained on LongMagpie data achieve better performance on long-context benchmarks (HELMET, RULER, LongBench-v2) compared with existing short or long-context instruction-tuning dataset, while p-Mix ensures competitive short-context capabilities. The generated data is proven to be of higher diversity with sample efficiency.", "questions": "1. While LongMagpie shows leading performance, some gains over strong baselines appear marginal. Also, data scaling yields modest improvements (+1.05 LongAVG from 190k to 450k samples). Could authors elaborate on the practical significance of these gains, perhaps highlighting scenarios or metrics where the advantage is more pronounced? Is it possible to do further scaling to observe more significant gains?\n2. Could the author provide a more granular breakdown of performance for benchmarks like RULER and HELMET across different context length ranges and tasks. This would clarify effectiveness at truly extreme versus moderately long contexts, and the effectivenes across tasks?\n3. Training uses equal total tokens (1B), but distinct sample counts may vary between datasets. Since SFT is sensitive to unique examples, not just token count, could authors clarify this comparison or discuss if a sample-count-controlled comparison was considered?\n4. How do authors plan to detect and mitigate biases inherited from the source LLMs in the generated data? Additionally, can the framework be extended beyond document-query relationships to cover other critical long-context tasks (e.g., summarization, complex structured extraction, multi-turn dialogue)?\n5. Any safely analysis of the generated dataset? It seems important to do filtering when synthesizing data using pretrained LMs.", "limitations": "The authors discussed the limitation adequately.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper effectively tackles the challenge of acquiring large-scale, high-quality long-context instruction data, which has been a major bottleneck for training powerful long-context LLMs.\n2. LongMagpie's core innovation lies in its \"self-synthesis\" approach, which leverages the inherent auto-regressive capabilities of aligned LLMs to generate contextually relevant queries from documents. This method eliminates the need for expensive human annotation or complex, pre-defined templates.\n3. The paper provides evidence that the synthetically generated queries are of greater diversity, which are crucial factors for effective instruction tuning.\n\nWeaknesses\n1. A fundamental limitation is that LongMagpie, by design, inherits biases from the source instruction-tuned LLMs used for data generation. If the base LLM has biases, these will likely be propagated into the synthetic instruction data.\n2. The current method primarily focuses on document-query-response triplets, which might not adequately cover tasks requiring more complex or open-ended outputs. The emphasis is heavily on information extraction and question-answering from documents. Analysis regarding the covered tasks is needed. Additionally, authors can provide concrete examples of the synthesized data so that readers can understand it more clearly. \n3. While overall performance is leading, looking at Table 1, some of the improvements over strong baselines (LongMagpie + UltraChat vs. LongAlign + UltraChat) on benchmarks like RULER and HELMET can appear somewhat marginal (e.g., gains of 1 points). This raises questions about the practical significance of some of the reported advances.\n4. Although the framework is designed to generate large amounts of data, the ablation study on data size shows that the performance improvement (e.g., +1.05 in LongAVG when scaling from 190k to 450k samples) can be rather modest. This raises a concern about diminishing returns when further scaling up the dataset, suggesting that the framework's scalability in terms of performance gain per additional data unit might be limited beyond a certain point.\n5. The paper presents aggregated scores for benchmarks like RULER and HELMET. A more detailed breakdown of performance across different context length ranges and tasks within these benchmarks would be highly beneficial. This would provide a clearer understanding of the method's effectiveness at truly extreme long contexts versus moderately long ones, and its effectiveness across different task types.\n6. The authors state that training involves a total of 1B tokens for both baseline datasets and LongMagpie, ensuring comparable computational cost. However, the number of distinct training instances (samples) for these 1B tokens can vary significantly between datasets. Since SFT performance can be heavily influenced by the diversity and quantity of unique examples, not just total token count, this discrepancy could lead to an unfair comparison, potentially disadvantaging baselines with fewer but longer instances. This aspect could be better clarified or controlled for in the experimental design. For example, LongAlign only contains around 10k instances, which is significanly less than those used in LongMagpie.", "was_revised": true, "final_justification": "Author addressed most of my concerns during rebuttal, and I appreciate the new results. Therefore, I decided to raise the rating to 4. However, three concerns still remain.\n\n1. The method inherits bias from the underlying pretrained LLMs. \n2. Building on the Magpie framework, which generates queries from domain-specific prompts using pretrained LLMs, LongMagpie replaces these prompts with long context documents. The novelty of this method seems somewhat modest to me. \n3. Modest improvement is achieved by scaling from 190k data to 450k data as noted in W4.", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article.\n\n---\n\n**Q1: \"some of the improvements over strong baselines (LongMagpie + UltraChat vs. LongAlign + UltraChat) on benchmarks like RULER and HELMET can appear somewhat marginal (e.g., gains of 1 points)...\"**\n\n**A1:** We would like to clarify that the best long-context performance is achieved using only LongMagpie data. The p-Mix strategy offers a broader option that caters to scenarios where stronger performance on short-text inputs is required. This design choice, to some extent, comes at the cost of slightly reduced performance on long-context benchmarks. However, for long-context tasks specifically, using only the LongMagpie-generated data yields the best results. Compared to other methods, we obtain the highest scores on HELMET (62.10), RULER (91.17), and LongBench V2 (34.4). \n\n\n**Q2: \"...Could authors elaborate on the practical significance of these gains, perhaps highlighting scenarios or metrics where the advantage is more pronounced?\"**\n\n\n**A2:** Following the partitioning method in [1], we provide metrics for different sub-tasks.\n\nHELMET and RULER Results:\n\n| Method      | Recall  | RAG     | ICL     | Re-rank | LongQA  | RULER|\n|:-----------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| ChatQA      | 93.34 | **66.47** | 80.36 | 23.74   | **37.25** | _89.82_ |\n| LongAlign   | 92.43   | 59.05   | 81.20   | 27.12 | 29.14   | 86.08 |\n| LongAlign + UltraChat   | _94.81_ | _63.81_   | _84.44_ | **30.48** | 31.36   | 89.49 |\n| LongMagpie  | **97.53** | 63.37 | **85.84** | _28.60_ | _35.16_ | **91.17** |\n\nLongBench V2 Results:\n\n| Method      | Multi-Doc QA | ICL     | Single-Doc QA | Code    | Long-dialogue History Understanding | Long Structured Data Understanding |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| ChatQA      | _25.6_        | _34.57_ | _36_           | 24      | 25.64                             | 30.3                             |\n| LongAlign   | 20            | 27.16   | 28.57          | _24_      | 17.95                                | 21.21                              |\n| LongAlign + Ultrachat    | 23.2          | 23.46   | 34.29          | 18     | _30.77_                                | **45.45**                          |\n| LongMagpie  | **28.8**      | **35.8**| **37.14**      | **28**  | **46.15**                            | _33.33_                          |\n\n\nThese results highlight that LongMagpie consistently achieves either the best or second-best performance across the majority of subtasks, leading to its overall superior performance. In particular, it demonstrates significant improvements in **Recall, ICL** on the HELMET benchmark, as well as in **QA, Code, and Long-Dialogue History Understanding tasks** on LongBench V2.\n\n\n**Q3: \"...Is it possible to do further scaling to observe more significant gains?\"**\n\n**A3:** Due to our limited computational resources and time, we are unable to scale to larger data volumes within the current rebuttal period. However, as we discussed in **A5**, we compared our method with LongAlign and demonstrated that LongMagpie exhibits stronger scalability and more consistent performance improvements as the data scale increases.\n\n\n\n**Q4: \"...This would clarify effectiveness at truly extreme versus moderately long contexts, and the effectivenes across tasks?\"**\n\n**A4:**  As outlined in **A2**, we provide a task-based categorization, and the reported metrics there are averaged over context lengths of 8k, 16k, 32k, 64k, and 128k. To further clarify effectiveness under truly extreme context lengths, we present the detailed results for the 128k context length:\n\n| Method      | Recall       | RAG          | ICL          | Re-rank      | LongQA       | RULER        |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| ChatQA      | 82.69        | **61.88**      | 87           | 6.05         | **41.75**    | 78.64      |\n| LongAlign   | _87.75_      | 50.29        | 87.4       | 8.64       | 31.14        | 78.06        |\n| LongAlign + Ultrachat   |87.56 | _58.88_ | _89.60_ | _9.87_ | 33.60 | _82.05_ |\n| LongMagpie  | **95.44**    | 58.29    | **90**       | **14.55**    | _38.14_      | **84.11**    |\n\n\nThese results show that LongMagpie continues to deliver leading or second-best performance even at extreme context lengths. We will provide a breakdown of results across different context lengths from 8k to 128k in the revised version.\n\n\n**Q5: \"...could authors clarify this comparison or discuss if a sample-count-controlled comparison was considered?\"**\n\n**A5:**  To ensure a sample-count-controlled comparison, we train on 190k samples across different methods. For ChatQA, we use its original 190k dataset;  For LongAlign, we follow its original construction strategy to generate a 190k version. Results are shown below:\n\n| Method      | Data Size | HELMET | RUELR | LongbenchV2 |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|\n| ChatQA   | 190k      | 60.23  | _89.82_ | 30.8        |\n| LongAlign   | 190k      | _60.63_  | 87.36  | **33**        |\n| Longmagpie  | 190k      | **61.29**   | **90.65** | _32.6_        |\n\nWe further scale up the LongAlign dataset to 450k samples to compare its scalability. Results are as follows:\n\n| Method      | Data Size | HELMET | RUELR | LongbenchV2 |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|\n| LongAlign   | 450k      | 60.62   | 88.77 | 33.2        |\n| Longmagpie  | 450k      | **62.1**   | **91.17** | **34.4**        |\n\nAs shown, LongMagpie consistently outperforms LongAlign on average, especially as the data scale increases. We attribute this to its ability to generate more diverse and higher-quality questions (as shown in Figure 2) through adaptive query generation, rather than relying on fixed prompt templates or seed questions.\n\nIn addition, prior methods often depend on domain-specific long-context data or long-context-capable LLMs, which hinders their scalability. For example, ChatQA synthesizes data using NarrativeQA and needs to be combined with LongAlpaca12k and OpenOrca to reach 190k samples. LongAlign requires long documents and long-context models for data synthesis, and also needs to be mixed with short-text instruction data. In contrast, LongMagpie uses only general short-document datasets (~1.6k tokens average, as shown in Figure 3) and a simple, scalable method, enabling efficient synthesis at scale without external instruction data.\n\n\n\n\n**Q6: \"How do authors plan to detect and mitigate biases inherited from the source LLMs in the generated data? \"**\n\n**A6:** We plan to adopt the following strategies to detect and mitigate biases inherited from the source LLMs:\n\n* **Cross-model verification**: Inspired by the multi-agent debate framework proposed by Google [2], we will generate QA pairs for each document using multiple LLMs, then conduct a debate among them. A separate judge model is used to evaluate the debated outputs and select the most appropriate responses. This process helps reduce the likelihood of propagating biases or errors from any single model.\n    \n* **Quality filtering**: We employ FsfairX-Llama3-RM-v0.1 to score and filter out low-quality generated data (As shown in Figure 2).\n    \n* **Safety filtering**: In addition, we perform safety checks using Llama-Guard-3-8B. Our analysis shows that **99.86%** of the data passes the safety criteria. Detailed safety statistics are provided in our response to Reviewer KGZd (as we discussed in **A7**).\n\n**Q7: \"...Additionally, can the framework be extended beyond document-query relationships to cover other critical long-context tasks (e.g., summarization, complex structured extraction, multi-turn dialogue)?\"**\n\n**A7:** We categorize the generated QA pairs into various task types, as shown below:\n\n| Category                                | Count   |\n|-----------------------------------------|---------|\n| Precise Retrieval                       | 201,306 |\n| Summarization                           | 91,118  |\n| Advice Seeking                          | 51,609  |\n| Planning or Reasoning (Multi-step Analysis) | 38,526  |\n| Comparative or Choice-Based Task        | 25,342  |\n| Math or Data Analysis                   | 8,679   |\n| Complex Structured Extraction           | 5,725   |\n| Creative Task                           | 3,339   |\n| Coding & Debugging                      | 2,999   |\n\n\nAs shown above, our framework already generates a substantial number of instances beyond traditional document-query pairs, including tasks related to summarization and complex structured extraction. \n\nIn addition, the generated task types in LongMagpie often align closely with document content—e.g., code documents yield code-related queries, and structured texts lead to extraction tasks. We will expand to more diverse domains and formats to broaden task coverage, with concrete examples to be included in the revised version.\n\nWhile we do not explicitly construct multi-turn datasets, our method improves performance on LongBench V2's Long-dialogue History Understanding task (see **A2**), indicating implicit benefits for multi-turn dialogue.\n\n**Q8: \"Any safely analysis of the generated dataset?\"**\n\n**A8:**  Our analysis shows that **99.86%** of the data passes the safety criteria. Detailed safety statistics are provided in our response to Reviewer KGZd (as we discussed in **A7**).\n\n\n---\n\n[1] Gao C, Wu X, Lin Z, et al. Nextlong: Toward effective long-context training without long documents. ICML 2025.\n\n[2] Du, Yilun, et al. Improving factuality and reasoning in language models through multiagent debate. ICML 2023."}, {"review_id": "AR3Z8KXF4u", "reviewer": "Reviewer_vum6", "summary": "The paper presents LongMagpie, a straightforward strategy for sampling long-context instructions from an aligned LLM. This approach extends the original Magpie method, where a language model is given a conditional system prompt along with special tokens and tasked with generating both a user query and its corresponding response. LongMagpie follows a similar setup but introduces documents into the prompt, encouraging the model to produce queries that focus on long-document understanding. The results indicate that the generated instructions effectively capture long-context phenomena, achieving state-of-the-art performance on long-context benchmarks. Furthermore, the authors propose a method for mixing long- and short-context instructions to better balance the model’s performance across both settings.", "questions": "- In the data mixing experiments, why was the combination of LongMagpie + Magpie not explored? Since both methods are closely related, it would be interesting to see whether combining them yields better or more balanced performance.", "limitations": "While the authors acknowledge several limitations, one important aspect is not addressed: the proposed method relies on access to a collection of high-quality documents to generate the long-context instructions. This requirement may restrict the applicability of the approach in low-resource languages or specialized domains where such document collections are scarce or difficult to obtain.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper is well written and easy to follow.\n- The main contribution is simple yet effective and potentially impactful.\n- The authors conduct several ablations and insightful analyses.\n\n### Weaknesses\n- The effectiveness of the second contribution, p-Mix, is unclear. If the goal is to achieve a balanced performance across both long- and short-context tasks, one might consider using the harmonic mean of the ShortAVG and LongAVG scores as a more representative metric. Based on this, the improvement of p-Mix (62.82) over No-Mix (62.46) appears marginal, making it difficult to draw strong conclusions about its impact. While p-Mix clearly shifts the trade-off between long and short performance, it is not evident that it offers a meaningful balance overall.\n- There are some concerns regarding the evaluation framework of the main contribution:\n  - What model was used to generate the instructions for the (short-context) Magpie dataset? If it differs from the model used for LongMagpie, what is the rationale behind that choice? For a fair comparison of the techniques, it would be more robust to keep the model architecture consistent across datasets.\n  - Table 4 highlights that the size of the dataset has a measurable effect on performance. However, the paper does not clearly specify how many data instances are present in the other evaluated datasets. How do their sizes compare to LongMagpie? A clearer and more detailed description of the experimental setup would improve the paper’s transparency and reproducibility.", "was_revised": true, "final_justification": "During the rebuttal the authors clarified some of the points raised in my review, however, the justification of the p-mix approach is still limited. I decided to keep my score, which was already high.", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.\n\n---\n\n**Q1: \"...While p-Mix clearly shifts the trade-off between long and short performance, it is not evident that it offers a meaningful balance overall.\"**\n\n\n**A1:** The p-Mix strategy offers a broader option that caters to scenarios where stronger performance on short-text inputs is required. As shown in Table 3, among all mix approaches, p-Mix achieves the best short-text performance while incurring the least degradation in long-context performance. For use cases that prioritize long-context understanding, we recommend directly employing our best-performing model trained solely on LongMagpie data.\n\n\n**Q2: \"What model was used to generate the instructions for the (short-context) Magpie dataset?\"**\n\n**A2:**  The instructions for the Magpie dataset were also generated using Qwen2.5-72B-Instruct, which is the same model we used in our work.\n\n\n**Q3: \"Table 4 highlights that the size of the dataset has a measurable effect on performance. However, the paper does not clearly specify how many data instances are present in the other evaluated datasets. How do their sizes compare to LongMagpie? A clearer and more detailed description of the experimental setup would improve the paper’s transparency and reproducibility.\"**\n\n**A3:** To ensure a sample-count-controlled comparison, we train on 190k samples across different methods. For ChatQA, we use its original 190k dataset;  For LongAlign, we follow its original construction strategy to generate a 190k version. Results are shown below:\n\n| Method      | Data Size | HELMET | RUELR | LongbenchV2 |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|\n| ChatQA   | 190k      | 60.23  | _89.82_ | 30.8        |\n| LongAlign   | 190k      | _60.63_  | 87.36  | **33**        |\n| Longmagpie  | 190k      | **61.29**   | **90.65** | _32.6_        |\n\nWe further scale up the LongAlign dataset to 450k samples to compare its scalability. Results are as follows:\n\n| Method      | Data Size | HELMET | RUELR | LongbenchV2 |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|\n| LongAlign   | 450k      | 60.62   | 88.77 | 33.2        |\n| Longmagpie  | 450k      | **62.1**   | **91.17** | **34.4**        |\n\nAs shown, LongMagpie consistently outperforms LongAlign on average, especially as the data scale increases. We attribute this to its ability to generate more diverse and higher-quality questions (as shown in Figure 2) through adaptive query generation, rather than relying on fixed prompt templates or seed questions.\n\nIn addition, prior methods often depend on domain-specific long-context data or long-context-capable LLMs, which hinders their scalability. For example, ChatQA synthesizes data using NarrativeQA and needs to be combined with LongAlpaca12k and OpenOrca to reach 190k samples. LongAlign requires long documents and long-context models for data synthesis, and also needs to be mixed with short-text instruction data. In contrast, LongMagpie uses only general short-document datasets (~1.6k tokens average, as shown in Figure 3) and a simple, scalable method, enabling efficient synthesis at scale without external instruction data.\n\n\n\n\n\n\n\n**Q4: \"In the data mixing experiments, why was the combination of LongMagpie + Magpie not explored? \"**\n\n**A4:** We conduct experiments using the combination of LongMagpie and Magpie. The results were roughly comparable to the LongMagpie + Ultrachat setting. Specifically, we observed a slight improvement in long-context performance, while the performance on short-context tasks decreased slightly.\n\n| Method      | HELMET | RUELR | LongbenchV2 | LongAVG | ShortAVG | LongAVG + ShortAVG |\n|:-----------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| Longmagpie + Ultrachat  | **62.11**  | 89.70 | 33  | 61.60| **64.10** | **62.85** | \n| Longmagpie + Magpie | 61.95    | **90.47**  | **33.4** | **61.94** |  63.17 | 62.56 | \n\n\n**Q5: \"...This requirement may restrict the applicability of the approach in low-resource languages or specialized domains where such document collections are scarce or difficult to obtain.\"**\n\n**A5:** For low-resource languages or specialized domains where document collections are scarce or hard to obtain, one potential solution is first to use large language models (LLMs) to synthesize domain-relevant documents, followed by employing LongMagpie to generate QA datasets. This approach can partially alleviate the issue. We believe that exploring the synthesis of documents remains an exciting and valuable direction for future research."}, {"review_id": "2TauzdlW6g", "reviewer": "Reviewer_KGZd", "summary": "LongMaggie introduces an automated framework for generating large-scale long-context instruction data without human annotation or predefined templates. The core insight is that aligned LLMs (e.g., Qwen, Llama) auto-regressively generate contextually relevant queries when provided with a document followed by special user-role tokens. The pipeline involves: (1) Extracting diverse documents (e.g., from FineWeb-Edu), (2) Generating queries via LLMs, (3) Filtering low-quality queries (rule/length-based), (4) Producing responses to form instruction triplets. The method extends to multi-document settings for complex reasoning and proposes p-Mix—a probabilistic data-mixing strategy—to mitigate short-context performance degradation. Evaluations on HELMET, RULER, and LongBench-v2 show state-of-the-art long-context performance. Models trained solely on LongMaggie data outperform competitors, while *p*-Mix hybrids maintain competitive short-context results.", "questions": "1. For query generation failure cases (Sec 2.2.1), how frequently do smaller models output non-query text? Could a classifier improve filtering beyond rule/length heuristics?\n\n2. Beyond QA, can LongMaggie support tasks requiring long-form outputs by modifying the generation protocol?", "limitations": "1. Architectural Bias: Synthesized data inherits limitations of source LLMs (Qwen/Llama), risking error propagation.\n\n2. Narrow Output Scope: Emphasis on document-query pairs overlooks tasks needing extended reasoning (e.g., multi-step analysis).\n\n3. Computational Cost: Heavy reliance on 70B models for high-quality data generation raises accessibility barriers.\n\n4. Ethical Gaps: No discussion of misuse potential (e.g., generating deceptive long-context content) or fairness audits.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Novelty: Leverages inherent LLM capabilities for query synthesis, eliminating costly human annotation/seed examples.\n\n2. Effectiveness: Achieves SOTA on major long-context benchmarks (HELMET: 62.10, RULER: 91.17).\n\n3. Scalability: Generates 450K high-quality instructions efficiently (1.6K tokens/sample vs. 10-13× for competitors).\n\nWeaknesses:\n\n1. Bias Inheritance: Acknowledges uncorrected biases from source LLMs.", "was_revised": true, "final_justification": "Thanks for the authors' response. I have read it and decided to maintain my positive score.", "rebuttal": "We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.\n\n---\n\n**Q1: \"For query generation failure cases (Sec 2.2.1), how frequently do smaller models output non-query text? \"**\n\n\n**A1:** When using the smaller Qwen2.5-7B-Instruct model to generate data, the success rate is 77.2%. This means that 22.8% of the generated texts are non-query text.\n\n**Q2: \"Could a classifier improve filtering beyond rule/length heuristics?\"**\n\n**A2:** \nTo compare the performance of a classifier and rule/length heuristics, we sample 1,000 outputs and use the Qwen2.5-72B-Instruct model as the classifier for comparison with rule/length heuristics. The prompt provided to Qwen2.5-72B-Instruct is as follows:\n```\nClassify the text below strictly as either QUERY or NON-QUERY.\nQUERY: Represents a direct question or explicit search intent (e.g., \"What is...\", \"How to...\", \"Find restaurants near...\").\nNON-QUERY: Represents statements, descriptions, continuations, or any non-interrogative content.\n```\n\nUltimately, only 48 outputs show inconsistent classifications between the two methods, yielding an agreement rate of **95.2%**. This suggests that the differences between the two approaches have minimal impact.\n\n\n\n**Q3: \"Beyond QA, can LongMaggie support tasks requiring long-form outputs by modifying the generation protocol?\"**\n\n\n**A3:** The core functionality of LongMagpie lies in its ability to generate diverse, document-adaptive queries. We categorize the generated queries into the following types:\n\n| Category                                | Count   |\n|-----------------------------------------|---------|\n| Precise Retrieval                       | 201,306 |\n| Summarization                           | 91,118  |\n| Advice Seeking                          | 51,609  |\n| Planning or Reasoning (Multi-step Analysis) | 38,526  |\n| Comparative or Choice-Based Task        | 25,342  |\n| Math or Data Analysis                   | 8,679   |\n| Complex Structured Extraction           | 5,725   |\n| Creative Task                           | 3,339   |\n| Coding & Debugging                      | 2,999   |\n\nThese queries support a wide range of downstream tasks, particularly those requiring long-form responses. For example, Planning or Reasoning tasks involving multi-step analysis can be addressed by models such as DeepSeek-R1, while Creative Tasks are well-suited for models optimized for long-form content generation.\n\n\n**Q4: \"Architectural Bias: Synthesized data inherits limitations of source LLMs (Qwen/Llama), risking error propagation.\"**\n\n**A4:** We plan to adopt the following strategies to detect and mitigate biases inherited from the source LLMs:\n\n* **Cross-model verification**: Inspired by the multi-agent debate framework proposed by Google [1], we will generate QA pairs for each document using multiple LLMs, then conduct a debate among them. A separate judge model is used to evaluate the debated outputs and select the most appropriate responses. This process helps reduce the likelihood of propagating biases or errors from any single model.\n    \n* **Quality filtering**: We employ FsfairX-Llama3-RM-v0.1 to score and filter out low-quality generated data. As shown in Figure 2, this filtering step effectively improves the overall quality of the dataset.\n    \n* **Safety filtering**: In addition, we perform safety checks using Llama-Guard-3-8B. Our analysis shows that 99.86% of the data passes the safety criteria. Detailed safety statistics are provided in **A7**.\n\n**Q5: \"Narrow Output Scope: Emphasis on document-query pairs overlooks tasks needing extended reasoning (e.g., multi-step analysis).\"**\n\n**A5:** We categorize the query tasks in **A3**, and our dataset already includes instances that involve extended reasoning (Planning or Reasoning), such as multi-step analysis.\n\n\n**Q6: \"Computational Cost: Heavy reliance on 70B models for high-quality data generation raises accessibility barriers.\"**\n\n**A6:**  Considering the computational cost, we will release our data in the revised version to make it more accessible to the community. We hope this will further promote the development of long-context models in the field.\n\n\n**Q7: \"Ethical Gaps: No discussion of misuse potential (e.g., generating deceptive long-context content) or fairness audits.\"**\n\n**A7:** We performed a safety analysis using the Llama-Guard-3-8B model to classify the generated content. The results are summarized below:\n\n| Category                    | Percentage (%) |\n|-----------------------------|----------------|\n| Safe                        | 99.8603        |\n| Specialized Advice          | 0.1147         |\n| Intellectual Property       | 0.0072         |\n| Non-Violent Crimes          | 0.0063         |\n| Hate                        | 0.0033         |\n| Indiscriminate Weapons      | 0.0028         |\n| Child Sexual Exploitation   | 0.0009         |\n| Violent Crimes              | 0.0009         |\n| Defamation                  | 0.0009         |\n| Elections                   | 0.0002         |\n| Sexual Content              | 0.0007         |\n| Code Interpreter Abuse      | 0.0007         |\n| Privacy                     | 0.0005         |\n| Sex-Related Crimes          | 0.0002         |\n| Suicide & Self-Harm         | 0.0002         |\n\nThanks to the safety alignment of the Qwen model used in generation, the resulting dataset is overwhelmingly safe, with **99.86%** of samples categorized as safe. This suggests that our pipeline can produce high-quality instructional data with minimal safety concerns.\n\n\n---\n\n[1] Du, Yilun, et al. Improving factuality and reasoning in language models through multiagent debate. ICML 2023."}]

Author rebuttal:

Meta-review: This submission proposes a self-synthesis framework LongMagpie for generating large-scale, high-quality long-context instruction data without human annotation or predefined templates. The approach leverages the auto-regressive capabilities of aligned LLMs: when presented with a document and special tokens indicating a user query, the LLM generates contextually relevant queries and responses, producing document-query-response triplets at scale. LongMagpie also introduces a probabilistic mixing strategy p-Mix combining long- and short-context instructions to balance performance across tasks. Experiments on HELMET, RULER, and LongBench-v2 demonstrate that the proposed method achieves sota results on long-context tasks while maintaining competitive short-context performance.

Strengths of the submission: 

1. self-synthesis attempt (iG7h, KGZd): the authors introduces a simple and effective approach that leverages aligned LLMs to generate long-context instructions automatically.
2. simplicity and clarity of approach (vum6): the proposed framework is conceptually straightforward and well-presented, making it easy to adopt.

Weaknesses pointed out by reviewers: 

1. bias inheritance and limited task coverage (iG7h, KGZd): since the proposed method relies on aligned LLMs for data generation, any existing biases or limitations in the base models are likely propagated. in addition, the current focus on document-query-response triplets restricts its applicability to broader long-context tasks.
2. modest improvements and scalability concerns (iG7h, vum6): on some benchmarks, performance gains over strong baselines are marginal, and ablation studies show diminishing returns when scaling up the dataset size. The impact of the p-Mix strategy on balancing long- and short-context performance is also limited. Reviewers also noted that comparisons across datasets and benchmarks could be more transparent, including detailed breakdowns by task type and context length, as well as clarifications around dataset sizes and training configurations.

Overall, the submission presents a technically sound and practical approach to an important problem. While there are valid concerns regarding scalability, bias inheritance, and evaluation clarity, the method’s simplicity, and empirical improvements make it a valuable contribution to the field. During the rebuttal, the authors managed to address most issues, making it with all positive ratings. All factors considered, this submission at its current format is given a positive recommendation.
Decision: Accept (poster)