Title: LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long-context Model

Synthesis Data
Abstract: High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLongMaggie introduces an automated framework for generating large-scale long-context instruction data without human annotation or predefined templates. The core insight is that aligned LLMs (e.g., Qwen, Llama) auto-regressively generate contextually relevant queries when provided with a document followed by special user-role tokens. The pipeline involves: (1) Extracting diverse documents (e.g., from FineWeb-Edu), (2) Generating queries via LLMs, (3) Filtering low-quality queries (rule/length-based), (4) Producing responses to form instruction triplets. The method extends to multi-document settings for complex reasoning and proposes p-Mix—a probabilistic data-mixing strategy—to mitigate short-context performance degradation. Evaluations on HELMET, RULER, and LongBench-v2 show state-of-the-art long-context performance. Models trained solely on LongMaggie data outperform competitors, while *p*-Mix hybrids maintain competitive short-context results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Novelty: Leverages inherent LLM capabilities for query synthesis, eliminating costly human annotation/seed examples.\n\n2. Effectiveness: Achieves SOTA on major long-context benchmarks (HELMET: 62.10, RULER: 91.17).\n\n3. Scalability: Generates 450K high-quality instructions efficiently (1.6K tokens/sample vs. 10-13× for competitors).\n\nWeaknesses:\n\n1. Bias Inheritance: Acknowledges uncorrected biases from source LLMs.\n\nQUESTIONS:\n1. For query generation failure cases (Sec 2.2.1), how frequently do smaller models output non-query text? Could a classifier improve filtering beyond rule/length heuristics?\n\n2. Beyond QA, can LongMaggie support tasks requiring long-form outputs by modifying the generation protocol?\n\nLIMITATIONS:\n1. Architectural Bias: Synthesized data inherits limitations of source LLMs (Qwen/Llama), risking error propagation.\n\n2. Narrow Output Scope: Emphasis on document-query pairs overlooks tasks needing extended reasoning (e.g., multi-step analysis).\n\n3. Computational Cost: Heavy reliance on 70B models for high-quality data generation raises accessibility barriers.\n\n4. Ethical Gaps: No discussion of misuse potential (e.g., generating deceptive long-context content) or fairness audits.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents LongMagpie, a straightforward strategy for sampling long-context instructions from an aligned LLM. This approach extends the original Magpie method, where a language model is given a conditional system prompt along with special tokens and tasked with generating both a user query and its corresponding response. LongMagpie follows a similar setup but introduces documents into the prompt, encouraging the model to produce queries that focus on long-document understanding. The results indicate that the generated instructions effectively capture long-context phenomena, achieving state-of-the-art performance on long-context benchmarks. Furthermore, the authors propose a method for mixing long- and short-context instructions to better balance the model’s performance across both settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well written and easy to follow.\n- The main contribution is simple yet effective and potentially impactful.\n- The authors conduct several ablations and insightful analyses.\n\n### Weaknesses\n- The effectiveness of the second contribution, p-Mix, is unclear. If the goal is to achieve a balanced performance across both long- and short-context tasks, one might consider using the harmonic mean of the ShortAVG and LongAVG scores as a more representative metric. Based on this, the improvement of p-Mix (62.82) over No-Mix (62.46) appears marginal, making it difficult to draw strong conclusions about its impact. While p-Mix clearly shifts the trade-off between long and short performance, it is not evident that it offers a meaningful balance overall.\n- There are some concerns regarding the evaluation framework of the main contribution:\n  - What model was used to generate the instructions for the (short-context) Magpie dataset? If it differs from the model used for LongMagpie, what is the rationale behind that choice? For a fair comparison of the techniques, it would be more robust to keep the model architecture consistent across datasets.\n  - Table 4 highlights that the size of the dataset has a measurable effect on performance. However, the paper does not clearly specify how many data instances are present in the other evaluated datasets. How do their sizes compare to LongMagpie? A clearer and more detailed description of the experimental setup would improve the paper’s transparency and reproducibility.\n\nQUESTIONS:\n- In the data mixing experiments, why was the combination of LongMagpie + Magpie not explored? Since both methods are closely related, it would be interesting to see whether combining them yields better or more balanced performance.\n\nLIMITATIONS:\nWhile the authors acknowledge several limitations, one important aspect is not addressed: the proposed method relies on access to a collection of high-quality documents to generate the long-context instructions. This requirement may restrict the applicability of the approach in low-resource languages or specialized domains where such document collections are scarce or difficult to obtain.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the need for high-quality, large-scale long-context instruction data for LLMs. \nBuilding on the Magpie framework, which generates queries from domain-specific prompts using pretrained LLMs, LongMagpie replaces these prompts with long context documents. This allows aligned LLMs to self-synthesize relevant document-query-response triplets by simply being presented with a documen, an empty user prompt and special tokens indicating start of users' prompt.\n\nThe method extends to multi-document contexts for more complex reasoning and introduces the \"p-Mix\" strategy to blend long and short-context instructions during training, preventing performance degradation on short-context tasks.\n\nExperiments show models trained on LongMagpie data achieve better performance on long-context benchmarks (HELMET, RULER, LongBench-v2) compared with existing short or long-context instruction-tuning dataset, while p-Mix ensures competitive short-context capabilities. The generated data is proven to be of higher diversity with sample efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper effectively tackles the challenge of acquiring large-scale, high-quality long-context instruction data, which has been a major bottleneck for training powerful long-context LLMs.\n2. LongMagpie's core innovation lies in its \"self-synthesis\" approach, which leverages the inherent auto-regressive capabilities of aligned LLMs to generate contextually relevant queries from documents. This method eliminates the need for expensive human annotation or complex, pre-defined templates.\n3. The paper provides evidence that the synthetically generated queries are of greater diversity, which are crucial factors for effective instruction tuning.\n\nWeaknesses\n1. A fundamental limitation is that LongMagpie, by design, inherits biases from the source instruction-tuned LLMs used for data generation. If the base LLM has biases, these will likely be propagated into the synthetic instruction data.\n2. The current method primarily focuses on document-query-response triplets, which might not adequately cover tasks requiring more complex or open-ended outputs. The emphasis is heavily on information extraction and question-answering from documents. Analysis regarding the covered tasks is needed. Additionally, authors can provide concrete examples of the synthesized data so that readers can understand it more clearly. \n3. While overall performance is leading, looking at Table 1, some of the improvements over strong baselines (LongMagpie + UltraChat vs. LongAlign + UltraChat) on benchmarks like RULER and HELMET can appear somewhat marginal (e.g., gains of 1 points). This raises questions about the practical significance of some of the reported advances.\n4. Although the framework is designed to generate large amounts of data, the ablation study on data size shows that the performance improvement (e.g., +1.05 in LongAVG when scaling from 190k to 450k samples) can be rather modest. This raises a concern about diminishing returns when further scaling up the dataset, suggesting that the framework's scalability in terms of performance gain per additional data unit might be limited beyond a certain point.\n5. The paper presents aggregated scores for benchmarks like RULER and HELMET. A more detailed breakdown of performance across different context length ranges and tasks within these benchmarks would be highly beneficial. This would provide a clearer understanding of the method's effectiveness at truly extreme long contexts versus moderately long ones, and its effectiveness across different task types.\n6. The authors state that training involves a total of 1B tokens for both baseline datasets and LongMagpie, ensuring comparable computational cost. However, the number of distinct training instances (samples) for these 1B tokens can vary significantly between datasets. Since SFT performance can be heavily influenced by the diversity and quantity of unique examples, not just total token count, this discrepancy could lead to an unfair comparison, potentially disadvantaging baselines with fewer but longer instances. This aspect could be better clarified or controlled for in the experimental design. For example, LongAlign only contains around 10k instances, which is significanly less than those used in LongMagpie.\n\nQUESTIONS:\n1. While LongMagpie shows leading performance, some gains over strong baselines appear marginal. Also, data scaling yields modest improvements (+1.05 LongAVG from 190k to 450k samples). Could authors elaborate on the practical significance of these gains, perhaps highlighting scenarios or metrics where the advantage is more pronounced? Is it possible to do further scaling to observe more significant gains?\n2. Could the author provide a more granular breakdown of performance for benchmarks like RULER and HELMET across different context length ranges and tasks. This would clarify effectiveness at truly extreme versus moderately long contexts, and the effectivenes across tasks?\n3. Training uses equal total tokens (1B), but distinct sample counts may vary between datasets. Since SFT is sensitive to unique examples, not just token count, could authors clarify this comparison or discuss if a sample-count-controlled comparison was considered?\n4. How do authors plan to detect and mitigate biases inherited from the source LLMs in the generated data? Additionally, can the framework be extended beyond document-query relationships to cover other critical long-context tasks (e.g., summarization, complex structured extraction, multi-turn dialogue)?\n5. Any safely analysis of the generated dataset? It seems important to do filtering when synthesizing data using pretrained LMs.\n\nLIMITATIONS:\nThe authors discussed the limitation adequately.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}