Title: Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long Context Ability; LLM Reasoning
Abstract: Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored. In this work, we hypothesize that current limitations in reasoning stem, in part, from insufficient long-context capacity, motivated by empirical observations such as i) higher context window length often leads to stronger reasoning performance, and ii) failed reasoning cases resemble failed long-context cases. To test this hypothesis, we examine whether enhancing a model’s long-context ability before Supervised Fine-Tuning (SFT) leads to improved reasoning performance. Specifically, we compared models with identical architectures and fine-tuning data but varying levels of long-context capacity. Our results reveal a consistent trend: models with stronger long-context capacity achieve significantly higher accuracy on reasoning benchmarks after SFT. Notably, these gains persist even on tasks with short input lengths, indicating that long-context training offers generalizable benefits for reasoning performance. These findings suggest that long-context modeling is not just essential for processing lengthy inputs, but also serves as a critical foundation for reasoning. We advocate for treating long-context capacity as a first-class objective in the design of future language models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study explores the impact of long-text processing capabilities on the reasoning performance of language models. Through controlled experiments, it was found that models with stronger long-text processing capabilities (such as a 128K context window) showed a significant increase in accuracy in mathematical reasoning benchmark tests like MATH500 and AIME, and this advantage also exists in short-text tasks. Analysis indicates that failures in reasoning cases are closely related to inadequacies in long-text processing, manifested as output repetition and contextual reference errors. The research validates the effectiveness of strategies such as expanding the RoPE theta parameter and model fusion, confirming that \"enhancing long-text processing capabilities before reasoning fine-tuning\" yields the best results.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. The study shows that even for short-input tasks, models with stronger long-text processing capabilities can significantly improve reasoning accuracy, breaking the traditional perception that \"long context is only suitable for long-input tasks\".\n\n2. By comparing models with the same architecture and fine-tuning data but different context lengths, the study effectively eliminates the interference of other factors and ensures that the conclusion focuses on the direct relationship between context length and reasoning performance.\n\n3. The proposed \"enhancing long-text capabilities before reasoning fine-tuning\" method (such as verification based on Qwen2.5-Math-7B-Instruct) provides a clear operational path for model optimization.\n\n# Weaknesses\n\n1. Figures 7 and 8 only mark the RoPE theta scaling coefficient and do not directly show the corresponding actual context lengths, making it difficult for readers to intuitively judge the relationship between training/test data lengths and model capabilities.\n\n2. The study did not deeply analyze \"why long-context training can improve short-input reasoning capabilities\", which makes the relationship between \"long-text capabilities → general reasoning gain\" insufficiently explained.\n\nMinor Suggestion: The experiment only targets 7B-8B parameter models. If the relationship between long-text capabilities and reasoning can be verified on larger models (such as 32B+), the universality of the conclusions will be enhanced.\n\nQUESTIONS:\nTo reinforce your core finding that \"long-context pre-training improves performance on short-context tasks\", it is recommended to add detailed ablation experiments on the following aspects: 1. Cross-domain generalization (not limited to math/code tasks); 2. Model scale sensitivity (such as 7B vs. 13B variants); 3. Robustness of context length configuration\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
This paper studies the relationship between an LLM's long-context capability and its ability to reason. The main idea is that improving long context capabilities can improve downstream reasoning performance. The authors modify existing LLMs to have varying length capabilities using ROPE theta scaling and merging, then fine-tune them on reasoning datasets Models with stronger long-context abilities (measured by NIH) achieve higher accuracy on reasoning tasks. This performance boost is observed even on reasoning problems with short inputs, suggesting that the benefits of long-context improvments are not limited to long test time inputs, but also short.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper is very nicely motivated and written, and the idea of rope scaling is nice
- The experiments are well executed
- The recipe discussion is very useful


Weaknesses
- Only evaluations are on math. What about other benchmarks like GPQA or MMLU?
- Although it's commonly used, I'm not convinced by the needle in a haystack benchmark for long context understanding, since it's primarily measuring retrieval capabilities. It would be nice to see other measures of long context reasoning capabilities. Regardless, the authors show downstream improvements with this proxy. 
- Any conclusions from Table 1 is a bit of a stretch since these are all different models

QUESTIONS:
- Legend in figures 7,9,10 are confusing. Are the line indicators shared across all 3 graphs?
- Why does rope scaling start to decrease performance after ~x16?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers an empirical investigation into an intriguing question: Does improving a model’s long-context ability during pretraining enhance downstream reasoning? Through carefully controlled experiments using RoPE theta scaling and model merging, the authors provide robust evidence linking long-context ability with reasoning skills. The results show that models with stronger long-context capacity consistently achieve higher accuracy on mathematical reasoning benchmarks such as MATH500 and AIME—even when evaluated on short inputs. Building on these findings, the authors propose a training strategy: first, extend the model’s long-context ability before applying reasoning-specific supervised fine-tuning (SFT). They validate this approach across multiple settings, highlighting that long-context modeling serves as a foundational factor in boosting reasoning performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Interesting perspective: such findings can be beneficial to the whole post-training community and shed light on the research about reasoning\n\n(2) Solid experiments: The paper presents a thorough empirical study, evaluating multiple publicly available LLM variants across a range of context window sizes (notably 32k and 128k) on established reasoning benchmarks (MATH500, AIME, GSM8K). \n\n(3) In-depth analysis of failure modes: The inspection of failed generations (Section 2.2, Figure 4) highlights common failure patterns (e.g., repetition, poor reference retention) linked to long-context limitations, providing qualitative depth to the quantitative findings.\n\nWeaknesses:\n\n(1) Overclaim on the conclusion: The research question this paper wants to discuss is about \"Does improving a model’s long-context ability during pretraining enhance downstream reasoning?\". However, it only focuses on math-related reasoning, which is only a very small portion of reasoning tasks. Therefore, the author should modify the claim and maybe limit it to math reasoning. Coding-based reasoning, multi-hop document reasoning, and social reasoning also belong to the scope of reasoning. Therefore, the conclusion is a little bit overclaimed.\n\n(2) Confounding variables not isolated: The specific mechanisms behind why long-context ability improves reasoning (beyond longer input handling) are insufficiently dissected. Such a deep reason lacks in-depth analysis. The correlation between long-context ability and reasoning accuracy (Figures 7, 9) is shown, but the causal pathway is not convincingly isolated from other training effects. A potential in-depth reason for that can be the concentration of attention on the whole context. Such in-depth reasons should be briefly discussed and verified.\n\n(3) Limited discussion on base model: To collect a solid conclusion, it is widely known that different model families like Qwen and LLaMA have different conclusions for reasoning training.[1] Moreover, since it is focusing on the reasoning tasks, Large Reasoning Models (LRM) like Deepseek-R1 and its distilled Qwen version or QwQ should be considered. I expect the conclusions on LRMs and different model families of LLMs would be different. If we can prove that it is a universal conclusion, the paper is much more augmented. We need to confirm that this training recipe is not a trick on specific models.\n\n[1] Shao et al. Spurious Rewards: Rethinking Training Signals in RLVR\n\nQUESTIONS:\nLine 113 and other parts: Lowercase and uppercase should be consistent in the section title.\n\nCan you provide some examples of both the short and long outputs? I’m particularly interested in the cases where the short version exceeds 10,000 tokens. What types of reasoning tasks are being performed, and what are the key differences between the short and long outputs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a study on the effect of improved long context abilities vis-a-vis long output reasoning for large language models (LLMs). The study proposes a hypothesis that improved long context can improve reasoning by demonstrating that accuracy on reasoning specific downstream tasks generally correlates with better (and longer) context processing ability, and that failures for both also follow similar patterns based on manual inspection. Using this as a motivation, the study then tests the said hypothesis by controlling the long context ability of base models, followed by conducting SFT on these base models on reasoning traces. This is demonstrated by results which generally agree with the proposed hypothesis: increasing the long context ability of the base model (via scaling the RoPE theta factor) generally leads to better post-SFT performance on reasoning benchmarks, even on shorter inputs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper is presented in a very clear manner. It is easy to follow, coherent and cogently presented.\n2. The direction is overall quite relevant: an attempt to understand how different LLM capabilities counteract / synergize each other.\n3. It makes intuitive sense that longer context understanding can potentially lead to sustained, coherent reasoning traces. This is an interesting perspective. \n4. I like the fact that both downstream task + long context ability numbers are reported in the results graphs. \n5. The conclusion is crisp w.r.t the recipe for reasoning SFT.\n\nWeaknesses:\n1. The biggest weakness is the difference between the proposed claim and what the hypothesis for this claim sets out to test. Concretely, the paper sets out to claim that longer context understanding can lead to better thinking, but may not be proposing the right hypothesis to test this claim. For example, Table 1 may not be an apples to apples comparison as there are many confounding factors between the left and right columns. Similarly, it makes intuitive sense that longer context will have better probability to get SFTd on longer traces - but what this indicate about the _current base model, as is, Pre SFT?_ \n\n2. It is surprising (and not intuitive) to observe that the output length of models is reducing as the 'long context improvement' is applied, even though the performance is increasing. This seems to indicate that there can be confounding factors in the context extension process, otherwise  one would expect output reasoning traces to also scale with improved input context, with the potential ability to reason more, and reason _consistently_.\n\n3. It's not clear if qualitative observations can tell us anything evidently, and may not be a sufficiently strong argument.\n\nQUESTIONS:\nBased on the weaknesses above and besides those, adding corresponding questions:\n1. Are there any results / intuitions around using these variants of the base models (with diff long context abilities) as init checkpoints, for say, RL? This inherently tests the ability of long reasoning, instead of going through the SFT route. Longer context may lead to better input understanding for SFT, but does it improve the underlying reasoning of the base model as is? Can we do evaluations on pre SFT models? Or, can we run RL on these models and understand if the same correlations between long input and long output hold? \n\n2.Why is the output length decreasing with the long context extension? Is there any qualitative insight?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}