Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
Authors: Van Yang
Zirui Liu
Hongye Jin
Qingyu Yin
Vipin Chaudhary
Xiaotian Han
Venue: NeurIPS 2025 poster ()
Keywords: Long Context Ability; LLM Reasoning
Primary area: deep_learning
TL;DR: uncovering the Role of Long-Context Ability in Reasoning Training

Abstract: Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored. In this work, we hypothesize that current limitations in reasoning stem, in part, from insufficient long-context capacity, motivated by empirical observations such as i) higher context window length often leads to stronger reasoning performance, and ii) failed reasoning cases resemble failed long-context cases. To test this hypothesis, we examine whether enhancing a model’s long-context ability before Supervised Fine-Tuning (SFT) leads to improved reasoning performance. Specifically, we compared models with identical architectures and fine-tuning data but varying levels of long-context capacity. Our results reveal a consistent trend: models with stronger long-context capacity achieve significantly higher accuracy on reasoning benchmarks after SFT. Notably, these gains persist even on tasks with short input lengths, indicating that long-context training offers generalizable benefits for reasoning performance. These findings suggest that long-context modeling is not just essential for processing lengthy inputs, but also serves as a critical foundation for reasoning. We advocate for treating long-context capacity as a first-class objective in the design of future language models.

Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
Wang Yang', Zirui Liu", Hongye Jin Qingyu Yin
Vipin Chaudhary', Xiaotian Han
1 CCsse Western Reserve University 2 University of Minnesota Twin Cities Texas A&M University
{wxy320, vipin, xcaaee edu, arliu@umn.edu, jby04108tamu.ed edu
Abstract
Recent language models exhibit strong reasoning capabilities, yet the influence long-context capacity on reasoning remains underexplored. this work, we hypothesize that current limitations in reasoning stem, in part, from insufficient long-context capacity, motivated by empirical observations such as i) higher con- text window length often leads stronger reasoning performance, and iī) failed reasoning cases resemble failed long-context cases. To test this hypothesis, we examine whether enhancing model's long-context ability before Supervised Fine-Tuning (SFT) leads to improved reasoning performance. Specifically, we compared models with identical architectures and fine-tuning data but varying levels of long-conted capacity. Our results reveal consistent trend: models with stronger long-context capacity achieve significantly higher accuracy on reasoning benchmarks after SFT. Notably, these gains persist even on tasks with short input lengths, indicating that long-context training offers generalizable benefits for rea- soning performance. These findings suggest that long-context modeling is not just essential for processing lengthy inputs, but also serves as a critical foundation for reasoning. We advocate for treating long-context capacity first-class objective in the design of future language models. Our code is anonymously available https: gighub.......... www. com/ussevvn/LLCMMe..
MATH500
AIME
Relation Between Long Context and Reasoning Ability
Accuracy (Poss(1)
89
50
Reasoning Accuracy
Long Context Ability
50
88
40
57
-4
87
88-
58
85
20
32k
12.8 Length
128k Model Length
32k
Model
(9k) x4 (21k) (29k) x16 x32 (21k) x64 (17k)
window length)
Figure 1: Impact of long-context capacity on mathematical reasoning. Left: Accuracy (Pass@) I) on MATH500 and AIME datasets for public models with 32k and 128k context lengths, showing consistent improvements in reasoning performance with longer context windows. The 32k and 128k LLMs refer three different public models. as shown in Table Right: Reasoning accuracy versus RoPE theta values, highlighting strong correlation between long-context capacity and reasoning performance. Increasing the RoPE theta value typically extends the effective context window length.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language models (LLMs) have recently demonstrated impressive reasoning capabilities across wide range of benchmarks [1-3]. Despite this progress, the underlying factors that influence such reasoning abilities remain only partially understood. One particularly underexplored dimension long-context ability-the the model's capacity to utilize longer reasoning path during inference-whin could affect the reasoning performances and the reasoning model fine-tuning.
While prior work has primarily focused on training paradigms and dataset quality to enhance reasoning [4-6], we hypothesize that a model's is reasoning ability is also fundamentally constrained by its Long-contexs capacity. This hypothesis is grounded in three empirical observations. First, models with extended context lengths (e.g.. 128k vs. 32k) consistently achieve higher accuracy on reasoning benchmarks such as MATH500 and AIME, suggesting a direct performance benefit from stronger long-context modeling (Table 1, Figure 1). Second, case studies reveal that failed generations often involve extremely long outputs, sometimes truncated at generation limits, and exhibit issues like repetition or incorrect cross-referencing- failure patterns strongly linked to inadequate long- context capability (Figure 2, Figure 3). Third, modern reasoning datasets now include many samples exceeding 8K or even 10K tokens substanalll longer than early CoT data-requiring models to learn from long, variable-length reasoning sequences (Figure 4). Together, these findings underscore long-context capacity as critical factor for reasoning ability, and motivate the central question: Does improving model's 's long-content ability during pretraining enhance downstream reasoning?
To rigorously investigate this, we conduct a controlled study comparing language models with identical architectures and fine-tuning data, but varying degrees of long-context pretraining. Our experimental results reveal consistent and compelling trend: models with stronger long-context capabilities consistently outperform their counterparts on reasoning tasks after SFT. Notably, these improvements extend to reasoning problems with short input lengths, suggesting that long-context training imparts generalizable cognitive benefits that go beyond simply processing long sequences. As shown in Figure I, BE--lAa----Instteee exhibits varying reasoning performance after training when equipped with different levels of long-contex, capability. Notably, reasoning ability tends to increase or decrease in accordance with the strength of the model's long-context capacity, suggesting direct correlation between the model's 's reasoning ability and long-context capacity.
Based on our experimental results, we thus propose Recipe for Reasoning Fine-Tuning which advocates for appropriately enhancing model's long-context capacity prior to reason- ing SFT for instance, by extending its context length to 128K tokens. Applying this recipe to Qwen2.5 5 5- -nstttruct, we observe substantial improvements: performance on MATH500 increases from an average of 85.04 to 88.70, and on AIME, from 15.00 to 28.00.
2 Motivation: behavioral evidence suggests a connection between long context and reasoning
In this section, we present a set of empirical observations that suggest a strong behavioral connection between model's long-context capability and its reasoning performance. Through controlled comparisons, token length analyses, and failure case studies, we observe that LLMSs with stronger long-contes abilities not only perform better on reasoning benchmarks but also handle diverse and extended reasoning sequences more reliably. These findings collectively highlight long-context modeling key factor and component in enabling strong reasoning ability.
2.1 Higher context window length often leads to stronger reasoning performance.
Recent advances in long-context modeling have enabled language models (LLMs) to process sub- stantially longer sequences. However, it remains unclear whether such long-context capacities yield tangible benefits for reasoning tasks. In this section, we empirically examine the relationship between long-context ability and reasoning performance. We collect a set of well-known open-source reason- ing models fine-tuned from pven/Qwen2. --7B-Instruch These models are categorized into two groups based on their long-context capacity: 32k and 128k tokens. We then evaluate and compare their reasoning performance on two math reasoning benchmarks: MATH500 and AIME.

---

The detailed results are reported in Table 1. Figure I presents that models with longer context lengths (128k vs. 32k) consistently achieve higher accuracy on mathematical reasoning benchmarks such as MATH500 and AIME. This suggests that the ability to encode and maintain longer contextual dependencies can directly translate into better reasoning capabilities. These results collectively highlight the importance of effective long-context training- not only for tasks involving long inputs, but also for general reasoning even when test-time inputs are relatively short.
Table I: Performance comparison on MATH500 and AIME benchmarks for some popular open-source reasoning models with different long-context abilities at 32k and 128k context lengths. Reason- ing models with enhanced long-context capacity (128k) generally exhibit improved performance, particularly on the AIME benchmark. Averages are reported in the bottom row.
Long Context Ability 32k
MATH500
AIME
I Long Context Ability at 128k
MATH500
AIME
OpenRI-Qwent --Qwen-BB
90.36
43.11
SrepSeek-RI-Diseee Seeill--wen-7BB
91.68
45.56
Open' TpennThinke--777
86,80
25.78
CeenMath-Nemotron
94,00
74.67
OpenThinker!
90.60
40.22
DeepMath-ZZeoo-77
83.48
14.67
OpenThiineer3-7B
93.72
64.00
AceReason Nemiston-
93,84
62.89
Avg (32k)
90.37
43.28
Avg (128k)
90.75
49.45
Questions: Alice chooses set A positive integers. Then Bob lists all finite nonemply sets B of positive integers with the property that the maximum element belongs A. Bob's list has 2024 Find the of the elements of
Questions:Thres spheres
mutually
externaly tangent. A plane intersects the sphores three congruent circles cantered at A, and C, respectively, and the centers the sphores all 56, Find AC
same side this plane. Suppose that
Failed Solution: Alright, so have this problem here: Alice chooses a set A of... which has 8 ones.
Failed Solution: Okay, me try to work through this problem step by step. Le... let's let's denote:
k=10, 8=10.
Let's define:
k=11:20355 ones.
Let's let's let's let's ss let's let's let's let's let's let's let's let's let's let's let's let's let's
k=12:2036,8 ones.
k=13:2037,99 ones.
let's let's let's let's is let's let's let's let's
Figure 2: Case Study: Repetition Failure. Two failure cases where the model produces clearly repetitive sentences in answers. Such repetition is common symptom insufficient long-context capability, leading to strange responses and degraded reasoning quality in extended sequences.
The expression 2 4. 1 equal 121, since mult muliplcatiio cartedd out before adsition. However, other than 121 this expression wo are awod change by inserting porentheses. For example, wa can obtain 144 writing ( (3) 4)) (5 - 1.d. In total, how many values can be obcained from the expression 2 3.d+5 1 by inserting parentheses?
Questions:
obtain values
Questions: Let / [K) be an odd function, and let g(x) be even odd, noither?
functioo, ((F(g(((g((()))
Failed Solution: Okay, so have this math problem here. It's about
Failed Solution: Alright, l've got this problem here: ... 2. ([(()) (-x)²
1. All multiplications first, then addition: 121.
3. f(g((()))) f(x*) which is odd.
2. Group 2 (3 (4 (5 + 1)]) + 127.
4. (f((((((()))) g(-x²) (-x*)
3. Group (2 (3 4)) (5 1): 144.
5. ((g(f(((((())) f(x' )
which is odd
4. Group 2 (33 (4 5) + 1): 42.
g(f(g(f(((f(x)))
g(-x'
5. Group 2 (3 4) (5 1)): 144....
Figure 3: Case Study: Contextual Reference Failures. Two failure cases where the model makes incorrect references to expressions introduced earlier the problem. These errors occur in the later stages of the response and reflect typical symptom of insufficient long-context capability.
2.2
Case study: failed reasoning cases resemble failed long-context cases.
We analyze failed cases of reasoning models and long-context inputs, revealing strong con- nection between reasoning ability and long-context capacity. Specifically, we examine the average input lengths of correct and incorrect predictions made by three reasoning mod- els DDeepSeek-R1-Disttlll.
5B, 7B, and 14B-on two math benchmarks: MATH500 and AIME. As shown Figure 4, incorrect predictions are typically associated with inputs exceeding 10k tokens, suggesting that these failures may stem from insufficient long-context handling.

---

Numina
OpenR1 MpenRi-MMtt-220k
DeepMath-103K
6000
80000
3000
Numberd
2000
20000
1000
2000
2000
3000
4000
10000
15000 20000 25000
10000
20000
30000
Token Length
Token Length
Token Length
13000
qwen-1.55
Aug Token Count
19227
20126
12500
10621
10175
Degghs Parces
23000
10303-
20000
7430
7453
B225
312?
3156
5000
2500
1.50
70
1
AIME22-24
MATH500
Figure 4: Top: Length distribution of three reasoning datasets. NuminaMath-Cof represents early chain-of-thought (CoT) data with short sequences, while MMenR1-Math-2205 and DeepMath-103K, generated by DeepSeek-R1, exhibit significantly longer outputs. Bottom-left: Performance of Diaeeee--Dssilled---eeeeee Qwen-1.58 the Needle-in-a-Haasttee
benchmark with 32K context. nottom-middle/rigee Average output lengths of correct and incorrect generations on AIME and MATH500 for Seepseee-DDstlleddee. 1.B,, 7B, and 14B. Incorrect answers consistently ex- hibit longer output lengths, indicating potential limitations of long-context ability in reasoning.
To further investigate this hypothesis, manually inspect subset of long-output failures and identify two recurring patterns, repetition and contextual reference failures, due to limited long- context capability. The first pattern involves excessive repetition, where the model loops the same sentence or phrase, failing to advance the solution, as shown in Figure 2. The second pattern arises in the latter part of the output, where the model incorrectly recalls earlier mathematical expressions from the problem, leading to flawed reasoning and incorrect conclusions, as shown in Figure 3. Both failure modes underscore the model's struggle to maintain coherence and accuracy over long sequences-a well-known limitation of inadequate long-context capacity that degrades and distorts in-context learning performance in LLMs.
2.3 Long and variable reasoning data necessitate long-context models.
Current reasoning models are typically fine-tuned on long chain-of-thought (CoT) datasets gener- ated by large-scale reasoning models. A key characteristic of these models is their tendency to produce long and variable-length outputs. As result, the resulting datasets exhibit broad length distributions, with many examples exceeding 10K tokens-far longer than early CoT-style data, This necessitates that fine-tuned models be capable of handling such long and diverse sequences. In Figure 4, we analyze the length distributions of three representative datasets: NuminaMathhCoo OptnR1-Math- 220KK, and DeepMath- 103K. NuminaMath- CoT, representing early CoT-style data, and ChepMathh10333 contain significantly longer reasoning sequences, with a large
primarily contains samples under 1K tokens. In contrast, OpenR1-Math-22 both collected from DeepSeek- proportion of samples exceeding 4K tokens and some even surpassing 10K tokens, which is totally different with early CoT datasets like NuminaMath-Con
There is a lack of research exploring how such increased sequence lengths interact with or de- pend on the model's long-context capability after the relcase Deepseck-RI. For instance, if model lacks sufficient long-contex, ability, training on long reasoning sequences may fail to yield expected improvements- or even negatively impact performance duneeee-uueee to the model's inability to effectively utilize the full input. While modern models are advertised to handle long sequences supports up to 32K tokens), their effective context length is often
(e.g., Qwen2. 10-1.5B-Instaan substantially shorter. As shown in Figure 4, we evaluate Qwen2.5- 5-1.551. 5B--BIIntte Instruct in-a-Haystack benchmark and observe that the model fails to maintain high accuracy across all cases in 32k contexts, indicating its limitations in effective long-context processing and long-context ability.
the Needle-

---

Table 2: Effective context length and long-context benchmark performance of LLaMA3-B3 Instrucct under different RoPE theta scaling factors. We report the estimated effective context length, Needle- un-a-Haystack (NIAH) retrieval accuracy at 32k, as well as performance on LongBench and RULER. Results show that scaling up to RoPE x 16 consistently improves long-context robustness across all benchmarks, but further scaling (e.g., x32 and x64) leads diminishing or even negative returns. Notably, the effective length surpasses the maximum sequence length (16k) of the current training dataset when the factor exceeds 4, which is sufficient for reasoning training.
RoPE theta
x1
x4
x8
16
x32
64
Effective Context Length
9k
21k
29k
32k
21k
17k
32k NIAH Score
0.00
3.75
58.30
77.05
58.86
35.00
LongBench Score
21.14
39.21
39.78
40.41
38.96
38.01
RULER Score
56.13
69.57
79.62
94.24
88.07
84.98
Llama-3-8 -intntucttt RoPE theta
Llama-3-884 ---Instttttt RoPE
Llama-3-4 Innssrucct
Dopth Percent
2 22 0 22355
Desth Percent
-00
Depeh
-15
-45
Liama-3-80-Inme Innntttctt RoPE theta 16
ll-m------nnstruct RoPE theta laama3--8--nssrusc RoPE theta x
Depth Percent
Desth Parcent
Dooth Percent
Figure 5: Needle-in-a--iee Haystaakk Results for LLLMA-3 -------------- Instruct.. Performance of LLAMA-3-8B- Instruct on the -eedlle-i----Haaytack benchmark with a 32K context under different RoPE theta
scaling factors. ROPE theta x refers scaling the original RoPE theta by factor of 16.
3. Empirical analysis: verifying the connection between reasoning and long-Context ability
In this section, we investigate the connection between a model's long-context capability and its reasoning performance. We begin by applying long-context extension strategies obtain models with varying levels of long-context ability at 32k tokens. These models are then fine- tie-ttnndd on different reasoning datasets using supervised fine-tuning (SFT), allowing us assess how different context capacities influence the effectiveness of reasoning training. Next, we further extend the contecontext capability to 128K tokens or beyond, order to examine whether extreme context lengths can provide additional gains in reasoning performance. Finally, we introduce recipe for LLM reasoning training: extend the long-context ability before reasoning finetuning and conduct an experiment to verify that this recipe is useful and effective. The overall training pipeline is illustrated in Figure 6.
3.1 Experimental setup
Long context ability extension strategy. We use two strategies to enhance long-context capability. The first is directly scaling the RoPE theta parameter by different factors, which has been shown to improve model's ability to handle longer sequences [7]. The second leverages model merging: we merge the target model with another model that possesses stronger long-context capabilities. In this setting, we carefully control the merge ratio to ensure that the base performance remains nearly unchanged, allowing us to isolate the effect of longecontext enhancement as the only influential factor.
dataset [8] and divide it into two categories based on response length: short samples (responses within SK tokens) and long samples (responses ranging from 8K to 16K tokens). For both categories, we sample 20K instances and perform correctness filtering to ensure that each response is factually accurate and correct. These two subsets are then used independently to fine-tune models to improve their reasoning ability.
Data processing for reasoning SFT. We utilize the OpenR1-Mate
5

---

Long Context
Reasoning SFT
Extention
Figure 6: Pipeline for verifying the connection between reasoning and long-contes ability. We first expand the model's long-com context capability to obtain variants with different levels of long-context capacity. Then, we perform SFT on reasoning datasets obtain reasoning-cnhamne models.
Training details. All models fine-tuned using four NVIDIA H200 GPUs. We employ the LLaMAFactory library with a batch size of 32, a learning rate of 1.0 x 10-5 and 3 epochs.
Long context evaluation. We adopt the Needle in Haystack benchmark provided by the OpenCom- pass framework to access long context ability. For simplicity and robustness, we use the aceuracy on the single-haystack setting as primary metric: correct response receives a score of +1, reperitive/degeneraae answer receives 1, and an incorrect but non-degenerate response receives O.
Reasoning evaluation. To further evaluate the model's reasoning ability post- estt-training, we use three math benchmarks: MATH500, AIME22- 24, and GSM8K. Following the evaluation methodology from DeepSeek-R1 R1, we adopt the pass @ 1(5) metric, where five responses are generated for each question and accuracy computed over all the responses. This provides more stable estimate of reasoning performance and abilities of Large Language Models after finetuning with datasets.
Long Context
Accurbcy (80
50.0
5
55.5
20
Long
52
Buit Cheta (curdaed Winclam tength)
DUFE theta (surcacc nindom langth))
(boft: Urela [ADIrincC
(a) MATH500
(b) AIME22-24
(c) GSM8K
Figure 7: Visualization of the relationship between 32k long-context ability and reasoning perfor-
mance across different model variants of LLaMA-3 3. 1-88---aatt -n-nrutttt (MATH500, AIME22-24, GSM8K). Short and Long refer to performance after fine-tuning on short and long reasoning datasets. Avg represents the average of the short and long tuned results.
three math benchmarks
Short
Context
Avg Daguutenng Lenghh
13000
27500
12000
60
25000
63
3002
Long
17500
x23
X32
tenght)
length
lengst)
(a) MATH500
(b) AIME22-24
(c) GSM8K
Figure Visualization of the relationship between 32k long-context ability and average output length across different model variants of LLaMA-3, 1-8B- Instruct on MATH500, AIME22-24 24, GSM8K.
3.2 How does long-context capacity affect reasoning SFT?
In this experiment, investigate how different levels of long-context capability affect the perfor- mance of reasoning SFT (Supervised Finetuing). We use LLaMA3 Instruct as the base model, which originally supports up to 8K tokens. To obtain models with varying long-context capabilities, we scale the RoPE theta value by the following different factors: 41 8, 16, 32, 64).
First, we evaluate the long-context ability of each variant using the Needle in-a---naataedd benchmark. As shown in Figure and Table 2, performance improves as the RoPE theta scaling factor increases from I to 16, reaching peak performance at 16. Beyond this point, further increases in theta lead to performance degradation. What's more, when the scaling theta is more than 4, the model's 's effective length is more than 16k, which is enough for the reasoning training for current datasets.

---

Table Effect of RoPE theta scaling on long-context and reasoning performance on MATH500. Evaluation of ne-a-Haystack benchmark. Base refers to the model's performance before SFT, while Short and Long denote results after fine-tuning on short and long reasoning datasets, respectively. Avg represents the average of the Short and Long performances.
with different RoPE theta scaling factors on the 32K Needle-
RoPE
32k long
Acc(%)
Average Output Length
theta
ctx ability
Base
Short
Long
Avg
Base
Short
Long
Avg
x1
0
24.40
50.68
58.92
54.80
746
12991
11187
12089
x4
3.75
20.40
52.80
61.16
56.98
754
9047
9949
9498
x8
58.30
18.96
53.12
62.24
57.68
767
8476
9673
9075
16
77.05
17.28
54.40
64.32
59.36
608
8782
9335
9059
x32
58.86
15.24
53.84
61.96
57.90
490
8489
9526
9007
x64
35.00
14.20
53.28
62.36
57.82
453
8605
9579
9092
Table 4: Cross-domain evaluation long-context ability on GPQA (science) and Livecode (code).
Task
Setting
GPQA Accuracy (%)
Livecode Accuracy (%)
x1
x4
x x16
x1
x4
x x16
32k NIAH Accuracy
0.00
3.75
77.05
0.00
3.75
77.05
before training
31.82
31.31
30.81
12.30
10.16
7.45
after training
37.27
39.19
41.92
25.34
27.95
32.27
Next, perform supervised fine-tuning (SFT) using both short and long reasoning datasets models with different RoPE theta settings. The detailed results are summarized in Table 3. In addition, we plot the relationship between each model's long-context capability and its reasoning performance. We also visualize how long-context ability correlates with the average output length. The results are shown in Figures 7 and 8. We observe that the best reasoning performance across all three evaluation benchmarks consistently occurs when the RoPE theta scaling factor is set to 16 coineidig with the strongest long-context capability observed in the Needle-in-a cnelll-in----lee Hay-stac task. In contrast, models with suboptimal theta settings exhibit reduced accuracy undergoing the same SFT procedure. These findings suggest that enhancing long-context processing ability can directly contribute to improved reasoning performance after reasoning SFT (Supervised Fine-tuning).
Furthermore, we observe that, in general, models fine-tuned long reasoning datasets outperform those fine-tuned on short datasets in both MATH500 and AIME. These long-form datasets typically contain more complex problems and richer intermediate reasoning steps, making them more effective for improving the model's reasoning ability. However, in order to benefit from such data, models must possess sufficiently strong and effective long-context capabilities process the extended inputs.
3.3 Generality and robustness of long-context gains to reasoning ability
We investigate whether the benefits of stronger long-context ability generalize across tasks, model families, and input lengths. Our results show that the improvements are not limited to math reasoning.
Cross-domain generalization. We fine-tuned LLaMA3-8B- -nnttrutt models with RoPE scales x1, x4, and x 16 on 20k samples from the science and code domains OpenThoughts3-11 2M. Evaluations on GPQA and Livecode (Table 4) show that stronger long-context ability consistently improves accuracy beyond math reasoning.
Across model families. We trained Phi-4 4 (14B) with the same RoPE settings. Consistent gains on MATH500 (Table 5) confirm that the benefit is not tied to a specific architecture scale. The results of AIME22-24 are in Table 10 and it has the similar performance like MATH500.
Short-input reasoning. On the short-input MMLU-STEM benchmark, models with stronger long- context ability also achieve higher post-training accuracy (Table 6). This indicates that long-context training not only preserves but can reinforce reasoning on short-input tasks.
7

---

Table 5: Performance of Phi under different RoPE scales on MATH500.
Model
RoPE
32k NIAH (%)
MATH500 (before, %)
MATH500 (after, %)
Phi-4
x1
52.27
79.52
88.62
Phi-4
x4
78.07
77.48
89.14
Phi-4
x16
84.77
73.20
89.90
Table 6: Effect of long-context ability on short-input reasoning (MMLU-STEM) with LLaMA3-8B.
RoPE Scale
32k NIAH (%)
MMLU-STEM (before, %)
MMLU-STEM (after, %)
x1
0.00
54.36
71.06
x4
3.75
53.85
73.04
x16
77.05
51.44
74.27
Overall, these results demonstrate that the gains from stronger long-context ability are robust: they extend beyond math to science and code, hold across LLaMA, Qwen, and Phi model families, and even benefit short-input reasoning.
3.4 Does extremely long context bring further gains on reasoning SFT?
In previous experiments, extended models to handle up 32K tokens and observed notable improvements in reasoning performance. A natural question arises: can even stronger long-context capabilities further enhance reasoning, is there a limit beyond which performance saturates or even degrades? To explore this. we conduct experiments using models with context length of IM tokens Brren2.5-7B-Isstruce 1M) far beyond the typical range of existing reasoning datasets.
We adopt a linear merging strategy to construct models with varying long-context capacities while minimizing changes to their base ability. Specifically, we merge two models with different context capabilities at various ratios to obtain models with intermediate long-context strengths. With selected merge ratio, we ensure that the base capabilities remain largely unchanged, isolating long-context capacity as the key variable. We apply this strategy to two models: Qwen2.5-7B- 1-225-5---nnstttt-lll with long context ability IM and Qwen2.5-7BBIInsrruu with long context ability at 32k.
We first evaluate the merged models on Needle-ire Bonedle-i---Haystere aama--Haytackk at 32k and find that long- long-context ability grows with the merge ratio, but near- perfect scores make it less discriminative (Appendix A). We therefore adopt more challenging 32k tasks such as Value Tracking and Question Answering, which better capture effective long-context processing by requiring stronger long-range reasoning.
We fine-tune the merged models on both short and long reasoning datasets, and evaluate on MATH500, AIME22-24 24, and GSMBK. We also examine the relationship between effective long-come long-context ability, reasoning accuracy, and output length (Figures and 10). Results show that moderate merge ratios (e.g.. 0.1, 0.7) yield strong effective long-context ability and high reasoning accuracy, while the 1M model (ratio 1 1.0) exhibits weaker effective long-context utilization and degraded performance.
3.5 Proposed reasoning recipe: extend context length first
Based our previous experiments, we propose a training recipe for improving reasoning capabilities via supervised fine-tur tuning :SFT):fire appropriately enhance the model's 's long-content capability, then apply reasoning- -pecrigigppppficccc SFT. This recipe aims to better prepare the model for long-form reasoning tasks, where both context length and reasoning complexity are critical. To validate this approach, we experiment with the Qwen2. 5-Math-777 Instruct model, which demonstrates strong mathematical performances but has limited context length of 4k tokens, which is a good example the recipe.
We enhance its long-con contecontext capability by multiplying the RoPE theta value by scaling factor of 16 and merging the model with Quen2. T7B-Instruct-11 using a merge ratio of 0.3. After the merging step. the long-context ability improves noticeably, as is shown in Appendix A. Subsequently, we perform SFT using both short and long reasoning datasets, and evaluate the fine-tuned models on two benchmarks: MATH500 and AIME22-24 -24, As shown in Table 7, across both short and long

---

Avg
Long
Sharl
32k Long Context Ability
Accurscy (ha)
18
6.7
1.5
Ropf thete (ranteen: isttdiow
Roff thete (norred window
Rodi thete (conkest windon
(a) MATH500
(b) AIME22-24
(c) GSM8K
Figure 9: Visualization of the relationship between 32k long-contens ability and reasoning per- formance across different model variants of Qwen2. 7B- IIB-Instrcct on MATH500, AIME22-24, GSM8K. A merge ratio of 0.1 indicates that the long-context variant (Qwen2. --B-IInstrctt 1M) contributes I 10% to the final merged model. Short and Long refer to performance after fine-tuning on short and long reasoning datasets. Avg represents the average of the short and long fine-tuned results.
Short
32k Context
6500- Tirre
20000
302.0
2530
5002
19000
2030
1500
0.0
0.7
thate (contExt window longth)
RoPE thota (consext window longon)
(contoxt longth)
(a) MATH500
(b) AIME22-24
(c) GSM8K
Figure 10: The relationship between 32k long-context ability and average output length across different model variants of Qwen2. 5- Instruct on three math benchmarks.
reasoning datasets, the model variant with RoPE theta scaled by 16 and a merge ratio of 0.3 consistently achieves the highest performance after reasoning SFT on short and long datasets.
4 Related Works
Reasoning models. The release of DeepSeek-RI R1 [9] has catalyzed widespread interest in open- source reasoning models [10-12]. Most existing approaches improving reasoning rely su- pervised fine-tuning (SFT) and rule-based reinforcement learning (RL) [13-16]. For instance, SkyThought [17] and Bespoke [18] fine-tune models using responses generated by larger reason- ing models such as QwQ-32B [19] and DeepSeek-R1. Other methods like S1 [20] and LIMO [21] highlight the importance of using compact yet high-quality reasoning samples to improve data effi- ciency. These efforts have led to the creation of numerous open-source reasoning datasets, including -aatR-Math-aa0
[8], DeepMath- 103K [4]. OpenThoughts [5], and others [6, 22, 23]. Notable models trained on these datasets include
and DeepMath-Zenn -er---ro--7 7B. Subsequent work [24, 25] has proposed techniques to further optimize SFT and RL pipelines, arguing that struc- tural aspects of reasoning data (e.g., step-wise decomposition) can be more impactful than content alone. More recent studies [3, 26- 26-28] aim improve the alignment and generalization of reasoning through better data construction, gradient control, and training optimization strategies. Despite these advances, an important factor remains underexplored: the role of long- cong-cotext capability in reasoning. While recent datasets and tasks increasingly involve longer sequences, existing work systematically investigates how a model's s ability to process extended contexts affects reasoning performance. Our work aims to figure out the relationship between them.
Long-context ability. Closed-source models such as GPT-4 [29], Claude [30], and Gemini [31] already support context lengths of 128K tokens or more, enabled by ad- vances in both pre-training and post-training [32-35].
In parallel, open-source mod- els iss-includigg LLaMA [36], Qwen [37, 38), and Phi [39]- also extended their ca- pacities, with some reaching 1M tokens, such as 2.e7-225-7B-Insrree LLaMA-3. 1.-MA-3.1-Nemma Nemotron-BB Wwweerrnnn---taattr-ee -anaLoon----Instruc [41]. To enable long context, several ap- proaches have been proposed. RoPE-based methods adapt positional encoding for extrapolation without full retraining, including Position Interpolation [42], NTK Scaling [7]. YaRN [43], and Selffi- tend [44]. Attention redesigns improve scalability or memory retention, such as StreamingI.LM [45], LM-Infinite [46], Inf-LLM [47], and Landmark Attention [48]. Another direction is input compres-
[40] and

---

Table 7: Validation the proposed fine-tuning recipe for reasoning. We evaluate the effectiveness of our proposed training recipe using Qwen2. -----ath------- 1ean--BBInsttae 103-Intruct, model with an initial long- context capacity limited 4K tokens. We first scale RoPE theta by a factor of 16, and then merge it with Qwen2. .ere2..5-BBBae (nrtB--nnstuutt111 using a merge ratio of 0.3. We observe consistent improvement in reasoning performance after each step, demonstrating the effectiveness of enhancing long-context capability prior to reasoning fine-tuning to get higher reasoning ability.
Acc(%)
Avg Output Length
Operation
Base
Short
Long
Avg
Base
Short
Long
Avg
MATH500
RoPE theta x1
81.88
86.28
83.80
85.04
2037
1022
2147
1584
RoPE theta x 16
65.16
87.68
88.72
88.20 20
4579
4213
5789
5501
0.7 x RoPE theta x16 + 0.3 IM-Model
74.12
88.28
89.12
88.70
2228
4948
6515
5731
AIMH22-24
RoPE theta x1
8.44
16.22
13.78
15.00
8257
3463
10418
6940
RoPE theta x 16
3.78
25.78
27.56
26.67
13411
15036
17321
16179
0.7 RoPE theta x16 + 0.3 x IM-Model
7.11
26.67
29.33
28.00
6859
20265
22919
21592
90-
Accurscy
Accurstcy
87
Roft: ax16
our Recipe
RoFE 8x36
Our reope
ROPE Bx36
Recip
(a) MATH500 (Short)
(b) MATH500 (Long)
(c) MATH500 (Avg)
26
8
20
22
17
14
14
HoFt ext
ROPE bx16
our zo0
Roft: ext
ROPE 8636
RG
RoFt ex
ROPE bx36
our ROO
(d) AIME22-24 (Short)
(c) AIME22-24 (Long)
(f) AIME22-24 (Avg)
Figure 11: Comparison of reasoning accuracy across different configurations. Results are reported on MATH500 (top row) and AIME22-24 (bottom row), with separate evaluation for short, long, and average context cases. Our proposed recipe (dark blue bar, dashed line) consistently outperforms the baseline RoPE theta x and the extended-context baseline RoPE theta x16 demonstrating that enhancing long-context capability prior to reasoning fine-tuning yields stable and significant accuracy improvements, enhancing models' reasoning ability.
sion, which reduces sequence length by summarization or filtering [49, 50]. Finally, recent work explores hardware-alignes designs, including sparse attention architectures [51] and MoBA [52].
Discussion and Conclusions
Our analysis is limited to 7B-8B models and does not cover larger scales. We also focus on supervised fine-tuning, leaving open how reinforcement learning interacts with long-context ability. Future work should examine these directions across model families and scales.
Limitations:
This paper investigates the overlooked yet crucial role of long-context capability in reasoning performance. Through behavioral analysis and controlled experiments, we demonstrate a consistent correlation between long-conter and downstream reasoning ability. Our results show that models with improved long-context capacity not only perform better on tasks involving lengthy inputs but also achieve higher reasoning accuracy even on short-form benchmarks. Furthermore, we find that enhancing long-context ability prior to supervised fine-tuning yields significant gains across multiple reasoning datasets. Based on these findings, we advocate for a training recipe: first, extend the model's 's long-context ability, then apply reasoning-spedde speciffcc fine-tuning.
10

---

Acknowledgments
This work was supported in part by NSF grants 21 12606 and 2117439. Further, this research made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing Case Western Reserve University (CWRU). We give special thanks the CWRU HPC team for their prompt and professional help and maintenance.
References
[1] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprini arXiv:2501112948 2025.
[2] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12553 2025.
[3] Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmaan, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, and Guoqing Zheng. Phi-4-reasoning technical report, 2025.
[4] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang. Wenxuan Wang. Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-1003. A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025.
[5] OpenThoughts Team. Open Thoughts. http://open-thoughts January 2025.
[6] Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state state-of-the oh-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.1689), 2025.
[7] Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023.
[8] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025.
[9] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and Ruoyu Zhang... Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
[10] Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm mastermind: A survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.012300 2024.
[11] Aske Plaat, Annie Wong. Suzan Verberne, Joost Brockens, Niki van Stein, and Thomas Back. Reasoning with large language models, survey. arXiv preprint arXiv:2407.15511 2024.
[12] Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, al. survey of frontiers in 1lm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint vrXiv:254400003 2025.
[13] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[14] Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint 244.12404.18922. 2024.

---

[15] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[16] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wen-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025.
[17] NovaSky Team. Sky-tl: Train your own ol preview model within S450. htts://novasky-
w.giihubbioo/pott/skyyy.. 2025. Accessed: 2025-01-09.
[18] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distilla-
tion. www.bespokelabs..o distillation, 2025. Accessed: 2025-01-22.
[19] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[20] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025.
[21] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025.
[22] Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Jash Mehta, and Toby Liang. Millions scale dataset distilled from r1-32b. http://wggnnniii..ceddtetsssrrrr SFT, 2025.
[23] Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabhah, Ran EI- Baniv,,v, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, and Ehud Karpas.. Llama-nemotron: Efficient reasoning models, 2025.
[24] Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo. Eric Tang. Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Llms can easily learn reason from demonstrations structure, not content, is what matters!, 2025.
[25] Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijic Wang, Junying Chen, Benyou Wang, et al. The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models. arXiv preprint arXiv:2503.0222 02875. 2025.
[26] Jiarui Yao. Yifan Hao, Hanning Zhang, Hanze Dong, Wei Xiong, Nan Jiang, and Tong Zhang, Optimizing tfhhouh reasoners via gradient variance minimization in rejection sampling and rl, 2025.
[27] Tianwei Ni, Allen Nie, Sapana Chaudhary, Yao Liu, Huzefa Rangwala, and Rasool Fakoor. Teaching large language models to reason through learning and forgetting, 2025.
[28] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild, 2025.
[29] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[30] Loredana Caruccio, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sundaramurthy, and Genoveffa Tortora. Claude 2.0 large language model: Tackling woal-world classification problem with a new iterative prompt engineering approach. Intelligent Systems with Applications, 21:200336, 2024.
12

---

[31] Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnel,, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
[32] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with ro-awareness. Advances in Neural Information Processing Systems, 15516444--1359, 2022.
[33] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 16039, 2023.
[34] Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song. Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for Ilm training. arXiv preprint arXiv:2410.10900 2024.
[35] Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence paral- lelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13/20, 2021.
[36] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, and Angela Fan... The llama 3 herd of models, 2024.
[37] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXi::2412.15112 2024.
[38] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, and Dayiheng Liu ... Qwen3 technical report, 2025.
[39] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024.
[40] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1mg technical report, 2025.
[41] Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, and Bryan Catanzaro. From 128k to 4m: Efficient training of ultra-long context large language models. arXiv preprint, 2025.
[42] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint 2rxiv:2306.15599 2023.
[43] Bowen Peng. Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00077 2023.
[44] Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang. Zirui Liu, Chia- Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend I1m context window without tuning, 2024.
[45] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv 2023.
13

---

[46] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 1991-4008, 2024.
[47] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training- free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
[48] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305. 16300, 2023.
[49] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yew Lin, Yuging Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839, 2023.
[50] Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.0620, 2023.
[51] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025.
[52] Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang. Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang. Zhilin Yang. Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13/ 13189, 2025.
[53] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey efficient reasoning for large language models, 2025.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: yes, we include the paper's contributions and scope in the abstract and intro- duction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
we add the limitations to the conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: include the full set of assumptions and a complete (and correct) proof Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: we include a section about the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
we provide open access to the data and code
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ /uwddee/CoeeSubmiee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: iooodeeummsssionn....
/ /nipss //ipsscccppubllli http:ccccppppppccccceee/cocpccce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: include section about the experimental setup Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: we include a stable metric to get results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:: we provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:::///eurip.... cc/ CawbbltcEEhicsuuic...
Answer: [Yes]
Justification: the research conducted in the paper conform with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: the paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: we do not involve LLMs to impact the methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https : htt////eeripppppee http///rpsscc//ooeeeeeeeeeeeeeee cc////oneeeeeccce//2/////11 for what should or should not be described.
21

---

A Other Results on Different Long-Context Ability
This section provides additional experimental results that complement the main findings by examining how different levels of long-context ability influence model performance across multiple reasoning benchmarks. While the main paper focused on reasoning accuracy trends, here we provide a broader view including auxiliary benchmarks, output length analysis, and merged-model experiments.
Effect of RoPE scaling. We first analyze the impact of varying RoPE 0 scaling factors LLaMA3-8B- Instruct. -.-MMA--B--Insruut..
As shown Table 8, models with stronger long-context ability gener- ally achieve higher reasoning accuracy across MATH500, AIME22 24, and GSM8K. The trend is most consistent up to RoPE 16, after which performance begins to plateau or slightly decline, suggesting diminishing returns when scaling beyond the effective context length of the training data. We also observe that output length tends to shrink as accuracy improves, consistent with our analysis in Section 3.2, where correct solutions are typically shorter and more concise.
Table 8: Effect of RoPE 0 Scaling on Long-Context and Reasoning Performance. Evaluation of LLLMM3-8B-- -Isttrut with different RoPE 0 scaling factors on the 32K Veedle-in-a-Haystacc benchmark. Base refers to the model's performance before SFT, while Short and Long denote results after fine-tuning on short and long reasoning datasets, respectively. Avg represents the average of the Short and Long performances. Length represents the average output length.
RoPE
32k long
Acc(%)
Length
theta
ctx ability
Base
Short
Long
Avg
Base
Short
Long
Avg
MATH500
X1
24.40
50.68
58.92
54.80
746
12991
11187
12089
X4
3.75
20.40
52.80
61.16
56.98
754
9047
9949
9498
X8
58.30
18.96
53.12
62.24
57.68
767
8476
9673
9075
x x16
77.05
17.28
54.40
64.32
59.36
608
8782
9335
9059
x32
58.86
15.24
53.84
61.96
57.90
490
8489
9526
9007
x64
35.00
14.20
53.28
62.36
57.82
453
8605
9579
9092
AIME22-24
X1
0.22
2.67
3.11
2.89
1895
27482
26706
27094
X4
3.75
0.67
3.78
5.11
4.45
2048
19961
20684
20322
X8
58.30
0.22
2.89
6.89
4.89
1723
17532
17547
17539
X x1
77.05
0.22
4.67
6.22
5,45
1617
18339
18015
18177
X32
58.86
0.00
3.56
7.11
5.34
1372
17876
18071
17973
X64
35.00
0.44
3.11
4.67
3.89
1108
17830
18251
18041
GSM8K
X 1
0
78.13
84.81
84.64
84.73
195
2244
4769
3506
X 4
3.75
75.54
85.64
85.88
85.76
190
1973
4010
2992
x8
58.30
73.03
85.90
85.79
85.85
184
1981
3812
2897
x16
77.05
70.39
85.64
86.38
86.01
182
2019
3936
2977
x32
58.86
66.35
85.41
86.14
85.78
180
2026
4179
3103
x64
35.00
62.21
85.56
85.25
85.41
182
2083
4177
3130
Extremely long context via model merging. To test whether extremely long contexts (e.g., 128k or 1M) provide further gains, we construct merged variants of Qwen2. 5-7B-Instruct by combining its base 32k model with the ultra-long version Qwen2.5. 5-7B- 111--Instuuct-11 1M. By adjusting merge ratios, we obtain intermediate models with controllable long-context capabilities, while preserving base reasoning ability. As shown in Figure 9, reasoning performance correlates with effective long-context strength: models with moderate merge ratios (e.g., 0.1, 0.7) achieve consistently strong accuracy, whereas the pure IM model shows weaker effective long-context utilization and degraded reasoning.
22

---

B Other Results on Different Merging Ratios
We further analyze how different merging ratios between base and ultra-long variants affect both retrieval ability and downstream reasoning. This provides insight into whether extremely long contexts (c.g.. 1M tokens) always yield benefits, whether moderate ratios strike better balance between long-context integration and base reasoning stability.
Table 9: Reasoning Performance After SFT with Different Merge Ratios. Qwen results are based on merging Qwen2. 5-2 Ies-Innsruuttt11 with Qwen2. 17BB-Instruct A ratio of 0.1 means that the long-context (1M) variant contributes 10% to the merged model.
1M Merge
32k long
Acc(%)
Length
Ratio
ctx ability
Base
Short
Long
Avg
Base
Short
Long
Avg
MATH500
0
78.1
75.00
83.48
84.84
84.16
638
5572
6545
6058
0.1
79.1
74.64
83.56
86.28
84.92
670
5625
6432
6028
0.7
79.5
72.80
83.88
84.88
84.38
630
5812
6563
6187
1.0
77.7
72.16
82.28
83.48
82.88
688
5897
6881
6389
AIME22-24
0
78.1
8.22
17.78
23.33
20.56
1051
19642
18592
19117
0.1
79.1
9.33
19.11
22.89
21.00
1313
19473
18178
18825
0.7
79.5
7.78
19.56
23.56
21.56
1416
20396
18142
19269
1.0
77.7
7.33
16.22
20.00
18.11
1750
20507
18680
19594
GSM8K
78.1
90.75
93.19
91.54
92.37
259
1429
3264
2347
0.1
79.1
90.75
93.15
92.16
92.66
252
1472
3197
2334
0.7
79.5
89.1
93.07
92.27
92.67
254
1488
2930
2209
1.0
77.7
89.13
92.84
92.34
92.59
254
1494
2915
2204
Heedle-in-a-Haystee (retrieval ability). We first evaluate the merged models on the 32K Needle- in-a-Haysen Hayytaay benchmark. As shown Figure 12, long-context ability generally increases with higher contribution from the IM-token variant. However, because most models achieve near cer--eerfect scores, the benchmark does not fully differentiate their effective long-context strength. This motivates evaluating more challenging reasoning datasets where differences manifest more clearly.
Reasoning benchmarks. We then fine-tune the merged models both short and long reasoning
datasets and evaluate them on three benchmarks: MATH500, AIME22-24, and GSM8K. Table 9 reports accuracy and output length. Three key observations emerge: (1) Moderate merge ratios (e.g., 0.1
or 0.7) yield the best overall reasoning performance, consistently outperforming both the base (0) and fully merged (1.0) variants. (2) Pure 1M models (ratio 1.0) exhibit degraded reasoning accuracy, despite higher nominal long-context capacity, suggesting weaker effective utilization. (3) Output lengths tend shrink reasoning improves, consistent with the observation that successful reasoning requires fewer redundant tokens.
C Additional Analysis on Output Length and Context Extension
Why is the output length decreasing with the long context extension? The observed decrease output length actually aligned with the ratio correct answers. Correct generations tend to be much shorter and more concise than incorrect ones. With long-context extension, the number of correct answers increases, thus reducing the overall average output length. This phenomenon has also been discussed in prior work [53].
23

---

qwen2-7b-insttruttooo---1
qwenz-77-inntruutt-0.1k
Depth Percent
Percent
21
31-
52
0.0
0.9
-0.5
-65
-1.0
100
-1.
176000
22n00
25000
23r899
31.00
32000
130000
17h009
21000
25000
20000
31.0d
32000
gwen2-7b-Instruce Ser.uucttt..-.lnmm
qwen2-7b-Insrruct 1.00
Depth Percent
05
23
31
40
00
50
20
63
1300
17800
21088
20000
28680
32,00
32000
13000
1768
21000
25n00
31,80
32000
Figure 12: 32K Needle-in- -ee--in-------eee
HayHHayttt stack evaluation for different merge ratios. Models with higher
long-context contribution achieve stronger retrieval ability, but performance differences saturate quickly.
Table 10: Performance of Phi -4 under different RoPE scales on AIME.
Model
RoPE
32k NIAH (%)
AIME (before, %)
AIME (after, %)
Phi-4
x1
52.27
14.89
47.56
Phi-4
x4
78.07
13.78
49.45
Phi-4
x16
84.77
10.22
50.17
To support this explanation, we report the average lengths of correct and incorrect generations across both short (0-8k) and long (8-16k) training settings for LLaMA3-8B under different RoPE configurations: RoPE x1 (8k context) and RoPE x x16 (Extended context)
Table 11: Effect of training length on output length under different RoPE scales.
RoPE Scale
Setting Accuracy (%)
Avg Length
# Correct
Avg
Wrong
Avg
x1
short long 58.92
50.68
12991
1267
3189
1233
23145
x1
11187
1473
5928
1027
20999
x16
short 54.40
8782
1360
3906
1140
14921
x16
long
64.32
9335
1608
6063
892
17265
As shown, models with longer context capacity (e.g., RoPE x x16) achieve higher accuracy while generating shorter outputs on average pprimarll because a larger portion of the outputs are correct, and correct answers are generally shorter.
24


Reviews:
[{"review_id": "xpUO9BBegZ", "reviewer": "Reviewer_EVh5", "summary": "This paper presents a study on the effect of improved long context abilities vis-a-vis long output reasoning for large language models (LLMs). The study proposes a hypothesis that improved long context can improve reasoning by demonstrating that accuracy on reasoning specific downstream tasks generally correlates with better (and longer) context processing ability, and that failures for both also follow similar patterns based on manual inspection. Using this as a motivation, the study then tests the said hypothesis by controlling the long context ability of base models, followed by conducting SFT on these base models on reasoning traces. This is demonstrated by results which generally agree with the proposed hypothesis: increasing the long context ability of the base model (via scaling the RoPE theta factor) generally leads to better post-SFT performance on reasoning benchmarks, even on shorter inputs.", "questions": "Based on the weaknesses above and besides those, adding corresponding questions:\n1. Are there any results / intuitions around using these variants of the base models (with diff long context abilities) as init checkpoints, for say, RL? This inherently tests the ability of long reasoning, instead of going through the SFT route. Longer context may lead to better input understanding for SFT, but does it improve the underlying reasoning of the base model as is? Can we do evaluations on pre SFT models? Or, can we run RL on these models and understand if the same correlations between long input and long output hold? \n\n2.Why is the output length decreasing with the long context extension? Is there any qualitative insight?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is presented in a very clear manner. It is easy to follow, coherent and cogently presented.\n2. The direction is overall quite relevant: an attempt to understand how different LLM capabilities counteract / synergize each other.\n3. It makes intuitive sense that longer context understanding can potentially lead to sustained, coherent reasoning traces. This is an interesting perspective. \n4. I like the fact that both downstream task + long context ability numbers are reported in the results graphs. \n5. The conclusion is crisp w.r.t the recipe for reasoning SFT.\n\nWeaknesses:\n1. The biggest weakness is the difference between the proposed claim and what the hypothesis for this claim sets out to test. Concretely, the paper sets out to claim that longer context understanding can lead to better thinking, but may not be proposing the right hypothesis to test this claim. For example, Table 1 may not be an apples to apples comparison as there are many confounding factors between the left and right columns. Similarly, it makes intuitive sense that longer context will have better probability to get SFTd on longer traces - but what this indicate about the _current base model, as is, Pre SFT?_ \n\n2. It is surprising (and not intuitive) to observe that the output length of models is reducing as the 'long context improvement' is applied, even though the performance is increasing. This seems to indicate that there can be confounding factors in the context extension process, otherwise  one would expect output reasoning traces to also scale with improved input context, with the potential ability to reason more, and reason _consistently_.\n\n3. It's not clear if qualitative observations can tell us anything evidently, and may not be a sufficiently strong argument.", "was_revised": true, "final_justification": "Based on the responses, I maintain my accept score.", "rebuttal": "Dear Reviewer EVh5,\n\nThank you for your support! Below, we provide our responses to your questions in detail.\n\n---\n## Question 1: Table 1 may not be an example to apply comparisons\nThank you for the insightful feedback. Table 1 is intended to illustrate that widely-used LLMs with stronger long-context capabilities tend to perform better on reasoning tasks. Since it is infeasible to control fine-tuning settings—such as the SFT dataset, we instead use a diverse set of popular open-source reasoning models that adopt the same base model (Qwen/Qwen2.5-7B-Instruct) but claim different context lengths, serving as representative examples to support it.\n\nTo further support this illustration, we have added several new models—spanning different organizations, training pipelines, and context lengths—into the comparison. The results below show consistent patterns across both 32k and 128k context settings:\n\n| Model                                   | Claimed Context Length | MATH500 (%) | AIME (%) |\n|----------------------------------------|----------------|-------------|----------|\n| OpenR1-Qwen-7B                         | 32k            | 90.36       | 43.11    |\n| OpenThinker-7B                         | 32k            | 86.80       | 25.78    |\n| OpenThinker2-7B                        | 32k            | 90.60       | 40.22    |\n| OpenThinker3-7B          | 32k            | 93.72       | 64.00    |\n| Average                            | 32k        | 90.37   | 43.28|\n\n| Model                                   | Claimed Context Length | MATH500 (%) | AIME (%) |\n|----------------------------------------|----------------|-------------|----------|\n| DeepSeek-R1-Distill-Qwen-7B           | 128k           | 91.68       | 45.56    |\n| OpenMath-Nemotron-7B                   | 128k           | 94.00       | 74.67    |\n| DeepMath-Zero-7B                       | 128k           | 83.48       | 14.67    |\n| AceReason-Nemotron-7B           | 128k           | 93.84       | 62.89    |\n| Average                            | 128k       | 90.75   | 49.45|\n\n| Claimed Context Length | MATH500 (%) | AIME (%) |\n|----------------|-------------|----------|\n| 32k        | 90.37   | 43.28|\n| 128k       | 90.75   | 49.45|\n\nAs shown above, models with stronger long-context abilities outperform those with shorter context in both math benchmarks.  We emphasize that Table 1 serves as an empirical motivation. Our main evidence is derived from controlled experiments where only the RoPE scaling is varied (see Table 3)\n\n---\n\n## Question 2: Can we do evaluations on pre SFT models? \n\nThanks for this insightful suggestion. In our submission, We have evaluated models before SFT. In Table 3, we have already conducted controlled experiments where the only changing factor is RoPE scaling. These results show that base model accuracy before SFT decreases as RoPE scaling increases, while accuracy after SFT increases with context capability—further supporting our claim (stronger long-context ability consistently leads to better reasoning performance).\n\nHere we show the partial results from Table 3. We could see the long-context extension method does not improve model ability before training.\n\n| RoPE Scale | 32k NIAH Score | Accuracy Before Training | Accuracy After Training |\n|------------|----------------|---------------------------|--------------------------|\n| ×1         | 0.00           | 24.40                     | 54.80                    |\n| ×4         | 3.75           | 20.40                     | 56.98                    |\n| ×8         | 58.30          | 18.96                     | 57.68                    |\n| ×16        | 77.05          | 17.28                     | 59.36                    |\n| ×32        | 58.86          | 15.24                     | 57.90                    |\n| ×64        | 35.00          | 14.20                     | 57.82                    |\n\n---\n## Question 3: Can we run RL on these models?\n\nThank you for raising this insightful question. To explore whether long-context variants of base models can support stronger reasoning in RL, we attempted reinforcement learning (RL) training on different configurations.\n\nWe used the Qwen2.5-7B-Instruct model and a merged variant (0.9 × Qwen2.5-7B-Instruct + 0.1 × Qwen2.5-7B-Instruct-1M) as initialization. For the RL algorithm, we adopted Dr.GRPO as our training strategy. Due to resource and time constraints, we conducted RL training on a small dataset of 5k math samples. \n\nAfter training, the average output length of the model remained below 2k tokens throughout. The final accuracy on MATH500 reached 74.72 and 74.80, respectively, for the two model variants. These results suggest that our current RL setup—with limited data and small-scale tuning—was insufficient to unlock complex strong reasoning and the relation. We consider this an exciting direction for future work and will include this discussion in the final paper.\n\n---\n\n## Question 4: Why is the output length decreasing with the long context extension?\nThank you for the question. The observed decrease in output length is actually aligned with ratio of correct answer. With the long context extension, the number of correct answer (which is shorter tipically) increases, thus the average output length decreases, which also been discussed in prior work [1].\nTo support this with evidence, we show the average lengths of correct and incorrect generations across both short (0–8k) and long (8–16k) training settings for LLaMA3-8B under different RoPE configurations:\n\n- RoPE ×1 (8k context)\n\n| Training Length | Accuracy | Avg Length | # Correct | Avg (Correct) | # Wrong | Avg (Wrong) |\n|--------------|----------|------------|-----------|----------------|----------|--------------|\n| short         | 0.5068   | 12991.38   | 1267      | 3189.36         | 1233     | 23145.07      |\n| long        | 0.5892   | 11187.43   | 1473      | 5927.55         | 1027     | 20999.25      |\n\n- RoPE ×16 (Extended context)\n\n| Training Length | Accuracy | Avg Length | # Correct | Avg (Correct) | # Wrong | Avg (Wrong) |\n|--------------|----------|------------|-----------|----------------|----------|--------------|\n| short        | 0.5440   | 8782.49    | 1360      | 3905.56         | 1140     | 14921.46      |\n| long       | 0.6432   | 9335.26    | 1608      | 6063.03         | 892      | 17265.13      |\n\nAs shown, models with longer context capacity (e.g., RoPE ×16) achieve higher accuracy while generating shorter outputs on average—primarily because a larger portion of the generations are correct, and correct answers tend to be shorter and more concise.\n\n[1]Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs"}, {"review_id": "3XkpQSZZTD", "reviewer": "Reviewer_k9kH", "summary": "This paper offers an empirical investigation into an intriguing question: Does improving a model’s long-context ability during pretraining enhance downstream reasoning? Through carefully controlled experiments using RoPE theta scaling and model merging, the authors provide robust evidence linking long-context ability with reasoning skills. The results show that models with stronger long-context capacity consistently achieve higher accuracy on mathematical reasoning benchmarks such as MATH500 and AIME—even when evaluated on short inputs. Building on these findings, the authors propose a training strategy: first, extend the model’s long-context ability before applying reasoning-specific supervised fine-tuning (SFT). They validate this approach across multiple settings, highlighting that long-context modeling serves as a foundational factor in boosting reasoning performance.", "questions": "Line 113 and other parts: Lowercase and uppercase should be consistent in the section title.\n\nCan you provide some examples of both the short and long outputs? I’m particularly interested in the cases where the short version exceeds 10,000 tokens. What types of reasoning tasks are being performed, and what are the key differences between the short and long outputs?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Interesting perspective: such findings can be beneficial to the whole post-training community and shed light on the research about reasoning\n\n(2) Solid experiments: The paper presents a thorough empirical study, evaluating multiple publicly available LLM variants across a range of context window sizes (notably 32k and 128k) on established reasoning benchmarks (MATH500, AIME, GSM8K). \n\n(3) In-depth analysis of failure modes: The inspection of failed generations (Section 2.2, Figure 4) highlights common failure patterns (e.g., repetition, poor reference retention) linked to long-context limitations, providing qualitative depth to the quantitative findings.\n\nWeaknesses:\n\n(1) Overclaim on the conclusion: The research question this paper wants to discuss is about \"Does improving a model’s long-context ability during pretraining enhance downstream reasoning?\". However, it only focuses on math-related reasoning, which is only a very small portion of reasoning tasks. Therefore, the author should modify the claim and maybe limit it to math reasoning. Coding-based reasoning, multi-hop document reasoning, and social reasoning also belong to the scope of reasoning. Therefore, the conclusion is a little bit overclaimed.\n\n(2) Confounding variables not isolated: The specific mechanisms behind why long-context ability improves reasoning (beyond longer input handling) are insufficiently dissected. Such a deep reason lacks in-depth analysis. The correlation between long-context ability and reasoning accuracy (Figures 7, 9) is shown, but the causal pathway is not convincingly isolated from other training effects. A potential in-depth reason for that can be the concentration of attention on the whole context. Such in-depth reasons should be briefly discussed and verified.\n\n(3) Limited discussion on base model: To collect a solid conclusion, it is widely known that different model families like Qwen and LLaMA have different conclusions for reasoning training.[1] Moreover, since it is focusing on the reasoning tasks, Large Reasoning Models (LRM) like Deepseek-R1 and its distilled Qwen version or QwQ should be considered. I expect the conclusions on LRMs and different model families of LLMs would be different. If we can prove that it is a universal conclusion, the paper is much more augmented. We need to confirm that this training recipe is not a trick on specific models.\n\n[1] Shao et al. Spurious Rewards: Rethinking Training Signals in RLVR", "was_revised": true, "final_justification": "The experiments on GPQA and coding make such reasoning not limited to math, making the whole claim more solid and more convincing.", "rebuttal": "Dear Reviewer k9kH,\n\nThanks for your valuable questions! Below, we address your concerns point by point.\n\n---\n## Question 1: The author should modify the claim and maybe limit it to math reasoning\n\nThank you for the insightful comment. Based on our additional experiments, our claim (stronger long-context ability consistently leads to better reasoning performance) holds on other reasoning tasks beyond math, including science and code reasoning.\n\nTo verify generality, we fine-tuned three Llama3-8B-Instruct models with different long-context capabilities (RoPE scale ×1, ×4, ×16) on 20k samples from the science and code domains of the open-thoughts/OpenThoughts3-1.2M dataset. The models were then evaluated on GPQA (science) and Livecode (code). Evaluation results show that stronger long-context ability consistently leads to better reasoning performance across these domains.\n\n| Task               |            | RoPE ×1 | RoPE ×4 | RoPE ×16 |\n|--------------------|------------------|---------|---------|----------|\n| 32k NIAH Score    | -                | 0       | 3.75    | 77.05    |\n| GPQA              | before training  | 0.3182  | 0.3131  | 0.3081   |\n| GPQA              | after training   | 0.3727  | 0.3919  | 0.4192   |\n| Livecode              | before training  | 0.1230  | 0.1016  | 0.0745   |\n| Livecode              | after training   | 0.2534  | 0.2795  | 0.3227   |\n\n---\n\n## Question 2: Confounding variables not isolated. In-depth reasons should be briefly discussed and verified\n\nThank you for the insightful comment. Controlling confounding variables—so that long-context ability is the only varying factor—has been a central goal of our experimental design. \n\nTo this end, we carefully evaluate model performance before and after extending the context window. As shown in Table, LLaMA3-8B-Instruct models with longer context window do not improve performance on reasoning tasks before SFT; in fact, performance often decreases. This indicates that mere window extension does not yield a reasoning boost. However, once the models are fine-tuned using the same datasets and hyperparameters, their reasoning abilities improve significantly in correlation with long-context capacity.\n\n| RoPE Scale | 32k NIAH Score |  Accuracy Before Training| Accuracy After Training|\n|------------|--------------------|--------------------|------------------|\n| ×1         | 0.00               | 24.40              | 54.80            |\n| ×4         | 3.75               | 20.40              | 56.98            |\n| ×8         | 58.30              | 18.96              | 57.68            |\n| ×16        | 77.05              | 17.28              | 59.36            |\n| ×32        | 58.86              | 15.24              | 57.90            |\n| ×64        | 35.00              | 14.20              | 57.82            |\n\nWe agree that exploring deeper mechanisms is important. One plausible hypothesis—that longer context ability improves attention concentration across the whole context—is indeed worth investigating. We will include this point in the discussion section and highlight it as a promising direction for future work.\n\n---\n## Question 3: Limited discussion on base model\n\nThank you for raising this important point. Based on our additional experiments as follows, our claim (stronger long-context ability consistently leads to better reasoning performance) holds across different model families, including LLaMA, Qwen, and Phi.\n\nIn the main paper, we already evaluated LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct, covering two widely used model series. To further verify the generality of our findings, we additionally trained Phi-4 models with varying long-context abilities (RoPE scale ×1, ×4, ×16) and evaluated them on two math reasoning benchmarks: Math500 and AIME.\n\nThese results further confirm that the benefit of enhancing long-context ability is not tied to a specific model architecture, but generalizes across different model families. We will clarify this in the revised version.\n\n| Model  | RoPE Scale |32k NIAH Score |Math500 before training | Math500 after training | AIME before training | AIME after training |\n|--------|------|------|---------------|-------------|-----------|----------|\n| Phi-4  | ×1  |  52.27     | 0.7952        | 0.8862      | 0.1489    | 0.4756   |\n| Phi-4  | ×4  |   78.07    | 0.7748        | 0.8914      | 0.1378    | 0.4945   |\n| Phi-4  | ×16 |  84.77     | 0.7320        | 0.8990      | 0.1022    | 0.5017   |\n\n---\n## Question 4: Can you provide some examples of both the short and long outputs exceeds 10,000 tokens?\n\nThank you for the question.  To provide further insight, we count how many correct and incorrect generations exceed 10,000 tokens under both short and long training with RoPE theta x 16 on MATH500: (due to limited word number of retbuttal, since the responses are too long, we can not paste responses here but we will include them in the updated version of our paper.) \n\n| Output Type | Short (# >10k) | Long (# >10k) |\n|-------------|----------------|----------------|\n| Correct     | 115            | 281            |\n| Incorrect   | 573            | 63             |\n\nWe observe that models trained under the long-data setting generate fewer wrong ultra-long outputs than models trained under the short-data setting. Our hypothesis is that long-data training helps the model learn how to reason more efficiently, thus avoiding unnecessarily verbose or incorrect completions to generate wrong answers.\n\n---\n## Question 5: Line 113 and other parts: Lowercase and uppercase should be consistent in the section title.\nThank you for pointing this out. We will revise the section titles to ensure consistent capitalization throughout the paper in the final verison."}, {"review_id": "dUAIuzYBMG", "reviewer": "Reviewer_EuNs", "summary": "This paper studies the relationship between an LLM's long-context capability and its ability to reason. The main idea is that improving long context capabilities can improve downstream reasoning performance. The authors modify existing LLMs to have varying length capabilities using ROPE theta scaling and merging, then fine-tune them on reasoning datasets Models with stronger long-context abilities (measured by NIH) achieve higher accuracy on reasoning tasks. This performance boost is observed even on reasoning problems with short inputs, suggesting that the benefits of long-context improvments are not limited to long test time inputs, but also short.", "questions": "- Legend in figures 7,9,10 are confusing. Are the line indicators shared across all 3 graphs?\n- Why does rope scaling start to decrease performance after ~x16?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper is very nicely motivated and written, and the idea of rope scaling is nice\n- The experiments are well executed\n- The recipe discussion is very useful\n\n\nWeaknesses\n- Only evaluations are on math. What about other benchmarks like GPQA or MMLU?\n- Although it's commonly used, I'm not convinced by the needle in a haystack benchmark for long context understanding, since it's primarily measuring retrieval capabilities. It would be nice to see other measures of long context reasoning capabilities. Regardless, the authors show downstream improvements with this proxy. \n- Any conclusions from Table 1 is a bit of a stretch since these are all different models", "was_revised": true, "final_justification": "The authors answered my comments and concerns sufficiently. I will maintain my accept score.", "rebuttal": "Dear Reviewer EuNs,\n\nThank you for your support! Below, we address your concerns point by point.\n\n---\n## Question 1: What about other benchmarks like GPQA or MMLU?\n\nThank you for the suggestion. We provide additional evaluation results on other benchmarks, including GPQA and MMLU. These results demonstrate that the performance improvement associated with stronger long-context ability is not limited to math reasoning, but also extends to benchmarks such as MMLU and GPQA.\n\nWe first evaluate LLaMA3-8B-Instruct models with different long-context capabilities (RoPE ×1, ×4, ×16) on the MMLU-STEM subset after fine-tuning on the Math dataset. The MMLU-STEM subset is used instead of the full MMLU for two reasons: (1) it focuses on STEM-related reasoning, which better aligns with our goal; (2) it avoids excessive evaluation time, as full MMLU includes over 10k examples.\n\nIn addition, we collect 20k samples from the science domain of the open-thoughts/OpenThoughts3-1.2M dataset and fine-tune the models, then evaluate them on GPQA. The results are summarized below:\n\n\n| Task       | Accuracy          | RoPE ×1 | RoPE ×4 | RoPE ×16 |\n|------------|------------------|---------|---------|----------|\n| 32k NIAH  Score  | -                | 0       | 3.75    | 77.05    |\n| MMLU  | before training    | 0.5436  | 0.5385  | 0.5144   |\n|            | after training    | 0.7106  | 0.7304  | 0.7427   |\n| GPQA       | before training    | 0.3182  | 0.3131  | 0.3081   |\n|            | after training   | 0.3727  | 0.3919  | 0.4192   |\t\t\t\t\t\t\t\t\n\nThese results demonstrate that the performance improvement associated with stronger long-context ability is not limited to math reasoning, but also extends to benchmarks such as MMLU and GPQA.\n\n---\n## Question 2: It would be nice to see other measures of long context reasoning capabilities.\n\nThank you for the valuable feedback. To address this concern, we further evaluate the LLaMA3-8B-Instruct model with six levels of 32k long-context capabilities (RoPE ×1 to ×64) on both Needle-in-a-Haystack (NIAH) and alternative long-context benchmarks, including LongBench and partial RULER tasks，which contain Multi Query, Value Tracking and Question Answering , better reflecting long-context reasoning beyond retrieval.\n\nThe results are summarized below:\n\n| RoPE Scale | 32k NIAH Score | LongBench Score | RULER Score |\n|------------|--------------------|------------------|--------------|\n| ×1         | 0.00               | 21.14            | 56.13        |\n| ×4         | 3.75               | 39.21            | 69.57        |\n| ×8         | 58.30              | 39.78            | 79.62        |\n| ×16        | 77.05              | 40.41            | 94.24        |\n| ×32        | 58.86              | 38.96            | 88.07        |\n| ×64        | 35.00              | 38.01            | 84.98        |\n\n\nWe observe a clear upward trend from RoPE ×1 to ×16, after which the performance begins to decline. These results indicate that the long-context capabilities associated with different RoPE scales are generally consistent across NIAH, LongBench, and RULER, thereby supporting our claim that stronger long-context ability consistently leads to better reasoning performance. \n\n---\n## Question 3: Any conclusions from Table 1 is a bit of a stretch since these are all different models\n\nThank you for the insightful feedback. Table 1 is intended to illustrate that widely-used LLMs with stronger long-context capabilities tend to perform better on reasoning tasks. Since it is infeasible to control fine-tuning settings—such as the SFT dataset, we instead use a diverse set of popular open-source reasoning models that adopt the same base model (Qwen/Qwen2.5-7B-Instruct) but claim different context lengths, serving as representative examples to support it.\n\nTo strengthen this illustration, we have added several new models spanning multiple organizations, training pipelines, and context lengths. The updated results are presented below:\n\n| Model                                   | Claimed Context Length | MATH500 (%) | AIME (%) |\n|----------------------------------------|----------------|-------------|----------|\n| OpenR1-Qwen-7B                         | 32k            | 90.36       | 43.11    |\n| OpenThinker-7B                         | 32k            | 86.80       | 25.78    |\n| OpenThinker2-7B                        | 32k            | 90.60       | 40.22    |\n| OpenThinker3-7B          | 32k            | 93.72       | 64.00    |\n| Average                            | 32k        | 90.37   | 43.28|\n\n| Model                                   | Claimed Context Length | MATH500 (%) | AIME (%) |\n|----------------------------------------|----------------|-------------|----------|\n| DeepSeek-R1-Distill-Qwen-7B           | 128k           | 91.68       | 45.56    |\n| OpenMath-Nemotron-7B                   | 128k           | 94.00       | 74.67    |\n| DeepMath-Zero-7B                       | 128k           | 83.48       | 14.67    |\n| AceReason-Nemotron-7B           | 128k           | 93.84       | 62.89    |\n| Average                            | 128k       | 90.75   | 49.45|\n\n| Claimed Context Length | MATH500 (%) | AIME (%) |\n|----------------|-------------|----------|\n| 32k        | 90.37   | 43.28|\n| 128k       | 90.75   | 49.45|\n\nAs shown above, models with longer context window tend to achieve better average performance on both Math500 and AIME. We emphasize that Table 1 serves as an empirical motivation. Our main evidence is derived from controlled experiments where only the RoPE scaling is varied (see Table 3).\n\n---\n## Question 4: Legend in figures 7,9,10 are confusing. Are the line indicators shared across all 3 graphs?\n\nYes, the line indicators are shared across Figures 7, 9, and 10 for consistency and ease of comparison. We will clarify this in the figure captions to avoid confusion.\n\n---\n## Question 5: Why does rope scaling start to decrease performance after ~x16?\n\nThank you for the question. While scaling the RoPE theta helps extend the model’s usable context window, this technique has its limits. Beyond a certain point, further increasing the scaling coefficient leads to degraded performance, as shown by the previous work[1].\n\nIn our experiments with LLaMA3-8B-Instruct, we found that RoPE ×16 achieves the best long-context capability, as reflected in both reasoning accuracy and NIAH scores. We speculate one of reasons is: when the scaling goes beyond ×16 (e.g., ×32 or ×64), the model’s internal attention dynamics begin to misalign with training-time inductive biases, leading to worse overall performance despite larger nominal context windows.\n\n\n[1] NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation"}, {"review_id": "IFQWOzfBKc", "reviewer": "Reviewer_Bec7", "summary": "This study explores the impact of long-text processing capabilities on the reasoning performance of language models. Through controlled experiments, it was found that models with stronger long-text processing capabilities (such as a 128K context window) showed a significant increase in accuracy in mathematical reasoning benchmark tests like MATH500 and AIME, and this advantage also exists in short-text tasks. Analysis indicates that failures in reasoning cases are closely related to inadequacies in long-text processing, manifested as output repetition and contextual reference errors. The research validates the effectiveness of strategies such as expanding the RoPE theta parameter and model fusion, confirming that \"enhancing long-text processing capabilities before reasoning fine-tuning\" yields the best results.", "questions": "To reinforce your core finding that \"long-context pre-training improves performance on short-context tasks\", it is recommended to add detailed ablation experiments on the following aspects: 1. Cross-domain generalization (not limited to math/code tasks); 2. Model scale sensitivity (such as 7B vs. 13B variants); 3. Robustness of context length configuration", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. The study shows that even for short-input tasks, models with stronger long-text processing capabilities can significantly improve reasoning accuracy, breaking the traditional perception that \"long context is only suitable for long-input tasks\".\n\n2. By comparing models with the same architecture and fine-tuning data but different context lengths, the study effectively eliminates the interference of other factors and ensures that the conclusion focuses on the direct relationship between context length and reasoning performance.\n\n3. The proposed \"enhancing long-text capabilities before reasoning fine-tuning\" method (such as verification based on Qwen2.5-Math-7B-Instruct) provides a clear operational path for model optimization.\n\n# Weaknesses\n\n1. Figures 7 and 8 only mark the RoPE theta scaling coefficient and do not directly show the corresponding actual context lengths, making it difficult for readers to intuitively judge the relationship between training/test data lengths and model capabilities.\n\n2. The study did not deeply analyze \"why long-context training can improve short-input reasoning capabilities\", which makes the relationship between \"long-text capabilities → general reasoning gain\" insufficiently explained.\n\nMinor Suggestion: The experiment only targets 7B-8B parameter models. If the relationship between long-text capabilities and reasoning can be verified on larger models (such as 32B+), the universality of the conclusions will be enhanced.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Bec7,\n\nThanks for you support! Below, we provide our responses to your questions in detail.\n\n---\n## Question 1: Need more details of the RoPE theta scaling coefficient and its corresponding context lengths\n\nThank you for the suggestion. We will update the x-axis labels in Figures 7 and 8 to include the corresponding effective context lengths for each RoPE configuration. This will make it easier for readers to intuitively understand the relationship between model settings and their actual long-context capabilities.\n\nHere are the x-axis labels we will update:\n\n|  |||||||\n| -------- | -------- | -------- |-------- |-------- |-------- |-------- |\n| Original x-axis labels  | x1 | x4 |x8 |x16 |x32 |x64 |\n| Revised x-axis labels     |  x1(9k) | x4(21k) |x8(29k) |x16(32k) |x32(21k) |x64(17k) |\n\n---\n## Question 2: Why long-context training can improve short-input reasoning capabilities\n\nThank you for pointing this out. We conducted the below experiment to support that enhancing long-context capability does not harm—and can even benefit—short-input reasoning tasks, which aligns with prior work such as YARN and Position Interpolation (PI).\n\nWe evaluate LLaMA3-8B-Instruct models with different long-context capabilities (RoPE ×1, ×4, ×16) on the MMLU-STEM subset, which consists entirely of short-input, STEM-related reasoning questions. All models were fine-tuned on the same Math dataset.\n\n| Task       | Accuracy          | RoPE ×1 | RoPE ×4 | RoPE ×16 |\n|------------|------------------|---------|---------|----------|\n| 32k NIAH  Score  | -                | 0       | 3.75    | 77.05    |\n| MMLU | before training    | 0.5436  | 0.5385  | 0.5144   |\n|            | after training     | 0.7106  | 0.7304  | 0.7427   |\n\nThe results show that stronger long-context capability correlates with improved performance even on short-input reasoning tasks. This suggests that long-context training is not merely an overhead for short tasks, but can reinforce reasoning in general.\n\n---\n## Question 3: More ablation study on cross-domain generalization, model scale sensitivity and context length configuration\n\nThank you for the thoughtful suggestions. We have conducted additional experiments to address each of the three aspects raised:\n\n1. Cross-domain generalization (long-context ability benefits not only math resoning but also non-math reasoning): We fine-tuned LLaMA3-8B-Instruct models (RoPE ×1, ×4, ×16) on 20k samples from the *science* and *code* domains of the open-thoughts/OpenThoughts3-1.2M dataset. The results show that long-context ability also benefits non-math reasoning tasks.\n\n| Task     |            | RoPE ×1 | RoPE ×4 | RoPE ×16 |\n|----------|------------------|---------|---------|----------|\n| 32k NIAH  Score  | -                | 0       | 3.75    | 77.05    |\n| GPQA     | before training  | 0.3182  | 0.3131  | 0.3081   |\n|          | after training   | 0.3727  | 0.3919  | 0.4192   |\n| Livecode     | before training  | 0.1230  | 0.1016  | 0.0745   |\n|          | after training   | 0.2534  | 0.2795  | 0.3227   |\n\n2. Model scale sensitivity (the effect of long-context ability on reasoning ability generalizes across model scales and architectures): We additionally evaluated the 14B-scale Phi-4 model, using RoPE ×1, ×4, and ×16 configurations. After fine-tuning on Math data, we measured performance on MATH500 and AIME. The consistent trends validate that the effect of long-context ability generalizes across model scales and architectures.\n\n| Model  | RoPE Scale |32k NIAH Score |Math500 before training | Math500 after training | AIME before training | AIME after training |\n|--------|------|------|---------------|-------------|-----------|----------|\n| Phi-4  | ×1  |  52.27     | 0.7952        | 0.8862      | 0.1489    | 0.4756   |\n| Phi-4  | ×4  |   78.07    | 0.7748        | 0.8914      | 0.1378    | 0.4945   |\n| Phi-4  | ×16 |  84.77     | 0.7320        | 0.8990      | 0.1022    | 0.5017   |\n\n3. Robustness of context length configuration (more evaluation benchmarks on long-context ability of models): To test the robustness of long-context configuration, we evaluated LLaMA3-8B-Instruct under six RoPE scaling levels (×1 to ×64) on multiple long-context benchmarks: Needle-in-a-Haystack (NIAH), LongBench, and RULER. We observe a clear upward trend on these benchmarks from RoPE ×1 to ×16, after which the performance begins to decline.\n\n| RoPE Scale | 32k NIAH Score | LongBench Score | RULER Score |\n|------------|--------------------|------------------|--------------|\n| ×1         | 0.00               | 21.14            | 56.13        |\n| ×4         | 3.75               | 39.21            | 69.57        |\n| ×8         | 58.30              | 39.78            | 79.62        |\n| ×16        | 77.05              | 40.41            | 94.24        |\n| ×32        | 58.86              | 38.96            | 88.07        |\n| ×64        | 35.00              | 38.01            | 84.98        |\n\nThese ablations support the robustness, scalability, and cross-domain generalization of our central claim."}]

Author rebuttal:

Meta-review: The paper hypothesizes that the reasoning ability of an LLM may also be affected by its long-context capabilities. The paper verifies this, demonstrating how failure modes in the former resembles modes in the latter,  and that even on inputs that are short, reasoning benefits from improvements in long-context. 

The hypothesis asked here is arguably a conceptually simple one, yet it is an important one to test. The reviewers find the paper clearly-written and well-motivated. The experiments are well-designed. A major concern had been that the analysis was limited to math reasoning tasks, but the authors responded with coding (Livecode) and science (GPQA) tasks which the reviewers are satisfied with.
Decision: Accept (poster)