R-Horizon: How Far Can Your Large Reasoning Model Really Go in Breadth and Depth?

## Abstract

Abstract Recent trends in test-time scaling for reasoning models (e.g., OpenAI o1, DeepSeek-R1) have led to remarkable improvements through long Chain-of-Thought (CoT). However, existing benchmarks mainly focus on immediate, single-horizon tasks, failing to adequately evaluate models’ ability to understand and respond to complex, long-horizon scenarios. To address this incomplete evaluation of Large Reasoning Models (LRMs), we propose R-Horizon , a method designed to stimulate long-horizon reasoning behaviors in LRMs through query composition. Based on R-Horizon , we construct a long-horizon reasoning Benchmark, comprising complex multi-step reasoning tasks with interdependent problems that span long reasoning horizons. Through comprehensive evaluation of LRMs using the R-Horizon Benchmark, we find that even the most advanced LRMs suffer significant performance degradation. Our analysis reveals that LRMs exhibit limited effective reasoning length and struggle to allocate thinking budget across multiple problems appropriately. Recognizing these limitations, we use R-Horizon to construct long-horizon reasoning data for reinforcement learning with verified rewards (RLVR). Compared to training with single-horizon data, RLVR with R-Horizon not only substantially improves performance on the multi-horizon reasoning tasks, but also promotes accuracy on standard reasoning tasks (+7.5 on AIME2024).
These results position R-Horizon as a scalable, controllable, and low-cost paradigm for enhancing and evaluating the long-horizon reasoning capabilities of LRMs.

## 1 Introduction

Recent advances in reasoning-focused language models, exemplified by OpenAI’s o1 (OpenAI et al., 2024) and DeepSeek-R1 (Guo et al., 2025), have established test-time scaling as a fundamental component for enhancing reasoning abilities in large reasoning models (LRMs). Specifically, test-time scaling enables long Chain-of-Thought (CoT) and induces sophisticated reasoning behaviors, leading to remarkable improvements on challenging reasoning tasks like mathematical reasoning (He et al., 2025b; Yu et al., 2025; Yue et al., 2025; Zeng et al., 2025), code generation (Luo et al., 2025a; Zeng et al., 2025) and agentic tasks (Team et al., 2025b, a).

By continuously expending computational resources throughout the reasoning process, models with longer reasoning trajectories achieve superior performance on various reasoning benchmarks (Muennighoff et al., 2025), yet this simultaneously exposes critical limitations in current training and evaluation paradigms.
Existing training and evaluation datasets (Cobbe et al., 2021; Hendrycks et al., 2021; Jain et al., 2024) primarily confine themselves to the reasoning of isolated problems, focusing on immediate single-horizon tasks where questions and answers remain independent of each other.
However, real-world scenarios often require an AI agent to reason, plan, and act over an extended series of steps, sometimes thousands or even millions, where inference must span across multiple sequential and potentially interdependent problems (Yao et al., 2024; Tao et al., 2025). Single-horizon evaluation paradigms cannot effectively assess the ability of a model to understand and respond to complex, multi-horizon tasks or scenarios that require a sequence of logical steps over a longer period of time.
Moreover, conventional reinforcement learning (RL) typically focuses on single, isolated problems, preventing models from developing long-horizon reasoning capabilities to tackle multiple problems through the RL process.
The incomplete picture of training and evaluation paradigms raises a fundamental question: How far can large reasoning models really go in breadth and depth?

In this study, we propose R-Horizon, a simple yet effective method to stimulate long-horizon reasoning behaviors in LRMs through query composition.
This method aims to construct dependencies and concatenate existing single-horizon tasks, transforming isolated problems into complex multi-horizon reasoning scenarios.
For instance, in mathematical tasks, we first extract key information from all problems, then establish dependencies by linking one problem’s answer to another problem’s critical information, requiring models to solve multiple problems sequentially to obtain all correct answers.
To address the limitations of current training and evaluation paradigms,
we leverage this method to establish an evaluation benchmark and training data to evaluate and enhance the long-horizon reasoning capabilities of LRMs.

We first establish R-Horizon benchmark, which comprises 6 representative datasets across mathematics, code generation, and agent applications (e.g. MATH500 (Hendrycks et al., 2021), LiveCodeBench (Jain et al., 2024), WebShaper (Tao et al., 2025)).
Through evaluating 25 mainstream LRMs, we find that even the most advanced LRMs suffer significant performance degradation on R-Horizon benchmark. The performance in multi-horizon reasoning scenarios falls substantially below the theoretical performance (Figure [1](https://arxiv.org/html/2510.08189v2#S0.F1)).
Further analysis reveals critical limitations of current LRMs that contribute to the performance gap:
(1) LRMs possess a limited effective reasoning length, with performance declining sharply once the thinking budget exceeds this threshold.
(2) LRMs exhibit constrained reflection scope—LRMs often reflect within the current problem, failing to identify errors from previous questions.
(3) The overthinking phenomenon (Chen et al., 2025) in LRMs prevents the appropriate allocation of thinking budget when facing multiple reasoning problems.

To address the absence of long-horizon problems in current RL training data, we leverage R-Horizon to reconstruct training datasets and design different reward functions, then conduct reinforcement learning with verified rewards (RLVR) with varying composed problems and reward schemes to investigate the impact of long-horizon reasoning data on the training process.
By employing mainstream RLVR algorithms GRPO (Shao et al., 2024) with R-Horizon, we observe that traditional RLVR provides limited improvements on multi-step reasoning tasks. In contrast, training with R-Horizon data is a highly efficient training approach that not only enhances single-problem performance more effectively but also rapidly improves performance on multiple problems.
Our analysis demonstrates that training with R-Horizon also improves response length efficiency and thinking budget allocation.
In summary, R-Horizon mitigates the current limitations of long-horizon reasoning in training and evaluation paradigms—offering a scalable, controllable and low-cost path to improve and evaluate the long-horizon abilities of LRMs.

## 2 Related Work

### 2.1 Test Time Scaling in Large Reasoning Models

The success of OpenAI’s o1 introduced a new scaling paradigm, test-time compute scaling, which improves performance through increasing inference computation (OpenAI et al., 2024).
However, recent studies reveal that LRMs may generate verbose reasoning trajectories with marginal accuracy gains. Chen et al. (2025) reveals the “overthinking” phenomenon, showing that LRMs generate significantly more tokens than conventional LLMs on simple arithmetic tasks, with minimal increase in accuracy.
To address this, Aggarwal & Welleck (2025) proposed length-controlled policy optimization, providing precise control over the length of the reasoning trajectories during generation. Yang et al. (2025b) developed a thinking-optimal scaling strategy, allowing models to flexibly adjust their reasoning depth according to the available test-time compute budget.
Recent studies have also focused on fine-tuning models to think efficiently according to task complexity (Hao et al., 2024; Liu et al., 2025; Fang et al., 2025; Arora & Zanette, 2025; Zhang et al., 2025).
In agentic tasks, overthinking also reduces performance while increasing inference costs (Cuadron et al., 2025).
Although previous studies indicate that overthinking leads to computational inefficiency with limited performance gains, our findings reveal that prolonged reasoning substantially degrades performance on compound multi-step reasoning tasks.

### 2.2 Effective Reasoning Length of Large Reasoning Models

Recent studies explore the effective reasoning length of LRMs in mathematical benchmarks (i.e., GSM8k (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021) and AIME (MAA, 2024, 2025)).
Su et al. (2025); Yang et al. (2025b); Wu et al. (2025b) investigates the relationship between reasoning length and accuracy. Su et al. (2025) finds that models fail to adaptively calibrate their response length according to the problem difficulty.
Wu et al. (2025b); Ghosal et al. (2025); Chen et al. (2024) demonstrate the existence of an optimal CoT length beyond which performance degrades.
By directly concatenating multiple independent questions, REST (Pan et al., 2025) reveals that LRMs fail to keep their performance under multi-context stress.
However, these tasks either focus on a single problem or concatenate independent problems without meaningful logical dependencies. In contrast, we design multi-dependent synthetic tasks to expose failure modes amplified by extended reasoning, consistent with findings that reasoning chains exceeding optimal length reduce accuracy.

## 3 R-Horizon

We propose R-Horizon, a method designed to stimulate long-horizon reasoning behaviors in LRMs via query composition. As illustrated in Figure [2](https://arxiv.org/html/2510.08189v2#S3.F2), R-Horizon supports the concatenation of three types of expanded questions and can be employed in both the training and evaluation stages to enhance and evaluate the long-horizon capabilities of LRMs.

Figure: Figure 2: The R-Horizon data composition pipeline is illustrated in (a)-(c). We leverage R-Horizon to construct a comprehensive long-horizon reasoning evaluation benchmark spanning 6 tasks and generate multi-horizon training data for long-horizon reinforcement learning.
Refer to caption: x2.png

### 3.1 R-Horizon Datasets Construction

For mathematical tasks, we adopt the sequentially composed concatenation to construct a dataset of multi-step mathematical problems with explicit dependencies that enforce sequential solving. The construction pipeline consists of two stages: seed problem filtering and expanded problem composition. For code and agentic tasks, we provide the construction process in Appendix [A](https://arxiv.org/html/2510.08189v2#A1.SS0.SSS0.Px1).

#### Seed Problem Filtering

Given an initial dataset $\mathcal{D}=\{(q_{i},a_{i})\}_{i=1}^{N}$
where $(q_{i},a_{i})$ is a pair of a question and an answer.
We apply the following filtering criteria to obtain a seed set $\mathcal{D}_{\text{seed}}$:

$$ $\mathcal{D}_{\text{seed}}=\left\{(q,a)\in\mathcal{D}\mid\left|I(q)\right|>0\land a\in\mathbb{Z}\right\},$ (1) $$

where $I(\cdot)=\texttt{extract_int}(\cdot)$ denotes extracting all integers appearing in the input text.

For each $(q,a)\in\mathcal{D}_{\text{seed}}$, we identify key variables from the extracted integers.
We then employ a model $M$ to verify each interger $m\in I(q)$ whether is a key variable:

$$ $K(q)=\left\{m\in I(q)\mid M(q,m)=1\right\},$ (2) $$

where $M(q,m)=1$ indicates that removing $m$ from $q$ renders the problem unsolvable.
Each filtered seed problem is then represented as a triple $(q,a,K(q))$.

#### Expanded Problem Composition

Given seed problems with annotated key variables, we construct dependency chains using Algorithm [1](https://arxiv.org/html/2510.08189v2#alg1). The algorithm ensures that each modified problem $q_{i+1}^{\prime}$ contains a placeholder variable $v_{i+1}$ that must be resolved through the dependency function $f_{i}(a_{i})=m_{i+1}$, requiring the solution $a_{i}$ from the previous problem. The augmentation step prepends the dependency specification to the problem statement, making the sequential constraint explicit. The final dataset $\mathcal{D}_{\text{composed}}$ consists of problem sequences that enforce strict sequential solving.

Figure: Algorithm 1 Dependency Chain Construction

### 3.2 R-Horizon Benchmark

We use R-Horizon to reconstruct existing evaluation datasets, combining different datasets through problem filtering and composition approaches, and design evaluation metrics for composed problems.

#### Evaluation Metrics

R-Horizon evaluates model performance by extracting all answers from the model’s response. Given a composed problem sequence $\mathcal{Q}=(q_{1},q_{2}^{\prime},\ldots,q_{n}^{\prime})\in\mathcal{D}_{\text{composed}}$, we extract the corresponding answer sequence $\hat{\mathcal{A}}=(\hat{a}_{1},\hat{a}_{2},\ldots,\hat{a}_{n})$ from the model’s response $\mathcal{R}$.
We use all-or-nothing scoring: correct only if all sub-problems are solved:

$$ $\text{Acc}(\mathcal{Q})=\begin{cases}1&\text{if }\hat{a}_{i}=a_{i}\text{ for all }i\in\{1,\ldots,n\},\\ 0&\text{otherwise.}\end{cases}$ (3) $$

We also propose a metric to estimate its theoretical accuracy. For each $(q,a)\in\mathcal{D}_{\text{seed}}$, we use the pass rate of these atomic problems to estimate the expected accuracy of composed problems through:

$$ $\text{Acc}_{\text{expected}}(\mathcal{Q})=\prod_{i=1}^{n}p_{i},$ (4) $$

where $p_{i}$ is the pass rate of atomic problem $q_{i}$. We use model-based extraction to handle diverse response formats (details in Appendix [E.2](https://arxiv.org/html/2510.08189v2#A5.SS2)).

### 3.3 Reinforcement Learning with R-Horizon

To investigate the formation of long-horizon reasoning capabilities and understand how multi-step dependent queries influence the reinforcement learning process, we employ R-Horizon datasets as training data for reinforcement learning from verifiable rewards (RLVR). We follow Skywork-OR1 (He et al., 2025a) RLVR pipelines while utilizing our constructed training data.

#### Group Relative Policy Optimization (GRPO)

We adopt GRPO (Shao et al., 2024) as our optimization algorithm, which eliminates the value function requirement of PPO (Schulman et al., 2017) by computing advantages in a group-relative manner.
For each question $q$, the behavior policy $\pi_{\theta_{\text{old}}}$ samples a group of $G$ response candidates $\{o_{1},\ldots,o_{G}\}$. We use GRPO with token-level policy gradient loss, which optimizes the policy model by maximizing the following objective:

$$ $\begin{aligned} \mathcal{J}&{}_{\text{GRPO}}(\theta)=\mathbb{E}_{q,\{o_{i}\}_{i=1}^{G}}\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\left\{\min\left(r_{i,t}\hat{A}_{i,t},\text{clip}\left(r_{i,t},1-\epsilon,1+\epsilon\right)\hat{A}_{i,t}\right)-\beta\mathbb{D}_{\text{KL}}\left[\pi_{\theta}||\pi_{\text{ref}}\right]\right\},\end{aligned}$ (5) $$

where $r_{i,t}=\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q,o_{i,<t})}$, $\epsilon$ and $\beta$ are hyperparameters, $\hat{A}_{i,t}$ is the advantage calculated based on the relative rewards of the outputs inside each group only, and $\mathbb{D}_{\text{KL}}$ denotes the KL divergence between the learned policy and a reference policy $\pi_{\text{ref}}$.

#### Reward Design

We design two reward schemes for multi-horizon training data:

$$ $R_{\text{last}}=\begin{cases}1&\text{if }\hat{a}_{n}=a_{n},\\ 0&\text{otherwise},\end{cases}\quad\text{and}\quad R_{\text{all}}=\begin{cases}1&\text{if }\hat{a}_{i}=a_{i}\text{ for all }i\in\{1,\ldots,n\},\\ 0&\text{otherwise}.\end{cases}$ (6) $$

Last-only reward $R_{\text{last}}$ provides feedback on the final answer only, while all-correct reward $R_{\text{all}}$ requires all intermediate steps to be correct.
This distinction allows us to study how different reward function influence the development of long-horizon reasoning capabilities.

## 4 Experiment

### 4.1 Evaluation Setup

#### Datasets

For mathematical tasks, we construct MATH500 (Hendrycks et al., 2021), AIME24 (MAA, 2024), and AIME25 (MAA, 2025) with multiple dependent queries, using $n\in\{1,2,4,8,16\}$ for MATH500 and $n\in\{1,2,3,4,5\}$ for the more challenging AIME datasets. For code tasks, we reconstruct LiveCodeBench (v5) (^*^**The time is ranged from August 2024 to May 2025.) (Jain et al., 2024) with $n\in\{1,2,3,4,5\}$. For agentic tasks, we use WebShaper (Tao et al., 2025) with multi-round tool calls for web search ($n\in\{1,2,3,4,5\}$). See Appendix [E.1](https://arxiv.org/html/2510.08189v2#A5.SS1) for details.

#### Models

We select 25 advanced LRMs to perform evaluation on our built R-Horizon benchmark,
including the R1-distill series models (Guo et al., 2025), Qwen series (Yang et al., 2025a) models, and Nemotron (Bercovich et al., 2025) series models.
Model details are in Appendix [E.1](https://arxiv.org/html/2510.08189v2#A5.SS1). We set the generation length to 64k tokens to avoid truncation. More inference settings are in Appendix [E.3](https://arxiv.org/html/2510.08189v2#A5.SS3).

### 4.2 Evaluation Result

Figure: Figure 3: Evaluation results of R-Horizon Benchmark.
Refer to caption: x3.png

#### Performance Degradation as the Reasoning Horizon Increases

As shown in Figure [3](https://arxiv.org/html/2510.08189v2#S4.F3), we observe that models across different categories experience performance degradation as the reasoning horizon increases.
Even the most powerful models, including DeepSeek-R1, Qwen3-235B-A22B-Thinking, and o4-mini, suffer from severe performance degradation as the reasoning horizon increases. For instance, on AIME25, DeepSeek-R1 drops from 87.3% ($n=1$) to 24.6% ($n=5$).
Additionally, we find that larger models exhibit less degradation when confronting composed problems, while smaller models experience more severe performance degradation. For example, R1-Qwen-7B drops from 93.6% ($n=1$) to 0% ($n=16$), which is 34.1% more than the 32B model.

#### Same Degradation Trends Across Different Model and Task Categories

We observe consistent degradation trends across tasks of varying difficulty and types. Models exhibit greater performance drops when facing more challenging tasks. For instance, Qwen3-235B-Thinking drops from 93.7% ($n=1$) to 69.2% ($n=5$) on AIME24, but experiences a steeper decline from 92.3% ($n=1$) to 29.2% ($n=5$) on AIME25. For code tasks, we find that the degradation trend is more severe compared to mathematical tasks, with smaller models (7B) struggling to complete multiple code problems. For web search tasks, we observe that many trained reasoning models have lost their ability to call tools, resulting in poor performance.

### 4.3 Reinforcement Learning with R-Horizon Datasets

Despite reinforcement learning bringing long CoT thinking capabilities to models, current mainstream LRMs still cannot achieve good performance on R-Horizon Benchmark.
We follow Skywork-OR1 (He et al., 2025a) to observe the changes in long-horizon reasoning capabilities of long CoT models before and after standard RL in Appendix [B](https://arxiv.org/html/2510.08189v2#A2).
We find that training with only single-problem data leads to slow improvement in models’ ability to handle composed problems. To investigate the impact of R-Horizon data on RL training, we construct composed training data through R-Horizon based on the original math training datasets.

#### Training Setup

We construct a data pool $\mathcal{D}_{\text{filtered}}$ from Skywork-OR1-RL training data using Problem Filtering (Section [3.1](https://arxiv.org/html/2510.08189v2#S3.SS1.SSS0.Px1)). To manage difficulty, we combine problems by pass rates, keeping $\text{Acc}_{\text{expected}}>0.25$. We train on R1-Qwen-7B and set maximum response length to 40k to prevent truncation and use the last-only reward $R_{\text{last}}$ as default, which provides feedback on the final answer only. Details are in Appendix [F](https://arxiv.org/html/2510.08189v2#A6).

#### Training with R-Horizon Datasets

Figure: Figure 4: Training curves comparing single and composed data on $\text{AIME24}_{\text{avg@8}}$ and reward.
Refer to caption: x4.png

We train R1-Qwen-7B using both original data and 2-query composed data. As shown in Figure [4](https://arxiv.org/html/2510.08189v2#S4.F4), compared to the original data, composed data significantly improves performance on composed problems (+17.4 on AIME24 (n=2)). Additionally, we find that training with composed problem data also substantially improves performance on the original tasks (+7.5 on AIME24). During the training process, the reward for composed data gradually increases and surpasses the reward for the original data.

#### Impact of Number of Composed Queries and Different Reward Schemes

To further investigate the impact of the number of composed problems, we construct four types of training data based on the number of composed problems: composed problem counts of (1, 2, 4, and a mixture of problems with counts 1, 2, 3, 4). We also study the effects of different rewards on composed data in Table [1](https://arxiv.org/html/2510.08189v2#S4.T1).

**Table 1: Results of different number of composed queries and reward function**
|  | MATH500 | AIME24 | AIME25 | AMC23 | Avg. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Origin | n=8 | Origin | n=2 | Origin | n=2 | Origin | n=2 | Origin | Multi |
| R1-Qwen-7B | 93.6 | 11.8 | 48.3 | 16.4 | 33.3 | 3.5 | 90.2 | 48.8 | 66.4 | 20.1 |
| Naive Training Data (n=1) | 95.6 | 8.4 | 57.9 | 16.7 | 47.9 | 5.1 | 95.9 | 55.0 | 74.3 | 21.3 |
| w/ composed queries (n=2) | 95.4 | 21.4 | 65.4 | 34.1 | 49.6 | 10.0 | 94.1 | 80.6 | 76.1 | 36.5 |
| w/ composed queries (n=4) | 94.6 | 50.6 | 62.9 | 34.8 | 45.4 | 8.1 | 91.9 | 79.1 | 73.7 | 43.2 |
| w/ composed queries (mixed) | 96.8 | 47.8 | 57.1 | 32.8 | 44.2 | 10.0 | 93.1 | 81.6 | 72.8 | 43.1 |
| w/ $R_{\text{all}}$ (n=2) | 95.0 | 26.8 | 64.6 | 38.8 | 48.8 | 11.9 | 95.0 | 83.4 | 75.9 | 40.2 |

All models trained with composed data demonstrate significant performance improvements on composed problems. Moreover, composed data also substantially enhances performance on the original datasets. For instance, composed problems with n=2 yield the largest improvements on AIME24 and AIME25. As the number of composed problems increases, models exhibit stronger capabilities in handling problems requiring more reasoning steps. Additionally, we observe that using $R_{\text{all}}$ as the reward function on training data with 2 composed problems outperforms $R_{\text{last}}$ when confronting scenarios with multiple problems.
More training dynamics are provided in Appendix [C](https://arxiv.org/html/2510.08189v2#A3).

## 5 Analysis

Our analysis covers evaluation results of the R-Horizon benchmark (Section [5.1](https://arxiv.org/html/2510.08189v2#S5.SS1)) and RL training results using R-Horizon datasets (Section [5.2](https://arxiv.org/html/2510.08189v2#S5.SS2)). Ablation studies on evaluation metrics, dependency relationships, and problem difficulty ordering are in Appendix [D](https://arxiv.org/html/2510.08189v2#A4).

### 5.1 Evaluation Result Analysis

#### Error Type Analysis

We analyze the error types of the evaluation result in Figure [5](https://arxiv.org/html/2510.08189v2#S5.F5).
We find that as the number of problems increases, Problem Reasoning Errors increase rapidly.
Adding simple dependencies between problems increases the overall reasoning difficulty, and the number of Dependency Reasoning Errors gradually increases with the number of problems, though the overall count remains relatively small. We observe that when facing multiple problems, models frequently terminate their responses prematurely, answering only a subset of the problems.

Figure: Figure 5: Error type distribution across different query numbers. Four error categories: Problem Reasoning Error represents reasoning errors made by the model for specific problems; Dependency Reasoning Error indicates the model correctly solved previous problems but made errors when calculating the dependencies; Early Stop indicates the model prematurely terminated generation after solving previous problems; Output Truncation indicates generation exceeded token limit.
Refer to caption: x5.png

#### Effective Reasoning Length of LRMs

As shown in Figure [6](https://arxiv.org/html/2510.08189v2#S5.F6), as the number of problems increases, the gap between the actual accuracy and theoretical accuracy of models becomes increasingly larger, indicating that models struggle to maintain their original performance as reasoning length increases.
We observe that the error position of models gradually declines and stabilizes within a certain range as the number of problems increases. Comparing R1-Qwen-7B and R1-Qwen-32B, we observe that larger models can reason over longer contexts, and each model has its own reasoning boundary.
For example, the 7B model’s error range is (4-6k tokens) while the 32B model’s error range is (8-10k tokens).

Figure: Figure 6: Analysis of accuracy and error position with R1-Qwen-7B and R1-Qwen-32B.
Refer to caption: x6.png

#### Reflection Frequency and Depth of LRMs

As shown in Figure [7](https://arxiv.org/html/2510.08189v2#S5.F7), the reflection frequency of models gradually increases with the number of problems and converges to a maximum value.
As the number of problems increases, the proportion of problems involving long-range reflection also rises, yet we find that more than half of the problems lack any long-range reflection process, which indicates that LRMs’ reflections are highly localized.

Figure: Figure 7: Reflection analysis on MATH500 dataset. Reflection Frequency refers to the average number of reflections per question. Long Reflection Rate refers to the proportion of questions whose reflection range exceeds the current question.
Refer to caption: x7.png

#### Thinking Budget Allocation of LRMs

As shown in Figure [8](https://arxiv.org/html/2510.08189v2#S5.F8), current models tend to allocate more tokens to early reasoning stages.
Even DeepSeek-R1 cannot effectively distribute the thinking budget reasonably to subsequent problems, indicating that current mainstream LRMs have not yet developed the capability to allocate thinking budgets according to reasoning horizon.

Figure: Figure 8: The thinking budget allocation for different query configurations (1-5 queries) across R1-Qwen-7B, R1-Qwen-32B, and Deepseek-R1 models on AIME24 datasets.
Refer to caption: x8.png

### 5.2 Analysis of Reinforcement Learning with R-Horizon

We analyze models trained with R-Horizon data versus those trained with original data using RL, as shown in Figure [9](https://arxiv.org/html/2510.08189v2#S5.F9). In Figure [9](https://arxiv.org/html/2510.08189v2#S5.F9) (a), we find that training with composed queries significantly improves model performance on composed tasks and can generalize to longer reasoning horizons. Additionally, we find it alleviates the overthinking phenomenon. Models generate shorter responses when facing multiple problems compared to models trained on original data in Figure [9](https://arxiv.org/html/2510.08189v2#S5.F9) (b), and training with composed problems enables models to learn more reasonable token budget allocation in Figure [9](https://arxiv.org/html/2510.08189v2#S5.F9) (d).
These results demonstrate that training with composed data promotes efficient reasoning, which is consistent with the findings of training dynamics in Appendix [C](https://arxiv.org/html/2510.08189v2#A3).
We also provide a case study in Appendix [H](https://arxiv.org/html/2510.08189v2#A8) to compare the reasoning behavior between standard training and training with R-Horizon datasets.

In Figure [9](https://arxiv.org/html/2510.08189v2#S5.F9) (c), as the number of problems increases, training with composed problems enables models to engage in longer reflections with increasing frequency, while the reflection frequency of models also increases more reasonably. This demonstrates that using R-Horizon facilitates longer-range reflection in models, thereby improving performance on long-horizon reasoning tasks.

Figure: Figure 9: Analysis of reinforcement learning effects with single and composed datasets. (a) Math500 performance comparison, (b) error position analysis, (c) reflection analysis, and (d) token budget allocation across multi-horizon scenarios.
Refer to caption: x9.png

## 6 Conclusion

In this paper, we present R-Horizon, a novel and efficient approach to stimulating long-horizon reasoning in LRMs through query composition. By composing simple problems into sequential, interdependent tasks, R-Horizon constructs multi-step reasoning datasets that serve dual purposes: evaluating LRMs’ long-horizon reasoning capabilities and enhancing their complex reasoning abilities during training. Our method establishes a foundation for future advances in complex reasoning data synthesis and the development of models with robust long-horizon reasoning capabilities.

## References

- Aggarwal & Welleck (2025)
Pranjal Aggarwal and Sean Welleck.
L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025.
URL [https://arxiv.org/abs/2503.04697](https://arxiv.org/abs/2503.04697).
- An et al. (2025)
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong.
Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025.
URL [https://hkunlp.github.io/blog/2025/Polaris](https://hkunlp.github.io/blog/2025/Polaris).
- Arora & Zanette (2025)
Daman Arora and Andrea Zanette.
Training language models to reason efficiently, 2025.
URL [https://arxiv.org/abs/2502.04463](https://arxiv.org/abs/2502.04463).
- Bercovich et al. (2025)
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe,
Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung, and Chris
Alexiuk.
Llama-nemotron: Efficient reasoning models, 2025.
URL [https://arxiv.org/abs/2505.00949](https://arxiv.org/abs/2505.00949).
- Chen et al. (2024)
Qiguang Chen, Libo Qin, Jiaqi Wang, Jinxuan Zhou, and Wanxiang Che.
Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought, 2024.
URL [https://arxiv.org/abs/2410.05695](https://arxiv.org/abs/2410.05695).
- Chen et al. (2025)
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu.
Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025.
URL [https://arxiv.org/abs/2412.21187](https://arxiv.org/abs/2412.21187).
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems, 2021.
URL [https://arxiv.org/abs/2110.14168](https://arxiv.org/abs/2110.14168).
- Cuadron et al. (2025)
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez.
The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks, 2025.
URL [https://arxiv.org/abs/2502.08235](https://arxiv.org/abs/2502.08235).
- Fang et al. (2025)
Gongfan Fang, Xinyin Ma, and Xinchao Wang.
Thinkless: Llm learns when to think, 2025.
URL [https://arxiv.org/abs/2505.13379](https://arxiv.org/abs/2505.13379).
- Ghosal et al. (2025)
Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi.
Does thinking more always help? understanding test-time scaling in reasoning models, 2025.
URL [https://arxiv.org/abs/2506.04210](https://arxiv.org/abs/2506.04210).
- Guha et al. (2025)
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt.
Openthoughts: Data recipes for reasoning models, 2025.
URL [https://arxiv.org/abs/2506.04178](https://arxiv.org/abs/2506.04178).
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hao et al. (2024)
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian.
Training large language models to reason in a continuous latent space, 2024.
URL [https://arxiv.org/abs/2412.06769](https://arxiv.org/abs/2412.06769).
- He et al. (2025a)
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou.
Skywork open reasoner series.
[https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaonser-Series-1d0bc9ae823a80459b46c149e4f51680](https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaonser-Series-1d0bc9ae823a80459b46c149e4f51680), 2025a.
Notion Blog.
- He et al. (2025b)
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu.
Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025b.
URL [https://arxiv.org/abs/2504.11456](https://arxiv.org/abs/2504.11456).
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset, 2021.
URL [https://arxiv.org/abs/2103.03874](https://arxiv.org/abs/2103.03874).
- Jain et al. (2024)
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.
URL [https://arxiv.org/abs/2403.07974](https://arxiv.org/abs/2403.07974).
- Liu et al. (2025)
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He.
Learn to reason efficiently with adaptive length-based reward shaping, 2025.
URL [https://arxiv.org/abs/2505.15612](https://arxiv.org/abs/2505.15612).
- Luo et al. (2025a)
Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepcoder: A fully open-source 14b coder at o3-mini level.
https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51, 2025a.
Notion Blog.
- Luo et al. (2025b)
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.
https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2, 2025b.
Notion Blog.
- MAA (2024)
MAA.
American invitational mathematics examination - aime.
In *American Invitational Mathematics Examination - AIME 2024*, February 2024.
URL [https://maa.org/math-competitions/american-invitational-mathematics-examination-aime](https://maa.org/math-competitions/american-invitational-mathematics-examination-aime).
- MAA (2025)
MAA.
American invitational mathematics examination - aime.
In *American Invitational Mathematics Examination - AIME 2025*, February 2025.
URL [https://maa.org/math-competitions/american-invitational-mathematics-examination-aime](https://maa.org/math-competitions/american-invitational-mathematics-examination-aime).
- Muennighoff et al. (2025)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
s1: Simple test-time scaling, March 2025.
URL [http://arxiv.org/abs/2501.19393](http://arxiv.org/abs/2501.19393).
arXiv:2501.19393 [cs].
- Nvidia et al. (2024)
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad
Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu.
Nemotron-4 340b technical report, 2024.
URL [https://arxiv.org/abs/2406.11704](https://arxiv.org/abs/2406.11704).
- OpenAI et al. (2024)
OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric
Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu,
Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez,
Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li.
Openai o1 system card, 2024.
URL [https://arxiv.org/abs/2412.16720](https://arxiv.org/abs/2412.16720).
- Pan et al. (2025)
Zhuoshi Pan, Qizhi Pei, Yu Li, Qiyao Sun, Zinan Tang, H. Vicky Zhao, Conghui He, and Lijun Wu.
Rest: Stress testing large reasoning models by asking multiple problems at once, 2025.
URL [https://arxiv.org/abs/2507.10541](https://arxiv.org/abs/2507.10541).
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms, 2017.
URL [https://arxiv.org/abs/1707.06347](https://arxiv.org/abs/1707.06347).
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
URL [https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300).
- Su et al. (2025)
Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie.
Between underthinking and overthinking: An empirical study of reasoning length and correctness in llms, 2025.
URL [https://arxiv.org/abs/2505.00127](https://arxiv.org/abs/2505.00127).
- Tao et al. (2025)
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou.
Webshaper: Agentically data synthesizing via information-seeking formalization, 2025.
URL [https://arxiv.org/abs/2507.15061](https://arxiv.org/abs/2507.15061).
- Team et al. (2025a)
5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli
Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang.
Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025a.
URL [https://arxiv.org/abs/2508.06471](https://arxiv.org/abs/2508.06471).
- Team et al. (2025b)
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu
Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu,
Weiyu Zhuang, and Xinxing Zu.
Kimi k2: Open agentic intelligence, 2025b.
URL [https://arxiv.org/abs/2507.20534](https://arxiv.org/abs/2507.20534).
- Wu et al. (2025a)
Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, Yanwei Fu, Qin Liu, Songyang Zhang, and Qi Zhang.
Reasoning or memorization? unreliable results of reinforcement learning due to data contamination, 2025a.
URL [https://arxiv.org/abs/2507.10532](https://arxiv.org/abs/2507.10532).
- Wu et al. (2025b)
Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang.
When more is less: Understanding chain-of-thought length in llms, 2025b.
URL [https://arxiv.org/abs/2502.07266](https://arxiv.org/abs/2502.07266).
- Yang et al. (2025a)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025a.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Yang et al. (2025b)
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei.
Towards thinking-optimal scaling of test-time compute for llm reasoning, 2025b.
URL [https://arxiv.org/abs/2502.18080](https://arxiv.org/abs/2502.18080).
- Yao et al. (2024)
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.
$\tau$-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024.
URL [https://arxiv.org/abs/2406.12045](https://arxiv.org/abs/2406.12045).
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang.
Dapo: An open-source llm reinforcement learning system at scale, 2025.
URL [https://arxiv.org/abs/2503.14476](https://arxiv.org/abs/2503.14476).
- Yue et al. (2025)
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan.
Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025.
URL [https://arxiv.org/abs/2504.05118](https://arxiv.org/abs/2504.05118).
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He.
Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
URL [https://arxiv.org/abs/2503.18892](https://arxiv.org/abs/2503.18892).
- Zhang et al. (2025)
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li.
Adaptthink: Reasoning models can learn when to think, 2025.
URL [https://arxiv.org/abs/2505.13417](https://arxiv.org/abs/2505.13417).

## Appendix A R-Horizon Datasets Construction for code and agentic tasks

#### Datasets Construction for Code Tasks

For code tasks, we adopt a composition approach similar to mathematical tasks, using data points from existing datasets as seed questions for composition. We continue to employ the Expanded Problem Composition process described in Section [3.1](https://arxiv.org/html/2510.08189v2#S3.SS1.SSS0.Px2).
However, unlike the sequential composition used for mathematical tasks, we apply a directly composed concatenation format for code tasks without adding explicit dependencies between problems.
This design choice is motivated by the fact that code tasks require sandbox execution to obtain answers, making it challenging to construct direct dependency relationships between problems and answers as in mathematical tasks.

#### Datasets Construction for Agentic Tasks

For agentic tasks, we incorporate web search tasks for evaluation. We decompose questions based on the structured data from WebShaper (Tao et al., 2025), which consists of “Target” (target variable) and “Variable” (intermediate variables). Specifically, for each question, we follow this processing pipeline:
We filter the original WebShaper dataset to obtain questions with varying complexity levels, ultimately selecting 50 questions. Each question’s associated URLs are accessed using a browsing tool, with browsing results stored for subsequent processing (URLs that cannot be accessed are filtered out). We employ Claude-Sonnet-4 to extract values for each variable $V$ from the web pages (variables that cannot be extracted are excluded). The original questions and variables $V$ are then assembled into a directed acyclic graph (DAG). Following topological sorting, we perform pruning to derive sub-questions and seed questions (questions with erroneous or duplicate decompositions are filtered out).
This process yields a final dataset of 50 questions, with each question categorized into 5 levels based on the number of variables (ranging from 1 to 5), resulting in a total of 250 seed problems.

## Appendix B How Reinforcement Learning improves long-horizon Reasoning

Despite reinforcement learning bringing long CoT thinking capabilities to models, we find that current mainstream LRMs still cannot achieve good performance on R-Horizon evaluation. To further analyze the relationship between long-horizon reasoning capabilities and RL, we follow Skywork-OR1 (He et al., 2025a), an effective and scalable RL implementation for long CoT models, to observe the changes in long-horizon reasoning capabilities of long CoT models before and after RL.

#### Training Setup

We follow the Skywork OR1 (He et al., 2025a) multi-stage training approach, gradually increasing context length across different stages. Once the model’s performance converged, we increased the context length in the subsequent stage. This approach led to significant performance improvements on benchmarks while also enhancing training efficiency. We employ 3 stage training with max response lengths increasing from 8k (0-600 steps) to 16k (600-1400 steps), and finally to 32k (1400-1680 steps). We train on the math subsets of the Skywork-RL dataset. Additional training settings are provided in Appendix [F](https://arxiv.org/html/2510.08189v2#A6).

#### Observations During Training Process

Figure: Figure 10: The AIME24, AIME25 performance for single query and 2-query settings and response length evolution during multi-stage training progression across 8k, 16k, and 32k context lengths. Vertical dashed lines mark stage transitions.
Refer to caption: x10.png

We find that RL training can improve model performance on composed problems, but the improvement is smaller than that on corresponding single problems (+36.6% on AIME24 and +9.1% on AIME24 $n=2$). Additionally, we observe that the improvement on composed problems shows no clear correlation with the increase in response length. When training at the 32k stage, although response length increases significantly, the model’s performance on both single and composed problems does not improve substantially.

## Appendix C Training Dynamics of RL with R-Horizon

We present the training dynamics of models trained with composed training data ($n=1,n=2,n=4$) in Figure [11](https://arxiv.org/html/2510.08189v2#A3.F11). The response length of models trained with composed data initially decreases and then increases as training progresses, ultimately reaching levels comparable to those trained with original data, with similar training time per step. This indicates that models require fewer tokens to solve each problem, demonstrating that training with composed data promotes efficient reasoning. However, the entropy loss of models trained with composed data decreases more rapidly than those trained with original data, which may limit the model’s capacity for effective exploration.

Figure: Figure 11: Training dynamics comparison across different training data compositions (n=1, n=2, n=4) showing response length, training time per step, and entropy loss evolution during the RL training process.
Refer to caption: x11.png

## Appendix D Ablation study

### D.1 Ablation on Dependencies

Figure: Figure 12: Comparison between multiple dependent and independent problems.
Refer to caption: x12.png

We compare the difference between multiple dependent problems and multiple independent problems. We remove the dependency construction step and directly concatenate multiple problems. We conduct experiments using R1-Qwen-7B on Math500, with results shown in Figure [12](https://arxiv.org/html/2510.08189v2#A4.F12). We find that the accuracy of both problem composition methods falls below the theoretical accuracy, and the accuracy of multiple sequentially dependent problems is significantly lower than that of multiple independent problems. This indicates that current models still have substantial deficiencies when handling multiple correlated problems.

### D.2 Ablation on Evaluation Metric

R-Horizon adopts an all-or-nothing scoring criterion $\text{Acc}_{\text{all}}$ to ensure models correctly answer all problems. An alternative evaluation metric $\text{Acc}_{\text{last}}$ considers a response correct if only the final problem is answered correctly. Theoretically, these two metrics should be identical for problems with sequential dependencies, as correctly answering the final problem requires sequentially solving all preceding problems. However, our ablation experiments reveal substantial differences between these metrics as the number of problems increases, as shown in Figure [13](https://arxiv.org/html/2510.08189v2#A4.F13) (Left). The probability of correctly answering only the final problem far exceeds the probability of correctly answering all problems. We observe an anomalous phenomenon: models can correctly solve subsequent problems despite incorrect solutions to preceding ones, indicating that models can produce correct answers even when problems should be unsolvable. We provide statistics on these anomalous cases in Figure [13](https://arxiv.org/html/2510.08189v2#A4.F13) (right). We hypothesize that this phenomenon is related to data contamination in models (Wu et al., 2025a).

Figure: Figure 13: R1-Qwen models showing anomalous behavior in sequential reasoning. Left: $\text{Acc}_{\text{all}}$ vs. $\text{Acc}_{\text{last}}$ revealing increasing divergence. Right: Anomalous sample counts where models correctly answer final problems despite preceding errors.
Refer to caption: x13.png

### D.3 Impact of Query Difficulty Ordering

We conduct an ablation study to examine whether the ordering of query difficulty affects model performance and thinking budget allocation.
Using the pass rate of R1-Qwen-7B as the reference metric, we define a query as easy if its pass rate exceeds 0.5 and hard otherwise.
We then compare the performance of both 7B and 32B models under different orderings of easy and hard queries (i.e., easy-to-hard vs. hard-to-easy).
Figure [14](https://arxiv.org/html/2510.08189v2#A4.F14) (b) and (c) show that all models fail to allocate thinking budget reasonably according to problem difficulty. More powerful models (DeepSeek-R1, R1-Qwen-32B) can benefit from difficulty ordering in Figure [14](https://arxiv.org/html/2510.08189v2#A4.F14) (a). We hypothesize that this is because placing difficult problems at the beginning leads models to allocate more token budget to difficult problems, thereby improving overall success rate, while smaller models (R1-Qwen-7B) show no significant benefit.

Figure: Figure 14: Ablation study on the impact of query difficulty ordering for R1-Qwen-7B, R1-Qwen-32B, and DeepsSeek-R1 models. (a) Performance comparison between easy-to-hard and hard-to-easy query orderings. (b) Thinking budget allocation in the easy-to-hard scenario. (c) Thinking budget allocation in the hard-to-easy scenario.
Refer to caption: x14.png

## Appendix E Evaluation Implementation Details

### E.1 Models and Datasets in R-Horizon Benchmark

#### Datasets Statistics and Evaluation Metric

We present the statistics and evaluation metric of the R-Horizon benchmark in Table [2](https://arxiv.org/html/2510.08189v2#A5.T2), showing the number of problems in the original datasets, extracted seed questions, and final composed datasets.

**Table 2: Dataset statistics and evaluation metric for R-Horizon benchmark**
| Dataset | Number of Problems | Metric |  |  |
| --- | --- | --- | --- | --- |
| Original | Seed | Composed |  |  |
| Mathematical Tasks |  |  |  |  |
| Math500 | 500 | 257 | 500 | Accuracy |
| AIME24 | 30 | 28 | 30 | Avg@32 |
| AIME25 | 30 | 28 | 30 | Avg@32 |
| AMC23 | 40 | 37 | 40 | Avg@8 |
| Code Tasks |  |  |  |  |
| LiveCodeBench | 279 | 279 | 279 | Pass@1 |
| Agentic Tasks |  |  |  |  |
| WebShaper | 500 | 117 | 50 | Avg@3 |

#### Model Details

In the R-Horizon benchmark, we evaluate the following open-source models. We present the model sources and their corresponding evaluation lengths (max new tokens for generation) as follows: DeepSeek-R1-0528 (64k), R1-Qwen-1.5B (64k), R1-Qwen-7B (64k), R1-Qwen-32B (64k), R1-Llama8B (64k), R1-Llama70B (64k) (Guo et al., 2025), Qwen3-8B (40k), Qwen3-32B (40k), Qwen3-235B-A22B-2507 (64k), Qwen3-235B-A22B-Thinking-2507 (64k), QwQ-32B (64k) (Yang et al., 2025a), Nemotron-Research-Reasoning-Qwen-1.5B (64k), Llama-3.1-Nemotron-Nano-8B-v1 (64k) (Nvidia et al., 2024), DeepScaleR-1.5B-Preview (64k) (Luo et al., 2025b), Polaris-1.7B-Preview (64k), Polaris-4B-Preview (64k) (An et al., 2025), Skywork-OR1-7B (64k), Skywork-OR1-32B (64k) (He et al., 2025a), OpenThinker3-7B (32k) (Guha et al., 2025), Efficient-R1-7B ($\alpha=0.2$) (64k) (Arora & Zanette, 2025), Laser-DE-L4096-7B (64k) (Liu et al., 2025), DAPO-Qwen-32B (64k) (Yu et al., 2025).

#### Prompt Examples

We present the prompt examples for math, code, and web search tasks in Figure [15](https://arxiv.org/html/2510.08189v2#A5.F15), Figure [16](https://arxiv.org/html/2510.08189v2#A5.F16) and Figure [17](https://arxiv.org/html/2510.08189v2#A5.F17).

Figure: Figure 15: Prompt Example for Mathematical Tasks

Figure: Figure 16: Prompt Example for Code Tasks

Figure: Figure 17: Prompt Example for Web Search Tasks

### E.2 Evaluation Metrics Calculation

For mathematical and agent-based WebShaper tasks, we utilize GPT-4.1 to extract answers from all problems and perform subsequent scoring. For code tasks, we first extract code blocks from the responses and assess their correctness via sandbox execution. The prompts used for scoring are presented in Figure [18](https://arxiv.org/html/2510.08189v2#A5.F18) and Figure [19](https://arxiv.org/html/2510.08189v2#A5.F19).

Figure: Figure 18: Answer Extraction Prompt for Mathematical Tasks

Figure: Figure 19: Answer Extraction Prompt for WebShaper

We also compare the consistency rate between using model-based answer extraction and rule-based “\boxed{}” pattern extraction in Table [3](https://arxiv.org/html/2510.08189v2#A5.T3). We find that as the number of problems increases, many models fail to accurately follow the output format, making model-based answer extraction more accurate for evaluation. Therefore, we uniformly adopt model-based answer extraction for mathematical tasks.

**Table 3: Consistency rate between model-based and rule-based extraction for R1-Qwen-7B on Math500**
| Composed Problem Num | 2 | 4 | 8 | 16 |
| --- | --- | --- | --- | --- |
| Consistency Rate (%) | 96.83 | 96.41 | 93.77 | 91.04 |

### E.3 Inference Hyperparameters

We set the maximum generation length for inference to 64k tokens. For models with maximum lengths below 64k, we set the max generation length to their max sequence length.
For inference hyperparameters, we set temperature to 1.0, top-$k$ to 10, and top-$p$ to 0.95. For the Qwen series hybrid reasoning models that switch between thinking mode and non-thinking mode, we consistently test their thinking mode.

## Appendix F Training Implementation Details

### F.1 Training Setup

We show the training hyperparameters for training with R-Horizon datasets in Section [4.3](https://arxiv.org/html/2510.08189v2#S4.SS3). The maximum response length is set to 40k tokens to prevent truncation.
Training is conducted exclusively on the mathematical components of the Skywork-RL dataset.
All training progress are fine-tuned by optimizing the policy loss [5](https://arxiv.org/html/2510.08189v2#S3.E5) with a constant learning rate of $1\times 10^{-6}$. We set the batch size to 256, mini-batch size to 128, and group size to 16. We employ a higher clip ratio of 0.265, target entropy of 0.2, sampling temperature of 1.0, and rejection sampling. Notably, we do not apply any KL loss in our training process.

We use the same training hyperparameters for standard RL training in Appendix [B](https://arxiv.org/html/2510.08189v2#A2).
We implement a three-stage training paradigm following Skywork OR1 (He et al., 2025a), where context length is incrementally expanded upon reaching performance convergence at each stage. This progressive approach, advancing from 8k to 16k and ultimately to 32k maximum response tokens, delivers both improved benchmark results and enhanced computational efficiency.

### F.2 R-Horizon Training Datasets

We initialize a filtered data pool $\mathcal{D}_{\text{filtered}}$ from the original Skywork-OR1-RL training data via the R-Horizon Problem Filtering process (Section [3.1](https://arxiv.org/html/2510.08189v2#S3.SS1.SSS0.Px1)). To control the problem difficulty, we compose problems according to their pass rates while maintaining $\text{Acc}_{\text{expected}}>0.25$ for all composed instances. We show the datasets’ statistics in Table [4](https://arxiv.org/html/2510.08189v2#A6.T4).

**Table 4: Dataset statistics for Skywork-o1-RL Data**
|  | Original | Seed | Composed (pass_rate$>$0.25) |
| --- | --- | --- | --- |
| Skywork-o1-RL Data | 48371 | 18015 | 18000 |

## Appendix G The use of large language models

Large language models were employed exclusively as writing aids to refine sentence clarity, format tables, and improve overall readability. They were not involved in the central research contributions, experimental design, or scientific content of this work. The authors bear full responsibility for all content presented in the paper.

## Appendix H Case Study

We provide a case study with an example prompt shown in Figure [20](https://arxiv.org/html/2510.08189v2#A8.F20), and compare model outputs on multi-horizon problems when trained with original data versus R-Horizon training data, as illustrated in Figure [21](https://arxiv.org/html/2510.08189v2#A8.F21) and Figure [22](https://arxiv.org/html/2510.08189v2#A8.F22). We observe that models trained with R-Horizon training data consume fewer tokens per problem, avoid excessive thinking budget allocation on individual problems, and successfully solve all problems.

Figure: Figure 20: Example Prompt for Case Study

Figure: Figure 21: Case Study for Model Trained with Original Data

Figure: Figure 22: Case Study for Model Trained with Composed Data

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * Figure 2: What are the circles with a red outline?
* 4.2, last paragraph: Why is AIME25 more challenging than AIME24?
* Figure 6: How can the error position be above the token length? Because wrong answers tend to be longer for smaller query numbers?
* How much can the six benchmarks (MATH500, AMC23, AIME24/25, LiveCodeBench and WebShaper) be considered out of distribution compared to the training dataset based on Skywork-RL?
soundness: 3
strengths: * with the increase in context lengths of modern language models, long-horizon reasoning is a relevant topic and the dataset seems to be good contribution
* the writing is mostly clear
* good evaluation in general and large number of RLMs tested
* good ablation study
* detailed appendix with good overview over additional results as well as the evaluation setup
summary: The authors propose a new dataset for long-horizon reasoning and then also train a model on their dataset with Reinforcement Learning (RL). The new dataset is based on six other datasets targeting math (MATH500, AMC23, AIME24 and AIME25), code generation (LiveCodeBench) and agentic tool usage (WebShaper). Long-horizon problems are generated by combining several independent problems into a single one, where the problems have to be solved (usually) sequentially since later problems rely on the results of earlier problems, thereby transforming them into a long-horizon task. 25 language models are then evaluated on the developed dataset and their results are analyzed. In addition the author also generate a training dataset based on the Skywork-RL dataset and use it to fine-tune an existing model with custom reward functions, which then is also evaluated.
weaknesses: * I found the motivation a bit lacking: the new dataset is somewhat artificial in its dependencies - a proper motivational example would be beneficiary in my opinion
* not a weakness per se, but I found the third chapter a bit too high-level without much details
* writing primarily in the evaluation is at times a bit imprecise or not entirely accurate:
  * 4.1:
    * AMC23 is not discussed
    * for MATH500 it seems that there is also data for 20 composed query numbers
  * 4.2, first paragraph: "R1-Qwen-7B drops from 93.6% (n = 1) to 0% (n = 16), which is 34.1% more than the 32B model" - reference is unclear (probably MATH500); Where does the 34.1% come from?
  * 5.1, Effective Reasoning Length of LRMs:
    * "gap between the actual accuracy and theoretical accuracy of models becomes increasingly larger" - not necessarily true on AIME24
    * "7B model’s error range is (4-6k tokens) while the 32B model’s error range is (8-10k tokens)" - context is missing, probably for MATH500
  * Reflection is not defined or at least explained, which also makes it unclear why it matters.
  * Sometimes "expected accuracy" and sometimes "theorectical accuracy" is used? I assume they are the same thing, so I suggest to use one term consistently.
  * Appendix D.3: "Figure 14 (b) and (c) show that all models fail to allocate thinking budget reasonably according to problem difficulty" - seems to be somewhat not true for R1-Qwen-7B

minor issues:
* introduction: no reference for CoT
* Figure 1: Theoretical Accuracy is not introduced at this point. (only on page 4)
* 2.2:
  * "Su et al. (2025); Yang et al. (2025b); Wu et al. (2025b) investigates..." - It should be "investigate", since it is a plural.
* Algorithm 1: "and Create" - lowercase create?
* 4.1: I believe, there is no whitespace before a footnote.
* 4.3, Impact of Number of Composed Queries and Different Reward Schemes: n=1 is not really a composed problem
* Figure 5: not very readable in terms of fontsize as well as chosen color for example for early stop and output truncation - I had to zoom in to 400% to find an instance for output truncation
  * similar issue for Figure 6, especially the theoretical accuracy as well as Figures 7 to 9
* Figure 6: missing whitespace before the brackets in the subfigure titles (also for Figure 9)
* sometimes it is "Math500" and sometimes it is "MATH500" in the evaluation figures
* Figure 8: x axis label might be misleading, maybe something like "2 Queries" or just "2" and so on would be better?
* Appendix F.1: "policy loss 5 with" - should probably be something like "policy loss (see Eq. 5) with"
* Appendix H: details on model, which generated the responses are missing, where the questions are coming from, etc.
* references:
  * consider the proper capitalization of the titles, at least for proper names and abbreviations to improve readability
  * place of publication of arXiv references can only surmised from the URL
  * consider adding the access date for web references
  * [Guo et al. 2025] - cited differently than the other arXiv references
  * [Luo et al. 2025a/b] - URL is not a proper link
  * [MAA 2024/2025] - consider using something like the xurl package to fix the formatting
  * [Muennighoff et al. 2025] - cited differently than the other arXiv references - maybe this could be the blueprint for the others

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: **Questions for the Authors:**
1. In **Figure 11**, the entropy of RL training decreases as *n* increases (e.g., *n=4* vs. *n=1*). Could the authors provide an explanation or hypothesis for why longer reasoning chains lead to lower entropy?  
2. Is it possible that the poor performance of models on **R-HORIZON** arises because they have not been exposed to similar long-horizon or compositional data during pre-training or intermediate training? Have you attempted any supervised fine-tuning (SFT) or cold-start experiments on such data?  
3. Could the improvement in original tasks after RL with R-HORIZON stem from the fact that the constructed datasets yield more balanced reward signals (i.e., less likely to produce all-correct or all-wrong rollouts), thereby improving rollout efficiency?  
4. The paper reports an increase in reasoning efficiency (shorter thinking length) after RL. Could this be because higher *n* naturally leads to longer responses, which increases the truncation ratio during RL training, thus implicitly encouraging more concise reasoning?
soundness: 3
strengths: **Strengths:**

1. The paper is well-written, clearly structured, and easy to follow.
2. It introduces meaningful long-horizon benchmarks and corresponding datasets that are valuable for advancing research on large reasoning models (LRMs).
3. The extended benchmarks effectively reveal the limitations of current LRMs in maintaining long-horizon reasoning.
4. The authors conduct reinforcement learning (RL) experiments on the constructed datasets, demonstrating notable improvements in sustained reasoning performance.
5. The paper provides comprehensive analyses and insights into the proposed data construction and evaluation methods.
summary: The paper proposes R-HORIZON, a benchmark and data construction framework for evaluating and improving long-horizon reasoning in large reasoning models (LRMs). Using Expanded Problem Composition (EPC), it links independent problems into multi-step dependency chains. Evaluating 25 LRMs across math, code, and agentic tasks, the authors find significant performance degradation as reasoning length increases. They further integrate R-HORIZON data into RL from verifiable rewards (RLVR) using Group Relative Policy Optimization (GRPO), showing that such training enhances models’ sustained reasoning ability.
weaknesses: **Weaknesses:**
1. While the proposed methods enable verifiable long-horizon reasoning evaluation, the constructed benchmarks appear somewhat artificial and may not fully reflect realistic reasoning scenarios.  
2. Similar efforts, such as *GSM-Infinity* [1], have explored benchmark construction for increasing reasoning complexity. The paper would benefit from a clearer discussion of how its approach differs from these prior works.

[1] GSM-Infinite: *How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?* (ICML 2025) https://arxiv.org/abs/2502.05252

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: None
soundness: 3
strengths: Novel and impactful evaluation paradigm.
R-HORIZON systematically measures reasoning depth and breadth, exposing weaknesses that single-horizon benchmarks cannot reveal.

Comprehensive empirical analysis.
The evaluation across 25 strong LRMs and multiple domains (math, code, agentic tasks) convincingly demonstrates consistent long-horizon degradation patterns.

Actionable training insights.
The reinforcement learning experiments with R-HORIZON data provide a scalable and cost-effective way to enhance long-horizon reasoning, with clear empirical gains and behavioral analyses (reflection, budget allocation).
summary: This paper introduces R-HORIZON, a benchmark and training framework designed to evaluate and enhance long-horizon reasoning in Large Reasoning Models (LRMs).
Existing benchmarks mainly assess single-step reasoning, where each question is independent. R-HORIZON addresses this gap by composing interdependent multi-step queries from existing datasets (e.g., MATH500, AIME24/25, LiveCodeBench, WebShaper).
The authors evaluate 25 state-of-the-art reasoning models (e.g., DeepSeek-R1, Qwen-3-235B, Gemini-2.5, Claude-Sonnet-4) and show that all suffer severe performance degradation as reasoning horizons increase.
Furthermore, they train models using R-HORIZON-based reinforcement learning (RLVR), demonstrating substantial gains on both composed and standard reasoning tasks (+17.4 on long-horizon tasks and +7.5 on AIME24).

Overall, the paper provides a new perspective on reasoning length, reflection behavior, and token budget allocation, revealing fundamental limits in current LRMs and proposing a practical solution for improvement.
weaknesses: Multi-step reasoning is indeed necessary; however, in this paper, simply combining several basic math problems offers limited reflection of contextual consistency in true multi-step reasoning. Tasks such as “1 + 2 = ?” followed by “previous answer + 4 = ?” can essentially be summarized into a single query like “1 + 2 + 4 = ?”, which weakens the intended long-horizon reasoning challenge.

Moreover, maintaining contextual consistency across multi-step reasoning is highly dependent on prompt design, yet the paper does not clearly explain how prompts are constructed or formatted for solving these composed problems.

The evaluation on closed-source models is insufficient, as only o4-mini was included.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. For the dataset construction for coding tasks, it's mentioned that _"unlike the sequential composition used for mathematical tasks, we apply a directly composed concatenation format ... without adding explicit dependencies"_. So (a) what is exactly this "directly composed concatenation format" (hard to tell simply from the given prompt in Appendix)? (b) isn't the point of R-Horizon benchmark to create interdependent, long-horizon reasoning tasks? I think the compromise made here undermines the overall validity (i.e. many of the analysis made in the paper, e.g. the "dependency reasoning error, applies to math only).
2. For the reward design, the distinction between the "last" and "all" rewards is a bit non-intuitive as I thought they should be very similar when the problems are inter-dependent. Is it even possible for a model to obtain positive "last" reward even when its "all" reward is 0 (the other direction is trivial)?
3. For **Figure 6**, how can the error positions be even larger than the output lengths (even query num is low)?
soundness: 3
strengths: - The paper is overall clearly motivated, well-written, and easy to follow.
- The lack of long-horizon, multi-stage/problem evaluation of LRMs is indeed a practical concern, and the paper's approach of using query composition to automatically combine single-turn queries within one interdependent problem is (an example of) an elegant solution.
- On top of the R-Horizon benchmark, the paper further conducts solid RLVR training on the composed dataset, which increases both single and multi-problem performance and further highlights the importance of equipping LRMs with multi-step compound reasoning skills.
summary: R-Horizon is motivated by existing LLM reasoning benchmarks' over-emphasis on _immediate, single-turn tasks_. It instead proposes a method to automatically generate _long-horizon, interdependent reasoning tasks_ via query composition, which enables it to curate the R-Horizon benchmark using existing (single-turn) datasets targetting math, coding, and agentic capabilities. The paper finds out that even the SOTA reasoning models suffer performance degradation as the number of composed queries (subproblems) increase. The paper then further suggests RLVR training over query-composed dataset, which demonstrates considerable improvement in both LRM's single- and multiple-problem performance. Extensive analysis and ablation studies provide important insights into the causes of performance degradation and why training with composed dataset works better.
weaknesses: - Even though the paper mentions that it constructs composed problems using both math, coding, and agentic datasets, most of the construction details and analysis are centering around the math criteria (see Q1 below). And the RL training is also solely done on math-related datasets.
- For the analysis on the RL training, it would be great to see similar "error type distribution" plot (as in Figure 5) for the model before & after the training. This helps gain further insights on _why & how the performance on multi-query problems_ increase over the training.
- While the writing is good overall, certain plots are hard to read due to ambiguous legends & captions. As an example, on Figure 8 the x-axis shows "Query 1/2/etc.", which I believe actually denotes the number of composed queries but can be misunderstood as query indices. Also certain plots have (unexplained) error bars/intervals while some do not.


Decision: Accept (Poster)