A State-Transition Framework for Efficient LLM Reasoning
Authors: Liang Zhang
Yu Zhao
Longyue Wang
Tianqi Shi
Weihua Luo
Kaifu Zhang
Jinsong Su
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
reasoning
efficient reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: While Long Chain-of-Thought (CoT) reasoning significantly improves Large Language Models (LLMs) performance on complex reasoning tasks, the substantial computational and memory costs of generating long CoT sequences limit their efficiency and practicality.
Existing studies usually enhance the reasoning efficiency of LLMs by compressing CoT sequences.
However, this approach conflicts with test‑time scaling, limiting the reasoning capacity of LLMs.
In this paper, we propose an efficient reasoning framework that models the reasoning process of LLMs as a state‑transition process.
Specifically, we first apply a linear attention mechanism to estimate the LLM’s reasoning state, which records the historical reasoning information from previous reasoning steps.
Then, based on the query prompt and the reasoning state, the LLM can efficiently perform the current reasoning step and update the state.
With the linear attention, each token in the current reasoning step can directly retrieve relevant historical reasoning information from the reasoning state, without explicitly attending to tokens in previous reasoning steps.
In this way, the computational complexity of attention is reduced from quadratic to linear, significantly improving the reasoning efficiency of LLMs.
In addition, we propose a state-based reasoning strategy to mitigate the over-thinking issue caused by noisy reasoning steps.
Extensive experiments across multiple datasets and model sizes demonstrate that our framework not only improves the reasoning efficiency of LLMs but also enhances their reasoning performance.

Published as a conference paper at ICLR 2026
A STATE-TRANSITION FRAMEWORK FOR EFFICIENT LLM REASONING
Liang Zhang' Yu Zhao?, Longyue Wang", Tianqi Shi?, Kaifu Zhang', Jinsong Su 1,,4
Weihua Luo?,
Schooo of Informatics, Xiamen University, China AAlibaba International Digital Commerce
Kee Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural
Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China "Shanghai Artificial Intelligence Laboratory
1 thangestu xmu edu en, jssuê xma. eew.ux.aeddu......m....m. edu.c..
(fengli, i. .y, wang I ongyue w ly, weihua, luowh Cal ibaba inc com
ABSTRACT
While Long Chain-of-Thought (CoT) reasoning significantly improves Large Lan- guage Models (LLMs) performance complex reasoning tasks, the substantial computational and memory costs of generating long CoT sequences limit their ef- ficiency and practicality. Existing studies usually enhance the reasoning efficiency LLMs by compressing CoT sequences. However, this approach conflicts with test-time scaling, limiting the reasoning capacity of LLMss. In this paper, we pro- pose an efficient reasoning framework that models the reasoning process of L.MMs state-transition process. Specifically, we first apply linear attention mecha- nism to estimate the LLM's 's reasoning state, which records the historical reasoning information from previous reasoning steps. Then, based on the query prompt and the reasoning state, the LLM can efficiently perform the current reasoning step and update the state. With the linear attention, each token in the current reason- ing step can directly retrieve relevant historical reasoning information from the reasoning state, without explicitly attending to tokens previous reasoning steps. In this way, the computational complexity of attention is reduced from quadratic to linear, significantly improving the reasoning efficiency of LLMs. In addition, we propose state-based reasoning strategy mitigate the over-thinking issue caused by noisy reasoning steps. Extensive experiments across multiple datasets and model sizes demonstrate that framework only improves the reasoning efficiency of LLMs but also enhances their reasoning performance.
INTRODUCTION
Chain-of-Thought (CoT) (Wei al. 2022 Kojima al. 2022 has become core technique for hancing the reasoning ability of large language models (LLMs) on complex tasks. Through prompt-
ing step-by-step reasoning. CoT enables L.LMs to decompose complex problems into simpler sub-
tasks, thus improving their problem-solving capabilities (Yao et al. 2023] Wang et al. 2023 Zhou
al. 2023 Wang et al. 2024). Recent studies, including OpenAl ol (OpenAI et al. 2024). QwQ
(Team] 2024), and DeepSeek-RI (DeepSeek et al. 2025). demonstrate that scaling up CoT length can further enhance the reasoning abilities of LLMs. However, since most current LLMs are built on
the Transformer architecture (Vaswani et al., 2017b), the computational complexity of their atten- tion grows quadratically with context length, and the memory overhead of their KV-cache increases linearly with context length. Hence, generating long CoT substantially increase the computational and memory cost of L.I.Ms, limiting their practical efficiency on complex reasoning tasks.
To improve the reasoning efficiency of LLMs,, previous studies employ prompting (Han et al. 2024
Ma et al., 2025a), supervised fine-tuning (SFT) (Liu et al.] 2024 Munkhbat et al.) 2025). or re- inforcement learning (RL) (Aggarwal et al.) 2025 Shen al. 2025) to encourage LLMs toward generating shorter COT sequences. However, these methods often impair the reasoning ability of
"Corresponding author.

---

Published as a conference paper at ICLR 2026
LLMs (Jin et al.
2024 Merrill et al. 2024). since CoT shortening conflicts with test-time scaling
OOpenA et al. 2024). To preserve the reasoning ability of LLMs, some studies (Ma et al. 2025b]
Kang et al., 2025; Xia et al.) 2025) express the CoT in more concise text (e.g.. by removing less important tokens or rewriting with GPT-44 to reduce its length. However, they risk losing critical reasoning information or reducing interpretability when simplifying long Co'T (Wang et al. (025c).
In this paper, we propose an efficient reasoning framework for LLMs, which models the reasoning process of LLMs as state-transition process. We regard a long CoT as sequence of reasoning steps, where LLMS perform a specific thinking pattern in each step, such as induction or reflection. Notably, each reasoning step contains two types of information: substantial linguistic information to ensure its fluency, and limited reasoning information to support subsequent reasoning or answer generation (Zhang et al.) 2025a Xia et al. 2025). Thus, our framework (Figure I first compresses the reasoning information from previously generated reasoning steps into matrix, termed as the reasoning state matrix. Then, based on the query prompt and the state matrix, LLMS can efficiently generate the current reasoning step and updates the state matrix accordingly. Specifically, tokens in the current reasoning step can directly retrieve relevant historical reasoning information from the reasoning state, without explicitly attending to tokens in previous steps. In this way, we effectively reduce both the computational complexity of attention and the memory overhead of the KV-cache. Crucially, our framework does not shorten or simplify the CoT sequences generated by LLMs, thus preserving their reasoning ability and interpretability.
To efficiently obtain the state matrix, design Mixed Attention Module (MAM) to replace the original attention module in LLMs, which consists of a Softmax-Attention (SA) submodule and Linear-Attention (LA) submodule. We use the original attention module of LLMs as our SA submodule, where each token can only attend to the tokens in the query prompt and those in its current reasoning step. the LA submodule, we adopt linear-attention mechanism (Katharopoulos et al. 2020) to capture the reasoning state of LLMs. Mcanwhile, with the linear attention, each token can directly retrieve relevant historical reasoning information from the reasoning state, Compared with other methods, such as CNN (Krizhevsky et al. (012)) and Q-Former (Li et al. (023), linear attention offers the following advantages in capturing the model's reasoning state: (1) As a variant of softmax attention, linear attention is naturally compatible with it, thereby reducing the risk of losing critical reasoning information during compression and ensuring the stability and efficiency of model training. (2) Recent studies (Yang et al. 2024b 2025d) have demonstrated that the state- update process of linear attention is essentially a gradient- gradient-descent learning procedure (i.e., test-time training), revealing its strong potential for handling complex reasoning tasks (Zhu et al. 2025).
Another challenge is that L.LMs often produce noisy reasoning steps, which may mislead subsequent reasoning steps and lead the overthinking issue (Chen et al., (025b) Yang al.. (025c). To mitigate this issue, we propose a state-based reasoning strategy. Given the model's 's state-transition process is gradient-descent process, we first apply the momentum method to accumulate gradients from completed reasoning steps, obtaining a global gradient. The global gradient indicates the global reasoning direction of LLMs. Thus, we employ to guide LLMs in completing the current reasoning step, ensuring they do not significantly deviate from the global direction.
To validate the efficacy of our framework, we conduct experiments on seven widely-used benchmark datasets. Experimental results show that our framework not only improves the reasoning efficiency of LLMs but also enhances their reasoning performance. Extensive ablation studies further demon- strate the effectivvness of various components in our framework.
2 PRELIMINARY
We first present brief background on linear attention, which lays the foundation for proposed framework. Meanwhile, we outline the specific form of the CoT sequences in our framework.
2.1 LINEAR ATTENTION
Softmax Attention (SA). Popular LLMs, such as Qwen 3 YYang et al. 2025a) and Llama 4 (Meta 2025). adopt decoder-onle Transformer architecture composed of repeated blocks of meadi-head softmax attention followed by feed-forward networks (FFNs) (Vaswani et al.] (017a). Given the input sequence X X=(x1,". [ the softmax attention can be formulated as follows:
2

---

Published as a conference paper at ICLR 2026
Token-based reasoning
Query Prompt; <think
<hink>> Final Answer
Segment
Thinking Pattem Type
Query Prompt;
tthinkk
<sp think, Vepy <ithikk>; Final Answer
State-based reasoning
Model Reasoning State
Query Prompt
think,
think;
think,
Final Answer
So
A comparison of traditional token-based reasoning with our state-based reasoning. think, denotes one reasoning step. In state-based reasoning, LLMs can efficiently generate think, only based on the query prompt and the reasoning state St-1.
Figure 1:
0: = El exp (. /vd) vi
qt. kt U2 = Wo. D2 WK, x4 We,
(1)
Σ exp 9: /va)
where Wo. WK. Wy are learnable weight matrices. The computational complexity of the soft- max function increases quadratically with the length of the context sequence. Moreover, softmax attention heavily depends the growing KV-cache recall historical information for sequence modeling, leading to substantial memory overheads, particularly in the long context setting.
Linear Attention (LA). Linear attention is variant of softmax attention, designed reduce its computational complexity and memory costs. Here, we first replaces the exponential function exp(-) in softmax attention with simpler kernel function o(-): exp(q: k d) + of ((k))((k)) Next, based on the associative property of matrix products, linear attention can be written
Ei (k.)) To U.
o(q.) Ei= TT;
O: Er= ((.k))))))))))))))))))()))
((q.) Ex= (kk))
Moreover, recent studies (Sun 2023 Yang et al. perform well even when (0.) is set to the identity function and the normalizer is removed:
2024a) have shown that linear attention can
k
Sc S
k
where S, is the state matrix storing historical information. By using S,, linear attention can perform sequence modeling in linear time while maintaining constant memory overhead.
The Test-Time Training (TTT) Perspective of LA. Recent works (Sun et al. 2024 Chen et al., 2025b) treat the state matrix S. in linear attention as a fast-adapting parameter, updated at every token via lightweight gradient descent. They further introduce an online learning objective for linear attention and provide its closed-fore form gradient descent solution:
Objective:
L(S) -Skk, vz),
SGD update: S. = S.-1 - Bu,k:
where () denotes the inner product, and /3 is the learning rate. When /3 is set to 1, the state update process of linear attention is equivalent to training the state matrix S, using the learning objective C(S). While current studies have yet to provide direct evidence that linear attention can improve the model's reasoning ability, its theoretical properties suggest significant potential (Zhu al. 2025
2.2 LONG CoT SEGMENTATION
A long CoT sample (Q. T, A) usually comprises three parts: query prompt Q, thinking sequence T. and a final answer A. Given the query prompt Q, LLMs first perform complex reasoning (T), and then generates the final answer A. During reasoning, LLMs engage in various thinking patterns (e.g., reflection and result verification) and switch between them using common transitional tokens
(e.g., "Waif", "Hmm") (Yang et al. 2025c
Wang al. 2025d). Recent work (Wang al. 2025h)
3

---

Published as a conference paper at ICLR 2026
X 4
Add & Norm
W
Linear Attention
Softmax Attention
FFN
Softmax
Add Norm
W
W
W/
W
W
Wx
W4
MAM
Figure 2: In our framework, the original softmax attention module in LLMs is replaced with our mixed attention module (MAM), thus improving reasoning efficiency and reducing memory cost.
further shows that LLMs usually transition thinking patterns at some high-entropy tokens, such as "Alternativey"" and "Maybe". Here, we refer to the completion of a thinking pattern by LLMs as a reasoning step. Notably, when performing the current reasoning step, LLMs only consider the reasoning information (e.g., conclusion) of previous completed steps without their linguistic information (e.g., grammar).
Following Wang et al. (20256), we first extract high-entropy transition tokens from the long CoT training set, which occur at the start of sentence. Next, we use these tokens to segment the thinking sequence T in each training sample into sequence of reasoning steps. Meanwhile, we cluster all reasoning steps in the entire training set, with each cluster corresponding to a thinking pattern (i.e., type). Finally, for each thinking pattern, introduce two special tokens to enclose its corresponding reasoning steps, as shown Figure T With the reconstructed training set, the trained LLMs can generate more structured thinking sequences. Meanwhile, these special tokens allow us track and control the reasoning processes of LLMs more accurately.
OUR FRAMEWORK
In this section, we provide a detailed description for our proposed framework. In our framework, we first design mixed attention module (MAM) to replace the original attention module in LLMs, il- lustrated in Figure 2 Using MAM, we can model the reasoning process of LLMs as a state-transition base- reasoning strategy in Sec- Finally, our training strategy presented in Section 3.3
process (see Section 3.1.. Then, further introduces
tion 3.2
3.1
MIXED ATTENTION MODULE
Our framework aims improve the reasoning efficiency of LLMs without sacrificing reasoning ability by modeling their reasoning process as state-transition process. To achieve this, we design MAM to replace the softmax attention module in LLMs, which consists of Softmax Attention (SA) submodule and Linear Attention (LA) submodule. To avoid the performance loss caused use the original softmax attention module of LLMs as SA submodule.
by this replacement, However, in the SA submodule, each token can only attend to the tokens in the query prompt Q and the previously generated tokens in its reasoning step. By doing so, reduce the computa- tional complexity of attention from quadratic O(C²) to linear OCC) and the memory usage of the KV-cache from linear O(C) to constant O(1), where C denotes the context length. This significantly improves the reasoning efficiency of model, especially in complex scenarios requiring longer thinking sequences. Moreover, the LA submodule applies linear attention mechanism to obtain the LLM's reasoning state matrix, which records the reasoning information from previously completed reasoning steps. Therefore, each token the current reasoning step can access relevant historical information from the state matrix without attending directly tokens in previous reasoning steps.
Given our LLM generates each token in every reasoning step using the same procedure, we take the generation of the i-th token de. in the (-th reasoning step example. Specifically, the I-th
4

---

Published as a conference paper at ICLR 2026
layer of our model, first feed 11,5-1 the output feature of the input token 21,i-1 at the (1-1)-th layer, into the MAM of the current layer. Then, 11,i-1 is passed to the two submodules in MAM:
In the SA submodule, the KV-cache retains only the KV vectors of the query prompt tokens and ones of the previously generated tokens in the current reasoning step, while those of tokens in com- pleted reasoning steps have been removed. Through the softmax attention mechanism (seen Eq. the input token -1,i- attends to tokens retained in the KV-cache, yielding an updated feature 1 h
In the LA submodule, after completing the first (-1 reasoning steps, our model state is updated to 5 Then, we use this state as the initial state - t-1 for the current reasoning step and update it token-by-token, yielding the current model state s: _ 1 S دド=
0 Next, utilize
(1) of the input token -;;1 to extract the relevant historical reasoning information the query vector q,
In the current reasoning step, the model usually relies more on historical reasoning information to generate token in the early stage, and this dependence gradually generated. Therefore, we obtain the output of this submodule via
0:-1 from _ 0:-1 1
(1)
(1) - - (
decreases more tokens
gating mechanism: (
1 =o (W (W, ((-)))
o where o() denotes the sigmoid function and W
a learnable weight (see Figure 2).
Finally, we combine the outputs of the two submodules and apply linear output layer to yield the final output of MAM: h 1 - W.
(h - As shown in Figure 2 the two submodules have identical structures, except that the LA submodule incorporates an additional gating weight W., To control its parameter size, we implement the LA submodule via the LoRA strategy (Hu al. 2022).
Subsequently, we process b using the FFN module to produce the output A ( for the /-th layer of our model. By repeating the above process L times, we obtain the final output feature n! of the input token #t-1, where L denotes the number of layers in our model. Next, N is fed into the model's prediction layer to produce the predicted distribution p(r) of the next token. Finally, our model generates the i-th token x, of the /-th inference step according x;= arg max, p(x).
Following the above procedure, our model sequentially generates tokens in the current reasoning step until reaching its end token ('Yez), which corresponds to the thinking pattern of the step (see Figure D After generating (lez), the state matrix of our model at the I-th layer is updated to s!!) which we denote as S(!)) and use as the initial state for the next reasoning step, where n, denotes the number of tokens in the i-th reasoning step. Meanwhile, we clear the KV vectors of all tokens in the
current inference step from the KV-cache.
3.2 THE STATE-BASED REASONING STRATEG
During reasoning, LLMs often produce noisy reasoning steps that may mislead subsequent ones, thus resulting in overthinking problems (Chen et al. 2025b Yang et al. 2025c). In our framework, such noisy reasoning step can deviate the model's state transitions from the correct reasoning trajec- tory. resulting in erroneous results (see Figure a)). a))). To mitigate this issue, propose state-based reasoning strategy, which guides model reasoning with a global reasoning direction.
In the reasoning process, the state transition of our model the 1-th layer can be formalized as: S +.... ... -S T where St denotes the model state after the LA submodule processes the token in the query prompt Q. Considering that the state transition process in the line attention is gradient descent process (see Section 2.1 we represent the total gradient contributed by each reasoning step as [Vi Vi,,-. | where V S - indicates the reasoning direction of the t-th step. The reasoning direction of noisy reasoning step often deviates substantially from those of other steps.
Therefore, we first aggregate the reasoning directions of all previous steps into a global reasoning direction V using momentum accumulation: V/-1= / V -ーー Σ V where V is initialized as zero matrix. Then, once the 1-th reasoning step is completed, we em- play the global direction V/_1 to correct its reasoning direction V/: (1=(1_a)V++ay) where a= max(Amax: { and T denotes the maximum number of reasoning steps (default: 40). Since
5

---

Published as a conference paper at ICLR 2026
FA
P(A,710)
Reasoning Direction
m
Add Norm
Add & Norm
S2
V1
V/2
FFN
FFN
Reasening Direction of Noise Ressoning Step
$
S/
Add Norm
Add & Norm
Refining Noisy Reasoning Direction via Giohal Ressening Direction
SAM
MAM
SV |r
X
Query Prompt ?
Quary Prompt ?
(1) The state-based reasoning strategy
(2) The training strategy
Figure 3: Illustration of our reasoning and training strategies. SAM is the softmax attention module.
more prior reasoning steps yield a more accurate global reasoning direction, we linearly increase the correction coefficient a up to predefined threshold Omax In this way, our model can fully explore diverse reasoning directions during the early stages of reasoning and then gradually converge toward the global reasoning direction. Finally, we use the corrected reasoning direction V to update the model state, S. +V
V alleviating the negative impact of the noisy step on subsequent steps.
To enhance the diversity of thinking patterns during reasoning, we apply special markers indicating thinking patterns guide the model toward adopting different thinking patterns consecutive steps. By doing so, we can further enhance the robustness and overall performance of our model.
3.3 MODEL TRAINING
We train our model using the long CoT training set constructed in Section To improve training efficiency while preserving the original reasoming ability of LLMs, we fine-tune only the parameters of the newly added LA submodulc and ones of the special tokens corresponding to thinking patterns. As shown in Figure (b). we jointly optimize our model with two loss terms: (1) the autoregressive loss CAR of our model on the training samples, and (2) the knowledge distillation loss LKD between the base model (the original LLM with softmax attention module) and our proposed model. Finally, our training objective is defined C-LAR+3LKa where denotes hyperparameter.
we first input a long CoT sample into our model to obtain the predicted probability distribution P(A, Q) over the thinking sequence T and the final answer A conditioned on the query prompt Q. To maintain the consistency between training and testing, we apply a customized mask matrix in the SA submodule to ensure that each token attends only the tokens in the query prompt A and those from its corresponding reasoning step. Next, our autoregressive loss term CAR defined as follows: LAR log P(A,TQQ)
Specifically,
Meanwhile, the same long CoT sample is fed into the base model to produce the predicted probabil- ity distribution PA Q.Q). Notably, our model is built upon the base model by replacing its soft- max attention module with our MAM. In the base model, each token attends to all previous tokens. Subsequently, we use the Kullback Leibler (KL) divergence between P(A, TIQ) and P(A.TQQ) as our distillation loss term - (Oon=KL(((AATT)) P(A, T(Q)). The LKD loss term is designed to effectively train our LA submodule for capturing global reasoning information.
EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Implementation Details. Our experiments are primarily conducted on the Qwen-2.5 series models (Hui et al., 2024). Meanwhile, we extract 95K high-quality samples from OpenRI-Math-225 to construct training set using the method described in Section 2.1 We initialize Qwen-2.5 using

---

Published as a conference paper at ICLR 2026
GSM8K
MATII-500
AMC23
AIME24
AIME25
AVG.
Method
Acct Tok ReL,
Acct
Tok ReL]
Acct
Tok ReLL Acct
Tok
ReL,
Acct
Tok ReL,
Acct
ReL.
Qwen2.5-1.55 Series
Ours (Base)
80.1 2086
76.2
78.8
3958
139.4 62.5 6392 242.9 20.0
13765
536.8
16.7
12684
504.7
51.5
300.0
+H2O
76.1 2116
71.7
75.4
3835
128.5
55.0 6230 209.7
13.3
13921
462.2
10.0
12802
428.6
46.0
260.1
+SepLLM
77.3 2230
75.8
76.0
3890
130.3 57.5 6573 220.2
13.3
13690
452.6
13.3
12690
423.9
47.5
260.6
LightThinker
78.8
2109
69.6
77.6
4060
134.8 60.5 6312 214.6
16.7
13827
461.6
13.3
12934 434.6
50.1
263.0
INFTYTHINK
79.5
2503
89.3
78.0
4750
170.2
62.5 7605 263.5
16.7
16318
566.2
16.7
15020
501.0
50.7
318.0
Ours
82.1 2116
69.7
81.2
3812
125.8
67.5 6460 213.2 26.7
13610
440.13
23,3
12910
416.0
56.2
253.0
Qwen2.5-7B Series
Ours (Base)
89.4
2649
96.4
87.4
4253
161.6
82.5
6704 242.9 40.0
12901
522.1
30.0
13204
548.1
65.9
314.2
+H20
83.5
2730 92.0
82.6
4389
147.1 77.5 6843 209.7 30.0
13274
448.9
23.3
13204
443.7
59.4
268.3
SepLLM
84.9 2582
92.3
84.2 4244
144.3 80.0 6511 220.2
33.3
13100
484.7
23.3
12972 435.9
61.1 275.5
LightThinker
85.6
2779
90.0
84.8 4321
145.2 80.0 6790 214.6
33.3
13307
449.0
26.7
13250
443.9
62.1 268.5
INFTYTHINK
87.9
3237
117.0
86.2 5050
184.8
82.5 8061 254.3
36.7
15094
508.7
30.0
15448
530.5
64.7
319.0
Ours
90.9 2603 87.2
90.0 41%6
140.6
85.0 6665 213.2 43.3 13017
434.8
33.3
13191
440.6 68.3 263.3
Table 1: The results of our model and baselines on mathematical reasoning benchmarks.
its corresponding distilled version of DeepSeek-RI (DeepSeek et al. 2025), and then train it on the training set to obtain our base model. Finally, our framework is built upon this base model.
Baselines. We compare our framework with two types of baselines: efficient reasoning methods and KV-cache reduction methods. In the first category, LightThinker (Zhang et al. (222a)) uses learn- able special tokens to compress the reasoning information of completed reasoning steps, improving the reasoning efficiency of LLMs. INFTYTHINK (aan et al., 2025) compresses the information from previous reasoning steps into a concise summary. In the second category, H2O (Zhang et al. 2023) reduces KV-cache by only retaining the KV vectors for small set of heavy-hitte tokens and recent tokens with a greedy eviction strategy. SapLLM (Chen et al. (025a)) only stores the KV vec- tors of the initial, neighboring. and separator tokens into the K KV-cache of LLMs. These baselines are trained with LoRA and use the same number of trainable parameters as our model.
Dataset & Metric. We evaluate our framework seven benchmarks, including five mathematical reasoning benchmarks (GSM8K (Cobbe et al. (2021). MATH-500 (Hendrycks et al. 2021). AMC23
AMC 2023), AIME24 (AIME 2024), AIME25 (AIME 2025)), scientific reasoning benchmark (GPQA Diamond (Rein et al. 2024), and code reasoning benchmark (HumanEval (Chen et al., 202TD. We use greedy decoding to generate the output of the target LLM. We report three metrics to assess the performance and efficiency of various methods: (1) Accuracy (Acc) denotes the percent- age of correct answers generated by the target model; (2) Token Number (Tok) refers to the average length of the CoT sequence; (3) Reasoning Latency (ReL) is defined as the average inference time per sample. Moreover, we measure reasoning latency on a single NVIDIA A100 GPU with a batch
size of 1, while enabling FlashAttention-2 (Dao 2023) with BF16 (Kalamkar et al. 2019).
4.2 MAIN RESULTS
The experimental results on mathematical benchmarks are presented in Table ! AVG. column, our framework outperforms all baselines in reasoning efficiency and attains the best overall performance. After carefully analyzing these results, we draw several conclusions:
As shown in the
First, our model significantly outperforms the base model in reasoning speed, particularly on bench- marks requiring longer COT. On the AIME24 benchmark, our model achieved 16-18% improve- ment in reasoning speed over the base model. This mainly attributed to the lower computational complexity of our MAM relative to the softmax attention module of the base model. Furthermore, our model consistently achieves superior performance over the base model across all benchmarks. These results highlight the strong potential of linear attention mechanisms for efficient reasoning.
Second, the baselines (except INFTYTHINK) and our model generate CoT sequences with similar length on each benchmark, as they are trained on the same dataset. While these baselines achieve higher reasoning efficiency than the base model, they exhibit worse reasoning performance. For in- stance, on the AIME25 benchmark, LightThinker improves reasoning efficiency by 10-12% over the base model, but its performance decreases by 3-6 points. These results suggest that naive KV-cache compression methods risk losing important reasoning information, thus limiting model performance.
7

---

Published as a conference paper at ICLR 2026
Method
GSM8K MATH-500 AMC23
AIME24
AIME25 AVG.
Ours
82.1
81.2
67.0
26.7
23.3
56.2
(l) w/o The state based reasoning strategy
79.3
78.8
65.0
20.0
20.0
52.6
(2) w/ Thinking patterns diversity
80.9
80.0
65.0
23.3
20.0
53.8
(3) w/o The distillation loss LKD
77.0
77.6
62.5
20.0
16.7
50.8
(4) w/o The model reasoning state
76.4
76.2
62.5
20.0
16.7
46.8
(5) w/ The gating mechanism
80.1
79.4
65.0
23.3
23.3
54.2
Table Ablation results of our model on mathematical reasoning benchmarks.
Third, unlike other baselines, INFTYTHINK uses text summaries of previous reasoning steps to record their reasoning information. While enhancing interpretability. this method reduces reasoning efficiency because it requires generating longer CoT sequences. Moreover, such discrete summa- rization may omit part of the implicit reasoning information within the model, leading to slight performance drop for INFTYTHINK compared the base model. However, we effectively avoid these two issues by utilizing the model's 's reasoning states to record reasoning information, enabling our model outperform INFTYTHINK both efficiency and performance.
4.3 ABLATION STUDY
We further conduct extensive ablation studies by removing different components from our frame- work to investigate their different impacts. As shown in Table 2 we only compare model with the following variants in terms of performance, as they share the same reasoning efficiency.
w/o The state-based reasoning strategy. In this variant, remove the state-based reasoning strategy from our framework. We observe an average performance drop of 3.54 points across all benchmarks for this variant. These results suggest that our reasoning strategy indeed enhance the robustness of LLMs to noisy reasoning steps, improving their overall performance. Moreover, this also indi- cates that, with suitable processing, our model states can intuitively reflect the quality of reasoning RL).
steps, which offers an interesting insight for future exploration (e.g., prooees-reward-beeee
w/o Thinking patterns diversity. We guide LLMs use different thinking patterns adjacent rea- soning steps via corresponding special tokens, thus ensuring diversity of the thinking patterns in the reasoning process. To test effectiveness, we remove this operation from our framework. As shown in Line (2), this variant exhibits a consistent decline in performance across all benchmarks. These findings imply that more diverse thinking patterns can enhance LLMs exploration ability, thereby improving the accuracy of their generated answers.
The distillation loss LKD- As described Section 3.3 we use LKD to train our LA submodule so that it can better capture global reasoning information. In this variant, LKD is excluded from training objective. From Line (3) in Table 2 of 3.6 3.6-6.6 points across all benchmarks. The main reason is that the autoregressive loss CAR may struggle to train the LA submodule to capture the global correlations between reasoning steps.
we note that this variant results in a performance drop
w/o The model reasoning state. In this variant, replace the model states in framework with sliding window of size 64 to store historical reasoning information. Although this approach is also compatible with the original softmax attention in LLMs, this variant yields the most significant per- formance drop compared with other variants. This mainly because this approach lacks reasoning capability as strong that of linear attention.
w/o The gating mechanism. This variant removes the gating mechanism from our LA submodule and directly sums the outputs of the two submodules in the MAM. This change leads to slight per- formance drop across all benchmarks. This confirms that our gating mechanism can more precisely control the amount of historical reasoning information each token requires.
4.4 FURTIER ANALYSIS
Analysis of Computational and Memory Costs. We conduct experiments to further compare the computational and memory efficiency of our model and the base model across varying CoT lengths. The experimental results are presented in Figure T a). Although our model exhibits similar reasoning efficiency to the base model for shorter CoT, it significantly surpasses the base model once
8

---

Published as a conference paper at ICLR 2026
60-
40
GPU Memory Usage (GB)
-5-
Oge Maneey
(herhauu Mensory
-30
Accuracy (%)
-0-
MATH-500
Accuracy
Latency
45-
20
CSMSK
OSMRK
10
35
0.6
0.7
0.8
0.7
Decoding
Length
8
(0)
4max
Figure 4: (a) shows the computational and memory efficiency of model and the base model. (b) and (c) present our model's performance with different values of hyper-parameters, 3 and O'max respectively. These experiments conducted Qwen2.5-1.5B.
the CoT length exceeds 4K. In particular, when the CoT length reaches 32K, our model achieves over 40% faster reasoning speed than the base model. Moreover, our model maintains a nearly constant memory usage across varying CoT lengths, whereas that of the base model increases linearly with CoT length. Theoretically, our model's advantages in computational and memory efficiency would become even significant when FlashAttention-2 is disabled.
Analysis of Hyper-Parammterrs We also investigate the impact of the two key and max: the performance of our model. As illustrated in Figure (c)- model exhibits low sensitivity to these two hyper-parameter when and Omax are set 0.2 and 0.4, respectively. We further analyze the choice of these two hyperparameter values as follows:
Meanwhile, our model attains the best performance
3 denotes the weight of the distillation loss term LKD. Setting 3 to an excessively small value (e.g.. 0.1) prevents our linear attention module from effectively capturing global rea- soning information, whereas an overly large value (e.g., 0.8) limits the potential of our model to outperform the base model.
In the state-based reasoning strategy, a determines the magnitude of correction applied to the reasoning direction of noisy reasoning steps. Hence, setting too low (e.g., 0.1) increases the model's 's sensitivity to noisy reasoning steps, whereas setting it too high (e.g.. 0.8) constrains the exploration capabilities of LLMs.
Moreover, within an appropriate range (c.g., 0.2-0.6), our model's performance insen- sitive to these hyperparameters.
2025 Aggarwal Analysis of More Baselines. Recently, many studies Cheng al. 2025 Wu et al.
& Welleck 2025 Li et al. 2025a Fatemi et al.) 2025 Li et al. [2025b] Lan et al. 2025b Lin et al.
2025 Yuan al. 2025 Dai et al. 2025a Xiang al. 2025 Qi et al. 2025a Zhang al. 2025b
Wang et al., 2025a Yong et al.] | Zhao et al. 2025] Chen et al.) 2025cl Qi et al. 2025b Yang al. 2025b) have focused on shortening the length of CoT to improve the reasoning efficiency of LLMs. However, our method aims to reduce both the computational and storage complexity of attention in LLMs, leading faster token generation. This indicates that our method improves the reasoning efficiency of LLMs regardless of the CoT length. Therefore, these baselines are both orthogonal and compatible with our method. Here, we select four open-source baselines as exam- ples to verify the compatibility of our method with these baselines: Task Arithmetic-baset Model Merging(TAMM) (Wu et al. 2025), SBT-D (Zhao et al. (025). I.1Maxx (Aggarwal & Welleck 2025). and enhances the reasoning efficiency these baselines by accelcrating the token generation speed of LLMs. Furthermore, RL-based baselines (e.g., AR) provide stronger base models for our method, thus further raising its performance upper bound.
(AR) (Qi al. 2025b). As shown in Table 3 our method further
In Appendix A and the gating mechanism within our LA submodule. In Appendix we present a more intuitive explanation of our framework and further examine its potential.
further analyze both the domain generalization capability of our framework

---

Published as a conference paper at ICLR 2026
GSM8K
MATII-500
AMC23
AIME24
AIME25
AVG.
Method
Acct Tok ReL. Acct
Tok ReLi
Acct
Tok
ReL1 Acct
Tok
ReL|
Accf
Tok
ReL.
Acct ReL.
Qwen2.5-1.55 Series
Ours (Base)
80.1
2086
76.2
78.8
3958
139.4
62.5 6392 242.9 20.0
13765
536.8
16.7
12684
504.7
51.5
300.0
+TAMM
78.0
1212
43.4
73.2
2265
82.1
55.0
2924 105.6
16.7
5702
215.8
16.7
5010
189.0
47.9
127.2
+SBT-D
77.8
1170
43.1
72.8
2105
76.7
57.5
3215 114.9
16.7
6409
246.4
16.7
6173
235.5
48.3
143.3
Ours
82.1
2116
69.7
81.2
3812
125.8
67.5 6460
213.2
26.7
13610
440.13
23.3
12910
416.0
56.2
253.0
1TAM
82.1
2116
69.7
81.2
3812
125.8
67.5
6460
213.2
26.7
13610
440.13
23.3
12910
416.0
56.2
253.0
SBBTD
79.9
1081
36.2
76.8
2110
70.3
62.5
3193
101.4
23.3
6458
212.6
20.0
6106
201.5
52.1
124.4
Token Budger
2k
4k
6k
13k
12k
Basc+LI-Mar 80.5
2077
77.4
79.0
4091
144.8
62.5
6103
233.5
20.0
13108
514.5
16.7
11904
475.5
51.7
289.1
Base+AR
79.8
2061
75.8
79.4
4083
143.6
62.5
6097
232.0
23.3
13108
514.5
20.0
12088
483.8
53.0
289.9
Ours Ours+LI-Mar LL-Maax 82.7
2103
69.5
81.4
3962
129.1
65.0
6025
199.2
26.7
12980
415.2
23.3
11871
382.0
55.8
239.0
Ours AR
82.7
2084
68.6
82.0
4079
131.2
67.5
6101
200.5
30.0
12980
415.2
26.7
12101
389.3
57.8
240.9
Token Budget
1k
2k
3k
6.5k
6k
Base LI-Max
77.8
1161
41.6
76.8
2171
80.3
57.5
3206
124.6
16.7
6606
260.7
16.7
6075
230.8
49.1
147.6
Base+AAR
78.8
1105
39.8
78.2
2075
77.8
60.0
3112
124.7
20.0
6587
260.3
16.7
6094
231.6
50.7
146.8
Muis+-i-Max
79.4
1130
36.3
79.2
2066
68.6
62.5
3147
103.1
23.3
6691
208.0
23.3
6030
198.1
53.5
122.8
Ours AA
81.9
1092
35.1
80.4
2107
69.1
65.0
3107
101.6
26.7
6603
207.4
23.3
6088
198.8
55.5
122.4
Table 3: The results of our model combined with baselines on mathematical reasoning benchmarks.
5 RELATED WORK
Efficient Reasoning, Although long CoT enhances the reasoning ability of LLMs, it introduces sig- nificant computational overhead. To alleviate this challenge, existing methods for CoT compression can be roughly divided into three categories. The first category leverages prompting (Han et al.,
2024 Ma et al.] 2025a), supervised fine-tuning (SFT) (Liu et al. [024] Munkhbaat et al.) 2025), and reinforcement learning (RL) (Aggarwal al.] 2025 Shen et al. 2025 Dai et al. 2025b) to encour- age LLMs to generate shorter CoT sequences. Since such CoT reductions conflict with test-time scaling (OpenAl et al., (2024), these methods often impair the reasoning ability of LLMs (Jin et al.) 2024 Merrill et al. (002)) The second category compresses information from multiple text tokens into single hidden vector, converting longer discrete CoT sequences into shorter continuous ones to improve the reasoning efficiency of L.LMs (Hao et al., [0224] Su et al. 2025). However, such methods typically require altering the input/output patterns of LLMs, increasing their vulnerability
to catastrophic forgetting and limiting their performance. The third category expresses the CoT in more concise text reduce its length while preserving the reasoning ability of LLMs (Ma al.)
2025b Kang et al. 2025 Xia et al. 20255. Nevertheless, when simplifying long CoT, these methods risk losing critical reasoning information or reducing interpretability (Wang et al., (025c).
Linear Attention. To address the quadratic computational cost of softmax attention (Zhang et al., 2018 2022). many linear attention methods have been proposed to improve training and inference efficiency. Early studies investigated kernel functions for linear attention (Kasai et al. 2021 Peng al. 202T). Subsequently, some studies introduced gating mechanisms into linear attention to more
effectively control the information in the model state (Yang et al.) [202aa] Qin et al. 2024 Lan al.)
2025a). Recent studies has provided theoretical evidence for the TTT property of linear attention
(Yang et al., 2024b, (un et al. 2024 Liu et al., 2025 Chen et al., 2025b). They focus exploring online learning objectives for linear attention to enhance its sequence modeling capability.
6 CONCLUSION
In this paper, we propose an efficient reasoning framework that models the reasoning process of L.MSs as a transirrnn process to enhance their reasoning efficiency. our framework, we de- sign an MAM to replace the original attention module in LLMs. The MAM uses linear attention mechanism to capture the model's reasoning state that stores reasoning information from previous reasoning steps. By utilizing the model state, we can significantly reduce the computational com- plexity and memory cost of attention, thereby improving the reasoning efficiency of LLMs. Further- more, we design a state- based reasoning strategy to avoid the overthinking issue caused by noisy reasoning steps. Extensive experiments across multiple benchmarks and model sizes demonstrate the efficiency and robustness of our framework.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
The project was supported by National Key R&D Program of China (No. 20222D0160501,, Natural Science Foundation of Fujian Province of China (No. 2024J011001), and the Public Technology Service Platform Project of Xiamen (No.35027202210049) We also thank the reviewers for their insightful comments.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no human subjects or animal exper- imentation was involved. All datasets used, including OpenRI-Math-220K, GSM8K, MATH-500, AMC23, AIME24, AIME25, GPQA Diamond, and HumanEval, were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care avoid any biases or discriminatory outcomes in our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made every effort ensure that the results presented in this paper are reproducible. The ex- perimental setup, including training steps, model configurations, and hardware details, is described detail in the paper. Additionally, we use publicly available datasets, such as OpenRI-Math-291 GSM8K, MATH-500, and HumanEval, to ensure consistent and reproducible evaluation results. We believe these measures will enable other researchers to reproduce our work and further advance the field.
LLM USAGE
In the preparation of this paper, we use Large Language Models (LLMs) solely to aid in writing and polishing the text, including improving clarity, grammar, and readability. LLMs are not used for generating scientific content, experimental design, analysis, or conclusions. All technical ideas, experiments, and results reported in this paper are entirely the work of the authors.
REFERENCES
Pranjal Aggarwal and Sean Welleck.
Controlling how long reasoning model thinks with reinforcement learning In Proceedings of COLM, 2025.
Ll:
Pranjal Aggarwal, Sean Welleck, Pranjal Aggarwal, and Sean Welleck, LI: Controlling how long reasoning model thinks with reinforcement learning 2503 2506.79633.0 2025.
AIME. American invitational mathematics examination (aime) aime 2024-i & ii, 2024. URL
-1-JiaaAAMM-- _024
AIME, American invitational mathematics exammmmiaattcseeexxx.atatiaiii (aime) 2025-i ii 2025. URL https:
h/huggiingfacc..coo http:///wwc..eeeeaatttatatp//nnn
AMC. American mathematics competitionnsns 2023. URL http////ttts///ttfoooooleee/ooo/ Soard__Soouooooo utions
com/ com/wwki/iinen /tt
Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo
Li, Weiyang Liu, and Chao Huang. SSepllm: Accelerate large language models by compressing one segment into one separator In Proceedings of ICML, 2025a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code, arXiv:2107.03372. 2021.

---

Published as a conference paper at ICLR 2026
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-kike lims In Proceedings of ICML, 2025b.
Zigeng Chen, Xinyin Ma, Gongfan Fang, Ruonan Yu, and Xinchao Wang. VeriThinker: Learning to Verity Makes Reasoning Model Efficient arXiv:2505. 174441,505./791,, 2025c.
Xiaoxue Cheng, Junyi Li, Zhenduo Zhang, Xinyu Tang, Wayne Xin Zhao, Xinyu Kong, and Zhiqiang Zhang. Incentivizing Dual Process Thinking for Efficient Large Language Model Rea- soning 16X1v2550..66315, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems arXiv:2110.. 14168, 2021.
Muzhi Dai, Shixuan Liu, and Qingyi Si. Stable Reinforcement Learning for Efficient Reasoning arXiv:2505./8086, 2025a.
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models arXiv:2505.07686. 2025b.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning vrXiv:2307.0869) 2023.
DeepSeck, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepscek-rl: Incentivizing reasoning capability in [lms via reinforcement learning arXiv:2501.12948, 2025.
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning arXiv:2504.05/S0 2025.
Tingxu Han, Zhenting Wang, Chumrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware Ilm reasoning arXiv:2412.18547, 2024.
Shibo Hao, Sainbayar Sukhbuatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space, arXiv:2412.06769 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. MMasuring mathematical problem solving with the math dataset In Pro- ceedings of NIPS, 202..
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of ICLR, 2022.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu,, et al. Qwen2.5 Technical Report 1rXiv:2409./2186 2024.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models In Proceedings of ACL, 2024.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das. Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of BFLOATI6 for deep learning training arXiv: /905. /2322, 2019.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of-thought without compromising effectiveness In Proceedings of AAAI, 2025.
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A Smith. Finetuning pretrained transformers into rnns In Proceedings of EMNLP, 2021.
12

---

Published as a conference paper at ICLR 2026
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention In Proceedings of ICML, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large
language models are zero-shot reasoners] In Proceedings of NIPS, 2022.
Alex Krizhevsky, Ilya Sutskever. and Geoffrey Hinton. Imagenet classification with deep convo- ]utional neural networks In Proceedings of NIPS, 2012.
Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, and Yu Cheng. Liger: Linearizing Large Language
Models to Gated Recurrent Structures In Proceedings of ICML,, 2025a.
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. UME-RI: Exploring
Reasoning-Driven Generative Multimodal Embeddings arXiv:251 2025b.
Chen Li, Nazhou Liu, and Kai Yang. Adaptive group policy optimization: Towards stable training and token-efficient reasoning arXiv:2503.15952, 2025a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models) In Proceedings of ICML., 2023.
Zheng Li. Qingxiu Dong, Jingyuan Ma. Di Zhang, Kai Jia, and Zhifang Sui. Selfbudgeter: Adaptive token allocation for efficient IIm reasoning WxXiv:2505.1227 2025b.
Junhong Lin, Xinyue Zeng, Jie Zhu, Song Wang, Julian Shun, Jun Wu, and Dawei Zhou. Plan and Budget: Effective and Efficient Test-Time Scaling on Large Language Model Reasoning (rXiv:2505.16122 2025.
Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners In Proceedings of ICLR, 2025.
Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn to skip steps? In Proceedings of NIPS, 2024.
Wenjie Ma. Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking 0r88582504.09888 2025a.
Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length compressible chain ofthouh tuning, urXiv:2502.0960) 2025b.
William Merrill, Ashish Sabharwal, William Merrill, and Ashish Sabharwal. The expressive power of transformers with chain of thought, In Proceedings of ICLR, 2024.
AI Meta. The Ilama 4 herd: The beginning of a new era of natively multimodal ai innovation checked on, 2025.
https://ai. meta.
Tergel Munkhbat, M Ho, Sco Hyun Kim, Yongjin Yang, Yujin Kim, and Se- Young Yun. Self- training elicits concise reasoning in large language models) 22Xiv:2502.20122, 2025.
Aaron OpenAl, Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card arXiv:2412.16720, 2024.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention In Proceedings of ICLR, 2021.
Penghui Qi, Zichen Liu, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization) arXiv:2505. 13438, 2025a.
Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization) arXiv:2505. 13438, 2025b.
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear mns with state expansion In Proceedings of COLM, 2024.
13

---

Published as a conference paper at ICLR 2026
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. GGqa: A graduate-level google-proof q&a bench- mark In Proceedings of COLM, 2024.
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. Dast: Difficulty-ddaptive slow-thinking for large reason- ing models arXiv:2503.04472, 2025.
Dilia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinging Zheng. Token as- sorted: Mixing latent and text tokens for improved language model reasoning arXiv:2502.03275, 2025.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang. Sanmi Koyejo, et al. [Learning to (learn test time): Rnns with expressive hidden states arXiv:2407.046220 2024.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, arXiv:2307.0862/1 2023.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need In Proceedings of NIPS. 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need In Proceedings of NIPS, 2017b.
Ante Wang, Linfeng Song, Ye Tian, Baolin Peng. Lifeng Jin, Haitao Mi, Jinsong Su, and Dong Yu. Self-consistency boosts calibration for math reasoning, arXiv preprint Mxiv::2033.08499 2024.
Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. LiteSearch: Efficient Tree Search with Dynamic Exploration Budget for Math Reasoning In Proceedings of AAAI, 2025a.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. [eyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning ar arXiv:2506. 2593099900 2025b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Seeff consistesst improves chain of thought reasoning in language models, In Proceedings of ICLR, 2023.
Yibo Wang, Li Shen, Huanjin Yao, Tiansheng Huang, Rui Liu, Naigiang Tan, Jiaxing Huang, Kai Zhang, and Dacheng Tao. RI-Comp:sss: Long Chain-of- Thought Compression via Chunk Com- pression and Search, ar05::2505. 16838, 2025c.
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He. Linfeng Song. Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of 2025d.
ol-like llms) arXiv:2501.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougle prompting elicits reasoning in large language models In Proceed- ings of NIPS, 2022.
Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, and Mingxuan Yuan. Unlocking efficient long-to-short Ilm reasoning with model merging arXiv preprint arXiv:2503.20641, 2025.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in Ilms, arXiv:2502. /2067, 2025.
Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile. Sang Truong, Chelsea Finn, and Nick Haber, Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning) arXiv preprint 2rxiv2250.00226, 2025.
14

---

Published as a conference paper at ICLR 2026
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Inftythink: Breaking the length limits of long-context reasoning in large language models
arXiv:2503.06692, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report 250.002005.09388, 2025a.
Chenxu Yang. Qingyi Si, Mz Dai, Dingyu Yao, Mingyu Zheng, Minghui Chen, Zheng Lin, and Weiping Wang. Test-time Prompt Intervention, arXiv preprint arXiv:2508.025/1 2025b.
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. Dynamic Early Exit in Reasoning Models, arXiv:2504. 15895, 2025c.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-effhciet training In Proceedings of ICML, 2024a.
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transform- ers with the delta rule over sequence length In Proceedings of NIPS, 2024b.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule In Proceedings of ICLR, 2025d.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models] In Pro- ceedings of NIPS, 2023.
Xixian Yong. Xiao Zhou, Yingying Zhang, Jinlin Li, Yefeng Zheng, and Xian Wu, Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Thhoree Lens arXiv:2505.1825: 18237, 2025.
Hang Yuan, Bin Yu, Haotian Li, Shijum Yang, Christina Dan Wang. Zhou Yu, Xueyin Xu, Weizhen Qi, and Kai Chen. Not All Tokens Are What You Need In Thinking arXiv:2505. 17827, 2025.
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network In Proceedings of ACL, 2018.
Biao Zhang, Deyi Xiong, Yubin Ge, Junfeng Yao, Hao Yue, and Jinsong Su. Aan+: Generalized average attention network for accelerating neural transformer Journal of Artificial Intellignnce Research, 2022.
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression drXiv:2502.15589 2025a,.
Xuechen Zhang, Zijian Huang, Chenshun Ni, Ziyang Xiong, Jiasi Chen, and Samet Oymak. Making small language models efficient reasoners: Intervention, supervision, reinforcement arXiv:2505.07961 2025b.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models In Proceedings of NIPS, 2023.
Haoran Zhao, Yuchen Yan, Yongliang Shen, Haolei Xu, Wenqi Zhang, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, and Yueting Zhuang. Let LLMs Break Free from Overthinking via Self- Braking Tuning 2505 14604, 2025.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models) In Proceedings of ICLR, 2023.
Rui-Jie Zhu, Tianhao Peng, Tianhao Cheng, Xingwei Qu, Jinfa Huang. Dawei Zhu, Hao Wang, Kai- wen Xue, Xuanliang Zhang, Yong Shan, et al. A survey on latent reasoning. urXiv:2507.0620) 2025.
15

---

Published as a conference paper at ICLR 2026
100
Ours Our(base)
I20 SeUULM
LightThinber INFTYTHINK
Gate Scoce in The LA Submodule
50
Accuracy
0.4
Score
0.4
02
00
GPOA diamoud
HrouzEval
(b)
Relative Position
Figure 5: (a) shows the performance of our model and the baselines on the GPQA Diamond and HvaanEval benchmarks. (h) illustrates the average gating scores in the LA submodule for tokens different positions within each reasoning step. These experiments are conducted on Qwen2.5-7B.
A FURTHER ANALYSIS
our training set contains only mathe- matical reasoning data. To examine the domain generalization of framework, we further test our model on the scientific reasoning benchmark GPQA Diamond and the code reasoning benchmark HumanFval. As shown in Figure(5(a) our model consistently outperforms all baselines on these two benchmarks. These results reveal several advantages of framework: (I) our training strategy ef- fectively prevents the negative impact of target-domain training on the base model's performance on other domains; (2) our framework can effectively transfer the performance and efficiency benefits in the target domain to other domains.
Analysis of Domain Generalization. As stated in Section 4.1
Analysis of Gating Scores. Here, further investigate how much historical reasoning information required by tokens at different positions within each reasoning step. To achieve this, we calculate the average gating scores of these tokens in the LA submodule. As illustrated in Figure ()), tokens occurring earlier in a reasoning step generally require more historical reasoning information from the model state than those occurring later. The main reason is that later tokens can directly obtain more information from earlier tokens via the SA sublayer, reducing their reliance on the model state.
B IMPLEMENTATION DETAILS
We train both our model and the baselines for epochs the training set, with batch size of 32. The learning rate is set to 4, with a warmup ratio of 0.03. The learning rate follows cosine decay schedule reach zero. We set the learnable LoRA parameters for both our model and the baselines to approximately 66.5M on Qwen2.5-1.5B and 147.3M on Qwen2.5-7B, respectively. To control training cost, we limit the maximum length long CoT samples the training set 16,384 tokens. Meanwhile, we train our model and the baselines on 8 NVIDIA A100 GPUs, each with 80GB of memory. Further implementation details are provided below:
First, minimize conflicts in thinking types, we employ K-means clustering algorithm with 128 clusters to annotate the thinking type of each step. larger number of clusters effectively reduce the likelihood of assigning reasoning steps with different thinking patterns to the same cluster. Notably, given the large number of reasoning steps (1.4M), we utilize MiniBatch K-Means clustering improve clustering efficiency. Meanwhile, We use cosine similarity as the distance metric, resulting in balanced distribution of reasoning steps across clusters (max: 16. kk; min: 4.3k).
Second, adopt the LoRA strategy to implement our linear attention module, where the rank r is used to control the number of learnable parameters. Specifically, we set the rank of W Ws. W. to 512, and the rank of W, 16.
16

---

Published as a conference paper at ICLR 2026
Comp: (
Token-based reasoning
Query Prompl; <think>
<tthnnk>; Final Answer
Msm: (
Thinking Pattern Type
Attention Matrices
Query Prompl; <think><s;
wht.t hempiikk-n-eee----tiiiei Final Answer
Comp.: ((L)
Mem. 0(1)
State-based reasoning
Model Reasoning State
Query Prompt
think-1
think-2
think-n
Final Answer
Attention Matrices
Figure 6: A comparison of traditional token-based reasoning with our state state-based reasoning. think; denotes one reasoning step. Comp. and Mem. represent the computational and memory complexity, respectively, with I. indicating the context length. In state-based reasoning. LLMs can efficiently generate think, only based on the query prompt and the reasoning state St-1-
Third, in linear attention, the dimension d of the model state matrix S. S;=St-1 paxd corresponds to the dimensionality of the key and value vectors in LLMs, which is determined by the model architecture itself.
Third only the newly introduced parameters (the linear attention module and special tokens) are trained, the training cost of our model reduced to 47.7% of that of the base model. Concretely, with 8 NVIDIA A100 GPUs and the Qwen-2.5-1.55 model config- uration, one training epoch requires 8.6 hours for the base model and 4.1 hours for our model.
C FURTHER ANALYSIS OF OUR FRAMEWORK
In Figure 6 we present a detailed comparison the reasoning processes between our model (state- based reasoning) and naive LLMs (token-basee reasoning). In the case of naive LLM, we provide with a query, after which it autoregressively generates the full CoT, where each token attends to all previously generated tokens. Meanwhile, the key and value vectors of all generated tokens must be stored in the KV-cache. Therefore, under this setting, the attention computational complexity scales quadratically with the CoT length L (i.e., O()))), while the memory complexity scales linearly (i.e., (L))).
In our framework, LLMs generate the entire CoT step by step. in units of reasoning steps: think; + ... + think,. Mcanwhile, we adopt linear attention mechanism to estimate the state of L.LMs during the reasoning process. As showed in Figure 6 (bottom), as LLM reasoning progresses (think) - ... - think,). its state undergoes continuous transitions (Sp - ... - S.)- Therefore, in our framework, the LLM's 's reasoning process is modeled Importantly, the state of our model performs two key roles:
transsttniii process.
First, the state records/stores the historical reasoning information of the reasoning steps that have been completed. Hence, our model can generate the next reasoning step (think;) based solely on the current state (S,-1) and the query, without depending on any previously completed steps (think
this way, we reduce the computational complexity of atten- tion in L.MSs to linear, (.L). Since previously finished reasoning steps are not needed for subsequent reasoning, the key and value vectors of the tokens in these steps are removed from the KV-cache. As result, the storage complexity of attention in LLMs is reduced to (1))
Second, the model state can be interpreted as a quantitative representation of the LLM's knowledge. Consequently, in our state-based reasoning strategy, correct the reasoning direction of noisy steps to prevent them from contaminating the LLMs' knowledge (state), thus improving the reasoning performance. Moreover, ing model states to unsupervisedly estimate process rewards for reasoning steps, enhancing the efficiency and effectiveness of L.MMs in RL training.
conducting research into
17

---

Published as a conference paper at ICLR 2026
MALE-S0C GSMIK
AMECS
-3
MATH S00
175
OSMBK
AMET1
GSMRK
ANCCS
100%, 90%
90%
70%
50%
50%
10% 45%
30%
20%
Figure 7: (a) and (b) show the performance and reasoning latency of our model the fraction of reasoning steps removed. (c) illustrates our model's 's performance after removing varying proportions of training samples. These experiments are conducted on Qwen2.5-1.5B.
Here, we also conducted an analytical experiment further validate the potential of our model state. Specifically, for each sample in the training set, the process reward of every reasoning step estimated by computing the cosine of the angle between the global reasoning direction V and the step's reasoning direction V, (i.e.. cos( VV V.)). Next, we remove reasoning steps with low process rewards from each sample to shorten the CoT length. Finally, we present the experimental results in Figure 7 We observe that removing less than 20% of the steps resulted in performance improvement for our model. This indicates that such simple process rewards can reflect the quality of reasoning steps. Moreover, when more than 25% of the steps are removed, our model's performance begins to decline. This may be attributed to the removal of critical steps in the CoT, which limits the model's 's reasoning capacity. Meanwhile, shortening the CoT length further improves the reasoning efficiency of our model.
In addition, further the data efficiency of our method. Specifically, we first calculate the global process reward a training sample the average of the process rewards from its reasoning steps. The global process reward reflects the extent to which a sample contains noisy reasoning steps. Then, we discard training samples with low global process rewards and retrain our model. As shown in Figure (7),, the performance of our model does not degrade when 60% of high-qualit training samples are retained. Meanwhile, removing 60% of the training samples (i.e., retaining 40 %) results in only slight performance reduction.
model state to estimate training- rrining-ssmmlle sample quality, thereby improving
D GENERALIZATION ANALYSIS OF THE COT SEGMENTATION METHOD
Indeed, our long CoT segmentation method is also general framework. The framework includes two primary operations:
Segmentation, which aims to segment long CoT into sequence of reasoning steps.
Annotation, focused on identifying the thinking type underlying each reasoning step.
Meanwhile, it adheres to two core principles:
First, segmented reasoning steps should maintain low linguistic dependency, mitigating information loss caused by KV-cache cleaning.
Second, conflicts among thinking types should be minimized to avoid detrimental effects from type incompatibllities.
The primary objective of the framework is to enhance the controllability of LLM reasoning pro- cesses. In this setting, we utilize special tokens, associated with the thinking types, to monitor and control the reasoning processes of L.Ms..
With sufficient resources, CoТ segmentation and thinking-tyre tetiinggtppeeeao annotation be conducted man- ually, yielding superior performance. An alternative viable approach is to employ powerful LLMs, such GPT-4o, to carry segmentation and annotation.
18

---

Published as a conference paper at ICLR 2026
Method
MATH-500
AMC23
AIME24
Qwen2.5-1.58 Series
Greedy Decoding
Acet
Tok
RC.rate]
Acct
Tok
RC.rate]
Acc7
Tok
RC_rate|
Ours (Base)
78.8
3958
0.31
62.5
6392
0.40
20.0
13765
0.50
Ours
81.2
3812
0.18
67.5
6460
0.23
26.7
13610
0.31
Sampling Decoding
pass @ lt
Tok
RC.rate]
pass@
Tok
RC.rate]
pass @
Tok
RC.ratel
Ours (Base) Ours
80.3
3780
0.19
63.6
6607
0.28
22.5
13880
0.26
83.5
3652
0.06
69.4
6533
0.08
29.1
13809
0.09
Table 4: The results of our model and baselines under different decoding strategies. RC_rate denotes the proportion of the question for which the model failed to reach the final answer due to repeated generation out of all incorrect questions.
GSM8K
MATH-500
AMC23
AIME24
AIME25
AVG.
Method
Acct
Tok ReI.j Acct ReL. Acet Tok ReI. Acct
Tok Rel., Accf
Tok ReI.j. Acct ReI.j
Qwen2.5-7B Series (B = 0.4, a 0.3)
Ours (Base) 89.4 2649 96.4 87.4 4253 161.6 82.5 6704 242.9 40.0 12901 522.1 30.0 13204 548.1 65.9 314.2
Ours
91.6 2580 86.4 90.4 4244 142.2 87.5 6602 211.7 43.3 12730 432.0 36.7 13310 442.2 69.9 262.9
Table 5: The results of our model and baselines on mathematical reasoning benchmarks.
Given resource limitations, we adopt more economical strategy: (1) segmentation is performed using high-entropy transition tokens appearing at the beginning of sentences, ensuring that each reasoning step remains semantically independent: and (2) we conduct thinking-type annotation via a clustering algorithm (K-means), while configuring it with a large number of clusters (128) to minimize inter-type conflicts. Nevertheless, our model achieves outstanding performance in the math domain while generalizing effectively to both scientific and code domains.
Furthermore, the details of the clustering algorithm we used are as follows:
Given the large number of reasoning steps (1.4M), we utilize MiniBatch K-Means cluster- ing to improve clustering efficiency..
We used larger number of clusters (128) to effectively reduce the probability of grouping reasoning steps with distinct thinking patterns into the same cluster.
We use cosine similarity as the distance metric, resulting in a balanced distribution of rea- soning steps across clusters (max: 16.1k; min: 4.3k). This effectively reduces the risk of LLMs "over-fitting" or funder-fitting" to thinking patterns.
E FURTHER ANALYSIS OF MODEL PERFORMANCE
Analysis of Decoding Strategies. As observed in DeepSeek-RI DeepSeek et al. 20025), temperature- baared sampling strategy effectively reduce the proportion of repeated generation, thus improving the reasoning performance LLMs. Here, we adopted the DeepSeek- R1 sampling strategy ((emperature=0.6, top-k=0.95, run 32 times) and presented pass @ in Table We observed performance improvements in both our model and the base model. The primary reason is that this sampling method significantly reduces the proportion of repeated generation (i.e., RC rate). To avoid overestimating the performance of our model and affecting the reproducibility of our work, adopted more deterministic greedy decoding method in the main experiment.
Analysis of Performance- PerfrmaaneeImprooeemnn Magnitude. Notably, the performance gains achieved by our model non-trivial:
First, in the Swven-255-1.55 scenario, model achieves average improvement of 4.7
points over the base model.
Second, although hyperparameters were selected for Qwen Qwen-2.5-1.5B, our model delivers average gain of 2.4 points over the hase model in the Qwen-2.5-7B configuration. Nat-
19

---

Published as a conference paper at ICLR 2026
urally, conducting more targeted hyperparameter tuning will further yield greater perfor- mance gains (4.0 points) for our model on Qwen-2.5-7B (See Table
Third, the main goal of our work to improve the reasoning efficiency of LLMs, while performance gains are an unexpected surprise.
F CASE STUDY
we present two case studies to illustrate our model's reasoning process. We note that both cases involve 8 and 9 distinct types of thinking, respectively. Moreover, the two cases also share several thinking types:
In Figure 8 and 9
type_6>: problem understanding.
<type >5>> problem interpretation.
<ypp 61 algebraic solution.
<tpe. 12>: solution verification.
We obtain the interpretations for these thinking types using GPT-4o. Moreover, observe that many reasoning types are general- and not restricted to the mathematical domain, such as <type 6>, <type 25>, and <typp 12>, This also indirectly illustrates the domainngeneralization capability of our framework.
20

---

Published as a conference paper at ICLR 2026
Question:
Task B-I....nnnA ship traveling along a river has covered $24 Vmathrm (-km) upstream and $28 \mmathm (-km)5 downstream. For this journey, it took half an hour less than for traveling S30 \mmathmm-kmm upstream and S21 limathe (-km)S downstream, or half an hour more than for traveling $15 Nmathhmm--kn upstream and $42 Nmmtthrm (-km]S S downstream, assuming that both the ship and the river move uniformly.'ni wtwpmmlly////////// Deneeeemmne the speed of the ship in still water and the speed of the river.
Reasoning sequence
<think> <type 6> Okay, so need to find the speed of the ship in still water and the speed of the river. Let me start by recalling that when ship is moving upstream, its effective speed is the speed of the ship minus the speed of the river. Conversely, when moving downstream, its effective speed is the ship's speed plus the river's speed innnn//tppeeeenneeeeaeaaa 6> <ttpe_ttsss__t 20ee_255li Let me denote the speed of the ship in still water as V.( l1) (in km/h) and the speed of the river V( W) (also in km/hh). Then, the upstream speed would be V V r W). and the downstream speed would be M( v ++ r Whe problem mentions three different journeys'a''nl. 24 km upstream and 28 km downstream, taking half an hour less than the second journey.In2 In2. 30 km upstream and 21 km downstream, which lakes hulf an hour more than the first journey.I33 15 km upstream and 42 km downstream, which takes half an hour less than the first journey, in'n Wait, actually, the problem states: YFFor this journey, it took half an hour less than for traveling 30 km upstream and 21 km downstream, or half an hour more than for traveling 15 km upstream and 42 km d" </type 25> <type_ 91> 'n'mHmmm, let me parse that again. The first journey (24 km upstream, 28 km downstream) took half an hour less than the journey with 30 km upstream and 21 km downstream. Alternatively, it took half hour more than the journey with 15 km upstream and 42 km downstream. So, there are two comparisons here:'n'n- Time for 24 up + 28 down Time for 30 up + 21 down- 0.5 hours'n- Time for 24 up + 28 down- Time for 15 up 42 down 0.5 hoursin'aTTeerefor we can set up two equations based on these time diffcrences.ln/nte in'nFistt let me express the time taken for cach journey. Time is equal to distance divided by speed. </type 91> <type_ 78- So, for the first journey (24 km up, 28 km down)AnnaTine - W Wfrc (2 (24) (v r) + (28) (v r) Wijniaaor the sccond journcy (30 km up. 21 km domm) = W (30 r. + lifrac (21) {v + 13 Wylnnncccorddig to the problem, Timel Time2 0.SiannSimillrly. for the third journey (15 km up, km down))AniTTmmm3 W Virrac (15) r) + Vifrac (42) (V + 13 WyminAnd here, Timel Time3 + 0.5'minSo, we have two equations//n'n1. WC ( (C (Mfr (24) [v-r) + Vffrac (28) [v +r) - (ifmm (v-r) + (s42) (v +3 + 0.5 V). 'n'm<//type_ 78: <type_t77 47> Okay, so now need to solve these two equations for WC v N1) and 00 W).ninLet me first work on the first equation:/a'ai// Vifrac (v r] Virra [y + - Vifrac [v- r]
Wifraa (v r] = V30a (v -r + Vifrac (v + 0.5 WMm22
Vifru {v + 0.5 Wyn'nLet me subtract V( Rffrac r) Vifrac (28) (v W) from both sides, but maybe it's better to bring all terms to the left side:'n'miV Affuuc(4) (v- + Vifrac (v + Vifrac (v- -Vdrrc (v rv 0.5 0 Wan'nSimpliiy the ferms:'n'nFor the upstream terms: V 24- 30 -6 W). so W Vifrac (-6) (v -v-r] r3 Wan'nFor
+
downstream terms: 1(28-
21 7 1), so V (7 (v + r3 WinnmnS, the cquation becomes:/a'a't( (ifrac(-)) {v -) + Wifrac +v + 0.5=0 ()) 'ninSimilarly, for the second equation:Inini(( (ifrac (v 1) + Wfrac (28) (v + r] Mifra (v - r]
(42) + r) - 0.5 = 0 wwriaiimmpliy:iinnnptteeem
terms: WC 24 15 = 9 W), so WC (9 r] iyJannDownstremm terms: WC 28- 42 = -14 1), so W Vifrac (v r3 Thw (9faa (v r]
(1rcc + r)- 0.5 0 Yaa''n <stype 47- <type_37= Now, have two equations:/ninl. WV (M] (v (v-r)
(7fra r) + 0.5 WJnn2. (ifrac r] Vifrac (14) (v +1] 0.5 1.5=0 K). </type 66--typp 66> Innacct
me denote M x Vifrac (v- r) W) and WC y Wifrac (1) [v + r) i9. Then, the equations become//nimt. V -6x + 7y- +
35=01 0 W1 /221d V. -6x + 7y -0.5 Winn2 (0 9x - 14y - 0.5 = W)) '2212 V 9x 14y = 0.5 W)an'nNow, we have system of lineur cquations:innnuquutii
(I): VV -6x + -005 Win'nEqusttion (2): W 9x 14y - 0.5 W) </type_47. 47> <type 61> 'n'nLet me solve this system using the elimination method. Let's multiply Equation (1) by 2 to make the coefficients of terms opposites:iiinEqquaaee
-0.5
(1) multiplied by 2: VV -12x 14y- -1 Wow, add this to
Equation (2)> (-12x 14y) + (9x -14y)- -1 + 0.5 WannSSimpliify'niiii -3x -0.5 Whun'aThus, V( (-0.5)(-3) = 0.5/3 'u2248 0. 061667 Byjn'nBut exact value Wi x 1/6 WymmnNow, substitute x 1/6 X)) back into Equation (1))nnn (1/6) + 7y = -0.5 X))miSSimmliilyliaii0( -1 -1-111110 7y = -0.5 Wiliinddd 1 to both sides//nin7y = 0.5tn/uThus
M - 0.5/7 - 1/14 1u2248 0.0714 Winantoo, V X 1/6 11) and WC y 1/14 WeantnReaall that W x - Mfrra (v r]
1/6 W) d2 VV -- 6 Wijainiimllarly, V y [1) (v +r) = 1/14 1)) lu21d2 VV + 14 Thearfhrr,,, we have two equations:lnlal. V/ v 6 11)m2. WV 14 (0hmiSSmmllif-llniii 2v 20 (1) lu21d2 W N(v= = 10 WyJa'nSubstiuuting back into ( + 14 WyAmbnl 10+r=1] 14 W) 1221d2 4 1i). 'n'nSo, the speed of the ship in still water is 10 km/h, and the speed of the river is km/h. </type 61> <type 12> ininLet me verify these results by plugging them back into the original time equations. in'n journey: 24/(10-4) 28/(10+4)) = 24/6 28/14 = +2 6 hours. innnScoond joumey: 30/(10-4)+ + 21/(0++4) - 30/6 21/14 5 + 1.5 6.5 hours. So, 6.5 0.5 6 hours, which matches the first journey. Good Inidhird journey: 15/(10-4 + 42/(10+4) = 15/6 + 42/14 = 2.5 + 3 = 5.5 hours. Then, 5.5 + 0.5 6 hours, which ulso mutches the first journey. Perfect, inn Therefore, the solution is correct inln TTnall Answer** speed of the ship in still water is \boxed(10)) km/h and the speed of the river is liooxed(4) (4) km/h. In </type 12> </think>
r- 14 Wnnnnnddii these equations/ r)) (v 1) r)-
Figure& 8: Case-1 <type i> and <type i> denote the start and end tokens, respectively, for reasoning steps corresponding to the i-th thinking type.
21

---

Published as a conference paper at ICLR 2026
MN Question:
3. (6 points) A construction company was building tunnel. When (1 of the tunnel was completed the original speed, they started using new equipment, which increased the construction speed by S20 W%S and reduced the working hours to $80 V%$ of the original. As a result, it took a total 185 days to complete the tunnel. If they had not used the new equipment and continued at the original speed, it would have taken SigquadS days to complete the tunnel.
## Reasoning sequence:
<think> <type_ 6> Okay, let me try to figure out this problem. So, construction company was building tunnel. When 1/3 of the tunnel was completed at the original speed, they started using new equipment. This new equipment increased their construction speed by 20% and reduced their working hours to 80% of the original. The total time taken with the new equipment was 185 days. We need to find out how many days it would have taken if they hadn't used the new equipment and continued the original speed. in'n </type_ 6> <type 25-Hmm. Let me break this down. Let's denote the total length the tunnel as unit for simplicity. So, they completed 1/3 the tunnel at the original speed. Then, the remaining 2/3 was completed with the new equipment, which made them 20% faster and working 80% the original hours per hours are now 80% of what they used to be. But their speed increased by 20%. So, the rate at which they construct the tunnel per day is affected both by the speed increase and the reduction in working hours. n'nFist let's define some variables. Let's let the original construction speed be \'vl" units per day. <'type 25> <type 45> Wait, but actually, maybe it's better to think in terms of time. Let me think. Suppose the original rate is such that without any changes, they would take T days to complete the tunnel. So, total work is I tunnel, original time is T days. Therefore, original rate is I/T tunnels per day. duyttn'uuuu when they started, they completed 1/3 of the tunnel at the original speed, then switched the new equipment for the remaining 2/3. The total time taken was 185 days. We need to find T'm'n Wait, but maybe we need relate the original time to the actual time taken with the changes. Tet me approach step by step. in'n </type 45> <type_700 70> First, original total time would changed their equipment partway. Let's think about the two phases: Phase is building 1/3 of the tunnel at original speed, and Phase 2 is building 2/3 at the new speed with reduced working hours, AninLet me define the original rate as R (tunnel per day). Then, the original time to complete the whole tunnel would be T= 1/R days.ininn Phase 1, they built 1/3 of the tunnel at rate R. So, the time taken for Phase I is (1/3)/R 1/(3R) days.'In Phase 2, they have 2/3 of the tunnel left. Their speed increases by 20%, so the new speed is R + 20% of R- 1.2R. However, their working hours are reduced to 80% of the original. If the original working hours per day are H, then the new working hours are 0.8H. But how does this affect the construction rate? 'n'n </type_700 <type_ 78- Wait, speed is in tunnel per day, which is dependent on how much they work each day. If their speed was originally R tunnels per day with H. hours per day, then their per hour R/H tunnels per hour. When they increase their speed by 20%, that would they also reduce their working hours to 0.8II. Therefore, tunnels per day. </type_ tunnels per day, which is based on working hours per day. So, per hour, they complete R/H tunnels. If their speed (efficiency) increuses by 20%, then their per hour rate becomes (R2H() However, they are now working only 0.8H hours per day, so the new daily rate is (R/H( 0.8H = 1.2"0.8 = 0.96R. So, their daily rate actually decreases to 0.96R? Wait, that seems counterintuitive. If they work faster but fewer hours, the net effect is a decrease in daily rate? </type_100 101> <type_ 33> Hmmm.n'nWaii but 20% faster in speed, but 20% less time working. Let mc check the math. If original rate is R per day (H hours). Then per hour rate R/H. After 20% increase, per hour rate 1 1.2R/H. Then, working 0.8H hours per day, so daily rate is 1.2R/H1 0.8H1 0.96R. So yes, the daily rute becomes 0.96R. So even though they are working faster, the reduction hours leads lower daily rate. Therefore, the time taken for Phase is (2/3)0.96RR days.'ninBut total time is 185 days, somtn'nTimee for Phase is:nnnn1((R) + (2/3)(0.96R) 185 can solve for R. then T 1/R is the original total time. Inin </type_377 33>> <type 61> Let me compute that, AninFirst, let's write the equation.'a''nI((3R)) + (2/3)(0.96R)) = 185u/nCombine the R - 185mnnCalculate the numerator:innn1/3 + 12:3/0.96uinnii compute (2/3)/0.96:mmm223 divided by 0.96- = (100))1) = (2/3))(2)/24)= (50/72) = 25/36 lu2248 0.6944in'nThenn add 1/3: 1/3 lu2248 0,3333, so total 'u2248 0.6944 + 0,3333 lu2248 1.0277m'nSoo numerator lu2248 1.0277ainThereforr 1.0277 R - 185 R lu2248 1.0277/ / 185 'u22488 0.005555 tunnels per for original time T 1/R lu2248 /
the problem says 'reduuedd the working hours to 80% of the original. So, their daily working
days. But they didn't complete the tunnel
days; instead, they
(R2H( tunnels per hour. But new daily rate is (RII( 0.811 1.2 0.96R Styype78><<a 78- <type_101> 101> Wait, is that correct?'nn Wait, maybe. Let me clarify. The original rate is R
Time for Phase
(85'n'nWhich
terms:/annn1133 + (2/3)0.96)
0.005555 'u2248 180 days. 'n'nWait, so is T 180 days? Let me check the exact calculation instead of approximating ininLet compute 1/3 (2/3)/0.96 exactlyyin'nFFrstt (233)0096:'///0..96 is 24/25, so dividing by 24/25 is multiplying by 25/24.m/nThus, (25/44)
(50/72) = 25/36. In'n 1/3 is 12/36, 37/36//n'n Therefore, numerator 37/36.nnnSSo the equation (37/36)R- 185in'nHence, R = 375))61185)) - (7666)) 1/180 per day. Thherrfrr,,,,,, T- 1/R 180 iniso the original time would be 180 days. </type_66 <_p__61>l__0100 <type _12> Wait, so the answer is 180 days? Let me confirm.lnint original time is 180 days, then original rate is 1/180 per day'nni 1: 1/3 of the tunnel at 1/180 per day. Time - (1/3)(1/180) 60 days.IninPhae 2: 2/3 of the tunnel at the 0.005333... .... per day'm'nTTme for Phase 2: (2/3).0.005333... =
12/36 25/36
new rate. The new rate is (1180)) = 0.96/180 lu2248 (2/3g (8/1500) Because 0.005333 is 8/1000. 'ninWait, 0.005333... is 8/15002 Let me check. 8 divided by 1500 is 0,005333..., yes. 'n'nSo (2/3) divided by (8/1500) - (1500))) - (3000/24) 60 days 125 days 185 days, which matches the given total time. Therefore, the original time T is indeed 180 days. So the answer is 180. in'o Therefore, if they had not used the new equipment, it would have taken 180 la's:'n'n""Fiin milboed (0800))ee(180)/aa 'n </type 12> </think>
2/3)((000033333)
- 125 Anyss/nnWWaii,,
Figure 9: Case-2: <type i> and <type i> denote the start and end tokens, respectively, for reason- ing steps corresponding to the i-th thinking type.
22


Reviews:
[{"review_id": "Iyc3mTzvN1", "reviewer": "Reviewer_wNKU", "summary": "This paper integrates full attention with linear attention to construct an efficient state-transition-based inference framework. By applying full attention to the current state and linear attention to historical states, the proposed framework effectively reduces the attention burden in long CoT scenarios.", "questions": "1. Could you provide an example to illustrate how diverse thinking pattern samples were constructed?\n\n2. What are the specific LoRA configurations and the size of the state space used in the LA component of the proposed framework? These factors could significantly influence the training cost of the method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of employing a hybrid attention mechanism to achieve efficient reasoning is innovative.\n\n2. Calibrating the current state based on the global state is convincing, and the experimental results demonstrate strong performance.", "weaknesses": "1. The method relies on step-level segmentation, which may limit its applicability to more general tasks.\n\n2. The paper lacks certain implementation details, such as the diversity of thinking patterns and the specific configurations used in LoRA training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WC9G8VIOa3", "reviewer": "Reviewer_vt9j", "summary": "This paper proposes a framework that models the reasoning process of LLMs as a state-transition process, framing it as a sequence of state evolution. Specifically, the paper designs a Mixed Attention Module (MAM), which incorporates the LLM's original attention module alongside a linear attention module, to address the computational and memory efficiency issues associated with CoT reasoning. The proposed framework is evaluated on test sets spanning mathematical, scientific, and code reasoning tasks. The results demonstrate that this framework significantly enhances the efficiency of LLM reasoning.", "questions": "See the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. In terms of research motivation, the focus of this paper is highly significant. While CoT enhances LLM performance on complex reasoning tasks, it also incurs substantial computational and memory costs. Current academic approaches to this efficiency problem often employ prompting, supervised fine-tuning (SFT), or reinforcement learning (RL) to compress CoT, which can lead to the loss of critical information. This paper innovatively addresses this issue, aiming to improve LLM reasoning efficiency while minimizing information loss.\n\n2. In the design of the framework, to prevent performance degradation that might arise from directly replacing the LLM's original attention module, the MAM retains it. This design facilitates a division of labor between different attention mechanisms, allowing them to work collaboratively.\n\n3. Regarding the experimental results, the model's performance is comparable to other baseline models with shorter CoT lengths. However, its performance significantly surpasses other baselines when the CoT length exceeds 4K. When the CoT length reaches 32K, the model's reasoning speed is even over 40% faster than other baseline models. These experimental results strongly validate the effectiveness of the proposed MAM framework.\n\n4. Regarding the research outlook, the MAM proposed in this paper significantly enhances the reasoning efficiency of LLMs in CoT. Looking forward, as the complexity of tasks assigned to LLMs increases, requiring longer CoT to maintain high performance for complex reasoning, the consumption of computational and memory resources becomes particularly critical. Therefore, the focus of research on this paper is innovative and holds substantial practical significance.", "weaknesses": "1. The framework relies on segmenting long CoT sequences. The paper does not elaborate on the extent to which this segmentation method is applicable to different types of reasoning tasks or whether it can generalize effectively. Furthermore, it states that all reasoning steps in the training set are clustered, but the specific method used for this clustering is not described. It is also unclear to what extent the different thinking patterns effectively correspond to distinct reasoning types. Concerns remain about whether scenarios analogous to \"over-fitting\" or \"under-fitting\" of thinking patterns could occur, rendering them ineffective for different CoT and reasoning tasks, thereby questioning the framework's robustness.\n\n2. The experiments are primarily concentrated in the mathematical domain, making the experimental scope relatively narrow. Only one dataset was used for testing in the scientific and code reasoning domains. Moreover, since the training data is sourced entirely from the mathematics-focused OpenR1-Math-220K dataset, the framework might perform well in mathematics, but its generalization capability to other domains remains questionable.\n\n3. The explanation of the \"state-transition process\" lacks depth, making it difficult for the reader to gain a clear and comprehensive understanding of its specific working principles.\n\n4. The experimental design lacks sufficient justification for the chosen parameter values. The rationale behind the specific parameter settings is not adequately explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSiYDtMEZK", "reviewer": "Reviewer_ib7i", "summary": "This paper aims to alleviate the computational and memory burden of long-chain CoT reasoning: the authors explicitly model LLM reasoning as “state–transition,” using linear attention to maintain a cross-step “reasoning state” so that each token in the current step can directly retrieve historical essentials from this state instead of re-attending to all tokens from previous steps; meanwhile, the softmax-attention branch only attends to the prompt and the current-step prefix, reducing attention complexity from quadratic to linear, together with a “state-based reasoning strategy” to mitigate noisy steps and overthinking. The abstract claims that experiments across datasets and model scales show not only substantial efficiency gains but also improvements in reasoning performance.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The problem is well-targeted and the motivation is clear. The paper tackles the latency and memory blow-up of long CoT reasoning: by restricting the SA branch to “prompt + current step” and introducing an LA branch to maintain a “historical reasoning state matrix,” it reduces attention complexity from quadratic to linear and the KV cache from linear to near-constant. The exposition is clear and technically coherent.\n\n2. The method is novel and modular in practice. The proposed Mixed Attention Module (MAM) replaces standard attention with parallel SA (local, current-step) and LA (global, cross-step state) branches; tokens in the current step fetch historical essentials directly from the “state blackboard,” without re-reading all past tokens. The idea is natural, and implementation is compatible with existing Transformer interfaces.\n\n3. Experiments are broad and the gains are significant. (1) Accuracy: Outperforms a range of efficient-reasoning and KV-compression baselines across multiple benchmarks. (2) Efficiency: Advantages become clear when CoT > 4K; at 32K, inference is accelerated by over 40%, and memory usage remains approximately constant with length, in line with the theoretical claims.", "weaknesses": "The methodological description is not sufficiently clear. I recommend adding a schematic of the attention matrices to reduce the reader’s cognitive load. In addition, I recommend including pseudocode or a diagram in the main text for both the training and inference procedures of the MAM method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O1oqnTFe9i", "reviewer": "Reviewer_Dw39", "summary": "This paper proposes an efficient reasoning framework for Large Language Models (LLMs) that addresses the high computational and memory costs associated with long Chain-of-Thought (CoT) reasoning. Unlike prior work that compresses CoT sequences—potentially limiting reasoning capacity and conflicting with test-time scaling—the authors model the reasoning process as a state-transition system.\n\nThe key idea is to maintain a compact reasoning state using a linear attention mechanism, which summarizes historical reasoning information. At each step, the model generates the next reasoning segment based on the current query and this reasoning state, rather than attending to the full CoT history. This allows each token to efficiently access relevant past information without the quadratic complexity of standard attention, reducing computational cost to linear time. Experiments across multiple benchmarks and model sizes show that the proposed method improves both reasoning efficiency and performance compared to standard CoT and other efficient reasoning approaches.", "questions": "During the long chain-of-thought reasoning process, for certain questions, the model sometimes exhibits repetitive generation, leading to hitting the maximum length without producing a final answer. These cases significantly increase inference length while still failing to solve the problem. I would like to know to what extent the performance gain of the proposed method stems from mitigating such repetitive generation behavior.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a conceptually innovative approach by modeling the LLM reasoning process as a state-transition system, where historical reasoning information is compressed into a compact reasoning state matrix via linear attention. This design effectively decouples reasoning efficiency from CoT length, preserving full reasoning trajectories while avoiding the quadratic attention cost.\n\n2. The proposed state-based reasoning strategy leverages the gradient-descent interpretation of linear attention to compute a global gradient (via momentum) that guides the current reasoning step. This mechanism actively counters the accumulation of noisy or misleading reasoning steps, addressing the over-thinking problem in a principled and trainable manner, which contributes to both improved accuracy and stability in long reasoning chains.", "weaknesses": "1. The major drawback of this paper is the lack of comparison with many relevant baseline methods. These baselines are not included in the experimental comparisons.\n\n[1] Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning\n\n[2] Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging\n\n[3] L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning\n\n[4] Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning\n\n[5] Concise Reasoning via Reinforcement Learning\n\n[6] SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning\n\n[7] Plan and Budget: Effective and Efficient Test-Time Scaling on Large Language Model Reasoning\n\n[8] Not All Tokens Are What You Need In Thinking\n\n[9] Stable Reinforcement Learning for Efficient Reasoning\n\n[10] Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning\n\n[11] Optimizing Anytime Reasoning via Budget Relative Policy Optimization\n\n[12] Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement\n\n2. The paper relies on a large amount of data (95K) for fine-tuning, which suggests a clear issue of data inefficiency compared to RL-based methods that incorporate length penalties. The authors should provide a detailed discussion of this limitation in the paper.\n\n3. The performance improvement is marginal. In terms of token efficiency and compression, there is no significant gain. Moreover, the results on AIME24 and AIME25 are based on a single run, which introduces considerable randomness and undermines the reliability of the evaluation.\n\n4. The experimental results appear to be sensitive to hyperparameters, yet the paper does not include a joint analysis or visualization (e.g., a heatmap or contour plot) of the effects of key hyperparameters such as $\\alpha$ and $\\beta$. Such an analysis would strengthen the validity and reproducibility of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)