ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
Authors: Shuang Chen
Hangyu Guo
Yimeng Ye
Shijue Huang
Wenbo Hu
Jiayu Chen
Manyuan Zhang
Haoxi Li
Song Guo
Nanyun Peng
Venue: ICLR 2026 Poster (2026)
Keywords: Entropy based Multimodal Adaptive Reasoning
Primary area: reinforcement learning
TL;DR: ARES is a multimodal adaptive reasoning framework that curbs overthinking on easy tasks and promotes deeper exploration on hard ones, achieving state-of-the-art performance efficiently.

Abstract: Recent advances in multimodal large reasoning models (MLRMs) have substantially
improved their ability to solve complex textual and visual tasks. However, these
models tend to *overthink* on
simple problems, producing unnecessarily lengthy reasoning traces, while
*under-exploring* on challenging ones, leading to missed solutions. To 
address this imbalance, we propose **ARES**, a unified open-source framework
for *adaptive reasoning* that dynamically allocates exploration effort based
on task difficulty. Our approach is motivated by two key empirical findings:
(i) while single-token entropy is noisy, *high window-entropy (HWE)
tokens* (token-level entropies averaged under a sliding window) can reliably capture reasoning-critical moments; and (ii) reducing HWE usage
benefits easy problems, while increasing it is essential for solving hard ones.
Building on these insights, ARES introduces a two-stage training pipeline. In the
*Adaptive Cold-Start* stage, we curate multimodal and textual data paired
with reasoning traces of length proportional to problem difficulty, equipping the
model with initial difficulty awareness. In the second stage, we develop
*Adaptive Entropy Policy Optimization (AEPO)*, which uses HWE tokens as
exploration triggers to decide *when to explore*, and a hierarchical entropy
reward with dynamic KL control to decide *how much to explore*. Extensive
experiments demonstrate that ARES achieves state-of-the-art performance and
reasoning efficiency across diverse mathematical, logical, and multimodal
benchmarks, while closing the gap to leading commercial systems under
significantly lower inference costs. The anonymous code repository is available at https://anonymous.4open.science/r/ARES-60728M.

Published as a conference paper at ICLR 2026
ADAPTIVE REASONING VIA SAFFICULTY-WWAR TOKEN-LEVEL ENTROPY SHAPING
ARES: MULTIMODAL
Shuang Chen ; Guo Yimeng Ye², Shijue Huang', Wenbo Hu', Jiay Chen",
Manyuan Zhang", Haoxi Li?, Song Guo?, Nanyun Peng'
University of California, Los Angeles TThe Hong Kong University of Science and Technology CCouubia University "The Chinese University of Hong Kong
ABSTRACT
Recent advances in multimodal large reasoning models (MLRMs) have substan- tially improved their ability solve complex textual and visual tasks. How- ever, these models tend to overthink on simple problems, producing unnecessarily challenging ones, leading
lengthy reasoning traces, while under-exploring missed solutions. To address this imbalance, we propose A ARES, a unified open- source framework for adaptive reasoning that dynamically allocates exploration effort based on task difficulty. Our approach is motivated by two key empiri- cal findings: (i) while single-token entropy is noisy, high window-entropy (HWE) tokens (token-level entropies averaged under sliding window) can reliably cap- ture reasoning- critical moments; and (ii) reducing HWE usage benefits easy prob- lems, while increasing it is essential for solving hard ones. Building on these insights, ARES introduces two-stage training pipeline. the Adaptive Cold- Start stage, curate multimodal and textual data paired with reasoning traces of length proportional to problem difficulty, equipping the model with initial diffi- culty awareness. In the second stage, develop Adaptive Entropy Policy Opti- mization (AEPO), which uses HWE tokens as exploration triggers to decide when explore, and a hierarchical entropy reward with dynamic KL control to decide how much to explore. Extensive experiments demonstrate that ARES achieves state-of-the-art performance and reasoning efficiency across diverse mathemati- cal, logical, and multimodal benchmarks, while closing the gap to leading com- mercial systems under significantly lower inference costs. The anonymous code repository is available at https / /gt thub. com whw.n 0728/ 0728/ARES.
INTRODUCTION
Leveraging long Chain-of-Thought (CoT) reasoning with reflection, Multimodal Large Reasoning Models (MLRMs) have showcased strong capabilities on both complex textual and visual tasks (Co- manici al., 2025; Guo et al., 2025). While incorporating long CoT reasoning substantially im- proves performance on complex reasoning tasks, it often causes models to generate unnecessarily lengthy reasoning even for easy tasks (Qu al., 2025). This substantially increases inference costs and latency, limiting MLRMs' usability in various weal-world scenarios.
Recent studies have explored both training- free (Han et al., 2025; Yang et al., 2025b) and training- based strategies (Zhang et al.. 2025; Arora & Zanette, 2025: Ling et al., 2025; Tu et al., 2025) to mitigate the issue of verbose long-thought outputs. While these methods alleviate overthinking, they often cause model performance degradation (Huang et al., 2025b). To balance model's efficiency and accuracy, adaptive reasoning mechanisms have become a promising research direction, which dynamically adjust reasoning effort to trade off performance against computational cost. Existing approaches attempt this by either curating cold-start data with varying difficulty (Wang et al., 2025d; Zhang et al., 2025; Yu al., 2025a) or using difficulty-awan penalties during RL (Huang et al., 2025a; Shen et al., 2025b). However, such methods frequently encourage exploration on difficult
"Equal contributions.

---

Published as a conference paper at ICLR 2026
problems, leading to unnecessarily verbose reasoning traces, and they still fail to deliver significant performance improvements.
To address these challenges, we seek to answer two foundational questions for adaptive reasoning mechanism: When should a model be encouraged to explore? How much reasoning effort should be allocated during exploration? To answer this, the paper presents detailed preliminary analysis of exploration during the RL stage of MLRMs, specifically investigating how exploration affects the overall model's performance and efficiency. We meticulously design experiments by constructing an easy, medium and hard set for study the trade off of RL exploration cost against accuracy. As illus- trated in Figure I, our findings can be summarized in two key observations. Firstly, high token-level entropy in MLRMs not only captures tokens relevant to reasoning but also often includes noise such as punctuation, formulas, or other superficial elements. By comparison, window entropy, the mean entropy computed over consecutive tokens, more reliably identifies pivotal decision points that guide the reasoning trajectory across multiple potential pathways. Secondly, for easy problems, reducing the number of high window entropy (HWE) tokens shortens the reasoning trace and improves perfor- mance. For complex problems, by contrast, increasing the number of HWE tokens promotes more thorough exploration, thereby enhancing the model's ability to solve challenging tasks. These find- ings highlight HWE tokens as an exploration trigger for adaptive reasoning in MLRMs, improving performance while reducing inference costs.
Motivated by these empirical findings, we propose an adaptive reasoning training approach called ARES, standing for multimodal Adaptive Reasoning via difficulty-aware token-level Entropy ward Shaping. Specifically, first, during the Adaptive Cold-Start stage, we instill an initial difficulty awareness by fine-tuning the model on data where reasoning length is explicitly correlated with problem complexity. Second, we introduce Adaptive-Entropy Policy Optimization (AEPO), which uses high enttroy regions to trigger exploration and difficulty hierarchical reward to control its depth. This dual-stage approach enables ARES to adaptively allocate reasoning effort, rewarding deep exploration on complex problems while penalizing overthinking on simpler ones.
In summary, our main contributions are follows:
We empirically demonstrate that high-window window-entropy (HWE) tokens can guide MLRMs in exploration and reveal distinct exploration behaviors between easy and hard tasks.
We introduce ARES with a high-quality cold-start pipeline grounded in our curated ARES- SFT- 224K dataset. Together with AEPO's token-level entropy-shaping RI. strategy, training on these datasets consistently improves model performance.
Extensive experiments demonstrate that ARES achieves superior performance and inference effi- ciency across a wide range of mathematical, general knowledge, textual, and multimodal reasoning benchmarks.
RELATIONS BETWEEN TOKEN ENTROPY AND REASONING DIFFICULTY 2
In this section, we show two intriguing findings involving the exploration and effectiveness of ML- RMs, paving the way for the proposed ARES in Section 3.
2.1 WINDOW ENTROPY SERVES AS A TRIGGER FOR EXPLORATION IN MLRMS
Recent works (Wang et al., 2025b; Zheng al., 2025) highlight that high-entropy tokens often mark the starting points of reasoning forks, where the model must branch into alternative continuations. Such points are critical for determining the depth of reasoning, since they open up new solution paths. However, as discussed in Appendix C, token-level entropy H, only captures local uncertainty at single step and often dominated by short-lived lexical ambiguities (e.g., punctuation, function words). Conversely, discourse markers that indicate major logical transitions (c.g.. "but," "how- ever") may appear with very low entropy, despite signaling important shifts in reasoning. Hence, relying solely single-token entropy is noisy and insufficient for reliably identifying when explo- ration should be triggered.
From cognitive perspective, humans rarely experience uncertainty as instantaneous hesita- tion on a single word. Instead, when encountering a conceptual bifurcation, an entire segment of reasoning tends to remain unstable. Motivated by this intuition, aggregate token-level en-
2

---

Published as a conference paper at ICLR 2026
tropies into sliding- frlding-window window statistic that captures the persistence of uncertainty across consecutive steps: Huw 1, - H-- This measure highlights regions where the model sustains high un- certainty over multiple tokens, thus providing a smoother and more semantically aligned indicator of reasoning-critical moments. Window entropy therefore reduces the noise of single-token signals while better localizing genuine reasoning bifurcations.
Our empirical analysis supports this intuition. As shown in Figure 5 in Appendix, window entropy achieves consistently higher F1 scores than single-token entropy for detecting reasoning-critig critical tokens. Moderate windows (4-8 tokens) offer the best trade-off: they smooth local noise while remaining focused enough to capture sustained uncertainty. In contrast, single-token measures are overly sensitive to lexical artifacts, and long windows (16-32 tokens) dilute the signal with low- entropy tokens. These results validate window entropy as a more robust diagnostic of reasoning uncertainty and motivate its role as a central trigger our Adaptive Exploration Policy Optimization (AEPO) framework (Section 3.2.1).
2.2 ENTROPY-DIFFICUME INTERACTION
Exploration should not be uniform across all problem difficulties. Intuitively, for easy problems, the model already possesses sufficient knowledge to arrive at the correct answer; excessive high- entropy branching in such cases likely lead to unnecessary verbosity even distract from the correct reasoning path. In contrast, hard problems inherently require sustained exploration: reaching the correct answer typically involves probing multiple solution paths, which manifests as longer responses and more high-entropy tokens.
We validate these intuitions by analyzing model generations using window-entropy statistics. As illustrated in Figure exploration interacts strongly with problem difficulty. In the easy set, samples with high-entropy counts below batch- addth-v threshold attain higher accuracy while producing shorter responses, indicating that suppressing unnecessary exploration is beneficial. By contrast, in the hard set, above-threshold samples achieve higher accuracy but at the cost of longer responses, showing that sustained exploration improves success challenging problems.
Conditioning on correctness further highlights this pattern. For easy problems, correct cases contain markedly fewer high-entropy tokens and shorter responses compared incorrect ones. For hard problems, correct cases typically involve longer responses and more high-entropy tokens than in- correct ones, consistent with the need for extended exploratory reasoning. Together, these results establish a clear entropy-difficull interaction: exploration should be suppressed for easy tasks but encouraged for hard ones.
These results establish a clear entrop--dificult interaction: on easy problems, suppressing explo- ration reduces verbosity and improves accuracy, while hard problems, encouraging additional high-entropy reasoning is crucial for success. This motivates our difficulty-aware exploration strat- egy, where entropy-based triggers are adaptively modulated across difficulty buckets to balance performance and reasoning efficiency. As further justified in Appendix L, we establish theoreti- cally that response length grows approximately linearly with the number of high-entropy tokens, validating entropy as a principled proxy for reasoning effort.
METHOD
We aim to endow a multimodal policy with the ability to adapt its reasoning depth: produce short answers for easy instances and longer, exploratory chains for hard ones. Our ARES method consists of two stages: cold-start curriculum (AdaCS) that preserves length-controllable modes, followed by a KL-regularized RL stage (AEPO) that couples token-level high-entropy window detector with a difficulty- -warr KL budget, which is illustrated in Figure 2.
3.1 ADACS: ADAPTIVE COLDSTART FINE-TUNING
Inspired by Wang et al. (2025d), during the cold-start phase, training with simple problems paired with short chains of thought (CoT) and difficult problems paired with long CoTs effectively enhances the model's 's difficulty awareness and facilitates the acquisition of key high-window-en -iidoow-entropy (HWE) tokens as well reflective reasoning capabilities. We first curate high-quality set of RLVR textual
3

---

Published as a conference paper at ICLR 2026
(a) Difficuity modullaes the exploratory effort along the reasoning path
(b) Quantitative analysis across Easy and Hard
Abore Gucdlold Ang. Act
EASY MEDIUM
SS
12
Below chresdeddd
57000
0.910
Rospunal Lorgih
Rexch
Probuciity
HARD
3000
Reach
04
2000
022
0:00kg 0w0
1000
Fuuyy
(i) Accuracy by enerpy threshold
Response length by difficulty
Easy
Medium
490
2000
390
2073
316.2
6000
5979
53007
(25)
4000
3000
2034.5
Easy:
Medium:
Hard:
Q:
Q: Whath i 1-5 (lee 18)
Qr Prase that the cam the add suumbers always rren?
High
A: Fintts wd
Let's
a
Hard
Easy
Hadd
suwver 12.
and take the
eeanbert... Hinn, the mealt to wds liu 21e
-Wall reull divibiiiity Z...
(6)) High entroyy taken count by
(iv) Response length by comeemess
ressindee, so
correttees
interaction in exploration. (a) Conceptual illustration: task difficulty modu- lates the reasoning trajectory. with easy problems requiring little exploration and hard problems benefiting from deeper branching. (b) Quantitative analysis: (i) for easy lasks. responses below the entropy threshold are both shorter and more accurate; for hard tasks, above-threshold exploration yields higher accuracy; (ii) response length increases significantly with difficulty; (iii) within cach difficulty, correct cases use fewer high-entrops tokens for easy problems but more for hard problems; and (iv) correctness further amplifics this trend in re- sponse length. Together, these results show that limiting exploration improves efficiency on easy problems, while encouraging additional exploration is crucial for solving difficult ones.
Figure 1:
data and multimodal STEM tasks, which serves as the foundation for the cold-start stage. More details regarding dataset sources and preprocessing can be found in Appendix 4.1. Different from prior work (Wang et al., 2025d) that discards samples with a pass rate of 1 and oversamples those with lower pass rates, our methodology is specifically designed to train the model to differentiate problem difficulty and generate responses of corresponding lengths. To achieve this, for each data source, we establish a target response length for every pass rate bracket, which is determined by the median response lengths of the easiest (pass rate- = 1) and hardest (pass rate = 0) problems. We then sample responses whose lengths are proximate to this target, while ensuring uniform sam- pling across all pass rate brackets. This strategy is motivated by the objective of instilling a strong, explicit correlation between perceived problem difficulty and the verbosity of the generated ratio- nale. Formally, for any intermediate pass rate p E (0, 1), the target response length is defined as Lungen (p) (1 p) L(0) + p- L(L), where (()) and L(1) denote the median token lengths of responses for problems with pass rates of O and 1, respectively, for a given data source. This design maximizes response length diversity across pass rates, thereby amplifying the distinction between easy and hard problems and enabling the model to better learn difficulty-awan reasoning strategies.
3.2 AEPO: ADAPTIVE-ENTROPY POLICY OPTIMIZATION
To overcome the imbalance between overthinking easy problems and under-exploring difficult ones (Section 2.2), we introduce Adaptive-Entropy Policy Optimization (AEPO). The central idea is to endow the policy with an adaptive exploration mechanism that decides both when to explore and how much to explore, guided by entropy signals. A detailed flowchart of AEPO is presented in Algorithm in the Appendix.
3.2.1 WHEN TO EXPLORE.
To operationalize window entropy as exploration trigger, we introduce bucket-wise high- entropy threshold. Specifically, for each rollout trajectory, we compute the token-level entropies (H1) and extract the thth percentile value the high-entropy threshold for that sequence. This choice is motivated by recent empirical findings (Wang et al., 2025b), which show that RLVR pre- dominantly reshapes the entropy distribution of tokens in the top 5% percentile range, while low- entropy tokens remain comparatively stable. Hence, the 95th percentile serves robust indicator
4

---

Published as a conference paper at ICLR 2026
DiffoultyAAwoss Serreseeeeee selective Modile
Deto Collection
Tixt
Date Gurstion
Mui timedal
Sys2
Adaptive Coldstort Tuning
Dynemic
[Eayy] O
Reference Medel
Budget Allocator
A
[Hand]
Policy Mode!
Online + Difficulty Bucksting
[Wed] O
Overall
Group Computation
Entropy Reword
Eosy
High Entropy sindow work
EASY MEDIUM HARD
Med fum
As
A
Hord
Verying difficulty colle for verying thinking depth
Training Stage Adaptive Entropy Policy Optimizatin((EEO)
Figure 2: Overall training pipeline of method. Stage 1 (Adaptive Coldstart Fine-Tuning): difficulty- aware selective data curation adaptive KL-guided fine-tuning establish strong initialization across text and multimodal inputs. Stage (Adaptive Entropy Policy Optimization. AEPO): online difficulty bucketing and entropy- aware rollout allocate reasoning depth dynamically, with high-entropy windows serving as branching points for exploration. Together, the two stages enable uncertainty-awer large language models.
fifiuult--ddattve reasoning for then averaged across trajectories in a mini-batch to yield stable batch-level cutoff Thigh:
of reasoning-critical uncertainty. To reduce noise at the single-sequence level, the thresholds
1 [D]
Σ
Quantile, as (H.())...
Thigh
FED
where D denotes the set trajectories in batch.
During training, this threshold is updated dynamically on batch-by-batch basis, allowing AEPO to adaptively align the exploration schedule with the evolving entropy distribution. At rollout time, whenever the windowed entropy Hew of segment exceeds Thigh the model branches additional trajectories from step I.. At each high-entropy window, exactly one additional trajectory is gener- ated, unless the total number of generated trajectories has already reached the maximum allowed. In this way, exploration is triggered only at sustained high-uncertainty regions, concentrating com- putational resources on reasoning-on -ritiicca moments while avoiding unnecessary branching at stable, low-entropy segments.
3.2.2 HOw MUCH TO EXPLORE.
Motivated from our preliminary studies, we next investigate the principle of how much exploration should be allocated once a reasoning-crittie critical region is detected. While window entropy serves as a reliable trigger for when to explore, the amount of exploration must be carefully controlled to avoid excessive reasoning on simple tasks and insufficient exploration on complex ones. To achieve this balance, AEPO integrates two complementary mechanisms:
5

---

Published as a conference paper at ICLR 2026
Hierarchical Reward Design. The goal of AEPO's hierarchical reward is adaptively regu- late the degree of exploration once reasoning-critien regions are detected. Building on the online difficulty buckets (Appendix H), integrate accuracy and entropy-basce shaping into single constraint-driven formulation that requires no additional hyperparameters.
Formally, for a trajectory with NHE high- tokens, we define the bucket-dependent target as the batch mean:
Narrau (d) Ba [Ni d], E (easy, medium, hard),
which is updated online at each iteration track the evolving distribution of exploratory behavior. To regulate deviations from this target, we introduce a closed-form Lagrange multiplier derived from batch statistics:
Aa max (0.
Beatch [Nu |
N NE
(d)
Varbasch [Nie
d]
This term automatically scales the strength of entropy shaping without requiring manually tuned weights or learning rates.
Let Ayy; d) NHE
N M (d). The shaping direction is defined
max(O, A), easy,
(a(A)
|A|
medium,
max AA,, d hard.
The qualitative effect of (A( differs across difficulty levels. For easy problems, positive devia- tions (A >> 0) correspond to unnecessary over-exploration and are penalized. For medium problems, both under- and over-exploration are discouraged, resulting in symmetric shaping curve around the target. For hard problems, negative deviations (A < 0) indicate insufficient exploration and are pe- nalized, thereby encouraging longer and more exploratory reasoning chains. We provide detailed visualization and discussion of these curves in Appendix (Figure 7).
Finally, the overall hierarchical reward is given by:
R(z, y: d) (z y) [acc(2, v) 0] Aa (a(i(A():)))
where hierarchical reward R(x, y; d) unifies the correctness term Rge y) with difficulty-awan entropy regularization. Specifically, the entropy penalty is applied only when the response y is incorrect, ensuring that already-correct solutions are not discouraged while encouraging exploration on failed attempts.
This formulation establishes a difficulty-aware intrinsic reward that suppresses unnecessary explo- ration easy tasks, stabilizes reasoning depth around batch-adaptive target on medium tasks, and promotes sustained exploration on hard tasks. Importantly, the entire scheme operates closed- form manner based solely on batch-level statistics, thereby achieving adaptive exploration control without introducing additional hyperparameters.
Dynamic KL Design. As established in Appendix M, the KL loss provides a valid thinking bud- get, while Appendix F further shows that a KL penalty inflates variance, motivating our exclusive of the KL loss. To realize adaptive exploration, AEPO employs only token adketivdepiive weight mechanism:
p (< 1), if Woolid
Br,
Ba Pts
P
(I)
1,
otherwise,
where Ba is the deffeullty-deendeen baseline and Pt relaxes the KL constraint inside validated high-entropy windows. This simple yet effective design ensures that KL divergence is tightly con- trolled easy tokens but adaptively relaxed at reasoning-critical segments, functioning token- wise thinking budget allocator.
Then we present the surrogate objective of AEPO, which is adapted from the GRPO and DAPO formulations discussed in Appendix D. The objective function defined as:
1o'1 min (0) Ace.
(A) Eg [oi)g m
T
1=1
(2)
clip(ss.(0), 1 -e,, 1 en) Av.)) Baior DII(TTOL: 88.e) Tree(- s(,)))

---

Published as a conference paper at ICLR 2026
where (0)
"corlof(=.e)* Si, = (q. 021), and A,, is the task+entropy shaped (token-level)
advantage induced by the hierarchical reward (Section 3.2.2).
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Datasets The training of ARES follows our proposed methodology, utilizing carefully curated datasets for each stage. The initial adaptive cold-start phase employs a dataset of approximately 224K samples, including both high-quality textual data and multimodal STEM tasks (Table 5). We then employ strong open-source cold-start model, Revisual-R1 (Chen et al., 2025b), to generate eight responses per sample and compute the pass rate. To establish a robust initial policy, textual problems are answered using the powerful DeepSeek-RI model (DeepSeek-A) et al., 2025), while multimodal reasoning tasks are addressed with Seed-1.5-VD (Guo et al., 2025). The subsequent RLVR stage with AEPO then utilizes the ViRL39K dataset (Wang et al., 2025a), collection of verifiable question-anseee answer pairs derived from existing multimodal datasets.
Baselines We benchmark ARES against three categories of baselines. First, we include lead- ing proprietary systems such as GPT-4.1 (OpenAl), 2.5--i------o---iiiii (Comanici et al., 2025), Claude-4-Sonnet (Anthropic), and -aahoo------Tiiiigg---i which serve as upper-bound references for multimodal reasoning. Second, we consider representa- tive lightweight open-source MLLMs, including Owwn2.5--L-33--Inste (Bai et al., 2025), FAST- 3B (Xiao et al., 2025), and deployment. Third, evaluate competitive 7B-scale open-source MLLMs, such as Qwen2.5-VL: 7B-Instruct, Open W.Thnkiehrre-11222 1.2-7B (Deng et al., 2025), Mar--Eurrka--ee---- MMR (Leng et al., 2025), ThinkLite-7B-VL (Wang al., 2025c), VLAA-Thinker-7B, VL-Rethinker-77 (Wang et al., 2025a), and Vision-GI (Zha et al., 2025). These represent widely adopted open-source alternatives that support research reproducibility. Most of the open-source baselines are fine-tuned from wwcn2.5-L-33B--- parability in architecture and training setup. Collectively, this suite provides a comprehensive cov- erage of both proprietary and open-source models across multiple scales.
(Guo et al., 2025),
(Chen et al., 2025a), which are suitable for efficient
(Meng et al., 2025),
or Qwen2.5-VLL V-L VL-7B-IIstruct, ensuring com-
4.2
MAIN RESULTS
As summarized Tables and 4, MRLMs consistently outperform chat models across complex rea- purpose benchmarks. This confirms that training stages
soning, mmowledge-intensiee and general-purpose such as cold-start and RLVR effectively enhance self-reflection and reasoning ability. However, we also observe a sharp drop in textual reasoning for many open-source MRLMs likely due to reliance on multimodal verifiable data without cold-start fine-tuning.
In contrast, ARES achieves strong gains both 3B and 7B scales. On multimodal benchmarks (Table 1), ARES-7B exceeds the best open-source models by +19.0 on Vision and +11.5 on MMMU-Pro. On textual reasoning (Table 4), it attains 61.7 on AIME25, where most 7B base- lines score below 3.3. These results confirm that ARES improves core reasoning ability rather than overfitting to multimodal tasks.
Finally, Table 2 and Figure 3 demonstrate the adaptivity of our training strategy. AdaCS (ARES- CS-7B) modulates response length by task difficulty, while AEPO (ARES-RI--7B) further enhances this effect- effect-extending reasoning for hard tasks (e.g., OlympiadBench, AIME25) and shortening it for easier ones (c.g., GSM8K, Math These results, consistent with our entropy-difficuly analysis (Section 2.2), show that AEPO improves both accuracy and token efficiency by encouraging exploration only when needed.
4.3 ABLATION STUDIES
To validate the contributions of the key components within the ARES framework, conduct series of ablation studies. We systematically isolate and evaluate the effects of our Hierarchical Reward Design (entropy shaping), and the Dynamic KI. Design in A ARES.
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison various MLLMs on diverse multimodal reasoning benchmarks. Within each model group (3B and 7B), the best results are highlighted in bold, and the second-best are under- lined. Scores in italics indicate that they are not reported in the original work and are obtained using the Vii (Duan et al., 2025) for evaluation. Math Verse-V, DynaMath-W and WeMath-S denotes the vision-only, worst, and strict settings, respectively.
Reasoning Benchmarks
Model
Math Verse-V
Math Vision
MaihVista DymaMath-W WeMath LugieVista MMMU MMMU-Pre CharXIV MMStur Avg
Close Searce
GPT-4.1
39.8
518
72.0
48.3
55.5
6388
75.0
65.0
55.8
71.2
6188
Gemini 2.5 Pro Thinking
81.2
55.3
83.8
57.1
73.0
75.2
82.0
76.5
69.3
79.7
73.8
Chande 4-Sonnct
05.1
54.6
70.4
46.9
63.0
64.4
74.4
00.7
58.4
06.9
62.6
Douhoo- 1.5 tinking wisipp---io-----o pro
80.4
68.7
85.5
60.3
78.0
71.8
77.9
67.6
63.4
78.2
73.2
38-ccale MLLMs
Qwen2.-VV 1.mm2.5-VL-11 1--VV---B---nsteue
33.0
21.2
62.3
17.0
17.9
35.8
53.1
31.6
25.3
51.1
34.8
FAST-30
32.2
26.8
f1..2
15.4
23.3
35.4
52.0
21.6
28.2
55.2
37.4
VI 11
36.4
24.4
61.0
18.2
33.8
38.5
49.7
33.7
28.2
53.4
37.2
ARES-38
48.2
44.2
66.8
24.6
40.8
43.5
56.8
45.2
34.4
56.7
46.1
Open 38 SUTA)
+11.0
+17.4
+0.6
+6.4
+7.0
+5.0
+3.7
+10.6
+6.
+1.5
+8.4
7B scale Models
Owea25 Instrtt
42.9
25.1
68.2
21.2
36.2
45.0
58.6
38.3
35.5
62.1
43.3
OpenVLThinkeel1.2222
40.7
25.9
72.5
21.2
37.9
41.4
58.7
42.9
39.7
62.9
44.3
MM Eureka Eureka-Qwen-TB
19.6
26.9
73.0
24.0
34.7
16.8
57.3
13.2
39.5
64.4
46.0
MMRI Math vO
45.1
3022
71.0
25.2
33.2
50.8
57.1
43.2
39.2
63.8
45.9
ThinkLite-7B-VL
45.3
32.9
75.1
22.0
26.3
40.7
55.5
41.3
39.3
63.4
44.2
VLAA-TRanker-7B
48.2
2644
68.0
22.4
29.2
48.5
34.6
41.6
36.1
64.1
44.0
VL.Rethinkee-78
49.1
523
74.9
27.4
27.8
44.5
56.7
41.7
39.8
62.7
45.
Vision G1
30.0
3133
76.1
27.2
29.0
50.2
53.4
41.2
11.0
66.0
66.5
ARES-7B
56.5
51.9
74.5
39.7
47.3
54.1
67.9
$4.8
47.0
65.3
55.9
A (Ours 7B Open SoTA)
16.5
+19.0
+1.5
+12.3
+9.3
+3.3
+9.2
+11.5
+6.0
4.7
+9.4
6000
0.00
ARES wis Entrgy
ERPO
0.70
DAPOO
4220
1000
Rogeomee
<750
0.50
4510
ARES ARIS
065
ABES
6.0
100
202 250
190
200
Steps
Figure 3: Training dynamics of accuracy (left) and response length (right). We observe that ARES achieves higher accuracy compared with its ablations (w/o KI., w/o Entropy) and baseline methods (GRPO, DAPO), while also producing shorter and more stable responses during training. These results indicate the joint use of KL and entropy shaping contributes to improving both correctness and conciseness.
4.3.1 EFFECTIVENESS OF HIERARCHER ARCHICAL REWARD DESIGN
To verify the impact of our hierarchical autropy-based reward, we conduct an experiment compar- ing model trained with only this component against the GRPO baseline. As shown in Table 3, this component alone yields significant accuracy improvement of +1.8 points, demonstrating that better depth and valid exploration lead to better performance. Additionally, this adaptive regulation clearly visible in Figure 3. While the accuracy for this model (labeled "ARES w/o KL") consis- tently tracks above the GRPO baseline, its response length shows a marked and steady decrease. This shows the model learning to curtail unnecessary reasoning (i.e., overthinking) on simpler problems. The strong performance gains difficult benchmarks like DynaMath- W (+3.2 points) further prove it correctly encourages deeper exploration when necessary. This evidence confirms the hierarchical reward's primary role adaptive regulator of reasoning depth.
8

---

Published as a conference paper at ICLR 2026
Table Accuracy and response length comparison across multimodal and textual benchmarks. re- port both accuracy (Acc) and average response length (Len) for five model variants (ARES CS Vanilla, ARES- CS- 7B. ARES-CS -SSS-SS-A Vanilla-GRPP GRPO, ARES CRES-CS-AE Vanilla-RA -L, and ARES-RL-77 on six benchmarks. Visualiza- tion of these results is provided in Figure (accuracy) and Figure (response length) in Appendix.
Multimodal
Textual
Model
OlympiadBench MathVerse-V
Math
AIME25
MATH500
GSM8K
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Acc
Len
ARES-CS-Vane Vaniila
54.2
9123.2
50.5
3432.1
72.1
1853.3
55.2 16361.6
91.4
2750.1
358.8
ARES-CS-7B
53.4
10281.5
52.8
3575.4
72.3
1712.3
55.0
17895.0
93.4
3011.8
92.3
332.3
ARES-CS- Vanilla-GRPO
PRES-CS-Vanilla-GG
55.8
10342.5
$2.3
4256.9
71.9
2432.5
58.2
18520.3
93.9
3185.2
93.8
395.4
ARES-CS CHES-CS-Vannlla-I.
56.4
11050.4
54.7
3327.8
73.5
1573.1
60.4
20135.4
94.1
2681.5
94.5
302.4
ARES-RL-7B
56.9
12893.4
56.5
3198.3
74.6
1494.5
61.7
22618.8
95.2
2257.7
95.7
278.9
Table 3: Ablation study of Dynamic KT. Loss and Entropy Reward. Building upon our Cold Start stage. Best results column are bold and second-best are underlined.
Ablation Setting
MathWerse-1
Math Vision
MathVista
DynaMath-W
Avg
Cold start only
52.8
49.7
72.9
27.4
50.7
Cold Start GRPO (haseline)
53.7
50.4
73.3
36.1
53.4
Cold Start + DAPO (baseline)
54.1
50.9
73.4
37.9
54.1
Cold Start Dynamic KL
55.2 (+2,4)
51.7 (+2.0)
73.7 (+0.8)
(+10,9)
(+4.0)
Cold Start Entropy Shaping
55,9 (+3.1)
31.3 (+1.6)
74.4 (+1.5)
39.3 (+11.9)
(+4.5)
ARES-78
56.5 (+3.7)
51.9 (+2.2)
74.6 (+1.7)
39.7 (+12.3)
55.7 (+5.0)
4.3.2 EFFECTIVENESS OF DYNAMIC KI. DESIGN
To verify that the dynamic KL mechanism allocates the exploration budget efficiently, we analyze its isolated contribution. The results in Table 3 show that adding only the dynamic KL component improves average accuracy by +1.3 points over the GRPO baseline. Its role as an efficient budget allocator is best illustrated in Figure 3. The full ARES model's accuracy curve ultimately converges to a higher final accuracy than all other variants. Concurrently, its response length curve shows the most pronounced reduction, indicating that the token-wise budget allocation pushes the model to find more efficient reasoning paths. This outcome validates that the dynamic KI. design successfully allocates the thinking budget, guiding the policy toward greater reasoning efficiency and higher accuracy.
Finally, combining the hierarchical reward with the dynamic KL mechanism results in the full ARES framework. As shown in Table 3, this complete model achieves the highest average accuracy of all configurations (55.7). This result corroborated by Figure where we see the full ARES model not only converges to the highest final accuracy but also achieves the most significant reduction in response length. This demonstrates the synergistic effect of our two components: by simultaneously regulating reasoning depth and efficiently allocating the exploration budget, ARES achieves both stronger accuracy and more efficient training.
5 RELATED WORK
5.1 MULTIMODAL LARGE REASONING MODELS
Reasoning is widely recognized as a foundational element of intelligent behavior (de Winter et al., 2024; Bi al., 2025). To meet the demands of realistic multimodal environments, Multimodal Large Reasoning Models (MLRMs) fuse information from heterogeneous modalities to support deeper thinking. Extending chain-of-thought (CoT) fine-tuning to instill stepwise reasoning, works such as LLaVA-CoT (Xu et al., 2025) and Lla (Thawakar et al., 2025) transpose this paradigm to the multimodal setting. Following the success of DeepSeek-Ri (DeepSeck-Al et al., 2025), a growing body of research leverages reinforcement learning (RL) to extend long-horizon reasoning in MLRMs (Peng et al., 2025; Shen et al., 2025a). Against this backdrop. our goal is to build an adaptive MLRM that decide how much reasoning effort allocate inference time.

---

Published as a conference paper at ICLR 2026
Textual Reasoning Benchmarks
Model
AIME24
AIME25 MATH500
MMLU Pro
BBEH
GPQA
Avg.
Closed Source
GPT-4.1
46.7
23.3
89.6
80.2
32.7
67.7
56.7
66.7
88.0
85.8
84.3
17.7
82.3
70.8
Claude-4-Sonnct
92.0
35.0
68.7
Doubao- 1.5-Thinking- Vision-Pro
96.8
39.7
71.2
3B Sc-scale MILMs
Qwen2.5-VL-33 -n-3n-nnsttctt Instruct
0.0
0.0
55.8
41.5
7.0
29.3
22.3
FAST-3B
6.7
3.3
55.4
34.6
8.6
25.3
22.3
VLAA-Thinkers
3.3
0.0
61.4
33.3
8.1
27.8
22.3
ARES-3B
41.5
34.8
86.2
55.8
15.0
41.4
45.8
A (Ours-Oue Open 3B SoTA)
+34.8
+31.5
+24.8
+14.3
+6.4
+12.1
+20.7
7B-scale MILMs
Qwen2.5-VL- WL--B-LLnsstee
6.7
0.0
66.0
50.5
10.8
36.9
28.5
VLTnVL.T Bickkerr-122--7 1.2-7B
0.0
0.0
63.2
49.3
13.6
31.3
26.2
MM-Eureka-QU A1-Burekaa-wwen71
13.3
0.0
65,4
53,4
12.2
41.4
31.0
MMR1-Math-a
3.3
3.3
66.6
51.8
11.4
32.3
28.1
ThinkLite-BB-VL
3.3
3,3
67.6
51.9
11.3
35.9
28.9
VLAAA-Thinke--78
6.7
3.3
68.6
51.2
11.7
28.8
28.4
VL-Rethinker-711
3.3
3.3
64.4
$2.0
12.9
31.8
28.0
Vision-GI
3.3
0.0
69.0
53.0
13.9
31.3
28.4
ARES-7B
61.7
95.2
67.0
18.9
49,5
(Ours-7B Open SoTA)
+51.7
+58.4
+26.2
+13.6
+5.00
+8.1
+27.2
Table Performance on textual reasoning benchmarks. AIME24 and AIME25 results are av- eraged over eight independent inference to reduce score variance. Results AIME24/25, MATH500, MMLU Pro, BBEH, and GPQA. ARES-3B and ARES-7B substantially outperform all open-source baselines their respective scales, achieving large average gains (A rows) and narrow- ing the gap to leading proprietary systems.
5.2 ADAPTIVE REASONING
Current large reasoning models such as OpenAl ol (OpenAI al., 2024) and DeepSeek RI (DeepSeek-Al al., 2025) show notable reasoning capabilities while often generate excessively lengthy reasoning for trivial questions while providing insufficient exploration for more challenging ones (Zhu & Li, 2025; Alomrani et al., 2025). To address this limitation, numerous studies have sought establish adaptive reasoning mechanisms. One line of research develops training-free ap- proaches based on prompt engineering (Jiang et al., 2025; Yang et al., 2025a). Complementary to these approaches, trining-based methods aim instill more strategic adaptive reasoning. Some studies curate variable-length datasets containing both concise and elaborate reasoning chains for supervised fine-tuning (SFT) (Yu et al., 2025a; Ma al., 2025), while others leverage reinforce- ment learning (RL) to encourage dynamic control over reasoning depth by integrate length penal- ties (Arora & Zanette, 2025; Tu al., 2025) or employ ffffiuult--warrness mechanism (Chen et al., 2025e; Xiang et al., 2025). However, most work still focuses on text-only reasoning, and these approaches often over-encourage exploration on hard problems. In this work, we take an ini- tial step toward multimodal adaptive reasoning framework, and treat entropy localized signal to encourage concise solutions for casy questions and sustained exploration on hard ones.
CONCLUSION
This paper introduces ARES, a framework designed to cultivate adaptive reasoning in MLRMs, ad- dressing their tendency overthink simple problems while under-exploring complex ones. ARES uses two-stage training curriculum: an Adaptive Cold-Start phase instills difficulty awareness, fol- lowed by our novel Adaptive-Enttropy Policy Optimization (AEPO). AEPO leverages high window- entropy to determine when to explore and hierarchical reward to control how much reasoning depth applied. Experiments confirm that ARES achieves superior performance and significantly improves reasoning efficiency, validating our adaptive, entropy-guided approach.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide the full implementation of our ARES framework in an anonymous code repository. Our models built upon the publicly available Qwen2.5-VL-3B and Sowen22.5-VL-7B--nnsttn
The datasets used for all training and evaluation stages, including the Adaptive Cold-Start and AEPO phases, curated from public sources. Furthermore, the Appendix provides comprchensive breakdown of our data curation process, experimental setup, and key hyperparameters to ensure our results can be replicated.
ETHICS STATEMENT
We are not aware of any major ethical concerns arising from our work. Our study is conducted entirely within the academic domain of multimodal reasoning, using only publicly available models and datasets for training and evaluation. No human subjects were involved, and our research does not introduce sensitive or potentially harmful capabilities. The objective of ARES is to advance research adaptive and efficient reasoning for the open-source community, and we do not foresee any direct societal risks from this work.
REFERENCES
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
Mohammad Ali Alomrani, Yingxue Zhang, Derek Li, Qianyi Sun, Soumyasundar Pal, Zhanguang Zhang, Yaochen Hu, Rohan Deepak Ajwani, Antomios Valkanas, Raika Karimi, Peng Cheng, Yunzhou Wang, Pengyi Liao, Hanrui Huang, Bin Wang, Jianye Hao, and Mark Coates. Reasoning on a budget: A survey of adaptive and controllable test-time compute in llms, 2025. URL https:
//arxiv. org/ aa/s//ba//// 2507. 02076.
Anthropic. Introducing claude URL https: / /ww. anthropic com www.m//wws//ee htwp///www.eeeeeeeeeeeetaa claude-4u 4.
Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025. URL
https://aat /atpii//aaiiviiiiia org/ abss /202222522222 04463.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 1xXx:2502.1392. 2025.
Jing Bi, Susan Liang, Xiaofei Zhou, Pinxin Liu, Junjia Guo, Yunlong Tang, Luchuan Song, Chao Huang, Guangyu Sun, Jinxi He, Jiarui Wu, Shu Yang, Daoan Zhang, Chen Chen, Liang- gong Bruce Wen, Zhang Liu, Jiebo Luo, and Chenliang Xu. Why reasoning matters? a survey of advancements in multimodal reasoning (vl), 2025. https: www.aawwwwwww..///////aa/ 03151.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1 rike reasoning large vision-language models. arXiv preprint arXiv:2504.1 11468, 2025a.
Jiaqi Chen, Jianheng Tang, Jinghui Qin. Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105. 100.:11175........ 14517,
2021.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746, 2022.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin. et al. Are we on the right way for evaluating large vision-lane langugge models? arXiv preprint arXiv:2403.20 20333, 2024.

---

Published as a conference paper at ICLR 2026
Shuang Chen, Yue Guo, Zhaochen Su, Yafu Li, Yulun Wu, Jiacheng Chen, Jiayu Chen, Weijie Wang, Xiaoye Qu, and Yu Cheng. Advancing multimodal reasoning: From optimized cold start to staged reinforcement learning. arXiv preprint arXiv:2506.0422:
2025b.
Weize Chen, Jiarui Yuan, Tailin Jin, Ning Ding, Huimin Chen, Zhiyuan Liu, and Maosong Sun. The overthinker's s diet: Cutting token calories with aifficulty-aware training, 2025c. URL https:
/ //arxiv.org aaaaab//22 22005 19217.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inder- jit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpek- tor, Nan-Jiang Jiang. Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kaw- intiranon, Tania Madrax-Weiss Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Ilai Deu- tel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Wagh- mare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj. Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mah- moudieh, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dim- itris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, An- drey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang, Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co- Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next genera- tion agentic capabilities. CoRR, 0bbs/2507.0626/1 2025. doi: 1044550/ARXIV.25000060 06266. URL https: //t:://doiii/ /tog 48550/ arxiv. 2507. 06261.
Joost C. F. de Winter, Dimitra Dodou, and Yke Bauke Eisma. System 2 thinking in openai's ol-preview model: Near-perfeet performance on mathematies exam. Computers, 13(11): 278, October 2024. ISSN 2073-431X. doi: 00.330//computer133110272 //dx. org/ 3390/ http:///00.3330//pttrrr111111110
URL http:
computers13110277
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjic Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. CoRR, abs/2501. 12948, 2025. doi: 10. 12948. URL https: //wii org/lo 10. 48 5500/000000 arxiv. 2501. 12948.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Rai-Wei Chang. Openvlthinker: An early exploration to complex vision- reasoning via iterative self- relf-improvement, 2025.
URL https:/ /arxivv...... httgg/bb///55/ 17352.
Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao Yuxuan Qiao, Mo Li, Amit Agarwal, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Xiaozhe Li, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Shengyuan Ding, Tianhao Liang, Zicheng Zhang, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An ooen-source toolkit for evaluating large multi-modality models, 2025. URL https: : / aarxivi vvv.vvv. wttg/ba/////a/ 11 11691.
12

---

Published as a conference paper at ICLR 2026
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: //githubb www.///ggigggaa/ec///oo/oo//o//e open-rl. 2l.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint arXiv:2506.04111 2025.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng. Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Si- hang Yuan, Sijin Wu, Tianheng Cheng. Weiwei Liu, Wenqian Wang, Xianhan Zeng, Xiao Liu, Xiaobo Qin, Xiaohan Ding, Xiaojun Xiao, Xiaoying Zhang, Xuanwei Zhang, Xuehan Xiong, Yanghua Peng. Yangrui Chen, Yanwei Li, Yanxu Hu, Yi Lin, Yiyuan Hu, Yiyuan Zhang, Youbin Wu, Yu Li, Yudong Liu, Yue Ling, Yujia Qin, Zanbo Wang. Zhiwu He, Aoxue Zhang, Bairen Yi. Bencheng Liao, Can Huang. Can Zhang, Chaorui Deng, Chaoyi Deng, Cheng Lin, Cheng Yuan, Chenggang Li, Chenhui Gou, Chenwei Lou, Chengzhi Wei, Chundian Liu, Chunyuan Li, Deyao Zhu, Donghong Zhong, Feng Li, Feng Zhang, Gang Wu, Guodong Li, Guohong Xiao, Haibin Lin, Haihua Yang. Haoming Wang, Heng Ji, Hongxiang Hao, Hui Shen, Huixia Li, Ji- ahao Li, Jialong Wu, Jianhua Zhu, Jiampeng Jiao, Jiashi Feng, Jiaze Chen, Jianhui Duan, Jihao Liu, Jin Zeng. Jinggun Tang. Jingyu Sun, Joya Chen, Jun Long, Junda Feng, Junfeng Zhan, Junjie Fang, Junting Lu, Kai Hua, Kai Liu, Kai Shen, Kaiyuan Zhang, and Ke Shen. See vl technical report. CoRR, abs/2505.07000 070022, 2025. doi: 104455WARIIV2500.0777 https:: /doi www.10........ 48550/ a 2505 07062.
URL
Tingxu Han, Zhenting Wang. Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen, Token- budget-aware IIm reasoning, 2025. URL https::///aaaaa //axivv. org/ abs///b/////a /
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11455 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 103.03874, 2021.
Shijue Huang, Hongru Wang. Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, and Yi R. Fung. Adactrl: Towards adaptive and controllable reasoning via difficulty- awwae budgeting, 18822.
wrg/abs/////ooooooooot...ot..srt 2025a. URL https::////// //tpxa/avvv/////////a/taa//aawa/
Shijue Huang, Hongru Wang, Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, and Yi R. (May) Fung. Adactrl: Towards adaptive and controllable reasoning via difficulty-aware budgeting. CoRR, 054/2505.18822 2025b. doi: 10.4855WARIV.2505..160 //doi..org/1/ 18850//aaXia 2505 18822.
URL https:
Guochao Jiang, Guofeng Quan, Zepeng Ding. Ziqin Luo, Dixuan Wang, and Zheng Hu. Flashthink: An early exit method for efficient reasoning, 2025. htt.tttss///axxiv. 13949.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint 1rXiv:1710.07:30 2017.
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Peter Chen, et al. Big-bench extra hard. arXiv preprint arXiv:2502.1911 19187, 2025.
Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Hang Zhang, Yuming Jiang. Xin Li, Deli Zhao, Fan Wang, Yu Rong, Aixin Sun, and Shijian Lu. Mmrl: Advancing the frontiers of multimodal reasoning. https: //// thub. com m LengSi weweeniingggnnnnieeeeeee cong/MM11, 2025.
13

---

Published as a conference paper at ICLR 2026
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. https: / //huggt waw.ac....... ca coo/datasete. Numi namath, 2024a. Hugging Face repository, 13:9.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024b.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kontyeewki, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robust- ness in visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14963-14973, 2023.
Zehui Ling, Deshu Chen, Hongwei Zhang, Yifeng Jiao, Xin Guo, and Yuan Cheng. Fast on the easy, deep on the hard: Efficient reasoning via powered length penalty, 2025. URL https: // //arxiv... org/ /obbs/2506 10446.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. 2021a.
arXiv preprint ar arXiv:2105.04165
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021b.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209, 14610, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
Xinyin Ma. Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length- compressible chain-of-thought tuning, 2025. URL 09601.
wwtp:/iwwwwg//a//////aaaaa 2502.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Bo- tian Shi, Wenhai Wang, Junjun He. Kaipeng Zhang, Ping Luo. Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring visual aha moment with rule-based large-scale rein- forcement learning. CORR, abs/2503.07365. 2025. doi: 10.48550ARXIV2250..7777 arwiv...vvvv. 2503.07363
URL
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint wrXiv:2402.148811 2024.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint w:Xiv:5504166811 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Perey Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arX1v:2501.19395 2025.
OpenAl. Introducing gpt-4.1 in the api. URL
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich,
14

---

Published as a conference paper at ICLR 2026
Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhhagg Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang. Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe. Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins. Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes. Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan. Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders. Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL https:/ /aaaaaivvvooo www.iv.org/aaaee /ttpaa/.../.2222222
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multi- math: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.000 001407 2024.
Yingzhe Peng. Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl, 2025. URL https: / arxiv. org/ abs/ 07536.
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multi- modal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407. 0122444 2024.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian- Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. CoRR, abs/2503.21614 2025. doi: 10.48550/ 2RX1V.2503.21614 URL httph:tt///dtt .org/11 485..4855//680 2.00. 2503.21614,
15

---

Published as a conference paper at ICLR 2026
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proot q&a bench- mark. In First Conference on Language Modeling, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma. Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: A stable and generalizable rl r1-style large vision-language model, 2025a. URL https:/ /ttsss//arxiii /awxixxxivvvooo../aaa abs/ / 2504. 07615.
Yi Shen, Jian Zhang. Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. Dast: biffiuult- slow-thinking for large reason- arg/abs///
ing models, 2025b. URL https: //tt://araae arxiv..vivv org/ abs /20. 04472.
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, and Salman Khan. Llamav-ol: Rethinking step- by-step visual reasoning in Ilms, 2025. URL https::///// // /arxivv.vvv./////////////////av/ org/ waa.b//5011 06186.
Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. Learning when to think: Shaping adaptive reasoning r1-style models via multi-stage rl,
2025. URL https: : /awxiv.vvv. org/ abs /
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning- arXiv preprint arXiv:2504.0888 2025a.
Ke Wang. Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math- vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169 2024a.
Shenzhi Wang, Le Yu. Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song. Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoming. CoRR, abs/2506.01939, 2025b. doi: 10. 48550/ARXIV.2506.01133 URL
org/ 10. 48550/ arxiv. 2506 11.506019900 01939.
Xiyao Wang. Zhengyuan Yang, Chao Feng. Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: gets-guided sample selection for data-efficient visual reasoning self- eelf-improvement. arXiv preprint ar5iv:2504.000 009944, 2025c.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266 95290,, 2024b.
Yunhao Wang, Yuhao Zhang, Tinghao Yu, Can Xu, Feng Zhang, and Fengzong Lian. Adaptive deep reasoning: Triggering deep thinking when needed. CoRR, abs/2505.20101, 2025d. doi: 10. 48550/ARXIV.2505. 201000. URL https: //wii.ii.//////// waalli..og//..o /10 48550/ www.v..w......... 20101.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal 1lms. Advances in Neural Information Processing Systems, 1111113569-1119999 2024c.
Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, and Nick Haber. Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning, 2025. URL https: atwav.iiiiv.. wttp:/iiiorr.aaoooooooeo.ooe aba//bb////55 05256.
Wenyi Xiao, Leilei Gan, Weilong Dai, Wanggui He, Ziwei Huang, Haoyuan Li, Fangxun Shu, Zhelun Yu, Peng Zhang, Hao Jiang, al. Fast-slow thinking for large vision-language model reasoning. arXiv preprint 2rXiv:2504.18455 2025.
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal I1m logical reasoning benchmark in visual contexts, 2024. URL https: /
wwtgg/bb/////e /2407 .04973.
16

---

Published as a conference paper at ICLR 2026
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by stp-by-ste.... 2025. URL URRJtttps//acxxivoo/////// 10440.
Chenxu Yang, Qingyi Si. Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit reasoning models, 2025a. URL / org/abaa 2504 15895.
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weip- ing Wang. Dynamic early exit in reasoning models. CoRR, abs/2504.15895, 2025b. doi: 10. 48550/ARXIV.2504.
URL https:/ //oi://///i.// org/ 10. 48550/ arxiv. 2504 15895.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387 2025.
Bin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. short chain-of of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models, 2025a. URL https:ttpt.c.. /ttp:///aavvvv.. org/abs
abs// /tcaab///5050 2505. 03469.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo. Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang- Dapo: An open-source IIm reinforcement learning system at scale, 2025b. URL https: //w.xiv.org/aa//////.. 14476.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.0025 2024b.
Yuheng Zha, Kun Zhou, Yujia Wu, Yushu Wang, Jie Feng, Zhi Xu, Shibo Hao, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. Vision-gl: Towards general vision language reasoning with multi- domain data curation. arXiv preprint arXiv:2508. 12680, 2025.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin. Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Chang. Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024a.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei. Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv e-prints, pp. arXiv-2407, 2024b.
Shengjia Zhang, Junjie Wu, Jiawei Chen, Changwang Zhang, Xingyu Lou, Wangchunshu Zhou, Sheng Zhou, Can Wang, and Jun Wang. Othink-rl: Intrinsic fast/slow thinking mode switching for over-reasoning mitigation, 2025. URL https: : aawiv. org/ abs 2506. 02397.
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhou- futu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return, entropy-eliciting explore, 2025. URL https: /ttps:///rxxvvv arxiv. org/ abs/
Jason Zhu and Hongyu Li. Towards concise and adaptive thinking in large reasoning models: A survey, 2025. URL https: : http//xxivvvv///////ee////aa...a org/ abs/2500 09662.
Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models, 2025. URL https: www.ivvvvvvv org/ abs /2411 00836.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
APPENDIX CONTENTS
LLM Usage Statement
18
B Experimental Settings
19
B.1 Benchmarks
19
B.2 Implementation Details
19
C Token-Level Entropy Measurement
19
RLVR Algorithms
20
D.1 Group Relative Policy Optimization (GRPO).
20
D.2 Dynamic sAmpling Policy Optimization (DAPO).
E Window-Entropy Aggregation
21
KL Penalty Inflates GRPO Advantage Variance compared KL Loss
22
G The Algorithm workflow of AEPO
23
H On-policy difficulty and buckets
23
I High-Entropy Tokens
23
Visual Analysis of Entropy Reward Design
25
K Visualization Results
26
Why High-Entropy Tokens Predict Reasoning Response Length
26
M Why KL Loss is a Valid Thinking Budget
29
Fesmery Justification of AEPO
30
Case Study
32
A LLM USAGE STATEMENT
In this paper, we employed a Large Language Model (LLM) in an assistive capacity for this manuscript. Its use strictly limited for grammatical accuracy and refining sentence structure to improve clarity and readability. All intellectual contributions, including the formulation of ideas, data analysis, and the composition of the manuscript, entirely the work of the authors.
18

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL SETTINGS
B.1 BENCHMARKS
To comprchensively evaluate the capabilities of our model, we selected a diverse suite of benchmarks designed to test wide range of reasoning skills. In the multimodal domain, we assess multimodal mathematical reasoning using Vatheerre (Zhang et al., 2024a), MathVision (Wang et al., 2024a), MathViste (Lu et al., 2023), DynaMath (Zou et al., 2025), and WeMath (Qiao et al., 2024). To evalu- ate broader logical and general-purpos multimodal capabilities, we incorporated Logic (Xiao al., 2024), MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), CharXIV (Wang al., 2024c), and MMStar Chen et al. (2024). For text-based reasoning, we measured the model's ' abil- ity to solve challenging mathematical problems with AIME24/25 (Li et al., 2024a) and MATH- and reasoning proficiency were
500 (Hendrycks et al., 1)21), while its general question-answering tested on GPQA (Rein et al., 2024), MMLU Pro (Wang et al., 2024b), and BBEH (Kazemi et al., 2025). For all evaluations, we report pass I accuracy, with the exception of the AIME24/25 bench- mark, for which performance is measured using average @ accuracy.
B.2 IMPLEMENTATION DETAILS
are trained using a two-stage strategy. The first stage is cold-start supervised fine-tuning (SFT), where we indepen- dently fine-tune the large language model (LLM) module for two epochs. This stage uses a batch size of 256, a sequence length of 32k, and a learning rate of 2 x 10-5, Following SFT, we further optimize the model using Adaptive Entropy Policy Optimization (AEPO). In each AEPO iteration, we sample 512 prompts from the training set and generate eight rollouts per prompt, yielding 4,096 trajectories. Rollouts are generated with temperature of 1.0 and top-p of 0.99, and the maximum sequence length is set to 20k tokens (a 4k token prompt and a 16k token response). For policy up- dates,
Our models, based on Qwen2.5 2- WLV-L--IIsstrn and -wven..5-VL33--Inse
a global batch size of 128 and the AdamW optimizer with learning rate of 1 x 10" and a weight decay of 1 x 10-2, without a learning rate warmup. To ensure training stability, we ap- ply online filtering discard samples whose mean overall reward falls outside the predefined range of [0.01, 0.99]. This process removes outliers that could distort advantage estimation and impede policy updates.
Compute Efficiency. Stable-MM-RI requires = 16 rollouts per query during the initial PAQM phase. While this incurs one-time preprocessing cost (approx. 1.5x standard epoch time), the resulting training stability allows for faster convergence. As shown in Figure ??, our method reaches peak performance in fewer steps than baselines, effectively amortizing the initial rollout cost.
Table 5: Textual and multimodal reasoning datasets source of ARES cold-start data.
Multinnoaa
Test-anly
Sample SSuree
Samples
Sumples Saurre
Somples
FigunQA
Supor-13
30K
Wig--Mrth--
[GAIR (ND
D.K
MAVIS (Zhang et al., 2004b)
2158
TuhNWW (La etall 20121
38K
Big Mutt U'Olbaakk 2025)
35%
sIK 1.1 (Mucsnighotf 2025)
GuiQA (Cham 20213
Uniten (Ches 2022)
16K
OpenThougo (Clala et al, 2025)
OpenMatliR (Moddkos 2025)
3.200K
GeomoryyKK 3K (1a at. 2021a)
2.1k
MattiMath (Peng et 2054)
300K
DeepMaath (ite 2025)
10.9
OrcaMarh (Mtmm 2004)
200K
BeegAA (L. et 202,0)
107K
Grammer (Clen et 20236)
30K
2p0k (lice, 2025)
2208
NumisaMahh CoT (1. 2024b)
859K
C TOKEN-LEVEL ENTROPY MEASUREMENT
To quantify the model's local uncertainty during generation, we compute the token-level entropy at step &
H1 Σ
Pr. log Poji
Pt To q, oct)..
where Te denotes the policy under parameters 0, is the input query, and Ocr is the prefix of gener- ated tokens up position L. - The probability vector pt E IR is obtained from the normalized logits 2; by temperature-scalee softmax transformation:
Pt Softmax (+)
with vocabulary size V and decoding temperature T controlling the sharpness of the distribution.
19

---

Published as a conference paper at ICLR 2026
Role in our setting. Unlike sequence-level measures (e.g., log-likelihood of the whole output), H2 captures the fine-gres ame-grained uncertainty at each generation step. During reinforcement learning, treat H2 an intrinsic signal of "reasoning hesitation": higher entropy indicates that the model considers multiple plausible continuations, whereas a lower entropy suggests that the model is confident about the next token.
Note that H, is tied to the distribution Pr. not to the realized token o, drawn from Pt- Hence two identical tokens generated different timesteps may correspond to different entropies, depending on the local distributional context. In this sense, token entropy should be interpreted as a property of the model's 's decision process rather than of the discrete outcome itself.
Qwen? 5--i--al--
MLAA-Tinke-11
Qwant S-T--TB-ltttms
VLAA-Thinkm--11
MM--Bmaa--11 AREESTB
ThirkLae--B-VL
VL-Rastinke--BB
70
65.9
61.7
$5.8
Accuracy (%4
50-
30-
10
MathVerse
MachYsion
DynaMach
MMMU
MMMU-PPo
CharXTV
AIME24
AIME25
MMLU Pro
Arg
Figure 4: Accuracy comparison across selected open-source reasoning models on nine multimodal and textual benchmarks. Each group represents 3B-scale and 7B-scale models evaluated under the same benchmarks. The rightmost column ("Avg..") reports the average accuracy over all selected benchmarks, showing the overall advantage of the proposed adaptive reasoning framework. Our ARES-7B achieves superior performance.
D RLVR ALGORITHMS
In this subsection, we provide additional details of the baseline RLVR algorithms used in periments, namely Group Relative Policy Optimization (GRPO) and Dynamic sAmpling Policy Op- timization (DAPO). Both methods extend traditional policy optimization frameworks to improve the stability and effectiveness of reinforcement learning for reasoning tasks, yet they differ in how they handle variance reduction, KL regularization, and sampling dynamics.
D.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO).
GRPO (DeepSeek-Al et al., 2025) extends conventional policy optimization by introducing group- wise normalization. Specifically, training samples are divided into K groups (Gi,..., . Gx ) based on input prompts or task-specific criteria. For each group Gi. both a policy and a frozen reference policy Te... are maintained. The GRPO objective can be written as:
Te( 1- c, 1, A(s. A ) To.e (1)x)
En Gae (v|x)
min
A(x, v), clip (z)z)
To..
where e controls the size of the trust region. The group-relative advantage A(x, y) is defined as:
Å(x, wi ((2,1,) mean ((r(,, .I,,..., r(x, HG)))
std (r((,()))))))).))))))))))))))))) r(z, vG))) +
This relative advantage stabilizes training by normalizing each sample's 's reward against the variance of its group. By leveraging intra-group normalization. GRPO encourages responses that are better than their peers, while maintaining diversity across groups
20

---

Published as a conference paper at ICLR 2026
0.90
Window
Window
0.85
Window
Window
Window 32
0.80
Single Token F1
E1 Score
0.70
0.65
0.60
0.55
0.50
0.45
50
60
70
90
Threshold (Percentile)
Figure 5: F1 Score vs. threshold percentile across different window sizes. Window-based entropy aggregation consistently outperforms single-token entropy, especially at higher thresholds.
D.2 DYNAMIC SAMPLING POLICY OPTIMIZATION (DAPO).
Building GRPO (DeepSeck-AI et al., 2025), DAPO (Yu et al., 2025b) removes the explicit KL penalty and instead incorporates several mechanisms for improved sample efficiency. First, intro- duces clip-higher strategy to reduce wver-ennalization of promising trajectories. Second, it adopts dynamic sampling, where response sampling probabilities are adaptively adjusted based on entropy signals. Third, DAPO employs token-level policy gradient updates with overlong reward shaping. The DAPO objective is formulated as:
G
lo'l
(D) B(g.a)<D, ( (+1))
min (ri(0), clip(r;(0), - CLows + Chight 4$)
DG
|00|
where (0) is defined in GRPO, and A is the token-level relative advantage. Unlike GRPO, which focuses on group-level normalization, DAPO shifts optimization to the token level, enabling finer-grained control and adaptive shaping of the learning signal. This makes DAPO strong base- line for RLVR without requiring value network.
Both GRPO and DAPO contribute to reducing variance and stabilizing policy optimization in RLVR. GRPO leverages group-relative normalization, while DAPO refines the approach with token-level updates and dynamic sampling. In our work, we adopt DAPO the primary baseline due to its strong empirical performance and its widespread use in prior RLVR literature.
E WINDOW-ENTROPY AGGREGATION
While token-level entropy H, provides a fine- fine-grained measure of uncertainty, single-token fluctu- ations are often dominated by lexical ambiguity or local randomness. To capture more coherent reasoning segments, we aggregate entropy over sliding window of length w:
H
H
This windowed statistic smooths out spurious token-level spikes and highlights regions where the model persistently maintains high uncertainty.
21

---

Published as a conference paper at ICLR 2026
A window is flagged high-entropy if its average entropy exceeds dynamic threshold 9. which is updated online to match the distributional scale the current batch, which is shown in Section 2.1. Intuitively, these windows correspond to stretches of the output where the model is genuinely un- certain about its reasoning trajectory, as opposed to momentary ambiguity on function words or symbols.
The aggregated entropy Hew therefore serves two purposes: (i) it localizes "hard thinking" phases within the output, providing natural signal for allocating additional exploration, and (ii) it stabilizes training by reducing sensitivity to token-level noise. In our AEPO algorithm, only tokens falling inside validated high-entropy windows are granted relaxed KL budgets, thereby ensuring that extra computation is spent precisely where reasoning effort is most needed.
F
KL PENALTY INFLATES GRPO ADVANTAGE VARIANCE COMPARED TO KL LOSS
For fixed prompt x, GRPO samples group of responses {yi) i.i.d. from na(: x). Let the lask- tenr-oyy shaped return be
Cent (ar, Vi)
S - Tace ys)
Ad
(5)
Zent
and the sample-averaged per-token reference KL estimator be
K;
A
kldr.i-
(6)
L.
We compare two GRPO implementations:
KL penalty (merge KI. into the return):
R4 S RKe
A
R4
R
R
R
j=1
KL loss (actor-only):
= = S.
A = R R, R R= Σ
Rg
and add the separate actor regularizer
1 7 E, 3, klde, which does not enter A,
Assume ((St, K.) are i...d. with finite second moments, and denote
o' = Var(S), of Var(K), Cov(S, poSOK.
Lemma 1 (Group-baselin advantage variance). For ii.d. R,,..., Ry with finite variance, the GRPO group-baseline advantage A; = Rí R satisfies
Var(A:) (1 +) Var(R)..
Proof. Write A; -1- R, 1. Ejt: Rí and use i.i.d. plus independence across indices to expand Var(A,); cross-terms cancel and equation 7 follows.
Proposition 1 (KL penalty inflates GRPO advantage variance). Under the assumptions above,
Var(A() Var(A,) = (1 (1-N) x) (N"O - 2 Cov(SS K))
In particular, if Cov (S, K) z 0 (empirically common), then O for any K. + O. More generally, unless K E (o, Cov(S, K)/ok) (a narrow interval when Cov(S,K) is small), one has Var(A,) > Var(A.).
(1- +) o2 ) >
Proof. By Lemma 1, it suffices compare Var( Var(R;) to Var (Ra). We have Var(R,) Var(S) o3 and
Var(Rt) Var(S, -k)) o3 x x20 2x Cov(S, K). Subtracting and multiplying by (1 }) yields equation 8.
kKi)
22

---

Published as a conference paper at ICLR 2026
evaluaa eeluateelteeratiiv
Con's interpreiataee AND
PRNUELLER rSCEES PRESTRM
Sccarityy
a
Xiustity
t
di differen
ex O
Srerence
C
e
ause
componintt
contradit
aecording
ction
qullize
ctratep
seciltiea
determine
rerter
a
secor
boo
iSSumption in
ass
sume
tealsy
mint
conc I concept
ratter
e
S
Lanteaally Tofe
Figure 6: A word cloud visualization of semantically filtered high-entropy tokens, where font size reflects relative frequency. These tokens (e.g. explain, assume, constraint, conclude) correspond to reasoning triggers that mark the onset of logical transitions, highlighting the interpretable basis of our entropy-based reward.
Consequences for gradient variance. Let the score-function term be G; =i= Vo log ( $1,)) and suppose, as is standard in variance analyses, that A and are weakly correlated. Then Var (A,G.) ~ Var(A E > Var(A, E|G = Var(A,G,))
whenever Var(A) > Var(A,) by Proposition 1. In the KL-loss implementation, the additional KL gradient is E, Be Vekldr, not multiplied by the group advantage and therefore does not inherit its variance amplification; it also targets the trust-region deviation per-token granularity, improving credit assignment.
Merging per-token KL into the return injects high- fig-frrquency, task-agnostic signal (K;) into the groupwise competition, thereby amplifying the variance the advantage and the policy-gradient estimator. Using KL as an actor-only regularizer decouples the trust-region control from the task/en- tropy signals, keeping the advantage low-variance and allowing a dedicated controller to track the KL budget.
G THE ALGORITHM WORKFLOW OF AEPO
In this section, we provide a detailed flowchart of our AEPO algorithm in the following diagram 1.
H ON-POLICY DIFFICULTY AND BUCKETS
Instance difficulty estimated online using group rollouts. With K-8 samples per prompt,
8
pass@8(x)
(korrect(y))) x)} xx))
A=1
and we assign buckets
easy, pass@8(z) 2 6,
d(z) medium, 3 pass@8(x)) 6,
(10)
hard, pass@8(z) 2.
Buckets are recomputed each iteration and will parameterize per-bucket budgets/targets our method.
I HIGH-ENTROPY TOKENS
In our analysis, we define the number of high-entropy tokens in trajectory by combining an entropy- based thresholding procedure with semantics-aware filtering step.
'This assumption is used for comparative purposes; the conclusion also holds under more general covari- decompositions.
23

---

Published as a conference paper at ICLR 2026
Algorithm 1: Edtptive-Entropy Policy Optimization (AEPO)
Input Reference model Traf; initial policy æHs prompts D:
group size window size w; rollout parameters (temperature, top-p);
entropy quantile = 0.95; relaxation factor < 1;
difficulty buckets (easy, medium un, hard} with KL targets {dat.
entropy weights { J. base KL weights {5a), and shaping functions Sa(-);
GRPO clipping parameter e; controller step Cs..
Output: Updated policy Rg
for each mraining iteration do
Sample mini-batch prompts {m foreach prompt x: do
// (1) Rc 1 out generation on and entropy computat ion
Generate G rollouts {o: TT(- |
Compute token entropies , and sliding-window means 1a
// (2) Dynamic thresholding
0: +- Quantile, ((H(e)) for each i;
0 + Σg 0g mi. + T( 0); NE Σ, mi.t
// (3) Difficuity estimation
Compute pass G accuracy P2; assign difficulty bucket d(z)) based on Ps
// (4) Sequence rewards with entropy shaping
c(
R. < (2, a.) Sa(a) (N HE: acc;) Ad(z)
1/ (5) Grntpecentered advantages with entropy bonus
R, R.+RR ---,, En=
10
11
fort L; do
ATT IRR/L:
Vi. Ad(r) o(H 0]+) ma, b:
Â., ARP thi
15
end
// (6) KL regularization with window-adaptive weights
kld,,: (Tal m s;,))
17
Be. + - "
11 (7) Actor update with clipped GRPO objective
18
(0) + To(or (LLe//Taaa04 s1,))
19
Lactor + +- о Σ. Tr
- -0)
20
LKL c Fd(x) ④ E. I, E. Be klda.
21
L Luctor + LkI.
Update 0 4- voL.
11 (8) controller update (non-windon ndoww tokens only)
Kleun & E. T, E, T(mi 0} kldi.
Ka(a)) clip Ra(=s an (KLeta//dse 1)), Kiinn Kmax
25
end
26 end
Semantic filtering. Rather than counting all tokens that exceed certain entropy threshold, we first restrict attention to fixed vocabulary of semantically meaningful reasoning triggers. This vocabulary constructed using a filtering prompt applied to GPT, which classifies each candidate token as either "reasoning-relevant" (True) or not (False). The retained tokens include transition markers (e.g., but, however), reasoning initiators (e.g., therefore, thus), and other structural markers that signal the onset of a reasoning step. Tokens with little semantic content (e.g.. A, B, digits, or domain-specific placeholders) are excluded. The entire resulting vocabulary can be visualized in Figure 6, where the word cloud highlights the most frequent reasoning triggers.
Entropy thresholding. Given token-level entropy values {He)T=> we compute the 95th per- centile of entropies within the current batch, denoted 60.95. A token o; is labeled high-entropy
H₂ 2 0g.95 and Or E Viems
24

---

Published as a conference paper at ICLR 2026
Easy (acemb flppol): Ereropy Component va targeted
Modium (acc-ó fispedt Entropy Component targei-16
Hard (accmO Hlppel): Ertropy Comporent targei
Gan
Figure 7: Effect of entropy shaping and KL regularization accuracy reward. We plot the moving-average accuracy reward over training steps under different ablation settings. Baseline methods like GRPO and DAPO either lack stable improvement plateau early. In contrast, our ARES variants consistently achieve higher accuracy rewards throughout training. Notably, com- bining both KI. regularization and entropy shaping yields the most stable and significant gains, demonstrating the necessity of the two components working together.
where Vsem is the fixed of semantically filtered tokens.
The number of high-entropy tokens in a trajectory
or) ) is thus defined as
NHE(v) = Σ 1|1 2 00.95 A or E Vsem].
t=1
This definition ensures that our entropy-basee reward focuses only on tokens that both exhibit un- usually high uncertainty and carry clear semantic signals of reasoning transitions. As a result, the entropy reward mechanism becomes more interpretable and more closely aligned with reasoning- relevant exploratory behavior.
J VISUAL ANALYSIS OF ENTROPY REWARD DESIGN
To better understand the effect of our entropy reward shaping, we provide a visual analysis in Figure 7. Each plot illustrates the entropy reward component as a function of the deviation A = NH N where NHE the number of detected high-entropy tokens and praagtt is the dnffiulttydependenn target.
Easy tasks. For easy problems, excessive high-entropy activity indicates unnecessary overthink- ing. As shown in the left panel, the entropy reward heavily penalizes positive deviations (A >> 0), while providing only mild encouragement when errors occur (acc = O). This ensures that correct responses remain short and efficient, while still allowing small degree of exploratory thinking if the model initially fails.
Medium tasks. For medium-difficult problems, the design is symmetric around the target. Mod- erate deviations from the target are tolerated, but excessive deviations in either direction are penal- ized. Importantly, when the model answers incorrectly, the entropy reward encourages it to increase reasoning length (i.e., generate more high-entrops tokens). This balances efficiency with robustness, preventing both under- and over-thinking.
Hard tasks. For hard problems, high entropy consistently rewarded regardless accuracy. As shown in the right panel, both correct and incorrect responses receive positive shaping when A >> 0. This reflects the intuition that difficult problems require longer and more exploratory reasoning chains, and the model should be encouraged sustain high-entropy reasoning segments.
Overall, the entropy reward curves realize a difficulty- shaping strategy: they suppress un- necessary verbosity on easy problems, regulate reasoning depth for medium ones, and strongly encourage exploratory thinking on hard problems. This design provides a smooth and interpretable mechanism for adaptive reasoning across the full difficulty spectrum.
25

---

Published as a conference paper at ICLR 2026
Accuracy Comparison multimodal textual benchmarks
120
ARFS
Easy
Medium
Hard
AKE5-RL-77
100
92.5
93.2
Accuracy (%)
30
72.1
74.6
90.5
52.8
51.4
55.2
55.0
GSMBK
Mathviata
MATOSOO
Mathlerse
Olmpudmench
AUME23
Benchmarks
Figure 8: Accuracy comparison across multimodal and textual benchmarks. Accuracy of three model variants BREES-CS-Vanilley ARES-CS-1 and ARES- RL--) on six benchmarks. Results are grouped into three difficulty levels (Easy, Medium, Hard). RL fine-tuning consistently improves accuracy across all benchmarks.
K VISUALIZATION RESULTS
In this subsection, we provide additional visualization results to complement the quantitative find- ings in the main text. Figures 8 and 9 illustrate the accuracy and response length dynamics across different benchmarks and difficulty levels.
As shown in Figure 8, RL fine-tuning PARES-RL-2H consistently improves accuracy over both cold-start variants (ARES-CS-Vanile
and ARES-CS-77 on all benchmarks. The gains evident across easy, medium, and hard categories, demonstrating that our proposed reinforcement learning approach enhances reasoning robustness without sacrificing performance simpler tasks. This confirms the effectiveness of incorporating entropy-aware objectives in stabilizing training and improving generalization.
Figure 9 further analyzes the average response length. We observe that RL fine-tuning reduces response length on easier datasets (e.g., GSM8K, Math Vista), reflecting improved reasoning ef- ficiency by eliminating unnecessary verbose reasoning. On more challenging benchmarks (c.g., OlympiadBench, AIME25), the response length of ARES-RL-7B increases compared to cold- start models, indicating that the model adaptively allocates more reasoning steps when required. trivial reasoning while allowing deeper exploration on difficult tasks highlighss the benefit of our antropy-shaped reward design.
This adaptive
Together, these visualizations provide clear evidence that our RL-based framework not only im- proves accuracy but also adapts reasoning length to problem difficulty, thereby balancing efficiency and effectiveness multimodal reasoning tasks.
L WHY HIGH-ENTROPY TOKENS PREDICT REASONING RESPONSE LENGTH
Given input æ. the policy सe generates response o (oI..... ol) with random stopping time = L. Let h2 = (I, 01:-11 be the history at step t and define the (possibly top-k normalized) token entropy
He = - Σ pa(u he) logpo(u hz).
(1I)
26

---

Published as a conference paper at ICLR 2026
Response Length Comparison across multimodal textual benchmarks
4B7G-C55-8
Easy
Medium
Hard
AR15-1L-3B
20000
Average Response Length
14.414
15000
131914
10000
9121.
3000
WILA
JINAS
20853
LARB
2007
2088
65MBK
Marbiista
MATHSOP
Marthboree
Olympietench
AIMEZ3
Benchmarks
Figure 9: Response length comparison across multimodal and textual benchmarks. We report the average number of generated tokens for three model variants (ARES-CS- RRES-CS-Vanilll, ARES-CS-7B. and AREE--L-77B RL training consistently reduces response length on most benchmarks, indicat- ing improved reasoning efficiency. In contrast, response length increases the most challenging datasets-AlMM:25 (textual) and OlympiadBench
hiiiimdddaahhhhhhhhhhhg.g the adaptive behavior of our RI. approach: trimming unnecessary reasoning easy problems while encouraging deeper exploration on difficult ones.
Response Length vs (b) High Entropy Number Iteration
(c) Accuracy Iteration
140
0.18
Taken
0:10
2.50
0.15
225
124
0:13 445
500
500
200
Figure 10: Training dynamics of ValLine GRPO on Coldstart Model: (a) average response length, (b) number of high-entropy tokens, and (c) accuracy, all measured across iterations. The trends indicate that the growth in high-entropy tokens is closely aligned with increases in response length and accuracy.
For window size w and threshold 0, define window entropy and a high- indicator
f+w-1
HEw
H
THE I(H 2 0).
(12)
w
and let the (validated) high-entropy count be
NHE
Σ THE
(13)
Our goal is to relate L to NHE (or the total
residence time).
two-state latent process. Assume an interpretable latent state S, E (R, VJ with: (i) Reasoning There exist constants HR > Hy and threshold E (Hy, HR) such that the detector has bounded enror
(R): exploratory/high- entropy; (ii) Verbatim (V): declarative/low-entte
Pr 1 S R) 1
Pr(T.E 0 S. V) 2 3
(14)
27

---

Published as a conference paper at ICLR 2026
with B E [0, 1). Let the R + V transition probability be q (0, 1}, and let the answer answer-emitting hazard in V be h E (0, (stopping can only occur in V).
Theorem (Linear relation with reasoning residence). Let TR = - T(5 R] be the total time spent in the reasoning state. Then there exist a, b > such that
E[L] + b [[]R].
(15)
Moreover, under equation 14 there exist constants C1 Cg 0 for which
< [[[NEE] < E[TR]
g 02 E[Ne]],
(16)
hence
E[L] d' - B NHE
for some d' 6' > 0.
(17)
Proof sketch. Within R, the process cannot stop; each reasoning excursion has geometric length with mean 1/4. Within V, the segment ends with geometric stopping hazard h (mean 1/h). The total length is a renewal sum of i..dd excursions plus terminal V segment, yielding equation 15. Bounded detector errors map TR NHE, proving equation 17.
stopping-ting time view via cntropy-dependee hazard. Assume the probability emitting the final answer at step is non-increasing function of entropy,
Pr(stop at t h₂) = X(H)), ((H) S 0. Then larger entropy sequences stochastically dominate the stopping time.
(18)
Theorem (High entropy delays stopping). For two trajectories with entropy paths {H and (1(²)) such that H1!0) < H!!2) for all one has E[L < [[ Moreover, for any threshold 0,
HH, 2 0} A(O)
ETL
E[NHE].
(19)
Proof sketch. By stochastic ordering under equation 18. higher entropy lowers the instantancous hazard and yields larger stopping times in the convex order, implying the expectation incquality. Since A(H)= S X(f) whenever H 2 8, each high-entropy step contributes least 1/X(0) units in expectation; summing gives equation 19.
Information-theoree lower bound. requires mutual information
Suppose answering with error over label set of size M
7 A log M hz(e) - log(M 1)
(20)
an (by Fano's inequality). Let i, = T(A;;, h4-1) be per-step information about the (correct) answer A. By data processing, &, < Hy. Empirically, non-reasoning steps convey negligible infor-
mation: there exists 17 < 1 such that i, S when He < 0. Let is max i(H). Then
It E[L]
It
4 S (L NHE) to NHE = NHE >
(21)
L8
which rearranges linear lower bound of E[L] terms of E[NHe].
Synthesis and testable predictions. The renewal argument (Theorem 1), the hazard view (Theo- rem 2), and the information budget bound jointly imply stable, monotone, near-linear relationship between reasoning length and high- activity:
E[L] RC a + [EHHE)
3>0.
(22)
This yields concrete diagnostics: (i) E[L| NHE] is monotone increasing; (ii) the slope 3 correlates
with the mean reasoning-cxcursinn duration 1/q; (iii) the estimated stopping hazard ((H) is de- creasing; and (iv) the per-step mutual information i, concentrates within high-entropy windows. As shown in Figure 10, the number of high-entropy tokens grows approximately in tandem with response length and accuracy, validating entropy principled proxy for reasoning effort. Yet, this relationship is not uniform across all problems: as we show next, the way entropy and reasoning length contribute accuracy differs substantially between easy and hard instances.
28

---

Published as a conference paper at ICLR 2026
M WHY KL LOSS IS A VALID Thinking Budget
We show that the KL loss used in AEPO is mathematically equivalent to enforcing budget on the policy's s deviation from the reference model, and that such budget provably controls the expected "thinking" cost (e.g., reasoning length
entropy- basyd cost Cen)).
For a prompt z, let (- x)) the policy over full responses o E V. Fe a)) the frozen reference, and r(z, o) task reward. Let c(2; o) 2 be thinking cost (e.g., response length, high-entropy window cost), and define the shaped reward f(z, 0) = r(z, o) Xc(2, o) with 2 0. Denote the (pathwise) KL divergence DEL (- x) || (- x)).
Constrained formulation. We pose the budgeted policy optimization:
max EEND E Bo
o1]]
s.t.
EgED DKL(TTO(1 x)) Free(- m:)) S 8,
(23)
where 5 > 0 is the thinking budget. The feasible set is convex because DEL (- (-I( is convex in its first argument.
Lemma 2 (Strong duality and KL-as-budget), Problem equation 23 admits strong duality. Its La- grangian is
[TT,, K) = Eg Bownn(-1)) o)] x Ez DKI(T( (x)x Feor(- |x)) -
2 0, (24)
and the optimal policy for fixed has the exponential-tilt form
Tt ro x)) x Tee((o x:) exp f(2,0)).
(25)
Moreover, at the dual optimum x² the budget active unless k* hits its boundary: Eg DKL( x) || (- x)) =
Proof. The objective is linear in and the constraint is convex with nonempty interior (Slater's condition holds by taking ग = Tref). Thus strong duality holds. Optimizing over .t. x)) under the simplex constraint yields equation 25 via standard exponential-famil calculus. Complementary slackness gives the budget activity.
Token factorization and actor-only KL loss.
For auto-regressive policies, by the chain rule,
[o]
DKI(To(- ()) Tree(- |x))
BoNTT
DKL(TO(:1 se) Tref( (-1 [s)))]
(26)
with 8, 01 Hence per-token KI. loss in the actor objective
Σ Br DRI(TO(: 0
| + Teor(- sx))
precisely discretization of the dual term in equation 23; weights 3, E (0, oo) allow token-adaptive emphasis (c.g., relaxing inside validated high-entropp windows). Because this term does not enter the advantage, controls deviation magnitude independently of the task/entropy signals that decide when to explore.
Budget tracking via dual updates. Let KL be moving-average estimate of the left-hand side of equation 23. The multiplicative update
+ clip ((( a. (KT./8 1)), Kinin: Kmmx
(27)
is stochastic dual ascent that drives KL + 8. Under standard Robbins-Monro conditions on ax and bounded variance, K2 + x a.s., and the primal iterates satisfy the budget asymptotically. Thus the actor-only KI. loss with controller implements operational thinking budget.
29

---

Published as a conference paper at ICLR 2026
From KL budget to thinking budget. We now show that bounding KL controls the expected thinking cost (c.g., c= L or = Cem)).
Lemma (Pinsker-type bound). For any bounded f V + R with |l s M,
Em [ ( < M V DKI ( |
(28)
Thus if f is the (clipped) reasoning length or normalized entropy cost, a global KL budget implies a corresponding bound the change of its expectation.
Theorem (Donsker-Varadhan control of moment budgets). For any 1j and measurable f.
1 log [x
B [f] 1 77
+ DKI (Ro((TllTTe))
(29)
17
Consequently, under the budget DKL. S 8.
E [f] I inf { log Br [en + $} } n30 r7
(30)
Proof. equation 29 is the Donsker-Varadhan variational inequality obtained by upper- hopper-ouundig the log-moment generating function via relative entropy.
Taking f = c(z, o) (e.g.. or Cem) shows that KL budget upper-bounds the expected thinking cost relative to the reference through the reference MGF, i.e., the policy cannot arbitrarily increase reasoning length or entropy cost without paying KI..
Mirror-descent / trust-region view. Maximizing E[r] under local KL ball DKL(T]] S yields the mirror-descent nnatural-gradien update
1
π & Hold exp( i), with R
(31)
V5
so 8 is precisely the trust-region radius: smaller (larger) budget forces smaller (larger) policy movement and therefore smaller (larger) allowable growth of e by Lemma 3/Theorem auto- regressive models, equation 26 further identifies the budget with tokenwise sum of deviations, making the KL. loss a time- adde-vvie budget meter of exploration.
Token-adaptive windows preserve convexity. Let [we} be nonnegative weights (e.g., we E E(p, 1] with p < 1 inside validated reasoning windows). The weighted budget Σ, W; DKL, ( [eff re- mains convex in { and admits the same dual treatment; hence the actor-only term x <2,, wi kld, is still a valid Lagrangian penalty for a Hindow-relaxed KL budget. This realizes the intuition: "relax KL where intend to think" without breaking the budgeting semantics.
(i) Strong duality turns the KL loss into the Lagrange multiplier of the reference-deviation bud- get; (ii) dual updates equation 27 track the target 8; (iii) information inequalities (Pinsker, Donsker-Varadhan translate KL budget into explicit upper bounds on the expected thinking cost. Therefore, the actor-only KL loss in AEPO is not merely regularizer: it is principled and opera- tional thinking budget.
N FISHER-GEOMENT JUSTIFICATION OF AEPO
This subsection formalizes why the proposed Adaptive-Entropy Policy Optimization (AEPO) is principled from the viewpoint information geometry. We show that AEPO is equivalent to token-reweightted catural-gradien update under per-diffiulty rrust region, and that the tropy auggeaeeeetted advantage and loken-adaptive KL together maximize improvement in the direc- tions that matter for reasoning.
30

---

Published as a conference paper at ICLR 2026
Setup. Given prompt the auto regresseees policy xe generates response (0I:L) with states 8; = (2.02)) Let r(z, o) be the task reward and Cent (Z, o) the entropy entropy-window window cost (Section 2.1).
For difficulty bucket d d(z)), AEPO solves the following constrained problem:
ENND Bo r(z,0) >d(z)) Cent (z,
max
1(0) Ba(a), DKI Te(. se) Teerl- 1s)))
(32)
s.t.
ENnD (1)x)()))))
- dd(z))
L(o)
Here L(o) denotes the response length and 8, (zz ocx). The token weight is Ba. Ba Pt. where Pe E (0, 1) if lies a validated high-entropp entropy window and os = otherwise.
KL a Fisher quadratic. Let F. denote the token-wise Fisher information matrix under the reference policy Frofi
F (ass)
V log (a [ st) V log (a st)
For small parameter displacement A0, the per-token KL admits the standard second-order ap- proximation
DKal To+ 81) Tref(- (s))) 1 AO A0 + o(1
(33)
Lemma (Weighted Fisher trust region). Under the approximation equation 33, the KI. con- straint in equation 32 is equivalent the quadratic trust region 1 AO F8 AO S δá, with the token-reweighted Fisher
F8
Bas Fe,
Ba,t BaPe. Pt (0,1)
(34)
Proof. Average equation 33 over with weights Bar and linearity.
Duga/Lagrangian form and natural gradient. inner () problem becomes
Introducing Lagrange multiplier rid 2 0, the
max A6
g
A
Kd
A0
A0,
g A Vo da (o)
(35)
whose maximizer is the natural -raddintt step
A
9
(36)
Proposition 2 (One- step improvement bound). Under equation 33, the first order improvement at Ag* satisfies
A
Mat A0
FeA0
9 F
(37)
2nd
Proof. Plug equation 36 into equation 35.
Why token-adaptive relaxation helps. Inside validated reasoning windows choose Pt 1. which reduces the local curvature contribution in F3 on those tokens. Simultaneously, the en- tropy-augmented advantage increases the corresponding components of only where Hew 2 0. Because the gain equation 37 increases when (i) g has larger energy and (ii) the metric penalty Fş smaller in the same directions, AEPO aligns a boosted gradient with a relaxed Fisher metric on reasoning windows.
Corollary 1 (Directional benefit of window relaxation). Let be unit vector supported on window tokens. If Ps is reduced on these tokens, then u. Fgu decreases while u increases under entropy shaping. Hence the directional contribution (u g)² /2kg u Fąu) to equation 37 strictly increases, improving sample efficiency reasoning segments.
31

---

Published as a conference paper at ICLR 2026
Why non-window KL for control. The KL controller adjusts Ri to match the per per-bucket target dd. we fed the controller with the already-relaxed wwindow-weighted KL, the true nat- ural length $ A0 (t E, A: could be under-estimated, risking drift. Using non-window (or wenn--eegghtewwiddoy control signal aad stabilizes the global trust region while keeping local relaxation intact.
Per-bucket decoupling: shape vs. scale. The token weights Ba. determine the shape of the metric Fg (where allow deviation), whereas the multiplier Ra determines seale (how much to deviate) for each difficulty bucket. This yields well cell-conditioned, two-time scale scheme: slow adaptation of Ra (budget tracking) and fast ascent in equation 35.
AEPO corresponds to per-difficulty, tekenhreeeeghted natural-gradien update
~ = Ki 1 ( ++E 3a Fi)
VoLir - Xd Cent
entropy shaped
with Aa.e = BaPt relaxing KL strictly within high-entropy windows and controlled by non-window KL signal. Proposition shows this maximizes F 9/(2a) by (i) amplifying g where the model is genuinely reasoning and (ii) reducing curvature along the same directions, while stable bucket-wise trust region ensures controlled deviation from Tyef.
O CASE STUDY
QUESTION
As shown in the figure, in the right triangle one acute angle is 60% and the side opposite this angle has length 6/3. Find the hypotenuse y.
6/3
60
Answer: 12
Tokens: 243
OUR ADAPTIVE MODEL SOLUTION FOR EASY QUESTION
Use the sine ratio for the 60" angle:
opposite hypotenuse
6/3
sin 60
Since sin 60
/3
we have
2.
3
6/3
6v3 2
12.
2
y
12
32

---

Published as a conference paper at ICLR 2026
QUESTION
The sum of the cireumferences of circles H, J, and K is 56T units. Their radii are marked as TH = = 2x, TK 4x, and the three circles are pairwise tangent externally. Find the distance KJ between the centers of circles K and
H X
4x
K
2x
J
33

---

Published as a conference paper at ICLR 2026
Tokens: 4173
OUR ADAPTIVE MODEL SOLUTION FOR HARD QUESTION
First, recall that the circumference of a circle is 2nr. The problem states that the sum of the circumferences of circles H, J, K is 56T, so
2 (T + rJ + rk) 56m
TH + r. TK 28.
At this point, I examined the diagram and noticed the labels 4x, I, 2r. I hesitated: are these radii, or are they distances between centers? If interpreted as center distances, then
KH 4x, HJ = x, =K= = 2x.
but this quickly fails because + 2x 3r 4x, violating the triangle inequality. Therefore, this interpretation cannot be correct.
Next, considered that these labels instead denote the radii:
TH
TJ 2x,
TK 4z.
Substituting into Tи rr + = 28 yields
x. + 2x + 4x
28
2. = 4. = 7x = 28
However, doubts persisted. For instance, if one misreads JK as 2x, then KJ would be 8, which contradicts the tangent-circle rule requiring
KJ = TK + = 4r + 2x 6x = 24.
Here went back and forth, testing whether the labels might be mixed as radii or distances, but each alternative led to inconsistencies. For example, assuming KH 4x distance forces гı 3x if tangent to rи = x, yet then HJ should be 3r not x, another contradiction. These cycles of reasoning strongly suggested that the only consistent interpretation is that the labels 4x, x, 2r represent the radii themselves.
Therefore, with
we compute the actual radii:
Гк 16, тH 4,
=
Since circles K and J are externalyy tangent, the distance between their centers is
K. = rk + = 16 8-
Finally, check consistency:
(TH + rr) 27 + 8 16) 2T 56T, matching the given condition exactly.
24
34

---

Published as a conference paper at ICLR 2026
QUESTION
In &W, ares RS and TV are congruent. Find the length of RS.
S
T
5x
9
R
W
2x + 3
V
QUESTION WITH APPROPRIATE NUMBER OF HIGH-ENTROPY TOKENS
difficulty: Easy Entropy reward: 0.0 acc reward: 1.0
Gen hntrro token: 20 (ereshold 28
overall reward: 1.0
Okay, let's see. need to find the length of RS circle @W. The problem says the ares RS and TV are congruent.
First, recall that in a circle, congruent ares correspond to congruent chords. Therefore, if are RS a are TV, then chord RS ~ chord TV.
The problem gives:
RS 5z 9,
TV 2r + 3.
So we set them equal:
5x 9 2x +3.
Solving:
3z - 9 3
= 3r = 12 x= 4.
Now check:
RS 5(4) 9= 11, TV 2(4) 3 11.
Wait, let me double-check. Are RS and TV the correct chords? Arc RS corresponds to chord RS, are TV to chord TV. Since ares are congruent, chords must be equal. Yes, consistent.
Thus, the required length
35

---

Published as a conference paper at ICLR 2026
QUESTION
In AABC, the side lengths are
14, 48, and
50. Find cos B.
B
a
A
C
b
QUESTION WITH EXCESS HIGHI-ENTROPY TOKENS
difficulty: Easy Gen high entropy token: Threshold(easy): 28
Entropy reward: -0.5 ace reward: 1.0
overall reward: 0.5
Okay, so I need to find cos B given triangle with sides 14, b 48, and 50. Let me recall the Law of Cosines:
2 a + 62 2ab C.
First, label the triangle properly. Side is opposite angle A, b opposite B, and opposite C. The diagram shows 2C 90". Therefore, c 50 must be the hypotenuse, and a 14,
48 the legs.
Now, angle B is at vertex B, between sides BC (a = 14) and AB (c = 50). The opposite
side is AC (b 48). Thus, by definition:
cos B adjacent
14
7
hypotenuse
50
25'
But me confirm with the Law Cosines, For angle
a² c² 6²
cos
2ac
Substitute carefully:
142
50 482
196 2500
2304
392
cos
2 1.14-50 50
1400
1400
25"
Wait, let me double-check. If a 14, b = 48, 50, then indeed 142 + 48 196 + 2304 2500 502 confirming it is a right triangle. Hence the direct trigonometric ratio already suffices.
... (several "wait" moments, checking whether sides match standard notation, omitted for brevity)...
Finally, both direct definition and Law of Cosines give the same result. Therefore, the value is consistent:
cos
0.28.
25
25
36

---

Published as a conference paper at ICLR 2026
QUESTION
Find x.
W
4
X W
F
10
37

---

Published as a conference paper at ICLR 2026
MEDIUM QUESTION WITH EXCESS HIGH-ENTROPY TOKENS (INCORRECT ANSWER)
difficulty: Medium Gen Gin. wiigheettrpyy.oooo token: 84 Thresholdmmedimm: 55
Entropy reward: 0.38 acc reward: 0.0
overall reward: 0.38
Okay, let's try to figure out how to find x. The problem involves a diagram with a green square and blue square connected. There is a point F with segment of length 10, and another segment labeled 4.
Wait, maybe the diagram is a cube net, since dashed lines often indicate hidden edges. If FW = 10 and W WW' = 4, perhaps x is the distance from F to W
Alternatively, maybe this is related vectors triangle. x, 10, and form sides of right triangle, then
2 = 10 + 42 116, v166 29.
But hold on, maybe the 10 and 4 are not perpendicular. Then by the Law of Cosines:
= + 4 2 10 cos 0,
but without this is indeterminate.
Next, let me try coordinates. Suppose F (0,0, 0). II (10, 0) and W =(10, 4.0, then
FW / 10a + 42 V116.
Alternatively, if W = (10, 0, 4), the distance is also V116.
... .severall further "wait" moments about whether this is a parallelogram, trapezoid, 3D projection are omitted here)...
Another thought: perhaps FW and WW' are adjacent edges of cube net, so is the straight- line distance from F to W'. Therefore, again,
V 102 42 22.29.
Yet, could be colinear? so, 10 diagram.
4 14. But that seems inconsistent with the
Maybe it's a right triangle hidden the net. If FW is one leg, ww another, then æ is the hypotenuse.
Finally, after multiple conflicting attempts, the reasoning converges to
x = 29.9.
38

---

Published as a conference paper at ICLR 2026
QUESTION
Find y.
X
60
15
y
MEDIUM QUESTION WITH FEW HIGH-ENTROPY TOKENS (INCORRECT ANSWER)
difficulty: Medium Gen Gen.high.oo entropy_toker 24 Threshold(medium;: 55
Entropy reward: 0.19 acc reward: 0.0
overall reward: 0.19
Okay, so I need to find the value of y in this triangle. The diagram shows right triangle with 60 angle, side of length 15, and the hypotenuse labeled y. Since it a right-angled triangle, the other non-right angle must be 30".
In a 30-60-90 triangle, the sides are ratio 1 : : 2. The side opposite 30 is the shortest.
If the side of length 15 is opposite 30 then the hypotenuse should be 15 30.
Alternatively, using cos 60%
15
cos 60
y = 30.
But, in the problem statement the side labeled 15 might not match this assumption. If it were opposite 60%, the calculation would change. (further checks labeling are omitted)
Finally, the reasoning settled on
y
30.
39


Reviews:
[{"review_id": "nEiDq4Mm60", "reviewer": "Reviewer_oeN6", "summary": "The paper proposes a framework for training multimodal large reasoning models (MLRMs) that dynamically adjust their reasoning depth according to task difficulty. Existing MLRMs tend to “overthink” simple problems and “under-explore” hard ones, leading to inefficiency and reduced accuracy. To overcome this, the authors introduce ARES, which combines two training stages: (1) Adaptive Cold-Start (AdaCS), aligning reasoning length with task complexity to instill difficulty awareness, and (2) Adaptive Entropy Policy Optimization (AEPO), which uses high window entropy (HWE) tokens as triggers for exploration and a hierarchical entropy-based reward to regulate reasoning depth. ARES introduces token-level entropy shaping and dynamic KL regularization to control when and how much to explore. Experiments show that ARES outperforms strong baselines on multimodal and textual reasoning benchmarks, improving both accuracy and inference efficiency, demonstrating that entropy-guided adaptive reasoning can balance performance and computation cost", "questions": "See the Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novel Conceptual Contribution: Introduces the use of window-level token entropy as an interpretable and quantitative signal to trigger reasoning exploration, bridging uncertainty modeling with adaptive inference.\n\n* Principled Methodology: The dual-stage pipeline (AdaCS + AEPO) offers a theoretically grounded and empirically validated mechanism for aligning reasoning length with difficulty.\n\n* Empirical Rigor: Extensive experiments across 10+ multimodal and textual reasoning benchmarks demonstrate consistent improvements over state-of-the-art open-source models at both 3B and 7B scales.\n\n* Efficiency–Accuracy Balance: The adaptive exploration mechanism leads to shorter reasoning chains for easy tasks and deeper exploration for difficult ones, improving token efficiency without performance degradation.", "weaknesses": "* Hyperparameter Sensitivity: Although claimed to be “hyperparameter-free,” several thresholds (e.g., 95th percentile entropy cutoff, window size 4–8) could affect stability and may require empirical tuning.\n* Although ARES demonstrates clear effectiveness at moderate-to-large scales (3B–7B parameters), all experiments are limited to the 7B model, with no quantitative evidence for models beyond this range (≥13B–70B). Consequently, the scalability of the entropy-shaping mechanism to very large foundation models remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z99qPpDLAs", "reviewer": "Reviewer_eWR6", "summary": "This paper proposes **ARES**, a two-stage, adaptive reasoning framework for multimodal large reasoning models (MLRMs). The key idea is to use **high window entropy (HWE)** (i.e., the average token entropy in a sliding window) as a reliable signal for *reasoning-critical moments*, and then adapt exploration accordingly. \n\nIn the stage of **Adaptive Cold-Start (AdaCS)**, the model is fine-tuned on curated textual/multimodal data where *reasoning length is coupled to problem difficulty*. With regard to **Adaptive Entropy Policy Optimization (AEPO)** in RL stage, ARES (a) uses an HWE-based trigger to decide **when to explore** (using a batch-level 95th-percentile threshold and (b) employs a **hierarchical entropy reward** plus **dynamic KL control** to decide **how much to explore**. The intrinsic shaping term penalizes over-thinking on easy items and under-exploration on hard items via bucket-dependent deviations from the batch mean number of HWE tokens.\n\nExperiments across many benchmarks (e.g., MathVista, MathVision, MMMU-Pro) show improved accuracy-vs-length trade-offs.", "questions": "See the `Weaknesses` part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and easy to follow, with a clear description of the method.\n2. The paper provides intuitive visual demonstrations to help better understand the paper.", "weaknesses": "1. The paper contains several design choices that feel biased and currently under-validated by ablations: some are custom (e.g., the rate-to-length coupling that sets the target length for pass-rate p via linear interpolation from all-correct/all-wrong samples, as well as the online difficulty buckets), and others follow prior work (e.g., the 95th-percentile threshold). It remains unclear whether the gains on downstream reasoning benchmarks are sensitive to these hyperparameters or to sequence length; please add robustness ablations to substantiate these specific settings.\n\n2. **Training cost comparison lacks**: the method uses a two-stage pipeline (cold-start + RL) and introduces an additional hierarchical reward in the RL stage. Under identical training data and hardware, please report the training overhead relative to alternative algorithms (time/memory/GPU-hours/tokens processed) to clarify whether the efficiency trade-off is favorable.\n\n3. **Fair comparison needed**: as noted in `Line 315–322`, the AEPO surrogate objective is essentially based on DAPO; therefore, a **DAPO-only** baseline trained under the same protocol should be included for a fair comparison, rather than only contrasting against a base model or showing `Fig.3`-style downstream results without the DAPO's designs (e.g., higher clipping).\n\n4. In `Tab.2`, the ablation shows that **ARES-CS-Vanilla** outperforms **ARES-CS-7B** on **OlympiadBench**, **AIME25**, **MATH500**, and **GSM8K**, with shorter reasoning on 3/4 benchmarks, which appears to contradict the intended effectiveness of the proposed rate-to-length sampling strategy; please consider running an RL stage starting from **ARES-CS-Vanilla** and compare it directly to **ARES-RL-7B** to verify whether the strategy is actually effective.\n\n5. The ablation in `Tab.3` does not report how **Dynamic KL Loss** and **Entropy Reward** affect response length (and its variability), making it impossible to judge whether these components truly achieve their stated motivation; please add response-length metrics and curves to the table/analysis.\n\n6. Finally, the central motivation is to enable **adaptive reasoning** that mitigates overthinking on simple problems and short thinking on hard ones, but a single response-length comparison in `Tab.2` is insufficient to validate this claim; please include targeted analyses (e.g., per-difficulty accuracy/length and trigger usage) to directly demonstrate adaptation rather than relying only on aggregate length differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DautQdpIMw", "reviewer": "Reviewer_A67V", "summary": "This paper proposes ARES, an adaptive inference framework for MLRMs, designed to address the issues of models “overthinking” on easy problems and “under-exploring” on hard ones. ARES employs a two-stage training pipeline:\n\n1. **Adaptive Cold-Start (AdaCS)**: Constructs a dataset where reasoning length correlates positively with problem difficulty, enabling the model to initially acquire difficulty-awareness.\n2. **Adaptive Entropy Policy Optimization (AEPO)**: Uses High Window Entropy (HWE) as an exploration trigger and introduces AEPO to adaptively govern both when to explore and how much to explore.", "questions": "1. **Implementation Details of the Exploration Mechanism**: Upon detecting high window entropy, the model “branches additional trajectories” (Section 3.2.1). However, the paper does not specify how many new trajectories are generated—is this number fixed or dynamically determined?\n\n2. **Choice of Window Size**: A window size of w = 4–8 is deemed optimal. But does this hold across different tasks (e.g., pure text vs. multimodal) or model scales? Is there an adaptive mechanism for selecting w?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. **Precise Problem Identification**: Clearly pinpoints the non-adaptive reasoning-length allocation in existing MLRMs and empirically reveals (Figure 1) the interaction pattern between entropy and problem difficulty.\n2. **Novel Method Design**:\n   - Proposes **Window Entropy** as a more robust exploration trigger compared to single-token entropy;\n   - Designs a **hierarchical, difficulty-stratified reward function**, applying distinct entropy regulation strategies for easy/medium/hard problems;\n   - Introduces a **dynamic KL weighting mechanism**, relaxing constraints within high-entropy windows to enable token-level “thinking budget” allocation.", "weaknesses": "1. **Limited Scale of Empirical Analysis**: The conclusion in Figure 1 regarding the “entropy–difficulty interaction” remains unclear whether the sample size and diversity are sufficient, casting doubt on the generalizability of the findings.\n2. **Weak Theoretical Support**: The proof in Appendix L—claiming a linear relationship between response length and the number of high-entropy tokens—relies on strong assumptions, and the resulting bound is overly loose, offering limited practical guidance. Moreover, the phenomenon itself is fairly intuitive, diminishing the theoretical contribution.\n3. **Clarity and Readability Issues**:\n   - The main text heavily depends on the appendix, disrupting reading flow;\n   - Mathematical symbols are introduced without clear definitions upon first appearance, raising the barrier to understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "26AZ77gYbL", "reviewer": "Reviewer_gA48", "summary": "The paper proposes an adaptive-length reasoning approach based on the relative magnitude of high-window entropy (HWE). A dataset comprising samples of alternating difficulty levels is constructed to facilitate cold start. During the reinforcement learning (RL) phase, the model leverages the relative magnitude of HWE along with a hierarchical entropy-based reward signal to determine whether to explore further and to what depth. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "The experiments show significant improvements on other metrics; however, performance decreases on MathVista. What is the underlying reason for this degradation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is logically clear, accurately expressed, and easy to read; the experiments are substantial in part, demonstrating the effectiveness of certain proposed methods.", "weaknesses": "1. Experiments\n\nThe paper premises that token-level entropy is noisy and argues that window-based entropy is therefore more advantageous. While the paper presents partial experiments on detecting reasoning-critical tokens, it lacks a quantitative analysis and fails to provide direct validation of the method's effectiveness on the main benchmark. Please provide comparative results on major standard benchmarks to substantiate the overall efficacy of the proposed approach.\n\nAdditionally, certain experimental results on Vision-G1 appear inconsistent with the originally reported values in the prior work (e.g., MM-Start: 66.0 → 63.1 on WeMath, etc.), which raises concerns about result reproducibility or evaluation consistency.\n\nThe paper lacks comparison with works on adaptive chain-of-thought reasoning; what are its advantages over them?\n\n[1] Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLMReasoning\n\n2. Methodology\n\nThe core idea of the paper hinges on using the relative magnitude of entropy to assess uncertainty. Although the method shifts from single-token entropy to high-window entropy (HWE), the fundamental concept remains closely related to a substantial body of prior work on entropy-based uncertainty estimation. This proximity to existing approaches diminishes the perceived novelty and originality of the proposed method.\n\n[1] Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models, 2024.11\n\n[2] Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity, 2025.05", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)