Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
Authors: Zhuoxu Huang
Mengxi Jia
Hao Sun
Xuelong Li
Jungong Han
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Modal Reasoning
Reinforcement Learning from Verifiable Rewards
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model’s policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.

Published as a conference paper at ICLR 2026
CONTROLLABLE EXPLORATION IN HYBRID-POLICY RLVR FOR MULTI-MODAL REASONING
Zhuoxu Huang'; Mengxi Jia Hao Sun", Xuelong Li Jungong Han
Aberystwyth University, Instittte of Artificial Intelligence (TeleAl), China Telecom, "Tsinghua University zhh6 ac.uu
ABSTRACT
Reinforcement Learning with verifiable rewards (RLVR) has emerged as a pri- mary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads entropy collapse, pol- icy degradation, or over-exploitation of suboptimal behaviors. This necessitates exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. this paper, we propose CalibRL, hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, distribution-awwe aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Mcanwhile, the asymmetric activation function (LeakyRel.U) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding con- vergence to erroneous patterns. Importantly. these designs help alleviate the dis- tributional mismatch between the model's 's policy and expert trajectories, thereby achieving more stable balance between exploration and exploitation. Exten- sive experiments across eight benchmarks, including both in-domain and out- of-domain settings, demonstrate consistent improvements, validating the effec- tiveness of our controllable hybrid-policy RLVR training. Code is available at https//gittub.cmmm//222225////1
guided manner and
INTRODUCTION
Recent Large Language Models (LLMs), such OpenAl-ol (Jaech et al. (024), DeepSeek-Rl (Guo al. 2025). and Kimi-1.5 (Team et al. (025), have achieved remarkable breakthroughs in complex reasoning by leveraging extended Chain-of-Thought (CoT) reasoning (Wei et al. 2022), showcasing unprecedented proficiency multi-step logical inference. Building on these adncess Multi-Modal Large Language Models (MLLMs), including Virgo (Du et al. 2025), Intern (Zhu ct al. 2025). MiMo-Vi VL (Xiaomi (025), and Ovis2.5 (Lu et al.) 2025), have further extended rea- soning into multi-modal domains, enabling complex visual reasoning, mathematical diagram inter- pretation, and cross-modal logical inference.
The success of recent models is largely attributed advanced Reinforcement Learning using Veri- fiable Rewards (RLVR). Despite this progress, recent studies have shown fundamental challenges: improvements in policy performance often come the cost of reduced policy entropy, creating bottleneck where entropy depletion constrains further progress (Cui et al., 2025). Conventional RL methods usually incorporate entropy regularization (Liu et al. 2025a Starnes et (023) to en- courage stochasticity, but the resulting high-entropy strategies rely on unguided random sampling, leading to inefficient exploration. This issue especially pronounced in the enormous state space of MLLMs and significantly limits learning efficiency. Recently, combining Supervised Fine- Teee-Tunnng
Equal contribution.
Corresponding author; Jungong Han & Xuelong Li

---

Published as a conference paper at ICLR 2026
(SFT) with Reinforcement Learning (RL) training enables the integration of expert knowledge with self-improvement mechanisms, which can increase policy exploration while providing a clearer tar- get distribution. Yet, neither the popular "SFT-then-RL" pipeline nor recent hybrid-policy frame- works that embed SFT supervision directly into RI. training (Yan et al. 2025 Ma et al. 2025 Dong et al. 2025 /Zhan et al. 2025) could provide stable and controllable exploration. In the sequential "SFT-thennRR" RL" paradigm, the initial SFT stage anchors the policy to a static demonstration distri- bution, thereby diminishing its exploratory tendency in subsequent RL training. As result, the policy struggles to adapt beyond the supervised baseline and fails to concentrate probability mass on novel, high-reward behaviors. Hybrid-policy methods that inject SFT supervision into RL suffer from distributional mismatch between the current policy and expert trajectories. This mismatch introduces high bias and variance, leading to unstable policy learning. The resulting instability in- terferes with exploration and accelerates entropy collapse, driving the policy toward either overly deterministic or excessively random behaviors.
To address this dilemma, we propose CalibRL: Hybrid-Policy RLVR with Controllable Exploration, a framework that redefines the role of expert supervision as the calibration baseline. CalibRL treats expert data distributional baseline- reference against which the model's on-policy behaviors are evaluated. From this perspective, our framework performs distributional calibration, explicitly designed to maintain sufficient policy entropy while guiding effective exploration. Responses assessed relative to the baseline distribution from the expert: underrepresented yet correct reasoning paths are selectively reinforced, maintaining rare but informative behaviors as valuable exploration signals, while overconfident but erroneous predictions are penalized more strongly to prevent mis- leading convergence. This calibrated treatment transforms expert supervision from rigid imitation signal into nuanced guidance mechanism that balances exploration and performance, ensuring that policy entropy is retained while exploration directed toward meaningful reasoning behaviors.
Crucially, controllable exploration is achieved by using two complementary mechanisms. An ad- vantage weighting serves as an indicator of the relative likelihood of a response within its group, naturally emphasizing the contribution of the rare sample to enforce distribution calibration. In par- allel, a asymmetric activation based on LeakyReLU leverages expert knowledge as a calibrated baseline to moderate overconfident updates while preserving their corrective direction, ensuring ex- ploration remains guided and stable. Together, these mechanisms regulate both the strength and direction of policy updates, enabling guided and stable exploration, ensuring expert supervision remains informative rather than restrictive. We conduct extensive experiments across eight bench- marks and demonstrate consistent superiority over previous hybrid-policy methods, which fail to improve upon the GRPO baseline in our multi-modal scenario. In summary, the contributions of this work are threefold:
We propose CalibRL, a hybrid-policy RLVR framework for controllable exploration in reasoning-orientee MLLMs, which leverages expert guidance stabilize policy updates and increase policy entropy in a guided manner.
Our CalibRL introduces two complementary mechanisms: advantage weighting, which emphasizes rare responses enforce distribution calibration, and a LeakyReLU-based asymmetric activation, which moderates overconfident updates while preserving their cor- rective direction.
We validate our method through extensive experiments on eight reasoning benchmarks, demonstrating substantial improvements over GRPO and state-of-the-ant art hybrid-police baselines, with consistent gains across both in-domain and out-of-domain tasks.
PRELIMINARIES AND RELATED WORKS
This section introduces preliminary knowledge related our work to facilitate understanding of the proposed method. We then review the most pertinent related work, identifying key limitations and outlining our research motivation.
2.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS (RLVR)
We start with definitions for training the LLM with RLVR. Given an input prompt q. and an LLM with parameters 0, the reasoning generation task with the LLM is framed Markov Decision
2

---

Published as a conference paper at ICLR 2026
Process (MDP) (Puterman 2014). The LLM is represented policy Tg. The output response from the LLM is represented as the trajectory т = (01, ....... or), where T is the response length.
At each generation step t [1, the MDP defines state 8t (q, 01, 02. ................................ ot-1), which represents the concatenation of the prompt and the response tokens generated up to step t 1. The initial state so q. For each generation step f, the MDP defines action de or mo(-1s)), representing the selection of Or as the next token with state s, as condition.
In the circumstances, the policy нe provides probability distribution the LLM's vocabulary for the next token prediction. With a spare verifiable reward A formulated as follows:
R(()) {
if the response 7 is correct,
(1)
otherwise. The training objective of RLVR is then formulated as follows:
O E, [R(T)),
where D is the prompts set. This objective is optimized with policy gradient optimization that aims to maximize the reward R over the prompts D.
In recent applications, the Group Relative Policy Optimization (GRPO) (Shao et al. 2024) has become the de facto choice owing to the success of Deepseek-RI (Guo al. 2025). The primary advantage stems from utilizing intra-group reward comparisons to derive the advantage for each trajectory. Given responses for each prompt q, the response group LG is validated by reward function and used to calculate the advantages. This process can be formulated follows:
Á., R(T:) mean(((((T))E..))
std(R((T)))))
The advantage signal guides the preference policy optimization, directing updates to increase the log-probability of tokens that exhibit high advantage values using a clipped PPO-style (Schulman et al. 2017) objective:
Irl min (O).
(o) = E Sowollo
(ri, - c) A..) (Tol(Te()))))))
where the (0) = els.))
Me (TLulsa)) represent the importance sampling ratio. The KL-divergence penalty HL (Tel[ in the original GRPO (Shao al., 2024 is used regularize the policy update. In recent implementations Yan et al. 2025) Dong et al. 2025), this KL. term is usually omitted when training models for long CoT reasoning, as the model's distribution is expected to diverge significantly from the initial policy, rendering this constraint unnecessary.
2.2 RELATED WORK
Reinforcement Learning for Reasoning Models. Recent advances in reinforcement learning have
significantly improved the reasoning ability of large language models (LLMMs) (Guo al., 2025
Jaech et al. 2024 Team et al. 2025; Xiaomi, 2025 Du et al. 2025 Zhu et al. 2025 Lu et al.
2025
Wang et 2025 These advancements have largely benefited from the emergence of Reinforce-
ment Learning with Verifiable Rewards (RLVR) frameworks (Shhulan et al. 2017 Shao et al.
2024
Liu al., 2025b Rafailov al., 2023 Hao et 2025 Hu et 2025 p. which leverage
verifabll signals to provide precise and stable reward shaping. Among these, the GRPO framework
Shao et al. 2024) has become particularly influential, introducing group-normalized advantage es- timation an efficient alternative to PPO (Schulman et al.) (017)) under verifiable reward settings in mathematical reasoning. Despite these successes, recent studies have highlighted important lim- itations of on-policy RLVR when applied to reasoning tasks. Zhao et al. (2025) show that RL post- training often amplifies behaviors inherited from pre- me--trinnng rather than fundamentally expanding reasoning capacity, leading to "echo chamber" effect. Yue et al. (2025) further demonstrate that while RL improves sample efficiency (c.g., pass@ 1). it does not substantially broaden the model's s reasoning boundary beyond that of the base model. Complementarily. Cui et al. (2025) provide an
3

---

Published as a conference paper at ICLR 2026
analysis of entropy dynamics, revealing that on-policy updates tend to concentrate probability mass on a narrow set of high-reward trajectories, causing premature entropy collapse and limiting explo- ration. Together, these findings suggest that although RLVR stabilizes training through verifiable feedback, it also risks over-constraining the policy and failing to sustain the exploration necessary for discovering genuinely novel reasoning strategies.
Hybrid-Policy Optimization Frameworks. To address the limitations of purely on-policy RL, hybrid-policy optimization methods have been proposed. combining reinforcement learning with supervised fine-tuning (SFT) on expert data Yan et al., 2025; Dong et al., 2025; Wu et al. 2025] Ma et al. 2025] Zhang et al. 2025). LUFFY Yan et al. 2025) introduces off aff-policy guidance via mixed-policy optimization with regularized importance sampling, while RL-PLUS (Dong et al.) 2025) develops a multi-importance sampling strategy combined with exploration-base
advantage shaping. Other approaches, such as ReL.FF (Ma et al.) 2025), interleave SFT and RI. updates or introduce template-augee auggeetee objectives, whereas CHORD (Zhang al.] 2025) employs phased training to balance exploration and exploitation. Despite their innovations, these hybrid frameworks commonly rely on direct log-likelihood maximization of expert data, which imposes a unidirec- tional optimization pressure toward expert distributions. This often accelerates entropy collapse by suppressing alternative responses, thereby constraining policy diversity and weakening exploration. These limitations motivate work: rather than treating expert data an absolute imitation target, we propose to reinterpret it as a distributional baseline, enabling relative calibration of on-policy behaviors in a way that sustains entropy while still providing directional guidance.
METHOD
In this section, reveal the entropy collapse acceleration process caused by the distribution gap between the expert data and the model behaviors. We then illustrate our controllable exploration under expert guidance, analyzing how it preserves policy entropy and directs exploration toward reliable behaviors.
3.1 LIMITATIONS OF DIRECT EXPERT OPTIMIZATION
Given dataset of expert demonstrations D {(g T. ) policies are typically trained imitate expert behaviors by minimizing the negative log-likelihood, which is formalized as follows:
Lexpert (g log ne expert [q:)]
(5)
The corresponding gradient update can be formalized as:
V VeCespen = -B expen Ve log TT ( [9:)],
which monotonically increases No (7 (g). This objective enforces unidirectional expert optimiza- tion: probability mass is consistently shifted toward expert responses, regardless of the model's 's cur- rent distribution. While this secures alignment, also narrows distributional support and suppresses diversity of outputs.
In the sequential SFT-then-RL. paradigm, this effect anchors the policy close to the expert distri- bution after SFT. Subsequent RI. updates are further restricted by importance sampling and ratio clipping (Equation 4 which penalize deviations from the SFT initialization. Consequently, explo- ration remains confined to the neighborhood of expert behaviors, hindering adaptation to reward signals and limiting the discovery of higher- rewaard reasoning paths (Figure I
hybrid-policy frameworks embed expert supervision directly into the RL objective, effectively warding higher likelihood of expert responses. While this guarantees continual expert alignment, normalization dictates that increasing To ( (q) necessarily decreases To(T * - [g), leading to overall entropy reduction, thereby accelerating entropy collapse and pushing the model toward overly deterministic expert-like behaviors:
H(To(-lg)) = = Σ To(Tl))log 4
In summary, both SFT-then-Re RL and hybrid- frameworks inherit the limitation of unidirec- tional expert optimization. By uniformly shifting probability mass toward expert trajectories, they
4

---

Published as a conference paper at ICLR 2026
constrain exploration, accelerate entropy decay, and fail to adaptively account for the model's hetero- geneous cognitive states. To address this limitation, we advocate a relative calibration perspective, treating expert data not absolute imitation targets but distributional baselines for evaluating on-policy behaviors.
3.2 CONTROLLABLE EXPLORATION UNDER EXPERT GUIDANCE
Our framework redefines the role of expert supervision by treating it distributional baseline rather than a strict imitation target. Given an input 9,, a model-generatee response 7, ?.olicy, and the corresponding expert response r. ह( we first define log- prgg-rrbbbblity gap as
To (T policy
A4 = log (e( nolicy q.) log na [
expert (q:) = log
(8)
Tgl expert
The quantity Ae, captures the model's relative preference between its own response and the expert's s. A positive value indicates the model favors its answer the expert's, while negative value indicates underconfidence relative to the expert.
+1 for correct responses and 8; -1 for incorrect sponses, as the actual reward, including the format reward, can exceed the [0. 1] range. To ensure rigorous definition that remains valid across broad range of scenarios, separate correctness signal rather than relying solely on normalized version of the actual reward. We then design an objective that enables controllable exploration through an asymmetric activation based on the LeakyReLU operator, and an ad advantage weighting that emphasizes rare but informative events. The objective written as
We introduce correctness signal
explicitly define the
Lexploration |ådl LeakyReLUU - -;; -A;;
where |A, is the absolute value of the group-wise advantage Å,, representing an advantage weight capturing group-wiss wwse rarity. Multiplying 8; with AC, ensures that the optimization direction always aligns with correctness.
Under this objective, correct responses that are underweighted relative to the expert reinforced, thereby broadening the support of the policy distribution and increasing entropy, while incorrect re- sponses that are overweighted are suppressed, thereby shifting probability mass away from them and preventing entropy collapse. The LeakyReLU introduces asymmetric gradient gating, which allows the model to amplify underrepresented reasoning patterns while reducing overconfident predictions. Once response's 's probability crosses the expert baseline, the slope parameter E (0, 1) controls the degree of further reinforcement or suppression, enabling exploration that both directed and regu- lated. In this way, expert supervision functions as relative reference rather than absolute target, guiding probability redistribution in a calibrated manner while preserving sufficient stochasticity for the model to explore novel reasoning strategies.
The weighting | also contributes to controllable exploration by calibrating the scale of the update according to group-wise rarity. Large values occur when a rare correct response emerges among mostly incorrect ones, amplifying its reinforcement incorrect response appears among mostly correct ones, the weighting increases its suppression. By modulating the update magnitude in this way, the model emphasizes but informative deviations while damping misleading outliers, ensuring that exploration remains selective and controllable.
exploration signal. Similarly, when rare
We then integrate the controllable exploration term into the GRPO objective obtain the final training objective
J(0) = E BonrD,nsiifh D.r H min (Oe)
(10)
clip ((),, e, 1 e) A..) XCexploration
where À balances standard PPO-style policy optimization and expert-guided exploration.
In summary, our controllable exploration framework combines correctness signals, asymmetric gra- dient gating via the LeakyReLU, and advantage weighting to modulate probability updates, selec- tively reinforcing underrepresented correct responses and suppressing overconfident errors while maintaining calibrated stochasticity.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 SETUP
Training dataset. We construct our training dataset from the ViRL39K collection (Wang et al., 2025). Specifically, we sample geometry problems from ViRL39K and generate detailed Chain-of- Thought (CoT) responses using GPT-44 (Hurst al. 2024). We then validate these responses across three criteria: correctness, format adherence, and logical coherence. This process yields 9,695 high- quality guestion-response pairs for training and 933 samples for validation. Notably, the validation set comprises samples that failed our CoT validation criteria, representing challenging cases where even GPT-4o struggled to produce satisfactory responses. We refer readers to the appendix for further training details.
Benchmarks and Baselines. We evaluate our method on a diverse suite of benchmarks covering both in-domain and out-of-domain (OOD) settings. For in-domain evaluation, we use the Geo3K (Lu et al., 2021) y and GeoQA (Chen et al. 2021) test sets, along with our self-constructed GeoEval benchmark that failed the CoT validation criteria. To examine generalization, we test on Verse (Zhang et al., 2024), Vision (Wang et al. 2024), and Math Vista (Lu et al. 2024), which involve broader mathematical reasoning and visual understanding tasks. We further incorporate samples from ViRL39K (Wang et al. 2025), which covers diverse fields such as Science (Physics, Chemistry, Biology) and Spatial Reasoning. We also involve the MMMU (Yue al. 2024) and the ScienceQA Lu et al. (2022). These benchmarks enable us to evaluate not only the effectiveness of our approach in geometry/mathematees but also its robustness across broader general domains.
For baselines, we compare against several representative approaches. GRPO (under customized training settings following Yan et al. (2025) and Dong et al. (2025)) serves as the standard rein- forcement learning baseline, while SFT+GRPO reflects the sequential paradigm of supervised fine- tuning followed by reinforcement learning. highlighting the issues of distribution shift and overly supervised constraints. We also include LUFFY (Yan et al. 2025), state-of-the-e hybrid -ybrid-policy optimization method, to test whether our approach better preserves entropy compared to direct ex- pert integration, and RLPLUS (Dong al. 2025), which adopts conservative update strategies to stabilize training and thus serves as a strong baseline. We reproduce those methods on our dataset, keeping all training settings consistent throughout the training process. Our experiments are mainly conducted with Qwen2.55-VL-2
(Bai et al.) 2025) as base models. During evaluation, we use MATH-VERIFY (Kydlíček) to score the Geo3K, GeoQA, GeoEval, Science, and Spatial Reason- ing benchmarks, while the benchmarks Math Verse, Vision, and Math Vista are evaluated with the open-source V..MEVA..KIT (Duan et al. 2024).
4.2 MAIN RESULTS
We first examine the training dynamics in Figure I taking GRPO the reference baseline. As seen in the figure, SFT-then-GRPO suffers from overly supervised constraints: although it starts with relatively high reward values, its entropy remains excessively high throughout training, reflecting a lack of meaningful exploration. As a result, the policy distribution quickly solidifies, reward learning stagnates, and performance remains consistently low, In contrast, RL-PLUS (Dong al. 2025) exhibits an opposite pathology. The entropy collapses too rapidly in the early phase and converges early to a certain pattern. This suppresses exploration and makes later reward learning ineffective, which leads to suboptimal reward curves and declining accuracy. The training dynamic shows that aggressive entropy reduction hinders sustained improvement. As also seen in Table
entropy comparison
entropy comparison
reward comparison
performance comparison
0.88
0.35
Grpe
0.7
Mane
0.30
poward score
6,30
Grce Rickas
entrggg
0.4
$8: 0/90
Oico Aigkks
0.20
0.2
Sh-Groe
0.35
10
Traning
Tonnig
Trahiing
Training
Figure I: Entropy, reward, and accuracy curves of different methods. We split the entropy compari- son into two panels for clarity.

---

Published as a conference paper at ICLR 2026
both the hybrid-policy frameworks (Dong al. 2025 Yan et al. benefits upon the GRPO baseline.
2025 fail provide consistent
In contrast, quantitative results in Table and Table 2 demonstrate method's superiority previous
methods. On in-domain geometry reasoning tasks, our CalibRL achieves an average performance gain of 5.45 percentage points the GRPO baseline, substantially outper- forming existing hybrid-policy methods, including LUFFY (10.84) and RL-PLUS (J4.8). For out- of-domain reasoning benchmarks, observe a consistent improvement of 2.61 percentage points over GRPO, while maintaining competitive advantages over LUFFY and RL-PLUS. Additionally, DAPO does not outperform GRPO in our setting. One possible explanation is that the higher clip- ping threshold leads to uncontrolled exploration, which further underscores the importance of our work. We further visualize the relative changes from the GRPO baseline Figure 2 Our method consistently improves upon the GRPO framework across both in-domain and out- -o-maf-domain scenar- ios, whereas existing hybrid-policy approaches exhibit varying degrees of performance degradation, underscoring the effectiveness of our approach in overcoming their limitations.
Table 1: Performance comparison in-domain geometry benchmarks.
in-distribution
out-of-distribtiion
Method
Avg.
GeoEval
Geo3K
GcoQA
GRPO
26.15
39.77
$2.52
39.48
SFT+GRPO
6,00
18.64
40.98
21.87
DAPO
25.19
40.93
52.52
39.55
LUFFY Yan et 2025
25.62
39.10
51.19
38.64
RL-PLUS Dong 2025
20.69
36.94
46.42
34.68
CalibRI.
33.44
40.60
60.74
44.93
Table 2: Performance comparison out-of-domain benchmarks. mark as 'Sei.' and the Spatial Reasoning benchmark as 'Sp.'.
We present the Science bench-
Method
Sci
Verse
Math Vision
Math
MMMU
ScienceQA
Avg.
GRPO
61.99
48.02
49.01
27.89
70.00
55.44
88.34
57.24
SFT+GRPO
54.33
38.19
35.10
23.36
54.70
54.67
86.36
49,53
DAPO
61.61
50.21
47.82
27.07
70.40
54.67
87.95
57.10
LUFFY Yan al. 2025
63.11
48.18
47.97
26.64
70.30
55.22
87.95
57.05
RL-PLUS Dong a. 2025
61.21
47.55
46.40
27.14
69.90
55.88
87,50
56.51
CalibRL
65.12
53.64
51.35
27.93
71.90
56.55
89.04
$9.36
Notably, the GeoEval valida- tion set comprises samples that failed our CoT validation criteria, representing challenging cases where even GPT-4o struggled produce satisfactory responses. The performance disparities this demanding benchmark are particularly revealing: while the SFT+GRPO catastrophically fails with only 6.00% accuracy, and hybrid-policy methods struggle match the GRPO baseline, our method achieves remarkable 33.44% accuracy. This substantial improvement such challeng- ing instances demonstrates our method's superior capability in handling complex reasoning sce- narios that typically confound existing approaches, validating the effectiveness of our proposed training strategy in maintaining robust performance difficult edge cases.
SFT GRPO
... LUFFY
... RL PLUS
CalbAL GRFO
inddomain
aut-of-domain
+1:30
+2.34
+1.00
-0.30
2"2007"
--222"
.2.83
2.61
4.55
-6.10
.7.66
10-
.0.3
-11.54
.1381
-15
GpoEval
Go03K
Geo
Sclence
Sodttal
Mathierse MatWisiinn Mottisst
Figure 2: Performance comparison showing relative changes from GRPO baseline across in-domain geometry and out-of- domain reasoning tasks.
To validate the generalizability of our method across different model seales and architectures, we conduct experiments on both smaller model (Qwen2.5VI_3B (Bai al. 2025)) and different

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison on different base models.
Method
GcoEval
Geo3K
GeoQA
MathVision
Math
Avg.
Qwen2.5VI.33B Bai al. 2025
GRPO
15.11
28.95
40.05
22.99
65.1
34.44
LUFFY Yan ct al. 2025
12.54
28.95
33.55
23.42
62.9
32.27
RI-PLUS Dong et 2025
12.00
27.79
35.15
24.98
63.4
32.66- 11778
CalibRL
46.15
65.8
37.09. (2.65
IntrenVL3-8B (Zhu et 2025
GRPO
29.47
45.09
57.43
29.07
65.8
45.37
LUPPY Yan 2025
13.83
15.14
48.67
29.07
62.0
33.74
RL-PLUS Dong a. 2025
19.61
39.27
41.38
29.44
66.8
39.30
CalibRL
31.08
48.59
58.89
30.06
68.5
47.42,
architecture (Intern (Zhu et al., 2025) ). As shown in Table 3 CalibRL achieves consistent
improvements in both settings. On the smaller Qwen2.5VL-3B model, method outperforms the GRPO baseline by 2.65 points, while other methods (LUFFY and RL-PLUS) show performance degradation of about 2 points. On the VL3-mV CalibRL maintains its advantage with a 2.05 point improvement over GRPO, whereas competing methods suffer substantial drops. These re- sults demonstrate that our controllable exploration mechanism generalizes effectively across vary- ing models, consistently delivering performance gains while other recent approaches struggle with cross-model transferability.
4.3 ABLATION STUDIES
Controllable Exploration. We first conduct an ablation study to validate the importance advan- tage weighting [Á]] by removing it from Equation which treats all responses equally regardless of their distributional context. As shown in Table 4 this leads substantial performance degradation across all benchmarks, demonstrating that advantage weighting is essential for our entropy control mechanism. The advantage weight |A,| enables targeted distributional calibration by amplifying learning signals for rare correct responses while suppressing rare incorrect responses, systemati- cally shifting probability mass toward underrepresented but valuable reasoning patterns. This rep- resents the core of our entropy preservation strategy, ensuring controlled exploration that maintains distributional diversity. Without this mechanism, the exploration signal becomes indiscriminate, compromising both learning efficiency and generalization performance.
Table 4: Ablation studies the controllable exploration objective. We present the Science bench- mark as 'Sci.' and the Spatial Reasoning benchmark as 'Sp.'. The highlighted row represents our optimal results. Bold and underlined values denote the best and second-best results, respectively.
in-domain
out-of-domain
Setting
Avg.
GenEval
Geo3K
GeoQA
Sci.
Sp.
Math Verse
Math Vision
Vista
w/o AA,]
25.19
28.95
57.56
61.76
47.71
43.63
26.71
70.90
45.30
w/
[
0=3.3
31.30
42.43
59.81
64.52
50.88
50.46
28.88
71.00
49.91
0=0.5
33.44
40.60
60.74
65.12
53.64
51.35
27.93
71.90
50.59
a=0.8
30.44
43,59
58.75
65.27
50.62
50.46
28.71
71.00
49.85
entropy comparison
entropy comparison
0.99
0.5
Grpo
1.5
0.8
entropy loss
entropy loss
0.3
0.1
10
15
20
30
40
5
10
15
20
30
40
Training Steps
Training Steps
Figure 3: Entropy evolution during training for different a values in our framework. We split the comparison into two panels for clarity. The curves demonstrate how a controls exploration strength.
8

---

Published as a conference paper at ICLR 2026
We further conduct an ablation study on the hyperparameter in Equation to evaluate its role in regulating exploration during training, with Table reporting the performance across different a values and Figure [ iillustrating the corresponding entropy dynamics. Methodologically, a controls how the LeakyRel.U mechanism scales the gradient of the exploration term. When the input to LeakyReI.U is negative, the gradient is scaled by a reducing the update strength and prevent- ing excessive promotion or suppression. In this way, regulates the balance between exploration and convergence in a controlled manner. The results confirm this principle. Small (=0.3) values promote aggressive exploration in early training but result in unstable optimization characterized by entropy oscillations and premature decay, failing maintain the sustained exploration necessary for effective learning. Large a values (20.8) over-constrain the exploration signal, resulting in rapid entropy decay after a brief initial peak, thereby failing to maintain the distributional diversity essen- tial for effective reasoning generalization. In contrast, an intermediate setting of = 0.5 achieves optimal exploration regulation by maintaining smooth entropy growth without destabilizing oscilla- tions, reaching sustained high-level exploration that enables effective learning of diverse reasoning patterns while preserving training stability, ultimately resulting in the highest overall performance and wins the majority of benchmarks.
These findings highlight that properly calibrated exploration control essential for reasoning perfor- mance. More importantly, our framework enables fully controllable exploration regulation, allowing systematic discovery of optimal balance points and effective navigation toward superior reasoning solutions across varying task requirements.
Balance Weight. We then test the performance of our method with different balance weights X, controlling the trade-off between standard policy optimization and expert-guided controllable ex- ploration in Equation []] We observe that small À values favor in-domain improvements but pro- vide limited generalization, as the model tends to overfit to training- like trajectories. Conversely, very large À values suppress policy learning and hurt both in-domain and out of-iomiin perfor- mance due to excessive reliance on expert supervision. A moderate balance achieves the best overall performance, yielding competitive in-domain results while substantially improving out-of-domain generalization.
Table 5: Ablation studies on the balance weight. We present the Science benchmark as 'Sci.' and the Spatial Reasoning benchmark as 'Sp.'. The highlighted row represents our optimal results. Bold and underlined values denote the best and second-best results, respectively.
in-domain
oft-oi-domain
A
Avg.
GeoEval
Geo3K
GeoQA
Sci.
Sp.
Math Verse
Math
Math Vista
0.01
29.47
38.10
58.36
64.49
50.10
49.21
28.72
70.80
48,66
0.03
30.44
42.10
61.41
63.68
51.14
50.69
27.86
72.10
49,93
0.05
28.83
41.10
57.82
64.23
49.01
49.21
27.80
71.10
48.64
0.1
33.44
40.60
60.74
65.12
53.64
51.35
27.93
71.90
50.59
0.3
23.58
26.96
58.36
61.70
48.02
42.06
25.82
70.10
44.58
0.5
25.62
31.28
55.57
61.04
47.14
43.07
27.01
70.30
45.13
Different entropy control. We also conduct ablations compare controllable exploration with several entropy regularization methods. Results are reported in Table In summary, only CalibRL. delivers consistent improvements across all evaluated benchmarks. We first compare our method with fixed-coefficient entropy regularization by setting the entropy coefficient to 0.01 according (Yan al., 2025 Dong al. 2025), which results in a degradation in performance. We then apply the widely used entropy-contro mechanisms based on KL and clip-covariance regularization (Cui al., 2025) on top of GRPO. The KL-Cov variant provides a slight improvement on some tasks but remains noticeably weaker than our CalibRI, while the Clip-Cov variant again results in a performance drop. Compared with conventional entropy-based methods, CalibRI. enhances policy entropy in guided manner that avoids unguided randomness and directs cxploration toward meaningful reasoning behavior. As a result, achieves more effective exploration than the compared entropy-bascd baselines.
Different activation functions. Adopting the leaky mechanism of the LeakyReLU function enables us to build a controllable cntropy-shaping mechanism, facilitated by its adjustable negative slope. We show the necessity of this design through ablations using other activation functions, where the ReLU, sigmoid, and Huber fully truncate negative values, provide no controllable scaling, and the

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison of different entropy control.
in-distribution
out-of-distribution
Method
Avg.
GeoEval
Geo3K
GeoQA
GRPO
26.15
39.77
52.52
39.48
+ entropy coefficient 0.01
22.62
40.27
47.48
36.79
+ KL-Cov (Cui et al. 2025
26.47
41.60
53.58
40.55
+ Clip-Cov (Cui et al. 2025
25.40
39.43
52.12
38.98
CalibRL
33.44
40.60
60.74
44.93
tanh, which cannot modulate the magnitude, none of which offer the required level of control. Re- sults are reported in 7 the three ReLU, sigmoid, and Huber either failed to improve the GRPO baseline or only provided a slight gain. the other hand, the tanh provides a relatively strong improvement, showing the importance of the negative value in the entropy-shaping mechanism. Finally, our design for the CalibRL consistently achieved the highest performance.
Table 7: Ablations on activation functions.
in-distribution
out-of-distribution
Method
Avg.
GeoEval
Geo3K
GeoQA
GRPO
26.15
39.77
52.52
39.48
CalibRL
w/ ReLU
26.47
33.11
57.96
39.18
w/ Sigmoid
26.69
32.61
60.08
39.79
w/ Huber
24.22
27.62
55.70
35.85
w/ Tanh
30.65
40.93
58.62
43.40
w/ LeakyReLLU(a= 0.5)
33.44
40.60
60.74
44.93
Different reference policies. We conduct ablations to compare different reference policies for com-
puting A Specifically, we replace the log (,(,, eapert (qs) with the reference policy log <o(>;"llg)) Ig:),
resulting in ce-probability gap AC; log (o.polyy
policy (4: log (T;", (4))
Results are reported Table S The expert baseline strongly suppresses the reference policy base-
line, showing the importance of the expert guidance in controllable exploration design.
Table Ablations reference baselines.
in-distribution
out-of-distribution
Method
Avg.
GeoEval
Geo3K
GeoQA
CalibRL w/ expert baseline
33.44
40.60
60.74
44.93
CalibRL w/ ref policy baseline
27.87
40.77
55.57
42.74
5 CONCLUSION
In this work, we investigated the fundamental tension between exploration and supervision in train- ing reasoning-oriented MLLMs and demonstrated that existing SFT-then-RL and hybrid-policy frameworks either remain overly constrained by supervised baselines or suffer from entropy col- lapse. To address this challenge, we proposed CalibRL: Hybrid-Polics RLVR with Controllable Exploration, a principled framework that reinterprets expert supervision as relative calibration rather than direct imitation, thereby preserving policy entropy while providing directional guidance. Through SeakyReLU-basee asymmetric activation and an advantage weighting mechanism, our method achieves explicit control over exploration, enabling the model to reinforce underrepresented correct reasoning trajectories while discouraging overconfident errors. Extensive experiments across eight benchmarks confirmed the effectiveness our approach, showing consistent improve- ments over GRPO and strong hybrid-policy baselines. We believe this work provides a step toward more generalizable reasoning in MLLMs, highlighting the importance of controllable exploration as key ingredient in future post-training strategies.
10

---

documented in the Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. Details of the experi- mental setup are provided in Section 4.1 and the full training configurations Appendix (Training Settings). Additional implementation details are included in the supplementary materials. Our codebase is built upon the verl framework (Sheng et al. 2024), with the main im- plementations located in the src/ directory. To further support reproducibility, we will release all training data and pretrained model weights upon publication.
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China No. 62441235, and is also supported by Beijing Natural Science Foundation (1.257005).
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502 13923, 2025.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingho Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benehmark towards multimodal numerical reason- ing. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513-523, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18/33vv12021 http:////./....1.neeeeeee findings-acl.46. URL ittDs://aclanthoooog... wwwpllooy..or/200000 wrtp/0/1112222 findings-acl. 46/
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.222 226117, 2025.
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Ying- wei Ma. Jue Chen, Binhua Li, et al. Rl-plus: Countering capability boundary collapse of Ilms in reinforcement learning with hybrid-policy optimization. arXiv preprint 0rXiv22508.00222 2025.
Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol- like milm. arXiv preprint arXiv:2501. 2022250110/94 0/904,400000 2025.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. Vlmevalkit: An open-source toolkit for evaluat- ing large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 11198-11201, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 1rXiv:2501.11948. 2025.
Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. On-policy with optimal reward baseline. arXiv preprint arXiv:2505.2355 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021.
Jingcheng Hu. Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner- zero: zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXhv:2503.24290, 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21276 2024.
11

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Hynek Kydlíček. Math-Verify: Math Verification Library. URL https://gitte com/
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9)99, 2024.
Jia Liu, ChangYi He, YingQiao Lin, Ming Yang. FeiYang Shen, ShaoGuo Liu, and Ting Gao. Etrll: Balancing exploration and exploitation in Ilm test-time reinforcement learning via entropy mechanism. arXiv preprint 1tti:2508..133560 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 25X3:25033.0788, 2025b.
Pan Lu, Ran Gong, Shibiao Jiang. Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chur Chun Zhu. Inter-gps gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference the 59th Annual Meeting of the Association for Computational Linguis- fies and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), 2021.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, al. Ovis2. 5 technical report. arXiv preprint arXiv:2508. 2025.
Lu Ma, Hao Liang Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, al. Learning what reinforcement learning can't: In- terleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527 2025.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 01153228-53744 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, et al. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint uxXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
12

---

Published as a conference paper at ICLR 2026
Andrew Starnes, Anton Dereventsov, and Clayton Webster. Increasing entropy boost policy gra- dient performance on personalization tasks. In 2023 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1551-1558. IEEE, 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12591 2025.
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint urXiv:2504.068: 00837, 2025.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhnttps://openeeviee: htgpa//rereeee.eeeeoooooooorrrrr
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Jinyang Wu, Chonghua Liao, Mingkuan Feng, et al. thouught-augment policy optimization: Bridging external guidance and internal capabilities. arXiv preprint arXiv:2505./5662, 2025.
L.MM-Crr--Team Xiaomi. Mimo-vl technical report, 2025. URL htttts:///aeii...oo
0556603363
Jianhao Yan, Yafu Li. Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng. and Yue Zhang. Learning to reason under off policy guidance. arXiv preprint arXiv:2504. 14945, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14622 2024.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint 1rXiv:2508.11400 2025.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912. 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
13

---

Published as a conference paper at ICLR 2026
A TRAINING PROMPT
As shown in Table 9 for multi-modal image-question input, we concatenate the question with our instruction, which guides the model to perform step-by-step reasoning and specifies the desired output format.
Table 9: Prompt template.
<image> question.
FIRST think about reasoning process as an internal monologue and then provide the final answer.
The reasoning process MUST BE enclosed within <think < //tink>> tags. The final answer MUST BE put in <answer> /answer>> tags.
B TRAINING SETTINGS.
We conduct our experiments on NVIDIA A800 80G GPUS. For fair comparison, we follow the standard GRPO training setup with several modifications as previous works (Liu al. 2025bb Yan et al. 2025 Dong et al., 2025). First, we disable the KL regularization term by setting its coefficient to zero and removing both the length normalization and the standard error normalization in the original GRPO loss. For rollout generation, we use a temperature of 1.0 and perform 10 rollouts per prompt. In the case of hybrid-policy RL. we additionally include one off off-policy rollout. The rollout batch size is set to 480, while the update batch size is 120. As the reward signal, we employ MATH- VERIFY (Kydlíček), combined with lightweight format reward of 0.1. For controllable exploration, we set the À weight to 0.1 as the default. The GeoEval data remained completely unseen during training. We trained all models for fixed number of steps without any early stopping, and checkpoints for comparison were selected at identical training steps across all experiments.
C ADDITIONAL EXPERIMENTAL RESULTS
General applicability beyond visual reasoning. In this work, we focus achieving stable entropy control in RLVR for VLMs, where the challenge becomes especially severe in the vast state space of MLLMs and substantially limits learning efficiency. While series of works like LUFFY (Yan) eet al. 2025) and RL-PLUS (Dong et al., 2025) have achieved promising results for LLMs, such results have not transferred MLLMs, leaving notable gap in the literature, one that our work
aims to fill.
However, our method can certainly be expanded broader applications beyond visual reason- ing. We extend CalibRL to pure text math reasoning tasks to demonstrate the effectiveness be- yond visual reasoning. Specifically, we train D0we22.5-VL-7B on 9k-sample subset of the MATH dataset (Hendrycks et al. (222]) and evaluate the model on the in-distribution benchmark MATH-500 Hendrycks et al. 2021) and the out-of-distriibtioo benchmark AMC (Li et al. 2024).
Table 10: Performance of the Qwen2.5-VL--7B model on math reasoning tasks.
Method
MATH-500 (in-distribution)
AMC (out-of-distribution)
Owen2.5-VL-7B
64.8
31,97
GRPO
68.5
31.59
CalibRL (ours)
70.2
32.08
Results are reported in Table 10 Training on pure text data, CalibRL yields stronger gains than GRPO MATH-500, demonstrating more effective in-distribution improvement. Moreover, like GRPO, which fails to enhance out-of-distribution performance, CalibRL continues to deliver benefits AMC, achieving strong generalization results.
14

---

Published as a conference paper at ICLR 2026
Scalability to larger model sizes. We also verify method the Qwen2.5-VL-32B model. We train the Qwen2.5-VL VLL32BB with GRPO and our CalibRL, respectively, using the same training data constructed for our main paper results. Results reported in Table II consistent improvements when scaled to a larger model.
Our method exhibits
Table 11: Performance of the Qwen2.5-VL-328 model.
in-distribution
out-of-distribttion
Method
Avg.
GeoEval
Geo3K
GeoQA
Qwen2.5-VL-322
37.94
48.24
60.87
48.60
GRPO
44.48
51.75
63.53
53.25
CalibRL((urs)
48.77
52.58
67.90
56.42
Analysis of potential length bias in AC;. We acknowledge that using the Al; may introduce a mild length preference. However, in method, the expert trajectory intended guide only correctness but also the answering paradigm-includde overly long excessively short responses. In this sense, a small length preference is aligned with our design goal of encouraging the model to follow the expert's 's response style.
stylistic preferences such as avoiding
Moreover, this bias does override correctness learning. The policy still optimized primarily by the main GRPO loss, which determines reward -araaddmaxmmzzing behavior. The exploration term is scaled by small and serves only to regulate the degree of deviation from the expert, rather than to dictate correctness. Thus, the optimization direction remains dominated by the policy objective, and we did not observe harmful interference with correctness.
To further analyze the influence of such length bias, we conduct an ablation by adding length normalization to AG; computation. Results are reported in Table The strong performance of the model without length normalization supports our original design.
Table 12: Performance on length norm.
in-distribution
out-of-distribution
Method
Avg.
GeoEval
Geo3K
GeoQA
CalibRI
w/o length norm w/ length norm
33.44
40.60
60.74
44.93
29.26
39.10
60.34
42.90
Effeet of weaker expert baselines. In main results, we GPT-4o the expert to generate CoT responses for training. To further verify the generalizability of CalibRL, we additionally com- pare GRPO and CalibRL when the training data are produced by Qwen2.5-VL. a weaker expert compared to GPT--4o. We keep the size the Qwen-generated dataset comparable the GPT-generated one (approximately 9k samples). However, because the expert models differ in capability, the filtered sets of correct responses are not identical, and we cannot guarantee that both methods are trained the same sampled data. Nonetheless, this setting still provides meaningful evaluation of the robustness and generality of CalibRL under varying expert quality.
which serves as
The results in Table 13 show that CalibRL consistently delivers substantial improvements over GRPO in both settings. As expected, the magnitude of improvement depends on the quality of the expert baseline. with a stronger expert providing a high-quality. generalizable, and informative reference. This highlights the importance of expert guidance in controllable-entrope RLVR and fur- ther demonstrates that CalibRI. can effectively adapt learning behavior to the quality of the expert model, leading to significant gains in training performance.
D COMPUTATIONAL COST AND SAMPLE EFFICIENCY
Our method adds one oof-py expert response per prompt to the standard GRPO group G policy-generated responses (where G 10 in our implementation). This introduces well-defined
15

---

Published as a conference paper at ICLR 2026
Table 13: Trained different expert data.
in-distribution
out-of-distribution
Method
Avg.
GeoEval
Geo3K
GeoQA
On Qwen data
GRPO
23.26
41.10 40.43
49.87
38.08
CalibRL
26.90
56.90
41.41 +3.33
On GPT data GRPO
26.15
39.77
52.52
39.48
CalibRL
33.44
40.60
60.74
44.93
incremental cost that we quantify below (see Table 14). The expert response requires one addi- tional forward pass to compute the expert log probability. Importantly, different from previous hybrid-policy methods, including LUFFY and RL-PLUS, the expert response does not participate in gradient computation. It serves only as reference baseline. This means while we have G+1 forward passes per prompt, we still only perform backward passes, identical to standard GRPO. Also, different from LUFFY, we require no additional reward for the expert trajectory, since we only include on on-policy samples in the advantage calculation. Additionally, expert data collection one-time offline cost, not incurred during training, rollout sampling overhead is zero.
Table 14: Per-prompt computational breakdown for typical group size of G.
Component
GRPO
CalibRI
Additional Cost
Rollout sampling
Forward passes
G
G+1
+1
Reward model queries
0
Backward passes
G
G
0
E
THEORETICAL GROUNDING OF |
Possible double-counting of advantage terms. Our training objective:
rr min (O)e(0)
(0)) = (.gDD.FNTO(l))
(11)
().)) A| LeakyReL.U s,Als,
In the main GRPO objective, A determines the update direction of the policy, while in the ex- ploration term, A is treated purely static importance weight. The two occurrences of the advantage influence different aspects the optimization: the GRPO term governs policy improve- ment, whereas the exploration term modulates the strength of trajectory-level exploration
exxploootioo correction.
The exploration loss gradient is:
d @exxloration = | LeakyRel.U((ssAAA) - V (ogg)))))))
(12)
00
while the GRPO gradient is:
a (0)A.))) A Vori.(0)
00
(13)
= = - (0) Vo log (a,,))
Both terms produce updates through (V, log To), but with different multiplicative coefficients (the GRPO term uses (Ar,)aa) while the exploration term uses (1As((=x]TeakyRee..) and scaled
16

---

Published as a conference paper at ICLR 2026
by A). The key observation is that these coefficients are additive in the total gradient, not multiplica- tive. The final gradient is:
VoJ Σ An, Ai,e-Tirr(0) Va log (a(s))) - A - LeakyyeeU(()) Ve log (a())). (14)
The advantage appears in two separate, additive gradient contributions that can constructively or destructively interfere depending on their signs, with the hyperparameter À controlling the relative magnitude of exploration correction versus policy improvement, preventing uncontrolled amplifi- cation. In our experiments, À = 0.1 ensures the exploration term provides a measured correction signal without dominating the GRPO updates.
The "rarity" interpretation of |A: We provide formal characterization of the intuition of using | value naturally capture group-wise "rarity".
First, consider several example groups of sequence- level rewards:
Group 1: [O.1,1,1], where the "0" answer is relatively rare within the group. The abso-
lute value the group advantage is (0.75.0.25,0225,0.233 The "0" has the largest |A], indicating that a reward of "0" is the group-wise outlier.
Group 2: [1,0,0,0], where the "1" answer is relatively rare within the group. This group resulted in the same absolute value of the group advantage: [0.75.0.25,0225.02255 also indicating that a reward of "1" is "rare" in this group.
The examples demonstrate that the frequency of correct/incorrect answers and |A,| correspond sym- metrically and automatically, without requiring additional heuristics. We further show a continuous change between |A,| and reward frequency in a group of 10 samples in Figure strictly monotonic mapping between rarity and magnitude.
The curve shows
Relationship and Frequency
Al A.
Bare
Rore [kl 1x
Balanced
Equal 15
[A]
122 zame
10
Number "1'sin group samples)
Kay inigiee Al wwwwettyyyyy.oooo....e.. propertiona trsqurncy When tow trequameyl Aigh
Figure 4: Relationship between | and reward frequency in group of 10 samples.
Furthermore, to analyze robustness under small additive reward noise (e.g., formatting bonuses), consider the perturbed reward
=(T))=(TT)) + oi Se.ee.
The corresponding advantage becomes
bhe = R' R'(T.) mcan( ('((T,) = - (R(T.) -MG) + (5, - 8),
where
Ng = mean (((T)))=)))
= mean (8,)
Thus, the perturbation to the advantage
N/ A 8,
17

---

Published as a conference paper at ICLR 2026
which is mean-centered noise term satisfying
|6, - 01 S 2e.
Since typically
(((T)) pGl O(1,
the ordering of |A,, which determines group-wise rarity, is preserved for sufficiently small e. In the common case where noise is approximately uniform across samples (c.g., shared formatting bonuses), we have ó; = 8, and the perturbation cancels:
N Ais-
Hence, mean-centered normalization makes the rarity signal e inherently robust to small reward noise.
F EFFECTIVENESS ANALYSIS OF CALIBRL
We obtain several statistical data points from our trained checkpoints contrast CalibRL with RL- PLUS and LUFFY and to reveal the structural differences brought by explicit calibration. Both LUFFY and RL-PLUS attempt to use experts without an effective calibration mechanism, which leads to either diluted expert signals or unstable confidence dynamics. In contrast, CalibRL intro- duces a calibration mechanism that stabilizes expert influence, as reflected in the following statistics.
As in Table 15 LUFFY collapses to single expert mode. The policy and expert distributions be- come nearly indistinguishable after training. The model treats expert responses as uniform style to imitate. It loses the ability to evaluate where expert guidance should matter and where explo- ration should continue. RL-PLUS shows the opposite tendency. The policy becomes prematurely certain. Expert responses also lose diversity and drift toward the policy. The model reinforces its own early preferences without checking them against a stable expert reference. converges quickly but without reliable calibration. In both cases, expert information does not form stable reference for exploration.
Differently, CalibRL is not designed directly learn fit the expert distribution, which would resemble an off-policy imitation objective. Instead, CalibRL. uses expert responses as a baseline that calibrates exploration rather than constrains generation. As shown in Equations g and g the distributional influence of expert samples is not monotonic or uniform increase in consistency with the expert distribution. Specifically, when the policy's 's rollout produces an incorrect answer, optimization pushes the policy closer to the expert sample distribution; however, when the policy itself produces correct rollout, the method suppresses the probability of expert samples. Thus, distributional shifts measured across all samples cannot directly reflect the calibration mechanism that CalibRL. performs.
The statistics in Table 15 support this distinction. CalibRL. maintains the broadest policy entropy, showing that model stays exploratory rather than collapsing into single expert mode as in LUFFY or becoming prematurely overconfident as in RL-PLUS. Expert responses also remain di- verse and on-policy, which provides a stable reference for calibration rather than a target distribution to mimic. CalibRL shows negative Al, indicating that the model consistently assigns higher likelihood to expert answers and uses this signal to navigate exploration toward better solutions. This calibrated behavior produces higher rewards and shorter, more precise outputs.
Table 15: Statistical data points from our trained checkpoints.
response length (expert length 382.19)
Method
A4
policy entropy
reward
Qwen2.5-VL-7B
0.1459
0.4401
0.2029
437.76
GRPO
0.4256
0,2508
0,5381
431.79
LUFFY
0,0881
0.4688
0,4980
414.58
RL-PLUS
0.3452
0.2580
0.4975
425.08
CalibRL
-0.8025
1.4968
0.5667
271.71
18

---

Published as a conference paper at ICLR 2026
G CASE STUDY
We select cases from the challenging Vision Wang et al.] 2024) benchmark and compare our response with the GRPO baseline, the SFT+GRPO baseline, the LUFFY Yan et 2025). and the RL-PLUS (Dong et al. 2025).
We first compare the response among our CalibRL, the GRPO baseline, and the SFT+GRPO baseline in Figure 5 The GRPO baseline, lacking explicit guidance, exhibits erroncous reasoning and hallucinatory requirements (highlighted in red). In contrast, the SFT+GRPO baseline is overly constrained by the supervised signals, limiting its exploratory capacity; as a result, it attempts to frame the problem in purely mathematical manner, which ultimately leads to ineffective reasoning. Distinet from both, our approach leverages effective guidance to explore the most efficient solution path, directly and precisely resolving the problem.
Similarly, in the case analysis shown in Figure 6 LUFFY falls into inefficient exploration, which results both visual understanding errors and flawed reasoning (highlighted red), ultimately failing to produce the correct answer. RL-PLUS leverages multiple importance sampling (Dong ctt al. 2025) to partially alleviate inefficient reasoning. However, it fails to fundamentally resolve the problem. In contrast, method maintains effective exploration while preserving expert guidance, thereby successfully and reliably addressing the task.
H THE USE OF LARGE LANGUAGE MODELS
We employed large language models in our work to check the grammatical correctness of the paper and to provide refinements to selected sentences for improved clarity and readability.
19

---

Published as a conference paper at ICLR 2026
You helpful axcistant. On Nadya's smartphone, the diagram shows how much time she spent last week on four of her apps. This week she halved time spent of these apps, spent the same amount of time as the previous week on the other two apps.
Prompl:
<image1s:
<imageZo:
Correct Answer: "D"
GRPO
SFT+GRPO
CalibRL
<Lhinko The diagram shows time spent four apps lost wesk. nced ideutify
Response:
Response:
nkh To salve this andlyze changes
Response:
<Lhink> The diegram shows distibbtio of sgent apys last week, the opticns represent the same distihution bet with the time on two apps being halved and the other two the new diagram Since
peoblem, in Nadya spent her apps. diagram shows the time speat on four apps last week. Let's denote the time spent on
two apps Nodya halved and which two she kept the same. The diagram in image? shows this options how time might
apps follows:
lockiing capresencs muiio, we neal to identify the option that shows hilived time for two and no Comparing the initial diagum, option indieates a reduction in time for two bazs while keep- unchanged.
App A: TA
Let's onalyze the options:
Anp B: T.
Optiin A: The time spent is halved for
App C:: C:TC
apps
for other
App
change
Ciption
spent
This Nadya haleed time spent
options
zil four apps.
on two of these apps, but spent the same
manches this description,
Cpption C: The time is halved for
amouns time as the previous week on
apps
for other
apps. We noed ilemify apps had Let's sssume that the two apps with halved time and C. Therefore, this week
other Thus, coneect unswer must </think>>
-iption
time spear the
two apps and lsaledd for the other two.
-Option E: The time
is halsed for
apos
other
Nadya spent Lg on App A and Te.
<//rsver>>
Since Nadys halved the time spent on two
C. The
App B and App
apps and kept the same for the other twn.
Tn Tn repectivelly.
the correct option must be one where two
The tonal time spent this week is:
hnlived and
TA Tn + 2
Ic. Tn
Or- tion C beth show that two bars are halved and are the same. However, Option shows the halved hars which consistent with diagram im- agel where the hars are shoner for twa </shinko
Looking the options, Option
The sotal time speni last week isc
Tp +
Tp
the percentage change in sal time spens, we compure the total time spent this week to the coaal time spent last week:
option
Parcentage change
A +7D 1T -T
11TH 1O1TD
<ansWcE>> </answer>
100
Simplifying the expression inaide the puneniheses:
I I IF +TD
TA +T+
+
Tx+7" +To
The negative sign indicates a decresse in the total time spent. The percentage de-
Tx+7 D
To the pereentage, would specific values T and To- but the problem rsks for the pereemage change. Without specific values, we can't compute
perenstage.
say that the time spent less thas last week doe to the halving of the time spent rwo apps. cit
Ken Penswerrs -25% c/anawer>
Figure 5: A case of CalibRL compared with baselines GRPO and SFT+GRPO.
20

---

Published as a conference paper at ICLR 2026
The diagram shows the floor plan of Renate's house. Renate enters her house from the terrace (Terrassc) and walks through every door the house exactly once. Which room does she end up in?
Prompt:
helpful
Correct Answer: "2"
Terrasse
LUFFY
RL-PLUS
CalibRL
solve romm Renate ends up in after walking through door exacily once. This is essentially problam of finding Ierian poth a graph, a node and each door an odge. Here's the step-by-stop ressoning: Idemiiy the
Response:
Response: <thinks
Response: <chink>
problem.
prohlens we need to determine
represents a grid-ikke structure: walk through every door exactly onee. Resate must start the terrace. walk through that al- retracing steps. The rooms must be travened in such a that they form puth that each
rom Resate ends up in after walking through every door exacily once. This
classic problem of finding path graph,
each
room
each roam
lows
mode and each door an edge.
Let's analyse the floor plan:
house has five 5.
labeled
(doors) graph.
grid, one room not have on cxiit conesponding to non- visited dooe. Here, room 2 appears to be without esit coreepuncling to unvisited done, and connected to three doors. The logical end point is room </chink>
Since
2. Count the degree (nambeer of edges) of each node.
There are doors connecting these romms.
Buler's thearem graph his Fulerian path only if exactly zaro or two nodes have an odd degree. All other modes must have degree.
Resate
To find the room Renati ends in, we nead to identify the mom that hox odd odgos In un Eolerian path, puth stans veries with on odd degros and ands at venes. with an odd degroe. All other even degree.
exactly two
censwer> </ansuer>
odd degraes, the Fuleria path will start
of these nodes and end the
other.
there
edges for rooms
grees, the Ealerian path cas start at any
Room 1:3 edges
mode and end mode.
Room edges
given diagram:
odless
rooms labeled as
Room edges
Room S: edges
"The terroc not room but
number of edges, an of ellges, Thersfore, Room only room that can he the andpoint of the Eulerian path.
Room 5 has
count the degroes of ench mode:
Room 1: odges
Room 2: edlpss
odges
up
odges
</think>
Room S: edles
All have
graph degroe, which Falerian path
<answar> <
that tum starl at any room and end any mom.
from terrace, it's the only room connected race). If she walks through every door exacily she will the last she used. In this case, she will end up romm is she starts from room 5 and wailks through each ronm exaetly Therofore, the room she ends up is room lx/thick>>
Since Renate
she
the
<ansvce> aaa/w>>>>>>>
Figure 6: A case of CalibRL compared with LUFFY and RL-PLUS.
21


Reviews:
[{"review_id": "33Pziu2QrW", "reviewer": "Reviewer_MeYV", "summary": "This paper studies the RLVR algorithm. The authors start by analyzing the drawbacks of existing paradigms of incorporating expert data and reward feedbacks into foundation models, and then propse a novel algorithm, CalibRL, that alleviate the problems. Finally, they show via visual-reasoning tasks, that the proposed algorithm improves the performance of reinforcement learning.", "questions": "1. Does the algorithm apply to other areas, such as math reasoning (without multi-modality)?\n2. In formula 9, why do we need to define another signal that labels the correctness? Cannot we just use a normalized version of the actual reward?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and easy to follow. It provides a nice summary of existing methods as well.\n2. The proposed algorithm is simple and novel.\n3. The empirical performance is strong.", "weaknesses": "1. The algorithm seems to apply further beyond visual reasoning domain, but is not discussed.\n2. The effectiveness of the algorithm is not verified for larger scales, like 30B. This seems too much to ask for though especially if the paper comes from academia.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "17rWcatF5t", "reviewer": "Reviewer_sgYA", "summary": "CalibRL reframes demonstrations as a calibration baseline inside RL with verifiable rewards, adding two simple mechanisms to a GRPO pipeline: a relative log‑probability gap between the on‑policy and expert responses passed through an asymmetric (LeakyReLU‑style) gate, and a group‑rarity weighting that scales updates by how uncommon a response is within a rollout group. The result is exploration that stays stochastic yet directed—rare correct behaviors are strengthened while overconfident mistakes are damped. Across eight visual‑math and multimodal reasoning benchmarks and several base models, the method consistently improves over a GRPO baseline and avoids the entropy collapse or aimless randomness observed in common SFT‑then‑RL and prior hybrid setups.", "questions": "1. Why do LUFFY/RL‑PLUS trail GRPO here? Any hints from your logs (entropy trends, clipping ratios, off‑policy mismatch) as to why these hybrids underperform? Sharing failure modes would be valuable to the community.\n\n2. Calibration evidence. Beyond entropy curves, could you report ECE or other calibration diagnostics on answer probabilities, to substantiate “distributional calibration”?\n\n3. Generalization beyond geometry. Have you tried training CalibRL outside geometry (e.g., coding or algebraic word problems) with appropriate verifiers (e.g., test oracles for code)? It would help demonstrate domain generality.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This proposed method introduces a clear hybrid‑policy objective that treats expert trajectories as a relative reference, which helps maintain entropy while steering updates toward verified behaviors. The design uses an intuitive pair of knobs—an asymmetric gate on the policy‑vs‑expert log‑prob gap and a group‑rarity magnitude—that integrates cleanly with GRPO and is easy to implement.\n\n2. Simple, general mechanism: The LeakyReLU‑gated is an elegant way to use demonstrations for relative guidance. It respects the on‑policy signal and works as a plug‑in to GRPO.\n\n3. Experiments demonstrate consistent gains on eight benchmarks, with especially large improvements on the hardest geometry split and solid out‑of‑domain boosts. The approach generalizes across different model sizes/architectures, and the ablations probe the important controls (gate slope and balance weight) to show how exploration is tuned.", "weaknesses": "1. GeoEval split clarity: The paper constructs GeoEval from validation failures of GPT‑4o CoT filtering, then reports it as a test benchmark with the largest deltas. Please clarify whether this split was ever used for hyper-parameter tuning or early stopping. If yes, results could be optimistically biased; if no, state this explicitly and detail safeguards.\n\n2. Baselines for entropy control: Since the contribution is controllable exploration, it misses comparisons to standard entropy‑regularized GRPO (fixed/annealed entropy bonus) or token‑level entropy‑regularized updates. This would isolate whether your calibrated, expert‑relative update is better than “just add entropy.”\n\n3. Ablation on the activation & reference. It would strengthen claims to compare LeakyReLU vs hinge, Huber, or sigmoid saturations, and to test “expert baseline” vs “reference policy baseline” (e.g., KL to SFT policy), to confirm that expert pairing and asymmetry specifically drive the effect.\n\n4. Compute/efficiency reporting. Please quantify incremental overhead (extra forward passes to score expert trajectories), wall‑clock time vs GRPO, and sample efficiency (accuracy vs total rollouts). This is particularly important for practitioners training on LLMs / VLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GI93ExSsrV", "reviewer": "Reviewer_5pSB", "summary": "I am not an expert in this domain, I will try to review this paper from my limited understanding.\n\nThis work introduces CalibRL, a reinforcement learning framework that improves exploration in multi-modal large language models. Unlike prior RLVR methods prone to entropy collapse, CalibRL uses advantage weighting and a LeakyReLU-based asymmetric activation to guide exploration with expert supervision while preserving diversity. This balanced approach stabilizes training and enhances reasoning performance, achieving consistent gains over GRPO, LUFFY, and RL-PLUS across multiple benchmarks.", "questions": "1. The proposed method does not seem inherently limited to VLMs. It could also apply to LLMs. Why is the paper specifically focused on VLMs?\n2. Why LeakyReLU-based activation yields controllable entropy growth? Do other similar activations have a similar effect?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow.\n2. It addresses an important and widely existing problem, that under the SFT-then-RL paradigm, the policy becomes tightly anchored to the expert distribution during the SFT stage. This causes exploration to be restricted within the local neighborhood of expert behaviors, making it difficult to adapt to reward signals or discover more optimal reasoning trajectories.\n3. The paper provides comprehensive and convincing ablation studies that support its claims.", "weaknesses": "1. Most evaluations are math benchmarks. The claim of general multi-modal reasoning would be more convincing with benchmarks involving richer visual, linguistic, or commonsense reasoning modalities (e.g., ScienceQA, MMMU, or multimodal dialogue tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "emhq1UN6xV", "reviewer": "Reviewer_EZpA", "summary": "The paper proposes **CalibRL**, a hybrid-policy RLVR framework for MLLM reasoning that reformulates expert supervision as a distributional baseline rather than a hard imitation target. For each prompt, the method defines a log-likelihood gap $\\Delta \\mathcal{l}$ between the model’s on-policy trajectory and an expert trajectory, multiplies it by a correctness indicator $s_i$, and **passes $−s_i\\Delta \\mathcal{l}$ through a LeakyReLU gate**. The resulting exploration term is further scaled by the **group-normalized advantage magnitude $∣\\hat{A}_i∣$** (interpreted as group-wise rarity), and added to a GRPO-style objective. Intuitively, CalibRL **reinforces underweighted correct** responses and **suppresses overconfident incorrect** ones while keeping policy entropy from collapsing. Across eight benchmarks (in-domain and OOD) and multiple backbones, the method reports consistent gains over GRPO and several hybrid-policy baselines, with ablations on $\\alpha$ (LeakyReLU slope) and $\\lambda$ (trade-off).", "questions": "1. Please **explicitly define $∣\\hat{A}_i∣$** used in Eq. (9): is it exactly the GRPO group z-score at sequence level (shared across tokens)? Do you apply clipping or normalization? How do you handle **std = 0** groups (all-correct/all-wrong)?\n\n2. How do you address **length effects** in $\\Delta \\mathcal{l}$? Any experiments with token-average log-prob or length-normalized variants, and their impact on long CoT tasks?\n\n3. Does CalibRL remain effective if the **expert baseline is weaker** (e.g., a smaller SFT model)? Please report performance vs. strength of the expert.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "i. **Clear and practical idea:** Uses the expert as a relative reference to guide exploration instead of enforcing imitation; integrates into GRPO with minimal code changes.\n\nii. **Readable and structured:** Algorithm and ablation setups are clear and reproducible.\n\niii. **Well-motivated gating:** The four-quadrant behavior (boost underconfident correct, damp overconfident wrong) neatly matches RL intuition and helps preserve entropy.\n\niv. **Consistent empirical gains** across diverse reasoning benchmarks and model sizes; ablations on $\\alpha$ and $\\lambda$ illustrate controllability of exploration.\n\nv. **Rare-event emphasis** via $∣\\hat{A}_i∣$ often improves signal quality in sparse-reward, group-sampled settings.", "weaknesses": "i. **$∣\\hat{A}_i∣$ is under-specified** near Eq. (9): the paper states it “captures group-wise rarity” but does not restate the exact computation (presumably GRPO’s group z-score) nor the degenerate case handling when the group reward variance is zero (all-correct or all-wrong).\n\nii. **Length bias in $\\Delta \\mathcal{l}$:** using sequence-level log-likelihood differences without explicit length normalization can penalize longer (yet correct) solutions; no targeted analysis is provided.\n\niii. **Sensitivity/fairness:** Results depend on group size $G$, sampling temperature/top-p, and the presence/absence of KL to a reference policy. These choices for baselines (e.g., LUFFY, RL-PLUS) are not exhaustively justified or tuned for parity; statistical significance over multiple seeds is limited.\n\niv. **Potential double-counting of advantage:** GRPO already weights updates by $∣\\hat{A}_i∣$,$t$; the exploration term scales again by $∣\\hat{A}_i∣$. Without gradient-norm breakdowns, it’s unclear whether this over-amplifies certain samples.\n\nv. **Theory is light:** The claim that $∣\\hat{A}_i∣$ encodes “rarity” is intuitive but lacks formal characterization (e.g., relation to group frequency, robustness to reward noise).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JXxekzpZya", "reviewer": "Reviewer_ux1J", "summary": "This paper addresses the challenge of entropy collapse and inefficient exploration in Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing multi-modal large language model (MLLM) reasoning. Existing methods often suffer from policy degradation or over-exploitation, while hybrid approaches combining RL with supervised fine-tuning (SFT) can lead to distributional mismatch and accelerated entropy collapse. To overcome these issues, the authors propose CalibRL, a hybrid-policy RLVR framework that enables controllable exploration by treating expert demonstrations as a calibration baseline rather than a strict imitation target. CalibRL employs two key mechanisms: distribution-aware advantage weighting, which emphasizes rare yet informative responses to preserve exploration diversity , and an asymmetric LeakyReLU activation function that uses expert log-probabilities to moderate overconfident updates while amplifying underrepresented correct reasoning paths. By calibrating policy updates relative to expert knowledge, CalibRL aims to maintain productive policy entropy and guide exploration effectively, balancing exploration and exploitation. Extensive experiments on eight benchmarks demonstrate that CalibRL consistently outperforms standard GRPO and state-of-the-art hybrid-policy baselines in both in-domain and out-of-domain settings.", "questions": "Please see paper weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the crucial challenge of balancing exploration and exploitation within Reinforcement Learning with Verifiable Rewards (RLVR), a key paradigm for training reasoning capabilities in multi-modal large language models (MLLMs) which often suffers from entropy collapse.\n2. The proposed CalibRL framework introduces intuitive mechanisms, namely distribution-aware advantage weighting and asymmetric activation using LeakyReLU calibrated by expert knowledge, which are conceptually straightforward to understand.\n3. The effectiveness and generalization ability of CalibRL are rigorously validated through extensive experiments across eight diverse benchmarks, encompassing both in-domain and out-of-domain (OOD) reasoning tasks.", "weaknesses": "1. The comparison is limited to GRPO and a few hybrid methods (LUFFY, RL-PLUS), omitting recent advancements in GRPO variants like DAPO which could offer a more competitive benchmark .\n2. While supplementary experiments are conducted on two additional models, the primary evaluation and analysis rely heavily on a single MLLM architecture (Qwen2.5-VL-7B), leaving the method's broad applicability across diverse MLLMs less certain .\n3. The paper does not quantify the potential additional computational overhead introduced by calculating the log-probability gap relative to expert responses and applying the asymmetric activation and advantage weighting compared to the standard GRPO baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)