NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
Authors: Longtian Qiu
Shan Ning
Jiaxuan Sun
Xuming He
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Reinforcement learning
Primary area: reinforcement_learning
TL;DR: A systematic multimodal RL framework that improves the policy exploration and advantage estimation.

Abstract: Reinforcement learning (RL) has shown promise in enhancing the general Chain-of-Thought (CoT) reasoning capabilities of multimodal large language models (MLLMs). However, when applied to improve general CoT reasoning, existing RL frameworks often struggle to generalize beyond the training distribution.  To address this, we propose NoisyGRPO, a systematic multimodal RL framework that introduces controllable noise into visual inputs for enhanced exploration and explicitly models the advantage estimation process via a Bayesian framework.
Specifically, NoisyGRPO improves RL training by: (1) \textbf{Noise-Injected Exploration Policy}: Perturbing visual inputs with Gaussian noise to encourage exploration across a wider range of visual scenarios; and (2) \textbf{Bayesian Advantage Estimation}: Formulating advantage estimation as a principled Bayesian inference problem, where the injected noise level serves as a prior and the observed trajectory reward as the likelihood. This Bayesian modeling fuses both sources of information to compute a robust posterior estimate of trajectory advantage, effectively guiding MLLMs to prefer visually grounded trajectories over noisy ones.
Experiments on standard CoT quality, general capability, and hallucination benchmarks demonstrate that NoisyGRPO substantially improves generalization and robustness, especially in RL settings with small-scale MLLMs such as Qwen2.5-VL 3B.

Noisy Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
Longtian Qiu', Shan Ning Jiaxuan Sun', Xuming He "ShanghaiTech University, Shanghai, China
Shanghai Engineering Research Center of Intelligent Vision and Imaging Lingang Laboratory, Shanghai, China (qiult, ningshan2022, sunjx2022, hexm @shanghaitech.edus edu.cc
Abstract
Reinforcement learning (RL) has shown promise in enhancing the general Chain- of of-Thought (CoT) reasoning capabilities of multimodal large language models (MILMs).. However, when applied improve general CoT reasoning, existing RL frameworks often struggle to generalize beyond the training distribution. To address this, we propose NoisyGRPO, systematic multimodal RL framework that intro- duces controllable noise into visual inputs for enhanced exploration and explicitly models the advantage estimation process via Bayesian framework. Specifically, NoisyGRPO improves RL training by: (1) Noise-Injected Exploration Policy: Perturbing visual inputs with Gaussian noise encourage exploration across a wider range of visual scenarios; and (2) Bayesian Advantage Estimation: For- mulating advantage estimation as a principled Bayesian inference problem, where the injected noise level serves prior and the observed trajectory reward as the likelihood. This Bayesian modeling fuses both sources information to compute robust posterior estimate of trajectory advantage, effectively guiding MILMs to prefer visually grounded trajectories over noisy ones. Experiments on standard CoT quality, general capability, and hallucination benchmarks demonstrate that NoisyGRPO substantially improves generalization and robustness, especially RL settings with small-scale MLLMs such as Qwen2.5-VL 3B. The project page is available at https:: / aaraaaanniii c30, .ee.30..gttii gittab..oo/proeeccppeeeeeee http:roeeeecc.ogoooooioooooooooa
Introduction
The success of OpenAl's 01 [34] and DeepSeck-R1 (7) highlights that activating Chain-of- Chain-of-Thought (CoT) reasoning abilities through Reinforcement Learning (RL) is a promising strategy for scaling model intelligence test time. In partieular, value-free RI. methods such Group Relative Policy Optimization 44 (GRPO) have demonstrated substantial improvements in LLM performance while avoiding the costly requirement of training separate value model. As recent works show [38] [3] 8, 30 [46). this efficiency in both data and computation makes GRPO an appealing approach for extension to complex visual language tasks such as visual math problem solving.
However, when applying GRPO to improve general chain-of-thought (CoT) reasoning multimodal large language models (MLLMs), observe notable limitations in the model's ability to generalize beyond the training distribution, which hinders the broad applicability of MLLMs. As shown in Figure I higher training rewards under the GRPO framework do not consistently yield better evaluation performance. We identify two primary factors contributing to this issue: (1) Limited Policy Exploration. RL in the generative model domain utilizes temperature sampling to generate multiple rollouts for policy exploration; however, as noted in DAPO [58], these rollouts often converge to near-identical outputs during training, leading to insufficient exploration and early deterministic
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Researd Stad
Training Ieeaaion
Training Iteration
Training Iteration
Figure 1: Performance and training statistics of three RL methods. GRPO with noise injection refers GRPO trained with noise-perturbed rollouts. The left plot shows evaluation performance on the MMStar benchmark over training iterations. The middle plot presents the standard deviation of rewards, reflecting the exploration degree of the policy. The right plot shows the accuracy reward, indicating how well the model fits the training data. Shaded areas represent the variance of the corresponding metrics.
policy. (2) Missing process supervision. Rule-based rewards judge only the final answer, leaving the intermediate reasoning steps unsupervised; the policy, therefore, learns shortcuts and visual hallucinations that not transfer out of distribution.
To address these challenges, we propose NoisyGRPO, a unified framework that enhances multimodal reinforcement learning, such as GRPO [44] by both promoting thorough exploration and ensuring robust policy optimization. Our key innovation is to inject controllable noise into visual inputs during rollouts, thereby encouraging diverse exploration and providing unbiased, trajectory-level measure of reasoning difficulty. However, while promoting exploration, noise injection induces an off- off-policy scenario: the behavior policy that collects trajectories generated with -istt-disturbed images differs from the target policy that operates on clean visual input. This distribution shift causes the value function to queried on aut-of-distribution states, which introduces bias into the advantage estimates and corrupts the policy gradient, as illustrated in Figure [ To tackle this issue, we introduce a Bayesian advantage estimation framework that treats the injected noise as a prior and the trajectory reward as the likelihood within principled Bayesian formulation, which allows us to explicitly calibrate the policy update according to trajectory-level noise and reward signals.
NoisyGRPO consists of two key components without additional computational overhead. First, Noise Injection augments policy exploration by applying Gaussian noise of various magnitudes (sampled from a uniform distribution) to input images in rollouts, maintaining trajectory diversity (indicated by the consistently high Reward Std shown Figure I and reducing hallucinations implicitly favoring visually grounded reasoning. The second component is Bayesian Advantage Estimation, which integrates the magnitude of the injected noise with observed trajectory rewards to yield more accurate advantage estimates. Specifically, we treat the likelihood uncertainty fixed constant while modeling the prior uncertainty as a function of the variance among rewards within the same trajectory group. The intuition is that if trajectory rewards are similar despite varying noise magnitudes, the prior uncertainty remains high; in contrast, larger reward variation across noise levels corresponds reduced prior uncertainty. By defining advantage estimation in this Bayesian manner, we obtain a posterior that adaptively incorporates both noise magnitude and trajectory reward statistics, leading to more accurate and robust advantage estimates.
To demonstrate NoisyGRPO's effectiveness, we evaluate its impact on improving MLLMs' general capabilities, which is more challenging setting than addressing domain-specific problems. Specif- ically, we evaluate NoisyGRPO from three dimensions: CoT quality [16], comprehensive MILMM capability [. [63) and hallucination [50]. Results show that NoisyGRPP consistently outperforms GRPO, particularly with small-scale MLLM-for instance, achieving a +4.4 improvement CoT quality on MME-COT and +3.7 gain in average performance on MMStar with Qwen2.5-VL 3B.
Our main contributions are summarized follows:
We propose NRisyGRPO, a reinforcement learning framework designed to enhance the multimodal Chain-of-Thougl (CoT) reasoning capabilities of MLLMs general tasks.
NoisyGRPO employs noise injection strategy to promote policy exploration, and introduces Bayesian advantage estimation method that mitigates the negative effects of noise by combining prior estimates from noise magnitude with observed response correctness.

---

Policy Update
0.47
heed
Policy Mode!
Reword
Beyetion Advartage
Function
7
Noise Mogritude - Ujo.1
Figure Overall pipeline NoisyGRPO. For each image-question pair, we sample noise and inject it into the image. The policy model generates rollouts based on the perturbed inputs, and the reward function evaluates them. We then compute the posterior advantage by combining the noise-based prior with the reward-basek observation.
Experimental results show that NoisyGRPO consistently outperforms GRPO across CoT quality, general capability, and hallucination benchmarks, with especially strong gains in RL with small- scale MLLMs.
2 Related Works
Multi-modal Large I Language Model Vision-language models (VLMs) have rapidly advanced
in their ability comprehend and reason over both visual and textual modalities and demonstrate
great improvements over downstream tasks [35 64 [2| 9 31 40). These models typically inte-
grate visual encoders with large language models (LLMs)
enable cross-modal understanding
and inference. Foundational works such as Flamingo 22 and BLIP-2 [20] established effective
strategies for aligning vision and language components. The LLaVA-series 25
18 and SPHINX-
series 122 10] further advanced the field by introducing visual instruction tuning, significantly enhancing multimodal capabilities. Large-scale models like GPT-4o 14 and Gemini 42] have general-purpose demonstrated strong general-pee purpose visual understanding through large-scale multimodal pertaining. To address scalability, mffture-of-expeets approaches- such as DeepSeek- VI.2 (55), Uni-MoF [2]] improve and MoVA - computational efficiency by selectively activating expert modules based on input characteristics. More recently, domain komain-specific methods such as Math-LLaVA [47 and MAVIS [60] employed mathematical visual instruction tuning improve VLMs' ability to inter- pret and solve complex multimodal math problems. At the same time, unified architectures like SEED-X [II] Chameleon [S]. Show-O (56), and the Janus series [54] 29 6 integrate both visual understanding and generation capabilities within a single framework. Despite these advances. most existing VLMs still struggle with robust visual reasoning, particularly in tasks that demand deep visual analysis and complex multi-step reasoning.
Reinforcement Learning in LLMs and VLMs Reinforcement Learning (RL) has emerged as key technique for enhancing the capabilities of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). Early advancements focused on Reinforcement Learning from Human Feedback (RLHF) (37], which aligns model outputs with human preferences M. More recent research has demonstrated that RL-based methods can significantly improve reasoning abilities. For instance, DeepSeek-RI 7 integrates rule-based rewards with Group Relative Policy Optimization (GRPO) 45 while Kimi-1.5 1.5 (49 employs a variant of online policy mirror descent- both achieving notable gains in reasoning performance, In the multimodal domain, RL has also shown promise. Direct Preference Optimization (DPO) [1 a simple yet effective RL method, has been applied to reduce hallucination in MLLMs 39
36 61.. GRPO has also been extended to vision-language models for specialized tasks such as geometry reasoning and object counting [38
13
g
30 46).
Though these works apply the GRPO framework to multimodal tasks, their focuses differ from For example, VLM-R1 [46] adapts it to detection tasks with task-specific rewards. contrast, we propose a principled framework that systematically improves multimodal RL algorithms like GRPO in multimodal settings by enhancing both policy exploration and advantage estimation.
3 NoisyGRPO
In this section, we introduce NoisyGRPO, a reinforcement learning framework designed to enhance the Chain-of- Chain-of-Thought reasoning capabilities of multimodal large language models (MLLMs). To
3

---

address the generalization limitations of vanilla GRPO when applied to MLLMs, NoisyGRPO encourages policy exploration via perturbing visual inputs and mitigates the resulting adverse effects through Bayesian advantage estimation. Specifically, in section 3.1 we present the training objective
of NoisyGRPO and highlight its improvements over the standard GRPO. section 3.2 we introduce
the exploration policy with injection noise on the visual input strategy. In section 3.3 we describe the
Bayesian advantage estimation method, which incorporates both the level noise and the trajectory reward to yield more accurate policy gradients. We provide an overall pipeline in Figure 2
3.1 Training Objective and Reward Function
Training Objective To highlight the key innovations of NoisyGRPO, we start by presenting its training objective and comparing it with vanilla GRPO [44]. NoisyGRPO differs in two main aspects: (1) Exploration Policy. We introduce visual noise during sampling encourage more diverse trajectory exploration. (2) Advantage Estimation. By modeling advantage estimation as a Bayesian inference problem, we combine the noise level and observed trajectory reward to compute a more accurate advantage for each trajectory.
Specifically, given sample (y, a, I) drawn from the dataset D-where q is the input query, is the ground-truth answer, and is the corresponding image-we sample G rollouts {or G using a noise-injected exploration policy л moise where each of is sampled under a different noise level applicd to the image. The corresponding group-wise advantage A is estimated based on both trajectory reward and noise level. The training objective of NoisyGRPO is defined follows:
JNoisyRRPO (0) 1)q.a.)) {or (o D (- (-lq.1)
Te (o THta (o) 9, clip [o
9.1)
111) To (of 1,
1 100111,,1t e.1+)) A+) -Ki (Te
min
G
where and Toas are the current and previous policies respectively, is the PPO clipping parameter, the KL penalty coefficient, Tref is a reference policy, and DRI. denotes KL divergence. The modifications from vanilla GRPO are marked in bold bold-specifically, the exploration policy π noise and the group-relative advantage A.. We claborate these components in the following sections.
Reward Function For the reward function, we adopt two types of rule-based rewards inspired by GRPO: Accuracy and Format rewards. The Correctness reward measures whether the model's s final answer is accurate, regardless of the correctness of intermediate CoT steps. The Format reward encourages the model to produce answers in the expected CoT style. It is worth noting that our training data contains many open-ended VQA samples to improve general CoT ability in multimodal settings. Unlike code or math tasks with short, matchable answers, these samples require more flexible reward. We use a text embedding model such as SBERT [43] to assess semantic similarity for correctness. Although this may introduce embedding bias, our advantage estimation 3.3 helps mitigate it. In conclusion, the reward is defined as follows:
Accuracy rewards: We utilize the SBERT model to compute embeddings for both the predicted and ground-truth answers, and define their similarity as the accuracy reward. To filter low- confidence cases, we set the reward to 0 when the similarity is below т. For yes/no and multiple- choice questions, we apply exact match: a correct match receives a reward of 1, otherwise O.
Format rewards: To ensure CoT reasoning, we introduce format reward that requires the model's reasoning process to appear between <think> and </think> tags.
3.2 Noise Injected Exploration Policy
To enhance the generalization of the multimodal RL framework, we introduce a injecti exploration strategy to construct our exploration policy noise as shown in Figure 2 Different from injecting noise into the entire policy model as in [15], we adopt a targeted and structured strategy by applying multi-step Gaussian noise to the input image / based on a diffusion-style forward process. For each rollout, we first sample a noise level n, 4400, 1), representing a normalized noise step. We then perturb using the corresponding diffusion process timestep [n, T), where T' the total number of diffusion steps. n, = 0 yields the original clean image, while n, = 1 corresponds

---

to the image at the final diffusion step, effectively resembling pure Gaussian noise. This produces noisy image I,, which is then used to generate the rollout of under the exploration policy π " resulting in a diverse set of responses {or Ji=1 conditioned on different noise intensities. Notably, noise injection is applied only during rollout collection to encourage exploration. Clean inputs are used during policy updates to ensure stable learning and preserve the accuracy of the learned policy gg.
3.3 Bayesian Advantage Estimation
Bayesian Modeling of Trajectory Quality After collecting rollouts using the noise-injected exploration policy, we formulate the trajectory quality as a latent variable ræ, which represents the underlying quality of sampled rollout o; obtained under noise-injected exploration. To integrate both prior knowledge from noise level and observation, such as the trajectory reward, we adopt a Bayesian modeling approach, treating r; Gaussian random variable. Specifically, we assume:
r: N- o²),
where μ4 denotes the mean of the trajectory quality r; and O² quantifies the uncertainty. To update this prior based on observed evidence, consider a Bayesian inference framework. Let A; be an observed variable that serves as a noisy indicator of the true quality r,. We assume the observation model is Gaussian:
i ~ ((ri, oi).
where of captures the reliability of the observation process. Under this setup, the posterior distribution of r; given F; remains Gaussian, and the posterior mean can be derived analytically as:
o² o²
= f.
+ o²
(1: Pa)-
44)
This posterior mean balances the prior distribution and the observation, weighted inversely by their respective variances. Intuitively, more confident observations (small o3)) shift the posterior closer to Rs while a more confident prior (small o²) keeps the posterior closer to Pi-
Advantage Estimation For a set of samples {(n,, o,, r where n; is the injected noise level, of is the generated trajectory, and r is the trajectory semantic reward, which combines both accuracy and format reward. For the prior estimation of trajectory quality ri. we set the noisy reward as r = 1 n, and treat it as the prior estimation. The observation is the semantic reward rç. We apply normalization to make the prior and semantic rewards comparable across the group of G samples. Given a set of scalar values { the normalization operation defined as:
Norm(rs, (z)) 2; mcan (f (25)ş-1)
(5)
(z,((z1)
We apply this normalization to both the prior and semantic rewards as follows:
P Norm(r7. (17))1)) A (
(6)
For clarity, we assume both normalized signals are corrupted by a Gaussian, which serves as a core assumption in our Bayesian estimation framework. An analysis of this assumption and its empirical justification is provided in the Appendix. Accordingly, we model the latent reward r. as a random variable drawn from Gaussian prior centered at f? while the observed semantic reward f is treated as a noisy observation of rj:
T: N (i),
To this end, the key challenge is to determine the uncertainty of both signals, which is represented by the of and o3. We fix of to a constant a, as the semantic reward is computed using a text embedding model whose bias is relatively stable. To modulate the prior uncertainty, we define the prior variance o to be inversely related to the variability of semantic rewards within the group. The underlying intuition is that f and F. are positively correlated, both reflect the underlying trajectory quality, and n; is sampled from a uniform distribution. Therefore, when the group-wise variance of f is m the certainty of the prior distribution is low. Specifically:
small,
0-2 = a2
02
(8)
7+ (st((1r:)(
5

---

Table 1: Results on Co quality benchmark MME-CoT. We report results across three evaluation
dimensions of CoT quality to comprehensively characterize the performance of NoisyGRPO.
CoT quality
CoT Robustness
CoT Efficiency
F1
Precision
Recall
Ave.
Stability
Efficacy
Avg.
Rel. Rate
Rel. Quality
LLaVA-OV-7B
30.9
50,9
22.2
-3.4
-3.8
-3.0
91.5
83.0
100.0
LLaVA-OV-72B
36.3
57.3
26.6
-0.2
0.3
-0.6
95.4
90.8
100.0
MiniCPM- V-2..6
39.8
57.3
30.5
-3.5
-2.2
92.8
85.7
100.0
Intern -L2.5-8B-MPO
43.0
60.4
33.4
0.6
0.3
0.9
94.7
89.3
100.0
Qwen2-VL-72B
56.2
77.3
44.2
-2.1
-6.5
2.4
96.5
92.9
100.0
GPT-4o
64.0
85.4
51.2
2.1
-1.0
5.1
96.0
92.0
100.0
Kimi k1.5
64.2
92.0
49.3
1.4
2.9
0.0
82.2
92.2
72.2
Reasoning Models with Qwen2.5 VL
3B
GRPO
36.2
53.0
27.4
-7.0
-7.5
-6.5
51.1
76.3
26.0
NoisyGRPO
40.6
57.3
31.5
-2.4
-2.4
-2.4
55.4
71.9
38.8
Reasoning Models with Owen2.3-Ve
7B
GRPO
46.8
64,2
36.9
1.6
3.8
-0,6
66.4
87.2
45.7
NoisyGRPO 47.7
64.8
37.7
3,9
7.8
-0.1
59.7
81.8
37.5
where y is a scale hyperparameter that modulates the influence of semantic reward variability on prior uncertainty. This formulation implies that when the semantic reward varies widely (i.e., larger prior distribution considered more reliable and thus assigned a smaller variance. Given the prior and observation, apply Bayesian update obtain the posterior mean reward:
standard deviation),
f.)
Finally, to compute the trajectory- wise relative advantage within each group, we normalize the posterior reward:
A = Norm(fr, ff))g=1).
(10)
4 Experiments
4.1 Training Data
To enhance the general Chain-of-Thouue (CoT) reasoning ability of MILMs through reinforcement learning, we adopt the visual question answering (VQA) portion of the MM-RLHF (22 training set training data. This dataset spans diverse domains, including conversations, safety, multiple- choice questions, captions, and commonsense reasoning. MM-RLHF applies clustering and filtering techniques to curate high-quality visual instruction-following dataset. In total, it contains 13k VQA samples: 1.2k yes/no questions, 1.3k multiple-choice questions, and 10k open-ended VQA questions. Notably, most of the training data lack structured answers; therefore, to compute the accuracy reward. While this may introduce some model bias, our Bayesian advantage estimation helps mitigate its impact.
embedding model
4.2 Implementation Details
We implement the NoisyGRPO based on the VLM-RI 44) reinforcement learning training framework. For the policy model, we choose the state-of-the-ar MLLMs Qwen2.5-VL 3B and 7B. For hyperparameters, we follow the default settings provided by VLM-RI except for the following changes. The number of sampled rollouts In noise injection, the upper bound a for the (4(0, σr) is set to 1. The a and y are set to 0.1 and 0.01. The threshold for the accuracy reward set to 0.6. As there no validation set, we choose all the hyperparameters based on the results on MMStar (5] The training takes 6 hours for the 3B variant and hours for the on a single node with 8A 100 GPUs.
set to 4 due to computational resource considerations.
4.3 Evaluation Benchmarks
To comprehensively evaluate the CoT reasoning capabilities of NoisyGRPO, we select benchmarks from three perspectives: (1) CoT Quality Evaluation. We use the MME-CoT [16] benchmark, which assesses the quality, robustness, and efficiency of multi-modal CoT reasoning. The benchmark

---

Table 2: Performance Comparison General, Hallucination, and Real-World Benchmarks. The CP, FP, IR, LR, ST, MA, and AVG denote Coarse Perception, Fine-gra line-grained Perception, Instance Reasoning, Logical Reasoning, Science & Technology. Math, and the average performance, respec- tively. The symbols " and & indicate that higher or lower values are preferred. AMB. Gen. and AMB. Dis. represent the generative and discriminative components of the AMBER benchmark.
MMStar
AMB.
AMB.
MMERW
CP. FP. IR. LR. ST. MA. Avg.
C. 4
Cov. 1 Hal. 4 Cog. 4
FIT
Acc
LL 7B
58.8 24.0 38.8 24.0 13.6 22.8 30.3
7.8
51.0
36,4
4.2
74.7
Qwen VL- VL-Chat
59.6 32.0 50.8 29.2 22.0 31.6 37.5
5.5
49.4
23.6
1.9
CogVI.M-Chat
66.8 36.8 49.2 31.2 23.6
11.6 36.5
5.6
57.2
23.6
1.3
Internl M 2
70.8 48.8 65.2 56.4 42.0
49.2 55.4
LLaVA-Next 34B
66.4 52.0 62.4
46.0
32.4
53.6 52.1
GPT4V
76.6 51.4 66.6
55.8
42.6 49.8 57.1
4.6
67.1
30.7
2.6
Main Results
Qwen2.5-VL.3B
70.3 49.1 59.6 55,1 38.5 59.7 55.4
7.6
69.9
55.7
5.8
89.6
42.1
+ SFT
70.3 49.6 59.0 55.4 39.2 60.4 55.7
6.9
70.4
48.8
3.9
88.8
44.0
GRPO
68.0 43.3 60.4 57.0 40.0 58.0 54.5
6.7
68.5
44.6
4.2
89.2
40.8
NoisyGRPO
69.5 53.2 66.6 60.7 37.1 62.3
58.2
67.7
44.3
3.4
90.3
44.0
Qwen2.5-VL. 7B
73.3 60.1 72.2 62.0 44.6 67.3
63.2
4.8
63.6
27.5
1.6
87.4
43.6
SFT
71.8 58.8 70.2 60.8 45.2
67.4
62.4
5,4
64.9
32.4
2.1
87.5
43.5
GRPO
73.7 57.2 70.5 64.2
44.1 65.7
62.6
4,9
64.3
28.9
1.7
88.0
42.3
NoisyGRPO
72.8 63.7 71.7 66.9 48.3 71.4 65.8
63.2
23.9
1.2
88.2
44.6
covers six categories of questions guestions-innludinng math, OCR, logic, science, space-time, and general scene-providing holistic view of CoT performance. (2) General Capability Evaluation. We adopt the MMStar [S] benchmark, which is constructed by carefully selecting high-quality samples from existing datasets such A12D [17], SEED Bench [99). Science 28). MMBench (26).
Math Vista (27). and MMMU [59] By removing low-quality samples, MMStar provides cleaner and more comprehensive testbed across six reasoning domains. (3) Hallucination Evaluation. We
employ the AMBER (50) benchmark to evaluate both the generative and discriminative hallucination
behaviors of the model, offering a thorough analysis of output reliability. Additionally, we include MME-Reat Nee-RRellooo World-Lrr [63] to assess VQA performance in real-world scenarios.
4.4 Baselines
To fairly demonstrate the effectiveness of NoisyGRPO, we compare against strong baselines across all evaluation settings. For the CoT quality benchmark, we compare NoisyGRPO with the and also report results from wide GRPO 44 algorithm under different scales of Dwen2.5-VL
range of MLLMs [8
51 proprietary models like GPTT-40
32 from 7B to 72B models, as well as 49 for reference. For the general reasoning and hallucination benchmarks, in addition to comparisons between NoisyGRPO and GRPO, we include SFT (Supervised Fine- (ine-Tunngg) baselines assess the quality and impact of training data, We also report the performance of Mate-of-thheart MLLMs for broader comparison.
57
6
33
49
23
3
52
53
24
33 and Kimi K1.5
4.5 Performance Analysis
Benchmark Results For the CoT quality evaluation, we report the results of NoisyGRPO and baselines on the MME-CoT [16] benchmark in Table We observe that NoisyGRPO consistently outperforms GRPO on most CoT quality metrics, with particularly significant improvements for the 3B model. Notably, NoisyGRPO also enhances CoT robustness for example, NoisyGRPO-7B achieves higher robustness scores than proprictary models like GPT-4o [33] and Kimi K1.5 [49]. However, observe that vanilla GRPO surpasses NoisyGRPO in CoT efficiency when using Qwen2.5-VL 7B. We attribute this to NoisyGRPO's design, which prioritizes grounded reasoning over reasoning efficiency.
For the general reasoning and hallucination benchmarks, Table presents comparison between NoisyGRPO and baseline methods. The results demonstrate that NoisyGRPO effectively enhances MLLM performance across multiple domains by strengthening CoT reasoning, achieving notable gains such as +2.8 and +2.6 in average MMStar (5) scores over the base model. However, we observe a performance drop in the Coarse Perception category for both model variants. We attribute this to the injected noise being insufficient to meaningfully perturb the visual cues necessary for coarse-grained
7

---

perception tasks, leading to suboptimal performance for this specific task. Furthermore, we observe that NoisyGRPO performs worse on the Cov, metric of the AMBER [50] benchmark, suggesting that the generated captions do not fully cover all elements of the image. We attribute this to NoisyGRPO's emphasis on grounded responses, which tends to produce more concise answers and, consequently, reduces overall coverage.
We also find that supervised fine-tuning (SFT) on the current 13k dataset fails to bring consistent performance gains, highlighting that the improvements stem from reinforcement learning rather than high-quality data alone. Furthermore, the performance improvements on MME-Real Rorld-LLde 663 further demonstrate the potential of NoisyGRPO for real-world applications..
Performance Over Training Iterations In addition to reporting the best model performance across various benchmarks, we further analyze the training dynamics of NoisyGRPO to provide a more comprehensive understanding of its behavior. Specifically, we examine the relationship between training iterations and performance on the MMStar benchmark, which evaluates MLLMs we compare NoisyGRPO with the
across six distinct reasoning dimensions. As shown in Figure 3 baseline GRPO in terms of performance evolution, as well as the progression of the importance
in Eq. 9 throughout training. We also report the average completion length produced by each method. From the results, we draw the following insights: (1) The importance weight decreases steadily throughout training, indicating that NoisyGRPO initially relies more on prior estimation driven by noise level, but gradually shifts to relying on correctness- basedd reward signals as training progresses. (2) NoisyGRPO consistently produces shorter completions than GRPO while achieving better performance, suggesting that its supervision mechanism over the CoT process encourages more concise and accurate reasoning steps. (3) Compared to GRPO, NoisyGRPP yields greater improvements on tasks requiring fine-grained visual understanding, such as Fine- grained Perception and Instance Reasoning, but performs slightly worse on knowledge-intensing tasks like Scene & Technology. This indicates that NoisyGRPO primarily enhances vision-language alignment in CoT reasoning, rather than boosting knowledge retrieval or external fact-based reasoning capabilities.
weight-defined +o
4.6 Ablation Study
Method Design To investigate the contribution of each component in our proposed NoisyGRPO
framework, we conduct an ablation study on the comprehensive MMStar [5] benchmark. As illustrated in Figure we compare the performance trajectories of four variants throughout training. The baseline GRPO [44 corresponds to the original policy optimization algorithm introduced in DeepSeek-RI (7]. The variant GRPO with Noise Injection incorporates our proposed noise-injected exploration policy on top of GRPO. Naive NoisyGRPO refers to simplified version of NoisyGRPO, where the dynamic weighting mechanism between the observation and prior is removed swreeddiipciilllly....... in Eq. [ we set 0² o2 thereby eliminating the adaptive fusion. From the results, we observe that naively injecting noise to perturb the visual inputs degrades policy learning performance, indicating that a well- designed mechanism is required mitigate the adverse effects of noise. Furthermore, the NoisyGRPO with dynamic prior-observation fusion consistently outperforms other variants, demonstrating the effectiveness of this design in enhancing the reasoning capability of MLLMs during bootstrapping.
Hyperparameter Sensitivity To assess the sensitivity of NoisyGRPO its hyperparameters, we conduct experiments analyzing the impact of two key parameters in the Bayesian Advantage Estimation module. As defined in Eq. g controls the confidence assigned the observation, while 7, defined in Eq. 9 serves as a scale hyperparameter that modulates how strongly the variability of semantic rewards influences the prior uncertainty. As shown in Figure 4 NoisyGRPO exhibits relatively low sensitivity to compared to y, indicating that the bias introduced by the embedding- based reward function smaller than that resulting from inaccurate prior estimation due to noise injection. We further extend the analysis to the reward- reward-temperaie temperattuur parameter т. As shown in Table NoisyGRPO maintains stable performance across a broad range of values for both hyperparameters on MMStar, AMBER, and MMERealworld benchmarks, consistently outperforming the vanilla GRPO baseline. These results confirm the robustness of NoisyGRPO and demonstrate that reasonable hyperparameter choices can be made without extensive tuning.
8

---

Iepattnce Hegh!
Carp Langgs
E.5
SonySano
6m0
Training fteration
0.34
0.52
Rocuracy
3.8
250
130
02 40
Traiinng
Traning
Training
Logital Ressoning
Schecc Technology
Acsinry
0.20
-30
500 100
Figure Performance over iteration and training statistics. We report the evaluation results of NoisyGRPO-31 on the MMStar benchmark to demonstrate the comprehensive capabilities of the MLLM. For both Importance Weight and Completion Length, the shaded regions represent the variance across samples.
Heperpanantter
Acoreay
Training Heration
Tranning Barstio
Tratning Barstion
Traaning Henston
(d)
Figure 4: Performance of Ablation Study. We report the average performance the MMStar benchmark across the following ablation settings: (a) Ablation of core design choices in NoisyGRPO; (b) Sensitivity to the hyperparameter controlling observation confidence; (c) Sensitivity to the hy- perparameter 7 modulating prior variance adaptation; (d) Comparison different noise distributions used in the noise-injected exploration policy.
Noise Distribution in Noise Injection To investigate the effect of different noise distributions on policy learning in NoisyGRPO, we experiment with three types of injected noise during training. Specifically, we compare: (1) Uniform noise, which is the default choice in our final method: (2) Gaussian noise sampled from normal distribution with mean 0 and variance 0.1; and (3) Fixed noise with a constant value of 0.5. As shown in Figure we observe that noise sampled from uniform distribution consistently leads to more stable and improved MI.LM performance. In contrast, both Gaussian and fixed noise result in unstable training dynamics. We attribute this to the fact that uniform noise provides broader and more balanced exploration space, while proposed Bayesian

---

Table Sensitivity analysis of hyperparameters 7, Results under varying hyperparameter settings are reported on MMStar, AMBER, and MMERealworld benchmarks. The best performance for each metric highlighted bold. The configuration highlighted in blue corresponds the setting used in the main results.
MMStar
AMBER
MMERW
CP.
FP.
IR. 60.4 57.0 40.0 58.0 54.5 6.7
LR. ST. MA. Avg. Cs Cov. t Hal. J Cog.
F1 f
Acc
Vanilla GRPO 68.0 43.3
68.5
44.6
4.2
89.2
40.8
NoisyGRPO with different T
7 0.5
66.1 50.9 62.5 56.2 37.5 65.9 56.5
6.7
64.3
37.5
2.5
89.5
43.0
r = 0.6
69.5 53.2 66.6
60.7
37.1
62.3
58.2
6.6
67.7
44.3
3.4
90.3
44.0
== 0.7
69.1 52.3 63.4 58.2
39.3
56.2 56.4 7.4
65.8
42.4
3.0
90.0
44.7
Table 4: Comparison of computational efficiency between GRPO and NoisyGRPO. The reported
numbers for both methods are tested on a single node with A100 GPUs.
Method Wall-Clock Time GPU-Hours Peak GPU Memory
Owen2.5-VL--3.
GRPO
6h12min
49.6
57 GB
NoisyGRPO
6h40min
53.3
53 GB
Qwen2.5-VL-7BB
GRPO
7h58min
63.7
75 GB
NoisyGRPO
7h52min
62.9
77 GB
Advantage Estimation effectively mitigates the adverse impact of high-variance noise during policy optimization.
Training Efficiency Comparison To evaluate the practical scalability and efficiency of Noisy- GRPO, we measure its training cost and resource usage in comparison with GRPO 44 Specifically, we report the wall-clock time, GPU-hours, and peak memory consumption for both methods under identical experimental settings. Here, NoisyGRPO and GRPO sample the same number of rollouts. As summarized in Table 4 the results show that NoisyGRPO maintains comparable efficiency to GRPO despite its additional sampling step, demonstrating its suitability for real-world deployment.
5 Limitation
While our proposed NoisyGRPP effectively enhances the Chain-of-Thought (CoT) reasoning ability of MLLMs, it also comes with certain limitations. Specifically, our method relies on the assumption that injecting Gaussian noise into the visual input perturbs the model's generation and that the degree of noise correlates with the correctness of the generated answer. This assumption may not hold in tasks such as coarse- coarse-grained perception, where detailed visual information is less critical. In such cases, the injected noise may fail to meaningfully affect the model's 's outputs, and the noise magnitude, used as prior in Bayesian advantage estimation, introduce significant bias. This can, in turn, negatively impact the learning of the policy in NoisyGRPO.
6 Conclusion
In this work, we present NoisyGRPO, a multimodal reinforcement learning framework designed to enhance the Mhaii-off-hhoughe reasoning capabilities of MILMs. Without incurring additional computational cost, we introduce a noise injection strategy to improve policy exploration and address the limited diversity of generations in vanilla GRPO. To counteract the potential negative impact of injected noise, we propose a principled Bayesian Advantage Estimation method, where the injected noise level serves a prior and the trajectory reward serves as the likelihood. This Bayesian modeling fuses both sources of information to compute a robust posterior estimate of trajectory advantage, enabling more accurate and robust policy optimization under visual uncertainty. Experimental results demonstrate the effectiveness of NoisyGRPO across diverse benchmarks, particularly under challenging RL settings with small-scale MLLMs.
10

---

7 Acknowledge
This work was supported by NSFC 62350610269, Shanghai Frontiers Science Center of Human- centered Artificial Intelligence, and MoE Key Lab of Intelligent Perception and Human- Collaboration (ShanghaiTech University). This work was also supported by HPC platform of ShanghaiTeech University.
References
[1] OpenAl Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, and Gpt-4 technical report. 2023.
[2] Jean- Sann-Bapiiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 555:23116-237553 2022.
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile languagl model for understanding, localization, text reading. and beyond. arXiv preprint arXiv:2308. 11000./29900 /2966, 2023.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xic, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint 2502 13923, 2025.
[5] Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? ArXiv, abs/2403.200 20330, 2024.
[6] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu. Lixin Gu, Xuehui Wang. Qingyun Li. Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang. Wang. Conghui He. Botian Shi, Xingcheng Zhang, Han Lv. Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Hui Deng, Jiaye Ge, Kaiming Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahu Lin, Yunfeng Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. ArXiv, abs/2412.05271, 2024.
[7] DeepSeck-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. 2025.
[8] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng. Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement. ArXiv, abs/2503. 17352, 2025.
[9] Chengjian Feng. Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open- vecabulayy detection using uncurated images. In European Conference on Computer Vision, 2022.
[10] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wengi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ArXiv, abs/2402.05933, 2024.
[11] Yuying Ge Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- granularrtt comprehension and generation. ArXiv, abs/2404. 14396, 2024.
[12] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming Hc, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. arXiv preprint arXiv:2209. 14169, 2022.
[13] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaoshen Cao, Zheyu Ye, Fei Zhao, 7he Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. ArXiv, abs/2503.06749, 2025.
[14] OpenAl Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, and et al. Gpt-4o system card, ArXiv, abs/2410.21111 2024.
11

---

[15] Maximilian Igl. Kamil Ciosek, Yingzhen Li. Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann, Generalization in reinforcement learning with selective noise injection and information bottleneck. In Neural Information Processing Systems, 2019.
[16] Dongzhi Jiang. Renrui Zhang. Ziyu Guo. Yanwei Li. Yu Qi. Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang. Chaoyou Fu, Peng Gao, and Hongsheng Li. Mme-cot: Benchmarking thain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. ArXiv, abs/2502.00621, 2025.
[17] Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. ArXiv, abss/660.00339,, 2016.
[18] Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang. Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv, abs/2408.03340 2024.
[19] Bohao Li. Rui Wang. Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. benchbench: Benchmarking multimodal Ilms with generative comprehension. ArXiv, abs/2307. 2023.
[20] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2: Bootstrapping language -magge pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[21] Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal Ilms with mixture of experts. IEEE Transactions on Puttern Analysis and Machine Intelligence, 47:3424-34399 2024.
[22] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Jiao Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. ArXiv, 0ts/2311.07575 2023.
[23] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. ArXiv, abs/23 10.03744, 2023.
[24] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu. and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023.
[26] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all-around player? arXiv preprint arXiv:2307.06281 2023.
[27] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yuc Li, Hannanch Hajishirzi. Hao Cheng, Kai-We Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. ArXiv, ahs/2310.00225, 2023.
[28] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022.
[29] Yiyang Ma. Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, Liang Zhao, Yisong Wang, Jiaying Liu, and Chong Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. ArXiv, abs/2411.07975. 2024.
[30] Fanqing Meng. Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. 2025.
[31] Sha Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with dision-language models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23507-23517, 2023.
[32] OpenAl. Vision openai api. https://ppata:ie http://ww..ppaaa...ooo/////////e
2023.
[33] OpenAL. Openai go system card, 2024. System Card for OpenAl GPT-4o.
12

---

[34] OpenAL. Openai ol system card, 2024. System Card for OpenAI ol.
[35] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 0rXiv2230.007193 2023.
[36] Yassine Ouali, Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos. Clip-dpo: lasion-language models as a source of preference for fixing hallucinations in Ivims. ArXiv, abs/2408. 10433, 2024.
[37] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[38] Yi Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. ArXiv, abs/2503.07555 2025.
[39] Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang. Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. ArXiv, abs/24033087330 2024.
[40] Longtian Qiu, Shan Ning, and Xuming He. Mining fine-grained image-text alignment for zero-shot captioning via text-only training. ArXiv, abs/2440.022344 2024.
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. ArXiv, abs/2305. 18290, 2023.
[42] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Bapttist Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser, loannis Antonogiou, Rohan Anil, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, abs/2403.0555 03,5300000,0 2024.
[43] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods Natural Language Processing, 2019.
[44] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Mun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300, 2024.
[45] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingehuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, 0tss/2002.03000 2024.
[46] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang. Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang. Ruochen Xu, and Tiancheng Zhao. VIm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint 250Xi::2004.0615. 2025.
[47] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang. See- Kiong Ng, Li Bing, and Roy Ka wei Lee. Math-llave Bootstrapping mathematical reasoning for multimodal large language models. In Conference on Empirical Methods in Natural Language Processing, 2024.
[48] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. ArXiv, abs/2405.09818,
2024.
[49] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, and et al. Kimi k1.5: Scaling reinforcement learning with 1lms. ArXiv, 1t..2001.12999 2025.
[50] Junyang Wang, Yuhang Wang. Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia. Haiyang Xu, Ming Yan. Ji Zhang, and Jitao Sang. An lim-free multi--iimesional benchmark for milms hallucination evaluation. ArXir, 0ass/2311.0739)) 2023.
[51] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Kegin Chen. Xuejing Liu. Jialin Wang. Wenbin Ge Yang Fan, Kai Dang, Mengfei Du Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024.
[52] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. ArXiv, abs/2311.03079, 2023.
13

---

[53] Xiao wen Dong, Pan Zhang. Yuhang Zang, Yuhang Cao. Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao. Wenwei Zhang, Yining Li. Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Mastering free-form text-image composition and comprehension in vision-language large model. ArXiv, abs/2401. 16420, 2024.
Mitemlm-xcomporn
[54] Chengyue Wu, Xiaokang Chen. Zhiyu Wu, Yiyang Ma. Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation. ArXiv, abs/2410. 13848, 2024.
[55] Zhiyu Wu, Xiankang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Viyang Ma, Chengyue Wu, Bing-Li Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yu mei You, Kaihong Dong, Xingkai Yu. Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: Mfxture-of-experss vision-language models for advanced multimodal understanding, ArXiv, (bss22412.10302 2024.
[56] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang. Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-a: One single transformer to unify multimodal understanding and generation. ArXiv, abs/2408.12522 2024.
[57] Yuan Yao. Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qi-An Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Zhou, Jie Cai, Xu Han, Guoyang Zeng. Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level mllm on your phone. ArXiv, ahs/2408.01 2024.
[58] Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong. Chi Zhang, Mofan Zhang. Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan Dai, Yuxuan Song. Xiang Wei, Haodong Zhou, Jingjing Liu, Wei Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yong-Xu Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at scale. ArXiv, abs/2503. 14476, 2025.
[59] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Su, and Wenhu Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9556 956769567.200 2023.
[60] Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, and Hongsheng Li. Mavis: Mathematical visual instruction tuning. ArXiv, htl/220000877355 2024.
[61] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi. Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang, Rong Jin, and Tien-Ping Tan. Mm-rlhf: The next step forward in multimodal I1 alignment. ArXiv, abs/2502. 10391, 2025.
[62] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li. Jianshu Zeng, Wulin Xie, Yang Shi. Huanyu Zhang. Junkang Wu, Xue Wang. Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao. Di Zhang, Liang Wang, Rong Jin, and Ticn-Ping Tan. Mm-rlhf: The next step forward in multimodal IIm alignment. ArXiv, abs/2502. 10391, 2025.
[63] Yi-Fan Zhang, Huanyu Zhang. Haochen Tian, Chaoyou Fu, Shuangqing Zhang. Jun Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang. Rong Jin, and Tien-Ping Tan. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? ArXiv, abs/2408. 13257, 2024.
[64] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vasion-language models. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[65] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. ArXiv, abs/2404.13000 2024.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
14

---

Answer: [Yes]
Justification: The the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are disscussed
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:(Yoe Yes]
Justification:
THe prrof is provided
Guidelines:
The answer NA means that the paper does not include theoretical results.
15

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The information is provided for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code and data will be released.
Guidelines:
16

---

reproducible, they The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
httpii//ggidde//odeuuee oos.ouooiissoioooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips http:/cccppblclccccccceeccc guides/ Coddeeubmsssse ssinnoiiccyyyyii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity. the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: It's reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
17

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: No ethics issue.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The impacts are discussed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The safeguards are described.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The data source paper are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets provided.
19

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
No crowdsourcing in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing.
Guidelines:
20

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https://neurip:: /ttp://pp.ccc//offfeeeee//////// for what should or should not be described.
21

---

A Preliminary
This section formalizes the reinforcement learning (RL) framework for post-training optimization of multimodal large language models (MLLMs). We introduce the mathematical foundation of Group Relative Policy Optimization (GRPO) 44 state-of-the-art RL approach for language model alignment. Furthermore, we provide an overveiw of Bayesian Estimation and offer detailed derivations of the key equations introduced in the paper, starting from their original definitions.
A.1
Group Relative Policy Optimization (GRPO)
GRPO
is a reinforcement learning (RI.) algorithm specifically designed to enhance the reasoning capabilities of Large Language Models (LLMs).. Unlike traditional approaches, GRPO eliminates the need for value function and instead estimates advantages in a group-relative manner. Given a question-answer pair (q. a), the exploration policy Teas samples group of G individual responses for The advantage of the i-th response is computed by normalizing the group-level rewards (ri)er
44
A, = GN(r,, fr,)g-)) ri mean mean((ry) (r))ك²)
(11)
std({r,)
where define GN as the function for group normalization. GRPO adopts clipped surrogate objective similar to PPO, with an additional KL divergence penalty to regularize the policy update:
(0)
(0) D.g.a)=
1
To(o; 9) A., clip( To(o: 9) 21 Ter. (os q)
1 c) A.) A.) 3DD
(12)
min
G
Tew (o: 9)
Here, Ty and Nal are the current and previous policies respectively, is the PPO clipping parameter, 3 is the KL penalty coefficient, Tief is reference policy, and DRi. denotes KI. divergence.
It is also worth noting that GRPO incorporates temperature- -base sampling to promote exploration. The presence of both high- and low-quality responses within group enables the group-relative advantage estimator serve as a meaningful training signal. However, as noted in DAPO (58), GRPO suffers from a gradient vanishing issue when prompts yield identical reward values (e.g.. all correct or all incorrect responses). the main paper, introduce NoisyGRPO, designed to address this limitation and improve the robustness of GRPO.
of enhancements
A.2 Bayesian Estimation
Bayesian estimation is fundamental mechanism for probabilistic inference that refines prior belief in light of new evidence. Given a prior distribution p(0) over a latent variable 0, and likelihood function p(y ()) representing the observed data y. the posterior distribution p(0 ( y) is computed using Bayes' rule:
p(0 v) p(v 0)p(0)
(13)
p(v)
Here, p(y) is the marginal likelihood (or evidence), ensuring that the posterior is properly normalized.
Under the assumption that both the prior and the likelihood are Gaussian, and the latent variable 0 is scalar:
((0) N0;; Po, oz), plu 10) N(:: o3),
(14)
the posterior p(0 v) is also Gaussian. Its variance ofos and mean Poost are given by:
2
0
1
2
1
Mpost
140 6
0
y
o²2
post
(15)
22

---

Mean Correctriics Rewwrrd Trend
Dismiiuution Correctness Scores Oifferent
Correctree croectrees Score Reward
RC Rioodd 34-
Danedy
0.30
67
Correctnnes Score
Noksy Reward
Figure 5: Correlation of correctness and noise level.
Mean Normlizzd Correctness Score Trend
Normalized Conrectneess Score
Daj button Comeesriss Scores Offfeent Kokey Reswess
carara
Lokg Eomuad
02
Doya0
6.0
No momi 2nd Conrecor Score
Volay
Figure 6: Correlation of group normalized correctness and noise level.
Alternatively, the posterior mean can be rewritten in the following equivalent form highlight its interpolation behavior:
(Mo - v) of of of
Hpust = +
(16)
This form illustrates how the uncertainty and information from both the prior and the observation are combined with weighted contributions. The variances of and o play a key role in determining the influence of the prior and the observed data, representing the uncertainty in the prior estimate and the observation, respectively. While this interpolated form is convenient for interpretation, the expression in Eq. (4) is commonly used in Bayesian analysis, as it explicitly shows the posterior as preiision-weighted average of the prior and observed values.
These insights into Bayesian updating inform our design of Bayesian advantage estimation in policy optimization, allowing us to better incorporate uncertainty from both the noise prior and the trajectory reward, described in the main paper.
B Distribution Analysis of the Noise-level and Trajectory Reward
B.1 Correlation between Prior and Observation
To demonstrate the rationality of our method design, we begin by analyzing the correlation between the noise level prior and the correctness reward(observation during training. In Figure we visualize the raw values of the noisy reward r E [0, 1] and the corresponding correctness scores e [o, 1]. Specifically, present both the distributions of their means and the overall numerical distributions. From these visualizations, we observe that when the noisy reward is below 0.7 (corresponding to cases where more than 30% of the steps in the image trajectory corrupted with noise), the expected positive correlation between higher noisy reward and higher correctness score does not hold based on the raw values alone.
However, as NoisyGRPO applies group normalization from GRPO during advantage computation, we further provide the score distributions after group normalization. As illustrated in Figure 6 the distribution of the mean value of the correctness score after normalization exhibits a clear and consistent trend: the higher the noisy reward, the higher the correctness score. This shift indicates
23

---

Deta Distribution
fnccertage desn Di Dnfi 59.45 $5.75 PM
Chiteres Jals
Dorstyy
Norryiiers heay Correcresst
Figure 7: Histogram and Q-Q plot of residual of observation and prior. The Gaussian in red is the Gaussian distribution with identical mean and variance to the residual.
Distribution
Li uddae Unti
E:5e PM
Dansty
Ordered
Theoretical Quanties
Figure 8: Histogram and Q-Q plot of residual of posterior and observation. The Gaussian in red is the Gaussian distribution with identical mean and variance to the residual.
that the inherent differences in sample characteristics and difficulty levels cause the effect of noise injection vary significantly across samples. Consequently, group normalization is essential to approach for stabilizing training.
B.2 Verification of Gaussian Assumption
Our method is built upon fundamental assumption regarding the underlying distribution of reward signals. Specifically, we assume that the prior estimation of quality follows a Gaussian distribution, i.e., r; (r
((r?",o)) o.),, where r the estimated mean quality and a is the prior variance. Meanwhile, the observed semantic reward is modeled as a Gaussian observation of the true latent reward: r? ~N(r,, o?).
However, since the true latent reward r; is not accessible, we cannot directly validate this assumption. To address this, we employ indirect analysis and statistical testing evaluate the plausibility of the Gaussian assumption underlying our method design. To this end, we construct two residual terms based on available quantities:
(Observation vs. Prior)
f P (Posterior vs. Observation)
Here, (1) captures the deviation between the semantic observation and the prior estimation before fusion, while c! reflects the adjustment made by the posterior with respect to the observation. both observation and prior are Gaussian assumed, and the posterior is computed using standard Bayesian fusion, then both residuals should approximately follow zero-mean Gaussian distributions with variances that analytically tractable. In the following, empirically examine the distribution of these residuals to test the validity of our assumption.
Residual of Observation and Prior We first analyze the residual between the observation and prior, defined as the delta between the correctness score and the noisy reward. This quantity reflects how much the observed semantic signal deviates from the prior belief before any posterior fusion.
24

---

Coreeraan mat 2.3
carbo Tha Thainige prodies preeter noul what raclse arould
S:00
3-:0
sauie bran a Bammon con virm auic viluutes
Tw secore liiatd playe uucern P. cie
400 dhen plcues
Guerteer
Trunday shanndan whic ditl indkra
Vobe Levil 30%
Nole Love:
recoect
Comoen
Covetnent
dlok the
<ot eodecarti Thunday tadkor hawe pooer Wedwedyy playdd occon vitaro
p. avd looor
2388 swa Thunsda suertio whick uctarias licuce secric bised
fand sodine
Knower with optien' letter
deoand Irut.
Figure 9: Illustration "A "Answer Correctness is Partial Observation"
To assess the distributional characteristics of this residual, we visualize in Figure 7 Jusing both histogram and a Quantile-Quantile (Q-Q) plot against a standard Gaussian distribution. Additionally, we conduct formal normality tests to statistically evaluate the Gaussianity of the residual. The results are as follows:
Shapiro-Win Wilk Test: W 0.983, p-value 7.416x10-22 The Shapiro-Win Wilk test evaluates whether sample comes from a normally distributed population. A W value close to indicates that the data close to normal; the low p-value here, due to the large sample size, suggests statistical significance but does not necessarily imply practically meaningful deviation from normality.
Kolmogorov-Smirnov Test: D 0.110, p-value 2.432 10-52, The Kolmogorov- Smirnov (K-S) test compares the empirical distribution of the sample to a reference Gaussian distribution. The statistic D measures the maximum distance between the two cumulative distribution functions. Again, while the small p-value indicates statistical rejection of normality, the K-S test known to be sensitive in large samples, and the shape shown in the Q-Q plot remains approximately Gaussian.
As can be observed from both the visualizations and the statistical test results, the residual approxi- mately follows a Gaussian distribution. Although the extremely low p-values may suggest deviations from strict normality due to the large sample size, the overall shape and symmetry of the distribution, as shown in the Q-Q plot support the plausibility of the Gaussian assumption for this residual term.
Residual of Posterior and Observation In this section, we analyze the distribution of the residual between the posterior and the observation, which corresponds to the difference between the posterior advantage and the correctness score. Similar the previous analysis, we present both histogram and Quantile-Quantile (Q-Q) plot in Figure to visualize the shape of the residual distribution.
Shapiro- Wilk Test: W 0.932, p-value 1.694 x 10-42
The Shapiro-Wiis Wilk test assesses the null hypothesis that the sample is drawn from a normal distribution. Although the p-value very small (due to the large sample size), the W value remains reasonably high, suggesting the distribution retains approximate Gaussian characteristics.
Kolmogorov-Smirno Test: D = 0.190, p-value 3.529 x 10
The Bolmogorov-Smiine Smirnov test measures the maximal deviation between the empirical distribution and reference Gaussian. Despite the low paluee (again, expected in large-scale settings), the moderate value supports the plausibility a Gaussian assumption.
Taken together with the results from the prsor-observation residual analysis, these findings provide empirical support for the validity our Gaussian assumption in modeling both the prior and the observation noise.
C Visualization
Answer Correctness Partial Observation One of our key motivations stems from the fact that the correctness of the final answer cannot fully reflect the overall rollout quality, especially in
25

---

1330
Naterto
ingeet wever Parls
V/ rovire
of
Polas Level 37%
Putte Lesd Ri
Nike We glt
hunly Wesggn Lu
Se Sespasse
enegrit vie res gios budl giow
Toucs Corry senven whet by corch the senecviiiit Mejjewerr
image
8008 14 Roadic B9/0
shows
might
Figure 10: Illustration of Training rollouts and trajectory reward.
Conmetrane 8.0
Coreitteen 2.348
Pooderter All veriege 0u500
Pastader dalte 6.812
Broperet
Whet sermand neet ranthe rofol wheree
Aht wired bees
obvere doney vodiin
samound pesero..
lervo, The imaue senr dipiet tue proving lrves berigrond
rocty fom
Lit tuo tpw, oljec scrad tees
Buluue
Prsterter Advaniage OLENI
Paderie datre Mage 03644
You some to mepp and dooe
Ui--k/00 lyyn rigeti vitie inrouue F
th ei ribiid actven
Thopeleetcom can rorett srene vepen other trens Vihelt
poar and an byl herars
Leven, car partodie rrect orge
Eacd yideec sooms hve typo olyaro rekiect tottrt hight
Figure 11: Illustration of Training rollouts and trajectory reward.
complex multi-step reasoning settings like Chain-of-Thoughe (CoT). To qualitatively illustrate this phenomenon, we provide a concrete example in Figure 9 In this example, NoisyGRPO samples multiple rollouts under different noise levels. Interestingly, all sampled answers correct with respect to the final answer, yet the quality of the intermediate CoT' reasoning varies noticeably. This demonstrates that relying solely on answer correctness as a training or evaluation signal may overlook significant variations in reasoning quality. Hence, a more nuanced reward structure, as introduced in our method, is necessary capture the semantics of the full reasoning trajectory.
Weight in Bayesian Estimation In this section, we qualitatively illustrate how the training process dynamically balances the weights between the prior and observation in our Bayesian estimation framework. Specifically, the weight assigned to the observation is defined as
+
where of and of denote the variances the noise prior and semantic observation, respectively.
We first demonstrate an extreme case where all the answers in the sampled trajectories are incorrect. In this situation, vanilla GRPO fails to provide meaningful training signals, as the semantic correctness reward becomes uniformly zero. However, as shown in Figure TO, NoisyGRPO adaptively reduces the weight of the observation (correctness score) zero, allowing the noisy prior alone contribute to the training signal. This enables the policy model to continue learning from the noise level's implication CoT quality, even in the absence of correct final answers.
In contrast, when the correctness scores exhibit high variance variance-meaninee meaning some answers are correct and others not, illustrated in Figure II NoisyGRPO assigns balanced weight between the prior and the observation. This allows the model to leverage both the semantic correctness and the injected noise level to assess the overall trajectory quality more accurately.
These examples highlight the importance of dynamic weighting in our Bayesian estimation, enabling stable and informative policy updates across varying rollout qualities.
Rollouts Sampling Comparison with Vanilla GRPO In this section, we compare the policy exploration characteristics of vanilla GRPO and our proposed NoisyGRPO during rollout sampling.
26

---

NoisyGRPO
<derkohhhe surting the sowe ans waeting liack wete vilenks
<Tink ahod Thehe seage
Whe smetime
<hine-Who the peerr
ske Jugue Ree 1. to
surting veasiigg a blace wetsut while baanndng on <lanswers
and
dee carfooad, The wate onks
bo auring oichenlisde
cor
trosghngget
ndvis darialke
gatc sabann yook,
viial veaken lo
weoding
noe apens They
name we bnd
bete mg
the skers bock yobkliur
reinad per perdering part
halrnl
rea tike cinovaic
Question:
Novis Level: 0%
Noise Leveli 30%
Noise Leveli 60%
Neise Level: 90%
Convey nighlights of image facusing on the person, their outfit, and their activity picture.
GRPO
Toad stam con
lopgue right
Ground Truth:
man in black wetsult is surting white surfboard with yellow logo, riding a wave in the ocean.
hat peseratte ae not The
te peliii Th dn
tute letiig Cil
cobert
dats setenc The canoll the
Chaabel chel ge lart
nun stmosn the
un quek
Figure 12: Comparison of rollouts generation between vanilla GRPO and NoisyGRPO.
NelsydeFO
Nalaysanno
What the moin object in the foreground of the image? </hninio
thhink
es thent camerced trces, buthir </tiilo
grees
wanswer
Quastion:
Queetia::
Whot focau the
What Foaetoae
GRFO
Optees
ERPO
Aettic no anyy di
image treffic aigt red dindie white
through which in no entry sign "he bookgr sand
GrGen
Brown
truas trequct tee
cth
color the plent Treae wil the se at momh prave
shove
kea plante
Groasd
sign
Trutt:
foutiiss coundl coler
conmars
Figure 13: CoT Inference comparison between GRPO and Noisy.
Specifically, we examine the differences between two exploration strategies: one that solely relies on temperature sampling to generate multiple rollouts, and another that introduces noise injection the inputs to diversify the rollouts.
As illustrated in Figure 12] we observe that rollouts sampled by vanilla GRPO tend to exhibit high similarity to each other. This limited diversity constrains the scope of policy exploration and may lead to suboptimal learning. In contrast, NoisyGRPO, by injecting structured noise during rollout generation, encourages exploration more diverse reasoning paths and CoT structures. This diversity allows the model better capture the relationship between reasoning quality and task performance, ultimately leading to more robust policy improvement.
Inference Comparison with Vanilla GRPO In this section, we present a qualitative visualization comparing the CoT inference behavior of vanilla GRPO and our proposed NoisyGRPO. As shown in Figure [3] we observe that rollouts generated by NoisyGRPO tend to have shorter chains of thought while maintaining comparable answer accuracy. This suggests that NoisyGRPO achieves more efficient CoT reasoning by avoiding unnecessary elaboration.
Moreover, we find that in some cases, NoisyGRPO generates intermediate question within the COT, which serves as a refinement or decomposition of the original input question. This strategy allows the model to clarify its internal reasoning path and arrive at more accurate final answer. In contrast, vanilla GRPO often produces longer, redundant reasoning steps that do not necessarily contribute better accuracy.
These observations indicate that incorporating noisy priors during training encourages the policy model to develop more concise and targeted reasoning strategies during inference.
27

---

Comparison with Rule-Based Reward To validate our choice of using the text embedding model SBERT as the reward model, we compare with a popular rule-based reward approach. Specifically, we adopt the average F1 score of ROUGE-1, ROUGE-2, and ROUGE-I. as the reward. The results are summarized T As shown, SBERT consistently outperforms the rule-based reward in our experimental setting, likely due to its ability to capture semantic similarity beyond exact token overlap, which especially important for evaluating open-ended answers where lexical variance is common.
Table Comparison SBERT-bases reward vs. rule-based ROUGE reward on MMStar, AMBER, and MMERealworld metrics. Method
MMStar
AMBER
MMERW
CP. FP.
IR.
LR. ST. MA. Avg.
Cs
Cov.
Hal.
Cog.
1 F1 f
Acc
Rouge reward 64,9 49.5 64.8 51.3 38.0
52.6
53.5
8.0
66.4
46.5
3,4
89.3
43.6
SBERT reward 69.5 53.2 66.6
60.7
37.1
62.3
58.2
6.6
67.7
44.3
3.4
90.3
44.0
D Scalability Analysis
To evaluate the scalability of NoisyGRPO with respect to training data size, we conduct experiments using subsampled datasets containing 3k and 6k examples, compared the original 13k dataset. Specifically, we randomly sample 3k and 6k subsets from the full training corpus and train both GRPO and NoisyGRPO under identical training steps ensure fair comparison.
As shown Table 6 NoisyGRPO maintains consistent gains over the GRPO baseline across all dataset sizes, demonstrating its robustness and scalability when exposed larger data volumes. Furthermore, performance improvements are observed as the amount of training data increases, confirming that NoisyGRPO effectively leverages additional data enhance policy learning. These results highlight that NoisyGRPO is scalable and data-efficient optimization framework, capable of leveraging additional training samples further enhance multimodal reasoning performance without loss of stability.
28

---

Table 6: Scalability analysis of NoisyGRPO. Performance comparison between GRPO and Noisy- GRPO trained with subsampled datasets (3k and 6k samples). NoisyGRPO consistently outperforms the baseline under all data scales, and its performance improves with larger datasets. Best results in each column are highlighted.
Method Size
MMStar
AMBER
MMERW
CP
FP.
IR.
LR.
ST.
MA.
Avg.
Cs J.
Cov.
Hal.
Cog.
J.
F1
Acc
GRPO
66.4
44.6 59.7 53.8 35.1 56.7
52.7
7.1
66.1
46.9
3.8
88.9
38,9
NoisyGRPO 3k
66.7
51.8
61.2 56.4 35.3
62.3
55.6
6.7
63.5
36.8
2.2
89.9
44.5
GRPO 6k
67.2
45.6
58.7
53.2
40.2
54.1
53.2
7.0
67.3
48.8
4.7
88.3
38.1
NoisyGRPO 6k
69.1
52.2
64.3
54.7
39.7
59.5
56.6
6.4
65.4
40.2
3.1
90.1
44.1
29


Reviews:
[{"review_id": "8Hz5Mm2aqV", "reviewer": "Reviewer_nP6a", "summary": "The paper proposes NoisyGRPO, an RL training framework based on the classic GRPO method. Specifically, the method introduces (1) Noise-Injected Exploration Policy, which adds Gaussian noise to the images to obtain diverse rollouts, and (2) Bayesian Advantage Estimation, which takes the injected noise level as a prior, and the observed trajectory reward as the likelihood, to serve as advantage estimation.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The overall motivation is reasonable: when training RL, the rollouts tend to converge as the RL training proceeds. By injecting noise into images explicitly, the rollouts' diversity is ensured. The novelty is sound.\n2. The reported model results seem to perform well across different benchmarks and different model sizes. \n3. The paper is quite easy to read.\n\n## Weaknesses\n\n1. For the Off-Policy Strategy & Lack of Justification (L.161-162), the authors apply noise injection *only* during rollout collection but use clean inputs for policy updates. This creates an off-policy strategy where inference and training conditions differ. The paper lacks a clear justification for why this approach works effectively. More ablation studies and analysis are needed to validate this design choice.\n2. The distinction between the two Bayesian advantage estimation strategies is unclear. One injects noise directly during advantage estimation, while the other injects noise into the image inputs during rollout collection. The paper fails to claim the fundamental difference or relative contribution of these two noise injection points.\n3. The comparison with the SOTA methods ([1-3]) is insufficient. Performance should be rigorously evaluated against these methods using mainstream multimodal mathematical reasoning benchmarks like MathVista [4], MathVision [5], MathVerse [6], and MMMU Pro [7]. (MMStar is noted as containing only a small subset and is therefore not comprehensive enough for this purpose).\n4. Inadequate Reward Function Analysis: The authors state they use SBERT to compute text similarity for open-ended answers instead of rule-based rewards (lines 143-146). However, they provide no ablation study demonstrating SBERT's superiority over traditional rule-based methods. Furthermore, no ablation analysis is presented for the threshold parameter `τ`.\n\n[1] MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning\n[2] SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement\n[3] NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation\n[4] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts\n[5] Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset\n[6] MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?\n[7] MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark", "was_revised": true, "final_justification": "My major concerns have been solved. I think this has made the paper acceptable.", "rebuttal": "**We appreciate the reviewer for the positive feedback and for recognizing the motivation, novelty, and clarity. \nWe appreciate your comments and will address your concerns point-by-point below.**\n\n---\n**W1:** *The Off-Policy Strategy & Lack of Justification*\n\n---\nWe sincerely thank the reviewer for this insightful and important question, which touches upon the core design motivation of our method. We address this concern through a three-step response: \n- We clarify how the policy update is conducted in our framework.\n- We justify our design by showing that it has already accounted for the mismatch between noisy rollouts and clean policy updates, in a way that aligns with the principle of importance sampling.\n- We present ablation studies comparing policy updates with and without noisy inputs, which confirm that using clean inputs for policy updates leads to more stable and effective learning.\n\n**Clarification of “using clean inputs for policy updates”** \nWe provide the objective function of NoisyGRPO in Eq. (1), the policy gradient is computed based on the following objective:\n\n$$\n\\nabla\\_{\\theta} J\\_{\\text{NoisyGRPO}}(\\theta) = \n\\mathbb{E}\\left[\n\\nabla\\_{\\theta} \\log \\pi\\_{\\theta}(o\\_i^n \\mid q, I) \\cdot w\\_i \\cdot \\tilde{A}\\_i^k\n\\right], \\quad\n\\text{where } w\\_i = \\frac{\\pi\\_{\\theta\\_{\\text{old}}}(o\\_i^n \\mid q, I)}{\\pi\\_{\\theta}(o\\_i^n \\mid q, I)}\n$$\n\nFor clarity, we omit auxiliary components such as the KL penalty and gradient clipping in the above expression, which are standard in practice but do not affect the core analysis here.\n\nHere, *“using clean inputs for policy updates”* specifically refers to evaluating both the **policy gradient term** $\\nabla\\_{\\theta} \\log \\pi\\_{\\theta}(o\\_i^n \\mid q, I)$ and the **importance weight** $w\\_i $ under the clean image $I$.\n\n**Design Justification**\nOur Bayesian Advantage Estimation accounts for the mismatch between perturbed rollouts and clean policy updates, **serving a similar role to importance sampling** in off-policy learning.\n\nSpecifically, our advantage $\\tilde{A}^k_i$ is computed by normalizing a weighted sum of two reward estimates as shown in Eq. (9). Here, $\\hat{r^n_i}$ is the group-normalized form of $r_i^n = 1-n_i$,  where $n_i$ denotes the noise level of the $i^{th}$ trajectory.\nAnd $\\hat{r^n_i}$ could **approximate the importance sampling under our framework**.\n\nTo aid understanding, we consider an extreme case for illustration, where we let $\\tilde{A}^k_i = r_i^n$. In this case, the policy gradient update becomes:\n$$\n\\nabla\\_{\\theta} J\\_{\\text{NoisyGRPO}}(\\theta) = \n\\mathbb{E}\\left[\n\\nabla\\_{\\theta} \\log \\pi\\_{\\theta}(o\\_i^n \\mid q, I) \\cdot w\\_i \\cdot r\\_i^n\n\\right]\n$$\n\nWe can reinterpret $r\\_i^n$ as an importance weight:\n\n$r\\_i^n  = \\frac {1 - n} {1} = \\frac {q(x)} {p(x)}$\n\nwhere $q(x)$ denotes the (clean) target distribution, and $p(x)$ denotes the noise-injected behavior distribution. This shows that **$r\\_i^n$  downweights noisy trajectories proportionally to their estimated noise level $n^{i}$**, effectively approximating an importance sampling correction between $q(x)$ and $p(x)$.\n\n\n**Ablation Experiment**\nAdditionally, we perform ablation experiments where the policy is updated using noise-injected images. As shown in the table below, this leads to a performance drop. This is because the evaluation scenario assumes clean input images, requiring the target policy to generate responses based on clean visual inputs. Therefore, **updating the policy with clean images during training ensures alignment with the evaluation setting**. These results validate our design choice.\n\n\n| Method                      | MMStar           |       |       |       |       |       |       | AMBER          |       |       |       |       | MMERealworld |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|                             | CP.  | FP.  | IR.  | LR.  | ST.  | MA.  | Avg.  | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | F1 ↑  | Acc          |\n| Update policy with noise image | 67.0  | 49.3  | 57.6  | 55.2  | 35.1  | 59.3  | 53.9  | 7.2   | 66.3  | 45.1  | 4.2   | 88.7  | 36.6         |\n| Update policy with clean image  | 69.5  | 53.2  | 66.6  | 60.7  | 37.1  | 62.3  | 58.2  | 6.6   | 67.7  | 44.3  | 3.4   | 90.3  | 44.0         |\n\n---\n**W2:** *The distinction between the two Bayesian advantage estimation strategies is unclear.*\n\n---\nApologies for the confusion. We would like to clarify that our method **does not** involve two separate Bayesian advantage estimation strategies. We only inject noise into the image inputs during the rollout collection phase. The Bayesian advantage estimation module does **not** involve any explicit noise injection.\n\nWe recognize two potential sources of confusion that may lead to misunderstanding our approach:\n\n- **Noise prior** in line 192: The noisy prior is treated as a prior estimation of the trajectory quality decided by the noise level during rollout generation. \n- **Gaussian noise** in line 186: In our Bayesian estimator, we model the semantic reward $\\hat{r}_s$ and noisy reward $\\hat{r}_n$ as being corrupted by Gaussian noise, which enables us to obtain a posterior estimation $\\hat{r}_i$ through Bayesian update.\n\nWe hope this could resolve the confusion. We will also revise the manuscript accordingly in the revised version.\n\n---\n**W3:** *Lack of comparison with the SOTA methods in multimodal mathematical reasoning benchmarks.*\n\n---\n\nWe thank the reviewer for raising this important point. Our work addresses a multimodal RL setting that is **orthogonal to** multimodal math reasoning, so extensive comparisons with multimodal math reasoning benchmarks are **beyond the scope of our work**.\n\nSpecifically, prior works [1–3] on multimodal math reasoning assume **verifiable and unbiased reward signals** (e.g., exact answer matching), whereas our work addresses a **more realistic scenario** where reinforcement learning operates under potentially noisy or biased rewards—a setting we refer to as **RL beyond verifiable rewards**. This setting is crucial for real-world tasks such as visual chat and image captioning, where ground-truth signals are often ambiguous or unavailable. **Our method is specifically designed for this setting**, incorporating **Bayesian advantage estimation** to effectively mitigate the impact of reward noise during policy optimization.\n\nFurthermore, our study aims to **systematically investigate** whether **NoisyGRPO** can address **two under-explored but crucial challenges** in preference optimization for multimodal reasoning: (1) insufficient exploration during policy learning and (2) lack of supervision over intermediate reasoning steps. These aspects are **rigorously evaluated through controlled experiments** (see Table 1 and Table 2), where we measure both alignment with human preferences and improvement in reasoning consistency.\n\nWhile we acknowledge that a full comparison on multimodal math benchmarks would further validate our findings, we leave comprehensive benchmarking as future work and appreciate the reviewer’s suggestion in this direction.\n\n---\n\n**W4:** *Inadequate Reward Function Analysis.*\n\n---\nThank you for highlighting the importance of reward design analysis.\n\n**Comparison with rule-based reward**\n\nTo validate our choice of using the text embedding model SBERT as the reward model, we compare it with a popular rule-based reward approach. Specifically, we adopt the average F1 score of ROUGE‑1, ROUGE‑2, and ROUGE‑L as the reward. The results are shown in the table below.\n\n| Method                     | MMStar           |       |       |       |       |       |       | AMBER          |       |       |       |       | MMERealworld |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|                           | CP.  | FP.  | IR.  | LR.  | ST.  | MA.  | Avg.  | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | F1 ↑  | Acc          |\n| Rouge reward  | 64.9 | 49.5 | 64.8 | 51.3 | 38.0 | 52.6 | 53.5  | 8.0   | 66.4   | 46.5   | 3.4   | 89.3  | 43.6         |\n| SBERT reward | 69.5 | 53.2 | 66.6 | 60.7 | 37.1 | 62.3 | 58.2  | 6.6   | 67.7   | 44.3   | 3.4   | 90.3  | 44.0         |\n\nAs shown in the table, **SBERT consistently outperforms the rule-based reward** in our experimental setting.\nThis improvement is likely due to SBERT’s ability to **capture semantic similarity beyond exact token overlap**, which is especially important for evaluating open-ended answers where lexical variance is common.\n\n**Hyperparameter $\\tau$ Analysis**\n\nWe first clarify that we set $\\tau$ based on a reasonable default, and we humbly argue that introducing new hyperparameters is inevitable when proposing a new framework. Furthermore, we conduct an ablation study to show the sensitivity of $\\tau$, and the results are presented in the table below.\n\n| Method | MMStar           |       |       |       |       |       |       | AMBER          |       |       |       |       | MMERealworld |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|        | CP.  | FP.  | IR.  | LR.  | ST.  | MA.  | Avg.  | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | F1 ↑  | Acc          |\n| vanilla GRPO | 68.0  | 43.3 |  60.4 |  57.0  | 40.0  | 58.0  | 54.5  | 6.7  | 68.5  | 44.6  | 4.2 |  89.2 |  40.8 | \n| $\\tau$=0.5  | 66.1 | 50.9 | 62.5 | 56.2 | 37.5 | 65.9 | 56.5  | 6.7   | 64.3   | 37.5   | 2.5   | 89.5  | 43.0         |\n| **$\\tau$=0.6**  | 69.5 | 53.2 | 66.6 | 60.7 | 37.1 | 62.3 | 58.2  | 6.6   | 67.7   | 44.3   | 3.4   | 90.3  | 44.0         |\n| $\\tau$=0.7  | 69.1 | 52.3 | 63.4 | 58.2 | 39.3 | 56.2 | 56.4  | 7.4   | 65.8   | 42.4   | 3.0   | 90.0  | 44.7         |\n\nThe results demonstrate that the method’s performance remains stable over a reasonable range of $\\tau$ values and all of the choices of $\\tau$ outperform the baseline. These indicate robustness to the hyperparameter choice.\n\n---\n**We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.**"}, {"review_id": "j8WQyLfr1f", "reviewer": "Reviewer_7V19", "summary": "The paper introduces NoisyGRPO, a reinforcement learning (RL) framework aimed at improving the chain-of-thought (CoT) reasoning in multimodal large language models (MLLMs). The method involves injecting controllable Gaussian noise into visual inputs to encourage exploration during RL, combined with a Bayesian advantage estimation that fuses this noise (as a prior) and observed rewards (as a likelihood) for more robust policy optimization. Through extensive experiments on established CoT quality, general ability, and hallucination benchmarks, the authors show that NoisyGRPO outperforms vanilla GRPO and other leading baselines, particularly enhancing the robustness and generalization of small-scale MLLMs like Qwen2.5-VL 3B.", "questions": "1. Methodological Differentiation: How does NoisyGRPO fundamentally distinguish itself from prior selective noise injection RL methods ([12]) and other recent exploration-boosting strategies in visual RL? What unique properties (empirical or theoretical) arise from using image-space noise + Bayesian estimation compared to similar approaches?\n2. Reward Signal Robustness: Given that SBERT-based rewards may be imperfect for open-ended VQA, have you evaluated reward reliability or robustness directly? Are there scenarios where NoisyGRPO fails due to reward misspecification or bias?\n3. Ablation on Scale and Efficiency: Can you provide concrete wall-clock time, GPU-hours, or memory usage for NoisyGRPO versus GRPO, especially on larger models, to support real-world deployment claims?\n4. Broader Applicability: Is NoisyGRPO effective for other modalities or tasks (e.g., text-only, audio-visual tasks), or for LLMs not relying on visual input? What about transfer to reinforcement learning settings outside vision-language?\n5. Error Analysis: For categories where NoisyGRPO underperforms (e.g., Coarse Perception, Coverage in AMBER), do you observe systematic failure modes? Can your Bayesian estimator be adapted or regularized to address these?", "limitations": "The authors provide an explicit limitations section addressing key assumptions and failure modes. Additional discussion on scalability and reward signal bias would further strengthen this.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Methodological Innovation: The work presents a clearly articulated extension to GRPO: noise-injected exploration at the input level, paired with a principled Bayesian advantage estimator. The approach is well-motivated by the identified limitations of policy exploration and trajectory evaluation in MLLMs RL.\n2. Solid Theoretical Foundation: The Bayesian estimator for advantage estimation is soundly motivated and derived (Section 3.3), where noise magnitude serves as a calibrated prior and reward as the observed likelihood, resulting in adaptive, statistically justified updates.\n3. Comprehensive Experimental Evaluation: The experiments span multiple relevant benchmarks (e.g., MME-CoT for CoT quality, MMStar for general reasoning, and AMBER for hallucination) and consider both small- and large-scale MLLMs. Table 1 and Table 2 report detailed results, showing consistent gains especially for smaller models, validating the claims.\n4. Results Analysis and Ablation: The paper provides insight through ablations (Figure 4) on the effect of both core method components and hyperparameters, as well as comparisons of noise distributions. Performance trends over training (Figure 3) provide further empirical support for the learning dynamics described.\n5. Clear, Well-Organized Writing: Despite some dense sections, the paper is mostly well-structured, with equations, pipeline figures (see Figure 2), and comprehensive result tables and plots.\n6. Practical Relevance: The findings address a genuine problem—improving generalization and robustness of RL-driven reasoning in MLLMs, which is valuable for both researchers and practitioners dealing with multimodal and open-ended settings.\n\n---\n## Weaknesses\n1. Limited Novelty Relative to Recent Works: While the combination of noise injection and Bayesian estimation is methodologically appropriate, the paper does not demonstrate that the specific form of injected noise (Gaussian/diffusion in visual input) and the exact Bayesian estimator extend far beyond prior ideas of selective noise in RL [12] and group-based RL advantage estimation. The differentiation from similar recent approaches (especially those cited in [12], DAPO [51], and VLM-R1 [39]) could be clarified further.\n2. Empirical Gains Are Sometimes Marginal: Though the main method outperforms baselines, in Table 2 the MMStar overall gains (“+2.8 and +2.6”) over the base model are present but not transformative. In some subcategories (e.g., Coarse Perception), NoisyGRPO underperforms GRPO, calling into question broad generality.\n3. Complexity of the Bayesian Estimator: The approach introduces several new hyperparameters (e.g., $\\alpha$, $\\gamma$ in the Bayesian update; normalization steps) whose role becomes somewhat intricate (see Figure 4), requiring careful tuning. While the ablation explores sensitivity, practitioners may face difficulty generalizing the method without significant validation infrastructure.\n4. Reliance on Embedding-Based Rewards: The SBERT-based reward may bias the optimization, and although the authors claim the estimator mitigates this, the evidence for such mitigation is limited to indirect observations. Section 3.1 concedes some rewards for open-ended tasks are imperfect, but analysis is not fully conclusive.\n5. Scalability and Practical Impact: Although the authors claim low additional computational cost, there is no direct measurement or discussion (e.g., training time, inference cost increases due to noise sampling and additional rollouts). It would be beneficial to quantify this, especially for larger models where efficiency could be more critical.\n6. Analysis Lacks Broader Perspective: The main results (Tables 1 and 2) focus on Qwen2.5-VL 3B and 7B variants, with only limited comparison to a wider array of models or tasks. Wider external generalizability or applicability remains somewhat speculative.\n7. Interpretability of Results: Some key claims, such as the link between the proposed method and reduced hallucination (Table 2, AMBER benchmark), would benefit from deeper qualitative error analysis beyond metric scores.", "was_revised": true, "final_justification": "Thanks to the author for their effort in the rebuttal. I would advocate for acceptance.", "rebuttal": "**Thank you for acknowledging the methodological soundness, theoretical rigor, and practical value of our work; we will address your concerns in detail below.**\n\n---\n**Q1 & W1** *Methodological Differentiation.*\n\n---\nThank you for raising this important question. \n**Our method is conceptually distinct from prior RL works.**\nBelow, we provide clarification on how our approach differs from prior methods such as SNI\\[12], DAPO \\[51], and VLM-R1 \\[39]. We will add these discussions to the related works in the revised version:\n\n**Differentiation over Selective Noise Injection[12]**\n\nWhile both SNI [12] and our method leverage noise injection in policy exploration to enhance RL generalization, our approach fundamentally differs from SNI in two key aspects, as outlined below.\n- **Off-policy handling for noise-injected policy exploration.** SNI [12] adopts a manually controlled weighted sum of gradients from noisy and clean rollouts (Eq. 7 in [12]), using a *fixed mixing ratio set by a hyperparameter*.\nIn contrast, our method models the controllable noise level as a prior within a Bayesian advantage estimation framework, enabling *trajectory-wise dynamic calibration* of advantages and more principled off-policy correction.\n- **Noise Injection Strategy.** SNI [12] applies noise indirectly through structural mechanisms like Dropout and Variational Information Bottleneck (VIB), which perturb intermediate feature representations. In contrast, our method injects noise directly into the input images during rollout generation—a vision-language-specific strategy that jointly enhances exploration and robustness to multimodal hallucination, which is not explored by prior methods.\n\n**Differentiation over DAPO and VLM-R1**\n\nWhile DAPO and VLM-R1 also build on the GRPO backbone, our method optimizes GRPO from a principled perspective. We provide justification as below:\n- **DAPO** proposes several improvements to enhance the stability and efficiency of training with GRPO, including Clip-Higher, Dynamic Sampling, Token-level Policy Gradient Loss, and Overlong Reward Shaping.\n- **VLM-R1** focuses on applying GRPO to REF and OVDet tasks, designing detection-specific reward functions tailored for these domains.\n\nIn contrast, the main focus of our method lies in addressing the challenges of *insufficient exploration* and *the absence of CoT-process rewards* in multimodal RL. Beyond targeting different tasks, our approach introduces a *principled framework* that tackles these issues through controllable noise injection and a Bayesian modeling of advantage estimation, enabling more effective trajectory-level advantage assignment and policy exploration in multimodal settings.\n\n---\n**Q2 & W4** *Reward Signal Robustness*\n\n---\nThanks for the insightful question! We first clarify our claim regarding the SBERT reward bias, then we address the concerns of reward robustness.\n\n**Clarification** We would like to clarify that our intention is **not to suggest** that the Bayesian estimator eliminates the bias inherent in the SBERT-based reward model. Rather, we aim to claim that **within our experimental setting**, which involves multimodal RL across verifiable and unverifiable data, **the Bayesian advantage estimation alleviates the optimization bias** introduced by noisy or misaligned reward signals, as emperically demonstrated in paper Section 4.6.\n\n**Reward Signal Robustness** The presence of bias in rewards for unverifiable data is **inevitable** in open-ended RL tasks. For instance, widely adopted solutions such as LLM-as-Judge [1] produce inconsistent or unreliable signals, as highlighted in [2].\n\nWe choose SBERT as our reward model due to its **strong semantic understanding** and its **computational efficiency**. We also **empirically validate** its effectiveness, showing consistent gains with NoisyGRPO. We acknowledge that **SBERT’s bias may lead to failure in some cases**, but emphasize that SBERT is a **pluggable component** and can be replaced with stronger or task-specific reward models as they become available.\n\nFurthermore, we acknowledge that **reward for unverifiable data** is an important future direction. We are currently investigating principled ways to diagnose and improve insufficient exploration and missing process CoT reward issues in multimodal RL settings.\n\nReference:\n\n[1] Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.\n\n[2] Scaling Laws for Reward Model Overoptimization.\n\n---\n**Q3 & W5:** *Ablation on Scale and Efficiency*\n\n---\n\nGood suggestion! First, we clarify that NoisyGRPO and GRPO **sample the same number of rollouts**. The wall-clock time, GPU-hours, and peak memory usage for both methods are summarized in the table below to support real-world deployment considerations. The numbers are tested on one node with 8 A100 GPUs.\n\n| Model              | Method     | Wall-Clock Time | GPU-Hours      | Memory Usage     |\n|-|-|-|-|-|\n| Qwen2.5-VL-3B      | GRPO       | 6h12min          | 49.6 A100 hours | Peak GPU 57GB    |\n|                   | NoisyGRPO  | 6h40min          | 53.3 A100 hours | Peak GPU 53GB    |\n| Qwen2.5-VL-7B      | GRPO       | 7h58min          | 63.7 A100 hours | Peak GPU 75GB    |\n|                   | NoisyGRPO  | 7h52min          | 62.9 A100 hours | Peak GPU 77GB    |\n\nThis suggests that NoisyGRPO maintains comparable efficiency with GRPO, despite its additional sampling step, making it suitable for real-world deployment.\n\n---\n**Q4 & W6** *Broader Applicability*\n\n---\n\nThank you for the insightful suggestion. Our current work is tailored to **vision-language reinforcement learning**, where challenges like limited exploration and reward sparsity are prominent. While we have not yet applied NoisyGRPO to text-only or audio-visual tasks, the core idea of **injecting controllable perturbation to policy exploration and leveraging the perturbation level as prior for advantage estimation** is **modality-agnostic and potentially transferable**.  We view this as promising future work and hope to validate the method’s robustness and generality in broader settings.\n\n---\n**Q5** *Error Analysis*\n\n---\nWe believe the observed limitations are due to benchmark-specific characteristics and constitute a minor issue in the broader context of our method’s performance.\n\n**Coverage in AMBER**\n\nWe argue that **AMBER Coverage has inherent limitations** and does not provide evidence of a systematic failure in NoisyGRPO. In AMBER's generative setting, each case has on average **5.5 ground truth captions**, and our method shows at most a **2.7% decrease in Coverage** compared to the strongest baseline—equivalent to merely **0.14 missing words per case**, which we consider a negligible trade-off given the substantial gains in hallucination reduction.\n\nMoreover, we observe that **many hallucination-mitigating methods exhibit similar behavior** on AMBER, where improving factual accuracy can naturally lead to slightly lower coverage scores. This suggests that the metric may not fully capture the quality of grounded yet concise descriptions. To illustrate this, we propose a **CHR** to measure how much the hallucination rate decreases relative to the drop in coverage. A higher value indicates more effective hallucination reduction with less sacrifice in coverage.\n\n$\\text{CHR} = \\frac{|\\Delta \\text{Hallucination Rate}|}{|\\Delta \\text{Coverage}|}$\n\nThis trend is not unique to our method — as the table shows, several recent strong baselines also experience similar trade-offs between hallucination reduction and coverage.\"\n\n| Method         | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | CHR↑  |\n|-|-|-|-|-|-|\n| LLaVA-1.5 7B   | 7.8   | 51.0   | 36.4   | 4.2    | -    |\n| HA-DPO[1]        | 7.2   | 33.6   | 19.7   | 2.6    | 0.96 |\n| CLIP-DPO[2]       | 3.7   | 47.8   | 16.6   | 1.3    | 6.19 |\n| Qwen2.5-vl-3B  | 7.6   | 69.9   | 55.7   | 5.8    | -    |\n| NoisyGRPO 3B   | 6.6   | 67.7   | 44.3   | 3.4    | 5.18 |\n| Qwen2.5-vl-7B  | 4.8   | 63.6   | 27.5   | 1.6    | -    |\n| NoisyGRPO 7B   | 4.2   | 63.2   | 23.9   | 1.2    | **9.00** |\n\nFrom the table, we can see that other methods on AMBER also exhibit a decrease in hallucination accompanied by a drop in coverage, while NoisyGRPO 7B achieves the best CHR score. This suggests that **NoisGRPO reduces hallucination more effectively than others**, while incurring a relatively smaller drop in coverage, as reflected by the highest CHR score among all methods.\n\nReference:\n\n[1] Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization, arXiv 2023, cited 148 times.\n\n[2] CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs, ECCV 2024, cited 26 times.\n\n**Coarse Perception**\n\nWe argue that this is **a minor issue** of NoisyGRPO, as our method achieves performance comparable to the baseline (within ±1% accuracy). This suggests that our approach remains effective even in challenging scenarios.\n\nThe observed limitation stems from the fact that **our assumption — the monotonic relation between noise and trajectory quality — does not always hold** in Coarse Perception tasks. These tasks rely on high-level abstraction where fine-grained visual details are less critical, so noise injection may not sufficiently perturb the model's output to expose quality differences.\n\nHowever, **this does not undermine our method**, as the Bayesian advantage estimator is designed to adaptively downweight the prior when its confidence is low. Specifically, when noise fails to differentiate quality, the semantic reward's variance decreases (Eq. 8), leading to higher uncertainty in the prior and a reduced weight in the final estimation (Eq. 9). This adaptive mechanism ensures stable performance even when noise injection is less effective.\n\n---\n**We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.**"}, {"review_id": "TLjYViP2f3", "reviewer": "Reviewer_RksJ", "summary": "This paper proposes NoisyGRPO, an extension of Group Relative Policy Optimisation (GRPO) that targets general chain of thought reasoning in multimodal LLMs. The authors combine two ideas, 1. Noise injected exploration to widen the exploration space and implicitly labels “easy” (clean) versus “hard” (noisy) trajectories. 2. Bayesian advantage estimation, a closed form Gaussian update to yield  a posterior that down-weights noisy but correct trajectories and guides the PPO style policy update. Experiments fine-tuned on  Qwen-2.5-VL compare against vanilla GRPO and show gains and some ablations on contributions from both factors are presented.", "questions": "- Comparison to DPO & R1: Can you add a run of DPO or VLM-R1 on the same data for context?\n\n- Text-only generalisation, if the visual stream is empty (pure language tasks), does NoisyGRPO degrade to GRPO or hurt?\n\n- wondering what the relative wall-clock vs. vanilla GRPO is when using equal numbers of rollouts per iteration?\n\n- Have you tried how performance scales if the 13 k VQA set is subsampled to e.g. 3 k or 6 k?\n\nLooking forward to the discussion in rebuttal, any thoughts on the questions or limitations would be nice.", "limitations": "> Dataset size and domain scope are small, results may not transfer to web-scale pre-training or to other modality reasoning.\n\n> Baseline coverage limited, no uncertainty estimates (std / CI) reported for main numbers (minor point).\n\n> Method currently addresses exploration, not reward sparsity, might fail if answers are long-form or require external tools", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- Conceptual novelty that turns  “add noise for robustness” trick into a principled prior inside GRPO, an elegant closed-form update I would say.\n- Consistent gains on three benchmark families, especially for smaller 3B models. I personally liked  seeing gains on the smaller regime.\n- Ablations separate noise-only, naïve fusion, and dynamic fusion, also include a parameter study.\n- There is some disclosure of computational costs involved during training time.\n- Since the nose only during roll-out, it's a sign of being cost-neutral.\n\n### Weakness\n\n- Prior assumes a monotonic relation between noise and difficulty that essentially breaks down for tasks where coarse cues suffice (acknowledged by authors).\n- Training corpus is tiny (13 k VQA pairs) and settings are  a bit narrow, no study on larger, diverse datasets or on language-only tasks.\n- Usual baselines for RL are not highlighted e.g. DPO, vanilla rejection sampling, andit  would be nice to include one value-based method.\n- Computation cost is still heavy for academic or other practitioners, a minor limitation, I would say!\n- Readability of Figure-2 and Table-2 can be better!\n- Somehow, the presentation feels rushed. It can be improved.", "was_revised": true, "final_justification": "Some of the additional results and clarifications make the paper stronger.", "rebuttal": "**We sincerely thank the reviewer for recognizing the conceptual novelty of introducing a principled noise-based prior in GRPO, the consistent performance improvements, the thorough ablation studies, and the transparent reporting of computational costs. We appreciate your encouraging feedback and address the remaining concerns below.**\n\n---\n**Q1 & W3:** *Comparison to DPO & R1*\n\n---\n\nThank you for the great suggestion. \nFirst, we clarify that VLM-R1, when applied to our VQA training data, does not introduce additional design beyond the GRPO algorithm and thus is the **same as our GRPO baseline**. For DPO, we empirically demonstrate that the **NoisyGRPO outperforms DPO baseline**. Specifically, we extracted the same number of training samples from MMRLHF and trained a DPO model using the recommended settings from LLaMA Factory. The results are summarized in the table below.\n\n| Method      | CP. | FP. | IR. | LR. | ST. | MA. | Avg. | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | F1 ↑ | Acc |\n|-------------|-----|-----|-----|-----|-----|-----|------|------|--------|---------|---------|------|------|\n| **DPO**     | 61.6 | 46.3 | 56.2 | 54.1 | 34.0 | 52.2 | 50.7 | 7.8  | 70.6   | 49.2   | 4.4   | 87.3 | 41.0 |\n| **NoisyGRPO** | 69.5 | 53.2 | 66.6 | 60.7 | 37.1 | 62.3 | 58.2 | 6.6  | 67.7   | 44.3   | 3.4   | 90.3 | 44.0 |\n\nFrom the results, we observe a **significant performance gap between the DPO baseline and NoisyGRPO** across multiple metrics. This discrepancy can be attributed to the higher data efficiency of NoisyGRPO, whereas DPO requires a larger amount of high-quality preference data to achieve competitive performance.\n\n---\n**Q2 & W2:** *Text-only generalisation*\n\n---\nAlthough text-only generalization is beyond the current scope of our work, we conducted exploratory evaluations on MMLU and found that **NoisyGRPO exhibits potential for generalization to the text domain.**\n\nSpecifically, we compared GRPO, NoisyGRPO, and the pretrained MLLM. \n\nBecause the RL training prompts explicitly require the model to perform Chain-of-Thought reasoning grounded on visual input, multimodal models often fail to follow instructions properly when images are absent. This results in a **high format error rate** (exceeding 60%) for both GRPO and NoisyGRPO on text-only samples. To mitigate this, we **only report performance on samples where the model produces valid (format-correct) answers**.\n\n| Model                          | Other | STEM | Humanities | Social & Sciences | Overall Acc | Format Error Rate |\n|-------------------------------|:-----:|:----:|:----------:|:-----------------:|:-----------:|:-----------------:|\n| Qwen2.5-VL-3B                 | 63.8  | 58.0 |    55.8    |       72.1        |    61.6     |         -         |\n| Qwen2.5-VL-7B                 | 74.8  | 66.8 |    60.1    |       79.1        |    69.0     |         -         |\n| **CoT Reasoning with Qwen2.5-VL-3B** ||||| |\n| GRPO                          | 73.9  | 62.0 |    53.3    |       73.2        |    66.4     |       77.3        |\n| NoisyGRPO                     | 71.1  | 56.9 |    68.1    |       74.3        |    68.9     |       85.5        |\n| **CoT Reasoning with Qwen2.5-VL-7B** ||||| |\n| GRPO                          | 76.3  | 70.4 |    63.0    |       79.1        |    72.6     |       61.8        |\n| NoisyGRPO                     | 83.3  | 81.0 |    68.8    |       82.5        |    79.8     |       75.3        |\n\nWe observe that, despite the high format error rates caused by the absence of visual input, **NoisyGRPO consistently outperforms GRPO on valid outputs across most categories**. This suggests that our method has potential for generalization beyond vision-language tasks to purely text-based scenarios.\n\nWe acknowledge that these results are preliminary, and we plan to conduct more extensive evaluations to better understand and improve text-only generalization in future work.\n\n---\n**Q3 & W4:** *Statistics for Training*\n\n---\n\nThank you for the insightful question. We clarify that NoisyGRPO and GRPO sample the **same number of rollouts per iteration**, making their computation cost directly comparable.\n\nThe table below summarizes wall-clock time, GPU-hours, and peak memory usage measured on a single node with 8 A100 GPUs:\n\n| Model              | Method     | Wall-Clock Time | GPU-Hours      | Memory Usage     |\n|-------------------|------------|------------------|----------------|------------------|\n| Qwen2.5-VL-3B      | GRPO       | 6h12min          | 49.6 A100 hours | Peak GPU 57GB    |\n|                   | NoisyGRPO  | 6h40min          | 53.3 A100 hours | Peak GPU 53GB    |\n| Qwen2.5-VL-7B      | GRPO       | 7h58min          | 63.7 A100 hours | Peak GPU 75GB    |\n|                   | NoisyGRPO  | 7h52min          | 62.9 A100 hours | Peak GPU 77GB    |\n\nWe acknowledge that the overall computation cost remains substantial, which is a common challenge in large-scale vision-language RL research. The 13K dataset size already pushes our computational budget limits; with additional resources, we plan to explore even larger datasets to further validate our approach.\n\n---\n**Q4:** *Training with Fewer Data*\n\n---\n\nGreat suggestion! We conduct experiments with subsampled data (e.g., 3k and 6k) and observe that NoisyGRPO maintains **consistent gains** over baselines, demonstrating its **scalability** in leveraging more training data to improve performance.\n\nSpecifically, we sampled 3k and 6k training sets from the original 13k dataset and conducted experiments. To ensure a fair comparison, we set the number of training steps for the 3k and 6k subsets to be the same as that used for training on the full 13k dataset. The results are shown in the table below:\n\n| Method                | MMStar           |       |       |       |       |       |       | AMBER          |       |       |       |       | MMERealworld |\n|-----------------------|------------------|-------|-------|-------|-------|-------|-------|----------------|-------|-------|-------|-------|--------------|\n|                       | CP.  | FP.  | IR.  | LR.  | ST.  | MA.  | Avg.  | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | F1 ↑  | Acc          |\n| GRPO 3k     | 66.4 | 44.6 | 59.7 | 53.8 | 35.1 | 56.7 | 52.7  | 7.1  | 66.1   | 46.9   | 3.8    | 88.9  | 38.9         |\n| NoisyGRPO 3k | 66.7 | 51.8 | 61.2 | 56.4 | 35.3 | 62.3 | 55.6  | 6.7  | 63.5   | 36.8   | 2.2    | 89.9  | 44.5         |\n| GRPO 6k     | 67.2 | 45.6 | 58.7 | 53.2 | 40.2 | 54.1 | 53.2  | 7.0  | 67.3   | 48.8   | 4.7    | 88.3  | 38.1         |\n| NoisyGRPO 6k | 69.1 | 52.2 | 64.3 | 54.7 | 39.7 | 59.5 | 56.6  | 6.4  | 65.4   | 40.2   | 3.1    | 90.1  | 44.1         |\n\n\nWe observe from the results above that NoisyGRPO’s performance improves as the training data size increases. Moreover, at each data scale, NoisyGRPO consistently outperforms GRPO. Therefore, we consider NoisyGRPO to be a scalable approach.\n\n---\n\n**W4:** *Prior assumes a monotonic relation between noise and difficulty*\n\n---\n\nWe would like to provide some additional clarification on this limitation. We believe this **assumption holds in most current vision-language scenarios**, as evidenced by the results in Table 2. For cases where the assumption may fail, such as coarse perception tasks, we observe that this failure does **not cause a significant performance drop** in NoisyGRPO — it achieves results comparable to the baseline (e.g., less than 1% decrease on coarse perception in Table 2). Therefore, we consider this a **minor limitation** that does not affect the main claims of our paper.\n\n---\n**We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.**"}, {"review_id": "FWiNfQCHf8", "reviewer": "Reviewer_TGQ6", "summary": "NoisyGRPO tackles the limited generalization of existing methods for MLLMs by injecting controllable Gaussian noise into input images during rollouts, thereby widening exploration and providing a trajectory-level signal of reasoning difficulty. To prevent this noise from destabilizing learning, the framework casts advantage estimation as a Bayesian update that fuses a noise-based prior with the observed reward, yielding a posterior that adaptively weights exploration versus correctness. Both components are lightweight. Across different benchmarks, it consistently outperforms vanilla GRPO, while bringing the most pronounced gains to smaller models. Ablations show that uniform noise combined with the dynamic Bayesian fusion drives these improvements, though the method can underperform on coarse-perception tasks where noise fails to perturb visual cues, highlighting a key limitation meaningfully.", "questions": "See Weaknesses above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. NoisyGRPO perturbs each input image with controllable Gaussian noise during rollout collection, significantly broadening the search space and promoting visually grounded CoT trajectories.\n2. A Bayesian advantage estimator treats the noise level as a prior and the trajectory reward as the likelihood, enabling dynamic down-weighting of noisy or unreliable rollouts and preventing learning collapse.\n3. NoisyGRPO achieves improvements on benchmarks while remaining lightweight.\n\nWeaknesses:\n1. The method promotes grounded answers but at the expense of descriptive breadth. Its captions score lower on AMBER Cov. metric, which penalizes omissions of image elements.\n2. On certain models, vanilla GRPO produces more concise CoT with comparable accuracy.\n3. Ablations show that naïve noise injection or sub-optimal γ in the Bayesian fusion can degrade performance, indicating that the gains are sensitive and depend on careful tuning.\n4. The accuracy reward relies on SBERT similarity, and all hyperparameters are selected directly on MMStar without a held-out validation set, raising concerns about embedding bias and benchmark overfitting.", "was_revised": true, "final_justification": "The authors addressed most of my questions and concerns. Considering that the overall quality and the novelty of the work, I recommend a rating of 4: Borderline Accept.", "rebuttal": "**We thank the reviewer for highlighting the strengths of our method, including the noise-driven exploration, Bayesian estimator, and lightweight design. We address the reviewer’s concerns below.**\n\n---\n**W1:** *Promote grounded answers but at the expense of descriptive breadth*\n\n---\n\nWe argue that **AMBER Coverage has inherent limitations** and does **not fully support the claim** that our method promotes grounding at the expense of descriptive breadth. In AMBER's generative setting, each case has on average **5.5 ground truth captions**, and our method shows at most a **2.7% decrease in Coverage** compared to the strongest baseline—equivalent to merely **0.14 missing words per case**, which is a negligible trade-off considering the benefit of reduced hallucination.\n\nMoreover, we observe that **many hallucination-mitigating methods exhibit similar behavior** on AMBER, where improving factual accuracy can naturally lead to slightly lower coverage scores. This suggests that the metric may not fully capture the quality of grounded yet concise descriptions. To illustrate this, we propose a **CHR** to measure how much the hallucination rate decreases relative to the drop in coverage. A higher value indicates more effective hallucination reduction with less sacrifice in coverage.\n\n$\\text{CHR} = \\frac{|\\Delta \\text{Hallucination Rate}|}{|\\Delta \\text{Coverage}|}$\n\nThis trend is not unique to our method — as the table shows, several recent strong baselines also experience similar trade-offs between hallucination reduction and coverage.\"\n\n| Method         | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | CHR↑  |\n|----------------|-------|--------|--------|--------|------|\n| LLaVA-1.5 7B   | 7.8   | 51.0   | 36.4   | 4.2    | -    |\n| HA-DPO[1]        | 7.2   | 33.6   | 19.7   | 2.6    | 0.96 |\n| CLIP-DPO[2]       | 3.7   | 47.8   | 16.6   | 1.3    | 6.19 |\n| Qwen2.5-vl-3B  | 7.6   | 69.9   | 55.7   | 5.8    | -    |\n| NoisyGRPO 3B   | 6.6   | 67.7   | 44.3   | 3.4    | 5.18 |\n| Qwen2.5-vl-7B  | 4.8   | 63.6   | 27.5   | 1.6    | -    |\n| NoisyGRPO 7B   | 4.2   | 63.2   | 23.9   | 1.2    | **9.00** |\n\nFrom the table, we can see that other methods on AMBER also exhibit a decrease in hallucination accompanied by a drop in coverage, while NoisyGRPO 7B achieves the best CHR score. This suggests that **NoisGRPO reduces hallucination more effectively than others**, while incurring a relatively smaller drop in coverage, as reflected by the highest CHR score among all methods.\n\n\nReference:\n\n[1] Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization, arXiv 2023, cited 148 times.\n\n[2] CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs, ECCV 2024, cited 26 times.\n\n---\n**W2:** *Vanilla GRPO produces more concise CoT with comparable accuracy*\n\n---\n\nWe acknowledge this observation. However, we would like to clarify that the results on the MME-CoT efficiency category **do not fully support the claim that vanilla GRPO produces more concise CoT**. First, MME-CoT evaluates CoT quality based on **relevance and reflection**, which do not comprehensively measure consciousness. Furthermore, we conducted a quantitative analysis of the **actual CoT lengths during inference**. As shown in the table below, NoisyGRPO consistently produces shorter CoTs across benchmarks:\n\n| CoT Length    | MMStar  | AMBER  | MM-RealWorld |\n| ------------- | ------- | ------ | ------------ |\n| GRPO          | 295     | 335    | 358                |\n| **NoisyGRPO** | **172** | **53** | **236**      |\n\nWe observe that NoisyGRPO generates significantly shorter CoTs compared to vanilla GRPO, while achieving better performance on most benchmarks. **From the perspective of CoT length as a proxy for conciseness**, we argue that **NoisyGRPO demonstrates a clear advantage in generating more concise CoTs.**\n\nMoreover, we provide justification for the shorter CoT length on AMBER: since all questions are fixed as \"Describe the image,\" NoisyGRPO typically reframes the task by posing a question about the image's main subject. This behavior is illustrated in the visualization in Figure 10 of the appendix.\n\n---\n**W3:** *The gains are sensitive and depend on careful tuning.*\n\n---\n\nWe would like to clarify that we did **not perform extensive hyperparameter tuning**, and we humbly argue that introducing new hyperparameters is often **inevitable** when proposing a new framework. To demonstrate the **robustness of our method to hyperparameter choices**, we conduct a sensitivity analysis of $\\gamma$, as presented in the table below. \n\n|       | MMStar           |       |       |       |       |       |       | AMBER          |       |       |       |       | MMERealworld |\n|---------|------------------|-------|-------|-------|-------|-------|-------|----------------|-------|-------|-------|-------|--------------|\n|         | CP.  | FP.  | IR.  | LR.  | ST.  | MA.  | Avg.  | Cs ↓ | Cov. ↑ | Hal. ↓ | Cog. ↓ | F1 ↑  | Acc          |\n| vanilla GRPO | 68.0  | 43.3 |  60.4 |  57.0  | 40.0  | 58.0  | 54.5  | 6.7  | 68.5  | 44.6  | 4.2 |  89.2 |  40.8 | \n| NoisyGRPO with $\\gamma$ = 0.015   | 66.0  | 54.1 | 63.0 | 52.4 | 36.2 | 59.1 | 55.2  | 6.8   | 69.8   | 45.2   | 2.9    | 89.3  | 44.7         |\n|NoisyGRPO with  $\\gamma$ = 0.01    | 69.5  | 53.2 | 66.6 | 60.7 | 37.1 | 62.3 | 58.2  | 6.6   | 67.7   | 44.3   | 3.4    | 90.3  | 44.0         |\n| NoisyGRPO with $\\gamma$ = 0.005   | 67.1  | 48.9 | 62.8 | 57.9 | 37.0 | 61.7 | 56.9  | 6.4   | 65.2   | 38.4   | 2.6    | 90.5  | 43.2         |\n\nThe results show that NoisyGRPO consistently outperforms the baseline across a small but meaningful range of $\\gamma$ values, **demonstrating the method’s robustness and indicating that its performance gains do not rely on fine-grained hyperparameter tuning.**\n\n---\n**W4:** *Concerns about embedding bias and benchmark overfitting.*\n\n---\n\n**Embedding bias**\n\nWe acknowledge the reviewer’s concern that SBERT may introduce bias as a reward function. This is indeed a **key consideration in our method design**. To mitigate this issue, our Bayesian advantage estimation explicitly incorporates a prior term that reflects the expected trajectory quality based on noise levels. This mechanism **dynamically balances** the SBERT-based semantic reward and the prior estimation, thereby **reducing over-reliance on the potentially biased SBERT signal** during training.\n\nMoreover, as detailed in our response to *reviewer nP6a*, Weakness 4, we conducted ablation studies using a rule-based reward function. The results consistently show that **SBERT outperforms the rule-based alternatives** in our experiments, further supporting its utility despite its known limitations.\n\n**Benchmark Overfitting**\n\nAs addressed in our response to previous W3, the parameter settings optimized on MMStar are **not optimal but still yield strong results for other benchmarks**. For example, $\\gamma = 0.015$ achieves the best performance on MMERealworld, while $\\gamma = 0.01$ performs best on AMBER. Nonetheless, our method **achieves comparable and strong performance** across these benchmarks, which demonstrates that **NoisyGRPO does not overfit to MMStar**.\n\n---\n**We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.**"}]

Author rebuttal:

Meta-review: The paper introduces NoisyGRPO, an extension to GRPO which introduces a noise to image inputs to improve generalization, robustness, and exploration in CoT reasoning in MLLMs. A bayesian advantage estimator down weights noisy trajectories for policy optimization. Overall, the work is timely given attention on research on multimodal reasoning, and the approach is intuitive and sound. There were various questions regarding the noise prior and advantage estimation, as well as a variety of other concerns wrt comparisons to prior works, additional ablations, etc. However, the authors seem to have provided a very satisfactorily rebuttal, and the reviewers (and I) were satisfied with the answers.

I therefore recommend the paper is accepted as a poster.
Decision: Accept (poster)