Title: NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Model

Reinforcement learning
Abstract: Reinforcement learning (RL) has shown promise in enhancing the general Chain-of-Thought (CoT) reasoning capabilities of multimodal large language models (MLLMs). However, when applied to improve general CoT reasoning, existing RL frameworks often struggle to generalize beyond the training distribution.  To address this, we propose NoisyGRPO, a systematic multimodal RL framework that introduces controllable noise into visual inputs for enhanced exploration and explicitly models the advantage estimation process via a Bayesian framework.
Specifically, NoisyGRPO improves RL training by: (1) \textbf{Noise-Injected Exploration Policy}: Perturbing visual inputs with Gaussian noise to encourage exploration across a wider range of visual scenarios; and (2) \textbf{Bayesian Advantage Estimation}: Formulating advantage estimation as a principled Bayesian inference problem, where the injected noise level serves as a prior and the observed trajectory reward as the likelihood. This Bayesian modeling fuses both sources of information to compute a robust posterior estimate of trajectory advantage, effectively guiding MLLMs to prefer visually grounded trajectories over noisy ones.
Experiments on standard CoT quality, general capability, and hallucination benchmarks demonstrate that NoisyGRPO substantially improves generalization and robustness, especially in RL settings with small-scale MLLMs such as Qwen2.5-VL 3B.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNoisyGRPO tackles the limited generalization of existing methods for MLLMs by injecting controllable Gaussian noise into input images during rollouts, thereby widening exploration and providing a trajectory-level signal of reasoning difficulty. To prevent this noise from destabilizing learning, the framework casts advantage estimation as a Bayesian update that fuses a noise-based prior with the observed reward, yielding a posterior that adaptively weights exploration versus correctness. Both components are lightweight. Across different benchmarks, it consistently outperforms vanilla GRPO, while bringing the most pronounced gains to smaller models. Ablations show that uniform noise combined with the dynamic Bayesian fusion drives these improvements, though the method can underperform on coarse-perception tasks where noise fails to perturb visual cues, highlighting a key limitation meaningfully.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. NoisyGRPO perturbs each input image with controllable Gaussian noise during rollout collection, significantly broadening the search space and promoting visually grounded CoT trajectories.\n2. A Bayesian advantage estimator treats the noise level as a prior and the trajectory reward as the likelihood, enabling dynamic down-weighting of noisy or unreliable rollouts and preventing learning collapse.\n3. NoisyGRPO achieves improvements on benchmarks while remaining lightweight.\n\nWeaknesses:\n1. The method promotes grounded answers but at the expense of descriptive breadth. Its captions score lower on AMBER Cov. metric, which penalizes omissions of image elements.\n2. On certain models, vanilla GRPO produces more concise CoT with comparable accuracy.\n3. Ablations show that naïve noise injection or sub-optimal γ in the Bayesian fusion can degrade performance, indicating that the gains are sensitive and depend on careful tuning.\n4. The accuracy reward relies on SBERT similarity, and all hyperparameters are selected directly on MMStar without a held-out validation set, raising concerns about embedding bias and benchmark overfitting.\n\nQUESTIONS:\nSee Weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes NoisyGRPO, an extension of Group Relative Policy Optimisation (GRPO) that targets general chain of thought reasoning in multimodal LLMs. The authors combine two ideas, 1. Noise injected exploration to widen the exploration space and implicitly labels “easy” (clean) versus “hard” (noisy) trajectories. 2. Bayesian advantage estimation, a closed form Gaussian update to yield  a posterior that down-weights noisy but correct trajectories and guides the PPO style policy update. Experiments fine-tuned on  Qwen-2.5-VL compare against vanilla GRPO and show gains and some ablations on contributions from both factors are presented.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Conceptual novelty that turns  “add noise for robustness” trick into a principled prior inside GRPO, an elegant closed-form update I would say.\n- Consistent gains on three benchmark families, especially for smaller 3B models. I personally liked  seeing gains on the smaller regime.\n- Ablations separate noise-only, naïve fusion, and dynamic fusion, also include a parameter study.\n- There is some disclosure of computational costs involved during training time.\n- Since the nose only during roll-out, it's a sign of being cost-neutral.\n\n### Weakness\n\n- Prior assumes a monotonic relation between noise and difficulty that essentially breaks down for tasks where coarse cues suffice (acknowledged by authors).\n- Training corpus is tiny (13 k VQA pairs) and settings are  a bit narrow, no study on larger, diverse datasets or on language-only tasks.\n- Usual baselines for RL are not highlighted e.g. DPO, vanilla rejection sampling, andit  would be nice to include one value-based method.\n- Computation cost is still heavy for academic or other practitioners, a minor limitation, I would say!\n- Readability of Figure-2 and Table-2 can be better!\n- Somehow, the presentation feels rushed. It can be improved.\n\nQUESTIONS:\n- Comparison to DPO & R1: Can you add a run of DPO or VLM-R1 on the same data for context?\n\n- Text-only generalisation, if the visual stream is empty (pure language tasks), does NoisyGRPO degrade to GRPO or hurt?\n\n- wondering what the relative wall-clock vs. vanilla GRPO is when using equal numbers of rollouts per iteration?\n\n- Have you tried how performance scales if the 13 k VQA set is subsampled to e.g. 3 k or 6 k?\n\nLooking forward to the discussion in rebuttal, any thoughts on the questions or limitations would be nice.\n\nLIMITATIONS:\n> Dataset size and domain scope are small, results may not transfer to web-scale pre-training or to other modality reasoning.\n\n> Baseline coverage limited, no uncertainty estimates (std / CI) reported for main numbers (minor point).\n\n> Method currently addresses exploration, not reward sparsity, might fail if answers are long-form or require external tools", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces NoisyGRPO, a reinforcement learning (RL) framework aimed at improving the chain-of-thought (CoT) reasoning in multimodal large language models (MLLMs). The method involves injecting controllable Gaussian noise into visual inputs to encourage exploration during RL, combined with a Bayesian advantage estimation that fuses this noise (as a prior) and observed rewards (as a likelihood) for more robust policy optimization. Through extensive experiments on established CoT quality, general ability, and hallucination benchmarks, the authors show that NoisyGRPO outperforms vanilla GRPO and other leading baselines, particularly enhancing the robustness and generalization of small-scale MLLMs like Qwen2.5-VL 3B.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Methodological Innovation: The work presents a clearly articulated extension to GRPO: noise-injected exploration at the input level, paired with a principled Bayesian advantage estimator. The approach is well-motivated by the identified limitations of policy exploration and trajectory evaluation in MLLMs RL.\n2. Solid Theoretical Foundation: The Bayesian estimator for advantage estimation is soundly motivated and derived (Section 3.3), where noise magnitude serves as a calibrated prior and reward as the observed likelihood, resulting in adaptive, statistically justified updates.\n3. Comprehensive Experimental Evaluation: The experiments span multiple relevant benchmarks (e.g., MME-CoT for CoT quality, MMStar for general reasoning, and AMBER for hallucination) and consider both small- and large-scale MLLMs. Table 1 and Table 2 report detailed results, showing consistent gains especially for smaller models, validating the claims.\n4. Results Analysis and Ablation: The paper provides insight through ablations (Figure 4) on the effect of both core method components and hyperparameters, as well as comparisons of noise distributions. Performance trends over training (Figure 3) provide further empirical support for the learning dynamics described.\n5. Clear, Well-Organized Writing: Despite some dense sections, the paper is mostly well-structured, with equations, pipeline figures (see Figure 2), and comprehensive result tables and plots.\n6. Practical Relevance: The findings address a genuine problem—improving generalization and robustness of RL-driven reasoning in MLLMs, which is valuable for both researchers and practitioners dealing with multimodal and open-ended settings.\n\n---\n## Weaknesses\n1. Limited Novelty Relative to Recent Works: While the combination of noise injection and Bayesian estimation is methodologically appropriate, the paper does not demonstrate that the specific form of injected noise (Gaussian/diffusion in visual input) and the exact Bayesian estimator extend far beyond prior ideas of selective noise in RL [12] and group-based RL advantage estimation. The differentiation from similar recent approaches (especially those cited in [12], DAPO [51], and VLM-R1 [39]) could be clarified further.\n2. Empirical Gains Are Sometimes Marginal: Though the main method outperforms baselines, in Table 2 the MMStar overall gains (“+2.8 and +2.6”) over the base model are present but not transformative. In some subcategories (e.g., Coarse Perception), NoisyGRPO underperforms GRPO, calling into question broad generality.\n3. Complexity of the Bayesian Estimator: The approach introduces several new hyperparameters (e.g., $\\alpha$, $\\gamma$ in the Bayesian update; normalization steps) whose role becomes somewhat intricate (see Figure 4), requiring careful tuning. While the ablation explores sensitivity, practitioners may face difficulty generalizing the method without significant validation infrastructure.\n4. Reliance on Embedding-Based Rewards: The SBERT-based reward may bias the optimization, and although the authors claim the estimator mitigates this, the evidence for such mitigation is limited to indirect observations. Section 3.1 concedes some rewards for open-ended tasks are imperfect, but analysis is not fully conclusive.\n5. Scalability and Practical Impact: Although the authors claim low additional computational cost, there is no direct measurement or discussion (e.g., training time, inference cost increases due to noise sampling and additional rollouts). It would be beneficial to quantify this, especially for larger models where efficiency could be more critical.\n6. Analysis Lacks Broader Perspective: The main results (Tables 1 and 2) focus on Qwen2.5-VL 3B and 7B variants, with only limited comparison to a wider array of models or tasks. Wider external generalizability or applicability remains somewhat speculative.\n7. Interpretability of Results: Some key claims, such as the link between the proposed method and reduced hallucination (Table 2, AMBER benchmark), would benefit from deeper qualitative error analysis beyond metric scores.\n\nQUESTIONS:\n1. Methodological Differentiation: How does NoisyGRPO fundamentally distinguish itself from prior selective noise injection RL methods ([12]) and other recent exploration-boosting strategies in visual RL? What unique properties (empirical or theoretical) arise from using image-space noise + Bayesian estimation compared to similar approaches?\n2. Reward Signal Robustness: Given that SBERT-based rewards may be imperfect for open-ended VQA, have you evaluated reward reliability or robustness directly? Are there scenarios where NoisyGRPO fails due to reward misspecification or bias?\n3. Ablation on Scale and Efficiency: Can you provide concrete wall-clock time, GPU-hours, or memory usage for NoisyGRPO versus GRPO, especially on larger models, to support real-world deployment claims?\n4. Broader Applicability: Is NoisyGRPO effective for other modalities or tasks (e.g., text-only, audio-visual tasks), or for LLMs not relying on visual input? What about transfer to reinforcement learning settings outside vision-language?\n5. Error Analysis: For categories where NoisyGRPO underperforms (e.g., Coarse Perception, Coverage in AMBER), do you observe systematic failure modes? Can your Bayesian estimator be adapted or regularized to address these?\n\nLIMITATIONS:\nThe authors provide an explicit limitations section addressing key assumptions and failure modes. Additional discussion on scalability and reward signal bias would further strengthen this.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes NoisyGRPO, an RL training framework based on the classic GRPO method. Specifically, the method introduces (1) Noise-Injected Exploration Policy, which adds Gaussian noise to the images to obtain diverse rollouts, and (2) Bayesian Advantage Estimation, which takes the injected noise level as a prior, and the observed trajectory reward as the likelihood, to serve as advantage estimation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The overall motivation is reasonable: when training RL, the rollouts tend to converge as the RL training proceeds. By injecting noise into images explicitly, the rollouts' diversity is ensured. The novelty is sound.\n2. The reported model results seem to perform well across different benchmarks and different model sizes. \n3. The paper is quite easy to read.\n\n## Weaknesses\n\n1. For the Off-Policy Strategy & Lack of Justification (L.161-162), the authors apply noise injection *only* during rollout collection but use clean inputs for policy updates. This creates an off-policy strategy where inference and training conditions differ. The paper lacks a clear justification for why this approach works effectively. More ablation studies and analysis are needed to validate this design choice.\n2. The distinction between the two Bayesian advantage estimation strategies is unclear. One injects noise directly during advantage estimation, while the other injects noise into the image inputs during rollout collection. The paper fails to claim the fundamental difference or relative contribution of these two noise injection points.\n3. The comparison with the SOTA methods ([1-3]) is insufficient. Performance should be rigorously evaluated against these methods using mainstream multimodal mathematical reasoning benchmarks like MathVista [4], MathVision [5], MathVerse [6], and MMMU Pro [7]. (MMStar is noted as containing only a small subset and is therefore not comprehensive enough for this purpose).\n4. Inadequate Reward Function Analysis: The authors state they use SBERT to compute text similarity for open-ended answers instead of rule-based rewards (lines 143-146). However, they provide no ablation study demonstrating SBERT's superiority over traditional rule-based methods. Furthermore, no ablation analysis is presented for the threshold parameter `τ`.\n\n[1] MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning\n[2] SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement\n[3] NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation\n[4] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts\n[5] Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset\n[6] MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?\n[7] MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}