NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
Authors: Xiangyan Liu
Jinjie Ni
Zijian Wu
Chao Du
Longxu Dou
Haonan Wang
Tianyu Pang
Michael Qizhe Shieh
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
Visual Reasoning
Reinforcement Learning
Data Augmentation
Primary area: deep_learning
TL;DR: NoisyRollout boosts VLM reasoning by mixing clean and noisy inputs during RL, improving generalization with no extra cost.

Abstract: Recent advances in reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy exploration to better scale test-time compute remains largely underexplored. In addition, VLMs continue to struggle with imperfect visual perception, which in turn affects the subsequent reasoning process.
To this end, we propose **NoisyRollout**, a simple yet effective data augmentation method that mixes trajectories from both clean and moderately distorted images during RL training. By injecting targeted diversity in visual perception and the resulting reasoning patterns, NoisyRollout promotes better policy exploration through vision-oriented inductive biases, ultimately leading to more robust reasoning behaviors. We further adopt a noise annealing schedule that gradually reduces distortion strength over training, leveraging noisy signals early on while ensuring training stability in later stages.
Crucially, our method is easy-to-adopt—**requiring no additional training cost and no modifications to the RL objective**. Extensive experiments on $2$ distinct training datasets demonstrate that NoisyRollout achieves state-of-the-art performance among open-source RL-tuned models across $5$ out-of-domain reasoning and perception benchmarks. Furthermore, we validate the effectiveness of NoisyRollout across model sizes ($7$B and $32$B) and data scales (from $1$K to $6$K), highlighting its generalizability and scalability.

NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
Xiangyan Liu
Jinjie Ni
Zijian Wu
Chao Du
Longxu Dou
Haonan Wang'
Tianyu Pang
Michael Qizhe Shieh
National University of Singapore
Sea Al Lab
Code
Collection
Abstract
Recent advances reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy explo- ration to better scale test-time compute remains largely underexplored. In addition, VLMMs continue struggle with imperfect visual perception, which in turn affects the subsequent reasoning process. We introduce Noisy Rollout, a simple yet ef- fective data augmentation method that addresses these issues by mixing training trajectories from both clean and moderately distorted images. This approach injects perceptual diversity, encouraging better policy exploration and leading to more robust reasoning. A noise annealing schedule gradually reduces distortion strength, aiding exploration early in training while ensuring later stability. Crucially, our method is casy-to-adopt- cedopt-requiring no additional training cost and no modifi- cations the RI objective. Extensive experiments on distinet training datasets demonstrate that NoisyRollout achieves state-of-the-an performance among open- source Rl.-tuned models across out-of-domain reasoning and perception bench- marks. Furthermore, we validate the effectiveness of NoisyRollout across model sizes (7B and 32B), data scales (from 1K 6K) and image augmentation types (Gaussion noise and rotation), highlighting its generalizability and scalability.
Policy Update
(I,q)
A
Ta))
Policy Model
Og:
Reward Model
Tmn
Group Computation
Az
Onitl
Trutl
An
(1,4)
Figure I: An Illustration of the NoisyRollout workflow. Solid lines depict the generation and use of clean rollouts from the clean (original) input (1,9), while dashed lines depict the generation and use of noisy rollouts from the corresponding noisy input qI. The distorted image is obtained by applying a distortion function i = Ta, (7) with distortion strength 0; The distortion level o controlled by a noise annealing schedule, which gradually decreases distortion during training. Rollouts from both sources mixed to form the final trajectories {o.)" rewards 1ri and advantages (A, [A,)1+* Crucially, policy optimization conditions only clean inputs (I, the corresponding noisy inputs (I, q) are used solely collect diverse rollouts for exploration.
*Equal contribution. 'Correspondence to Tianyu Pang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Improvemenn over Basaline
+1
+2.
MathVerse MathVision
MathVista
WeMath
HallusionBench
Average
0penMihhiniee-77
Bireka-Owen-88
NosyRaloutt78 (Ouns)
on 5 out-of-domail benchmarks, covering both visual reasoning tasks (from Math Verse to WeMath) and visual perception task (HallusionBench). Both Swee2.5-VL-GGREE (denoted with +) using vanilla GRPO with only K training samples from Geometry3K, The exact accuracy of NoisyRollout-7B is annotated above each corresponding bar in parentheses.
Figure 2: Accuracy improvement over Qwen2.5-VL.
and NoisyRollout-7B are fine-tuned by ourselves
Introduction
Scaling test-time compute-often referred to as reasoning- reinforcement learning (RL) has emerged as promising axis for advancing model intelligence [25, 11]. While this idea has been primarily explored in the context of large language models (LLMs) [21, 90], the vision-language model (VLM) community is also actively investigating this direction [46, 56, 53). Recent endeavours suggests that VLMs can also benefit from RL-driven scaling of test-time compute [23, 44, 72, 49, 86).
However, scaling test-time compute via RL requires more than sheerly generating longer outputs [45], and VLMs face unique challenges in this process. A key challenge is effective policy exploration, enabling policies to discover behaviors that generalize well beyond training data [87, 82)-an area largely underexplored in VLM research. Traditional practices, such as increasing rollout temperature to promote decoding diversity [89], often introduce superficial variability without meaningfully directing policies toward more robust or informative behaviors.
Moreover, VLMs inherently struggle with imperfect visual perception [40, 78], which negatively impacts subsequent reasoning processes [93, 97, 27]. Despite this, recent efforts [46, 53, 13] tend to adapt RL methods directly from the LLM domain. Such approaches often fail to take these perceptual challenges into consideration, thereby hindering the efficient development of visual reasoning capabilities through RL training.
Tackling the challenges of policy exploration and perceptual limitations in VLMs during RL train- ing, we propose Noisy Rollout, a simple yet powerful data augmentation technique for VLMs that introduces meaningful rollout diversity. Specifically, for cach training sample consisting of input image I and a corresponding text query the old policy (To...) ) produces two sets of rollouts based on the original clean image and moderately distorted version of the same image, respectively.
While the current policy (Tτ) updated solely by conditioning on the clean image and text query pair (I, q). the two sets of rollouts form group, collectively contributing to computing the reward baseline and normalized advantage in Group Relative Policy Optimization (GRPO) [64]. This hybrid rollout strategy enables the policy to achieve more targeted and efficient exploration, ultimately leading to more robust visual reasoning via RL through two key mechanisms:
Successful reasoning trajectories from noisy inputs with distorted images reveal alternative, potentially more robust reasoning strategies, improving reasoning generalization to harder or out-of-domain visual conditions.
When the same query yields different outcomes for clean and distorted inputs, the resulting reward differences expose percepiual discrepuncies that affect reasoning. These discrepancies act as implicit contrastive signals, helping refine the model's visual perception during reasoning by constraining the negative perceptual exploration space.

---

While incorporating noisy rollouts can facilitate more effective and efficient exploration, it may also introduce instabiliiyy in policy gradient estimation. To further enhance scalability and training stability, we employ a noise annealing schedule that gradually reduces the strength of image distortions over training. Such a strategy mitigates distributional mismatch between the evolving policy and the noisy trajectories generated from it when conditioned on clean inputs- an issue that often arises in later training stages- while retaining the benefits of noisy signals during the early phases of training.
We conduct extensive experiments to validate the effectiveness of NoisyRollout. Trained with only 2.1K samples from the 33 [47] dataset using Qwen2.5-7BB -----t--ttttt [4], Figure 2 shows that NoisyRollout achieves superior performance across 5 out oot-of-ddf of-domain visual reasoning and per- ception benchmarks [48, 19, 93, 74, 57] (Math Verse 53.2% Math Vision 28.5%, and HallusionBench 72.1%). It outperforms both open-source RL-tuned models [53, 77] and those utilizing large-scate srnge-ccale supervised fine-tuning (SFT) before [83, 92, 13). Furthermore, it consistently surpasses its direct baseline (vanilla GRPO) on both in-domain and out-of-domain tasks, all within fixed total rollout budget. Crucially, these out-of- to 32B) as well as training corpora and data scales (c.g., MMK12 [53] with 1K to 6K samples). These empirical results, combined with its simplicity and lightweight characteristics, establish NoisyRollout as potentially scalable approach.
improvements generalize across different model sizes (e.g..
2 NoisyRollout: A Free-Lunch with Noisy Reinforcement Learning
We introduce NoisyRollout, data augmentation method that enhances visual reasoning in VLMMs during RL training, particularly by improving the rollout diversity for better policy exploration. NoisyRollout achieves this by incorporating hybrid rollout strategy that leverages reasoning trajec- tories from both clean and distorted images, and a noise annealing schedule that progressively reduces distortion strength. These designs require no additional training cost and integrate seamlessly with standard GRPO implementations. A simplified overview is provided in Figure 1 and Algorithm 1.
GRPO. Group Relative Policy Optimization (GRPO) [64] originally developed to improve mathematical reasoning in LLMs but can also be effectively adapted to enhance visual reasoning in VLMs. For given input pair (I,q) q) consisting of an image and text query from the training set pD. a rule-based outcome reward function r(T, q.o) is adopted to avoid reward hacking. This function assigns r(1, q. o) if the generated response correctly addresses the query (as verified by parser) with the required format, and r(1, o) 0 otherwise. For each input, the old policy Teaa generates n response rollouts. The baseline reward is then calculated as mean(r), where r (ri)-. = {r(1, q, o,)]7 represents the rewards for all rollouts. The normalized advantage for the i-th rollout is defined as A, mean(r) Derived from PPO [62], the GRPO objective function is: std(r)
(o) = -( (11.a))
(o; I.
To I.q)
1 - e, 1 1 )a:) (1)
A, clip
min
n
Teaa (o; 1.
Ta (o, 1,
where no is the current policy, c > 0 sets the clipping range. We omit the KL divergence constraint DKr. / following recent practices in Meng et al. [53] and Liu et al. [45].
Hybrid rollout strategy. Building upon GRPO, NoisyRollout introduces a hybrid rollout strategy to enhance the rollout diversity. For each input pair (T, q). we generate an augmented version of the image I through noise transformation function T. parameterized by distortion strength a, i..., i = (I). As illustrated in Figure 1, the old policy Te.ia produces two sets of rollouts: nı responses conditioned on the clean input (I, q). and n2 responses conditioned on the corresponding noisy input (ī, q). All rollouts from both clean and distorted images are then combined into a single group for reward calculation, yielding r = {r:} = fr(I, q. } U fr(I, q, oz) ) 1 Crucially, the policy update step remains conditioned solely on the clean image and query q for better policy exploration. We defer the discussion of optimizing noisy and clean trajectories on their
+
3

---

Algorithm NoisyRollout: Noisy Reinforcement Fine-Tuning
1: Input: Current policy тe. old policy Too dataset pD, training steps maxx clean rollout number Wi, noisy rollout number 112, clip parameter e, initial noise strength og. noise scheduler 17(-), noise transformation function T())
2: for = 1 to mmax do
3: Sample batch (7, q) PD
4: Set noise strength a, maa, t, mmax)
Annealing schedule
5:
Generate distorted images / = To.
6:
Sample {o,) from mo (0 I, q)
Clean rollouts
7:
Sample (ox t7 from To.a (o I. q.q)
> Noisy rollouts
8:
Compute rewards r; = rl1, q. for all E {l...., ., + n22
9:
Compute advantages A, r meau(r) std(r)
where (m1
10:
Update policy using:
11:
J(t) = E +12
Dete +22 min L.
A. clip( na(o. 1 1, 1- +))
Fez (o.17.q)
Re
12:
- 0 - (0)
Update conditioned on clean images only Update old policy parameters
13:
Oold <- 0
14: end for
corresponding inputs to Appendix B. The NoisyRollout objective function is defined as:
(()
R (T (T.a)-pp.00))))
(ox Felaa (-17.a)
To(o; 1.9) A,, clip To(o; 1,9) 1,9)
1 1 )a.) (2)
Σ
112
To (o, [1.q)
Toa (o;
Noise annealing schedule. Applying fixed-strength distortions throughout training often leads to training instability, primarily due to distributional mismatch between noisy rollouts and the evolving policy. To mitigate this, we introduce a noise annealing schedule 17(-) that gradually
reduces the distortion strength over time. Specifically, training step t, the noise level is defined
as a₂ = 77( t, f, where onp is the initial noise strength and fmmx denotes the total number of training steps. As shown in Figure 1, the distorted image is then generated as = Ta (7).
Consequently, this schedule keeps diverse and informative supervision signals early in training, when the policy is constrained by its perceptual capacity. As training progresses, the noise level a is gradually reduced, narrowing the gap between noisy rollouts ((ox) |u ([*//kmi+1) and the trajectories that Te..a (-IZ, q) would typically produce. This decay helps mitigate abrupt distribution shifts after policy updates, which can arise from unstable or high-variance policy gradients. Over time, rollouts generated from (7, q) become progressively more "on-policy" w.r./ the clean-input-coneee Te (-IT, q). fostering smoother transition from exploration exploitation in later training stages.
policy
Summary. NoisyRollout aims to improve the visual reasoning abilities of VLMs by enhancing rollout diversity to enable more effective policy exploration during RL training. Built on top of GRPO, it introduces hybrid rollout strategy and a noise annealing schedule. These additions require no extra training cost and preserve the original RL objective. This design offers several benefits:
Robust reasoning: Positive trajectories' from distorted inputs offer alternative, and potentially more robust reasoning paths, improving generalization to challenging or out-of-domain visual conditions. Contrastive perceptual signals: When clean and distorted inputs yield divergent outcomes for the same text query, the resulting reward differences shape better perceptual exploration space, serving as implicit contrastive signals that refine the model's perceptual behaviors during reasoning.
Stable training dynamics for better exploitation: The noise annealing schedule enables smooth transition from carly-stage noisy signals fully on-policy learning, mitigating distributional mismatch and ensuring stable convergence as the model gradually improves its perception and reasoning. This provides solid foundation for further exploitation in the later stages of RL training.
We regard trajectory positive if it receives reward of 1.

---

Table Performance comparison of VLMs with moderate parameter sizes on a suite of out-of-domain benchmarks. Accuracy scores (%) are reported for all benchmarks for clarity. Models marked with "e" are evaluated using our evaluation suite. For R models, the corresponding reason in templates are used by default, while " indicates results obtained using the di rirrttansser template. Data sizes used for SFT and RI. are annotated in blue and red, respectively. The best value in each column is shown in bold, and the second- is underlined.
Model
Data Size MathVers.
Matovision MathVista WeMath HallusionBench
Open-sounce
Intem
39.5
19.7
64.4
67.31
LLLVVAOOne LL.VV--OnVision-P7 [35]
26.2
63.2
48.41
Kini-VL-16B [29]
44.9
21.4
68.7
66.21
URSA-8B [50]
45.7
26.2
59.8
Mulberry-BB [84]
63.1
Rl-related (reinforcement learning with verifiable reward)
RI-VL-7B [92]
260K+10K
24.7
63.5
Vision-RI-7B [23]
200K+10K
52.4
73.5
RI-OneVision-7B: [83]
10KK+10K
46.1
22.5
63.9
62.1
65.6
OpenVL.Thinker- [13]
35K+15K
48.0
25.0
71.5
67.8
70.8
MM-EurckasQOwe--7B [53]
15K
50.5
28.3
71.5
65.5
ADORA-7B* [20]
2.1K
50.1
27.6
71.1
67.1
53.1
ThinkL.te-7B-VL- [77]
11K
50.2
27.6
72.7
69.2
71.0
VLAA--Thinke--71 [5]
25K
49.9
26.9
68.8
67.9
68.6
Qwen2.5-VL-P7 Bwen2.5-VL-7B-Insttuett [4]
46,2
25.0
67.5
63.1
64.6 (71.2))
Vanilla GRPO (n 12)
2.1K (Geometry3K)
50.8
27.3
70.5
67.4
NoisyRollout"
2.1K (Geometry3K)
53.2
28.5
72.6
69.6
72.1
Vamilla GRPO* (n 12)
6.4K (MMK12)
51.8
29.4
73.2
70.2
70.3
NoisyRollout" (n] 6, ng
6)
6.4K (MMK12)
53.0
30.6
74.5
70.3
72.2
Table 2: Performance comparison of VLMs with large parameter sizes suite of out-of of-mmnn benchmarks. The notation and evaluation protocols are consistent with those described in Table
Model
#Data MathVerse Matonnsion MathVista WeMath HallusionBench
Flose-source
GPT-4o [24]
30.4
63.8
69.0
71.4!
38.0
67.7
71.6!
Kimil.5 [28]
38.6
74.9
Open-source
Intern VL.2.7. Instruct WVQ-72B-Preeveew [58]
$1.7
32.2
72.3
72.9t
35.9
71.4
Qwen2:5-VL-72B-Insaeee
38.1
74.8
71.91
RI-related (RL-tuned with verifiable reward)
MM-Eureka-Zenn-881 [53]
9.4K
48.9
26.6
64.2
MM-Earcka-Qwwe--325 [53]
17K
56,5
39.8
76.7
76.7
71.4
Owen2.5-VL-322B-B1 [4]
58.5
37.6
76.5
74.0
66.6
Vanilla GRPO" (n
2.1K (Geometry3K)
39.2
76.1
72.3
NoisyRollout" 11g
2.1K (Geometry3K)
58.9
39.9
77.8
77.2
73.5
Vanilla GRPO* (n = 88)
6.4K (MMK 12)
58.9
40.0
76.7
76.9
72.1
NoisyRollout" (m] 4, ng 4)
6.4K (MMK12)
59.3
41.6
77.4
77.6
73.2
Experiments
Dataset. use EasyRI [96] as our reinforcement learning training framework, which built on verl [65] and specifically designed for VLMs. Our experiments utilize two datascts: Geometry 3K [47], focused on geometric problem solving, and MMK [53], covering diverse K-12 math topics. These datasets comprise 2.1K and 6.4K training samples respectively. We processed them by converting all questions from multiple-choice to free-form format to prevent reward hacking and model guessing.
Evaluation. We mainly evaluate model performance along two dimensions. First, we assess out- of-domain generalization across five benchmarks: four visual reasoning benchmarks, including MathVerse [93], Math Vision [74], Vista [48], and WeMath [57], as well one visual perception
5

---

benchmark, HallusionBench [19]. Second, we evaluate the in-domain performance of NoisyRollout by comparing it with the vanilla GRPO baseline on the Geometry3K test set.
Moreover, we develop an evaluation suite for consistent assessment of our trained checkpoints and most open-source R1- 1--related checkpoints using vLLMM [31] for accelerated inference (marked with in Tables and 2), while adopting reported results for others. 2 We employ greedy decoding for model inference and use Gemini-2.0-Flashh-00 [17] as the judge model to parse generated responses.
Implementation details. Following prior work [53, 77], we initialize our policy models with which exhibit strong foundational capabilities well-suited for subsequent RL training. All experiments are conducted using 8 A100 GPÜs (40G for 7B model, 80G for 32B model). We keep the vision encoder frozen for training stability and parameter efficiency. For other general RI--related hyperparameters, we adopt the default settings from EasyR I: a global batch size of 128, rollout batch size of 512, rollout temperature of 1.0, and learning rate of 1e-6. To prevent token-length bias, we compute the policy loss using the token -men aggregation strategy." For coiisyoollutsspeciifi configurations, we adopt Gaussian noise as the default image distortion strategy, and apply a
annealing schedule:
1Ie--M(-ii/aa (1-T+C-MC-nT
a2 = () t, maax ao
(3)
where determines the midpoint of the annealing curve and À controls its steepness. Figure 7 illustrates the visual effects of applying different levels of Gaussian noise to clean image. We defer the discussion of unsuccessful image distortion strategies (e.g., cropping), noise annealing strategies (e.g., power, exponential), and proportions of noisy rollouts in total rollouts to Appendix A. The reasoning and direct-
templates used in experiments are shown in Appendix H. Additional implementation details regarding the number of training steps/epochs and the hyperparameters for image distortion and noise annealing presented in Appendix J.
3.1 Main Results
Result 1: Out-of-domain generalization. When trained on the 3K dataset using Qwen2.5- VL-7B- Instrutt, NoisyRollout not only improves in-domain performance (Figure 3, lower left sub- plot), but more importantly, demonstrates strong out-of-domain generalization. As shown Table 1, NoisyRollout achieves superior performance across five visual reasoning and perception benchmarks, consistently outprforming the vanilla GRPO baseline in every case. This advantage is further illustrated in Figure 3, which presents detailed comparisons across benchmarks as training progresses. Specifically, NoisyRollout achieves 53.2% on Math Verse, 28.5% on Math Vision, and 69.6% on WeMath, surpassing existing R1-related baselines and even outperforming GPT-4o.
Moreover, while Qwen2.5-7B 2-B.5--BB-VL-Intrrre -nsstrutst''s perception accuracy on HallusionBench drops from 71.2% to 64.6% when switching from di achieves 72.1% with the reasoning prompt (compared to vanilla GRPO's 's 69.8%). The final subplot in Figure 3 further confirms that NoisyRollout enhances perception quality during reasoning. achieving higher Bradley-Terry win rate over vanilla GRPO (See Appendix C for details). These results indicate that our hybrid rollout strategy enhances visual perception by promoting better policy exploration through vision-oriented inductive biases.
to reasoning templates," NoisyRollout
Result 2: Sample efficiency. NoisyRollout demonstrates exceptional data efficiency by generalizing with only 2.1K training samples from Geometry3K, whereas comparable models require signifi- cantly more data or even additional SFT as warm-up training. For example, Table indicates that OpenVVTThin VLThinker--P 7B needs 35K SFT samples and 15K RL samples but reaches only 48.0% on Math- Verse and 71.5% on Math Vista. This efficiency stems from NoisyRollout's use of noisy training sig- nals that foster targeted exploration during RI., enabling effective generalization from limited samples.
Robustness across training datasets and model sizes. NoisyRollout consistently im- proves upon vanilla GRPO, demonstrating strong robustness across model sizes and training datasets. As shown in Tables and 2, the 7B model trained on MMK12 12 achieves gains of 1.2%, 1.3%, and
Result 3:
Whiile we closely follow system (or format) prompts from relevant codebases or papers, minor result discrepancies may occur due to differences in judge models inference engines, which we consider acceptable.
"The code implementation can be found at verl.
"This degradation caused by the reasoning template has also been observed in previous studies [6. 27J.

---

Training Reward
MathVerse
Vishvision
HalluzienBench
12.3
0870
6.60
282
12.3
24
12.0
Accumcc
Accuracy
0.42
6.40
0.20
282-
0003
0.30
20
60
60
252
40
20
40
Accuracy
MisthViste
Perception Quality
0.57
6.30
72.0-
13.0
Seddeyyeery Win Rate
0.35
64C
0.0
In-Domain (Geometry3K)
Out-of-Domain (Reasoning)
Out-of-Domain (Percoption)
across in- domain and out-of-domain scenarios with the same total rollout number (12). The X-axis in all subplots represents RL training steps. First column: Reward comparison on the in-domain dataset during training. Second and third columns: Comparison on four out-of-domain visual reasoning benchmarks. Last column: Evaluation of visual perception capabilities, where the upper subplot directly compares their perception performance on HallusionBench and the lower subplot presents the model-ranked Bradley-Terry win rates w.r.f. the perception qualities of their reasoning traces.
Figure 3: Comparison of NoisyRollout and vanilla GRPO on Qwen2.5-VL-7B-Im
1.9% over GRPO on MathVerse Math Vista, and HallusionBench, respectively. Similarly, the 32B model trained on MMK12 surpasses GRPO by 0.4%, 0. 0.7%, and 1.1% on the same benchmarks.
Notably, in certain benchmarks like Math Verse and Math Vista, the performance gains of Noisy Rollout vanilla GRPO are smaller for the 32B model than for the 7B model. This is likely because the 32B model's initial policy was already fine-tuned via RI, whereas the 7B model's was not.
3.2 Ablation Study: More Effective Rollout Diversity with Noisy Trajectories
Setup. Unless otherwise specified, all ablation studies in this and the following subsection use Geometry3K as the training dataset on -e.-..--VV-B--nnsneeee In this part, we aim to examine the effectiveness of our NoisyRollout from the perspective of rollout diversity, key factor for effective policy exploration in RL training. Here, we define rollout diversity as the average pairwise cosine distance between trajectory embeddings, where higher values indicate greater diversity. We randomly sample 256 instances from the Geometry3K training set. For each sample, we generate either 12 trajectories in vanilla GRPO or a combination of nı = 6 and n2 = 6 trajectories NoisyRollout, then encode them with embedding model. We track both diversity and accuracy across training steps (Figure 4) and evaluate final performance on in-domain and out-of-domain benchmarks (Table 3). We use vanilla GRPO with rollout temperature of 1.0 as the control group.
Result. As shown in Figure 4, NoisyRollout enhances rollout diversity in early training stages compared to the control group, similar to increasing rollout temperature in vanilla GRPO from 1.0 to 1.2. This initial diversity boost, though accompanied by lower starting accuracy, ultimately leads to higher final training accuracy. Moreover, both NoisyRollout and higher-temperature vanilla GRPO show diversity decreasing below the control group in later training stages.
Table 3 reveals that NoisyRollour with temperature 1.0 consistently outperforms vanilla GRPO across all temperature settings (0.8 to 1.4), as well as mixed- texeeeetemerrturee variants. Moreover, when applying temperature 1.2 to both upproaches, NoisyRollout still demonstrates significant improvement over vanilla GRPO. These results indicate that NoisyRollout introduces more targeted and effective diver- sity than simply adjusting temperature parameters, which increases diversity in less focused manner.
'https://qwen git.. io/bl og/ http//wen22..22 - wttp://........e.eeeeeeeeeeeeeee sentenceeeeeeeeeeeeeeeeeeee contence-transtornee transtrrrnssseeeeeeeeo aal 1-------iiiii Mi--1M-6--1222
co/
7

---

Table 3: Performance comparison under different rollout temperature settings, with the total number of rollouts fixed at 12. In vanilla GRPO, "nx66 1.0, n(6) 1.2" indicates 6 rollouts with temperature 1.0 and another with temperature 1.2. In NoisyRollout, "n (6) 1.0" denotes 6 rollouts per sample generated from clean input (T, q with temperature 1.0, while (22 1.0" denotes 6 rollouts per sample from noisy input (I, q) with temperature 1.0. "Geo3K" represents the test set of Geometry3K dataset. "Avg." represents average accuracy (%) across six benchmarks.
Method
Rollout Temperature
Geo3K MathVerse MathVision Math WeMath HallusionBench Avg.
n(12) 0.8 50.1
50.5
26.7
69.9
65.8
70.1
55.5
n(12)
51.4
50.8
27.3
70,5
67.4
69.8
56.2
n(12) 1.1
50.4
50.2
27.7
70.4
68.1
69.4
56.0
Vanilla GRPO
n(12) 1.2
53.2
51.2
27.1
69.3
68.3
70,9
56.7
n(12) 11.4
51.4
50.6
25.8
70.1
69.0
69.6
56.
m(6) : 1.0, a(0) 1.2
50.8
50.7
26.8
70.1
67.4
68.2
55.7
NoisyRollout Ri (6) 1.0, (6) : 1.0
54,9
53,2
28.5
72.6
69.6
72.1
58.3
(6) : 1.2, (6( 1.2
53.4
52.6
28.3
72.9
70.9
70.9
58.2
Training Accuracy (%)
Avg. Rollout Diversity (%)
Training Accuracy (%)
Avg. Rollout Diversity (%)
Nobts lout (tmmp114))
SAPO (tamp-i Ion 1a GRRO drmpp1 @o GRFO 0emrp-1 20
Sove la GRFG Demp-110
30.0
NoisyRollout v.s. Vanilla GRPO
Vanilla GRPO with Different Temperatures
Figure 4: Comparison of accuracy and diversity metrics (%) across RL training steps (0 40). The left two subfigures contrast NoisyRollout versus vanilla GRPO (both with temperature 1.0), while the right two demonstrate the effects of different temperature settings (0.8, 1.0, 1.2) vanilla GRPO.
3.3 Ablation Study: Impact of Hyperparameters and Module Design
Noise annealing. As shown in Figure 5, removing noise annealing causes the in-domain performance of method drop sharply around
Table 4: Ablation study on the noise annealing strategy.
Method
Geometry3K 0OD Avg.
Qwen2.5-VL S--.---V--BB--ttrru
39.4
53.3
training step 45. This drop is due to divergence caused by distributional mismatch-aan an issue discussed in Sec- + NoisyRollout
+ Vanilla GRPO
51.4
57.2
+ NoisyRollout w.o. Noise Annealing
43.9
58.0
54.9
59.2
tion 2 and further illustrated by the training dynamics in the same figure. Additionally, Table 4 shows that disabling noise annealing leads to lower performance in both in-domain and out-of-domain settings (43.9% and 58.0%. respectively), compared our standard setting with noise annealing (54.9% and 59. 99).%)..o..... These results further highlight the effectiveness of noise annealing.
Table 5: Empirical validation with additional augmentation types.
Method GRPO
Augmentation Math Verse MathVision MathVista
WeMath HallusionBench
Average
None
50.8
27.3
70.5
67.4
69.8
57.2
NoisyRollout
Gaussian noise 53.2 (+2.4)
28.5 (+1.2)
72.6 (+2.1) 69.6
(+2.2)
72.1 (+2.3)
59.2 (+2.0)
NoisyRollout
Rotation
52.5 (+1.7)
28.1 (40.8)
71.9 (+1.4)
68.1 (40.7)
70.2 (+0.4)
58.2 (+1.0)
Image augmentation type. To validate that the benefits of NoisyRollout are not limited single type of distortion, we evaluate its performance using rotation as representative geometric transformation, in addition to our default Gaussian noise. The results, presented in Table show that both augmentation strategies outperform the vanilla GRPO baseline. Specifically, using rotation boosts the average score from 57.2% to 58.2%. While Gaussian noise yields superior results with an average score of 59.2%, the meaningful gains from rotation demonstrate the generalizability of our approach. This suggests that the core mechanism of NoisyRollout enhanccing policy exploration through diverse visual inputs is robust and effective across different augmentation techniques.
8

---

Policy Clip Fraction (%)
Training Reward
Test Accuracy Reward
Varila GRPO
horoyaalbur Noke fenesing
0.50
0.54
Nooyyollou
0.5
0.48
1.0
0.45
0.40
0.5
030
10
20
20
50
60
10
20
30
40
30
50
10
20
30
30 00
Training Steps
Figure 5: Comparison of NoisyRollout w: and w.o. noise annealing, and vanilla GRPO in terms of training dynamics (policy clip fraction and training reward) and accuracy on the in-domain test
Out-of- Domain Average Performance (%)
Table 6: Ablation study on the impact of initial noise steps (og). "OOD Avg." represents average accuracy (%) across five out-of-domain benchmarks.
Venill GRPO
60.0
59.4
59.0 0-
50.6
Noise Step Geometry3K OOD Avg.
58.0
$7,9
0
51.4
57.2
$7.0
100
52.7
57.4
56.0
300
53.4
57.7
56.1
400
54.6
58.1
1.1K
2.1K
4.1K
6.4K
Data Size
500
54.9
59.2
Figure 6: Performance comparison on MMK12
550
39.6
57.7
when scaling up the training data size.
600
Diverged
Data scale. Although Geometry3K is high-quality training dataset, its limited size (2.1K samples) prevents a thorough investigation of the scaling behavior of NoisyRollout compared to vanilla GRPO. To enable such analysis, additionally consider MMK12, which contains 6.4K samples after pre- processing. Figure shows NoisyRollout consistently outperforms vanilla GRPO across various data scales, ranging from 1.1K to 6.4K. Notably, the performance gains do diminish as the dataset size increases, suggesting that NoisyRollout has strong potential for use in large-scale training regimes.
Initial noise step. We evaluate the impact of noise strength by varying the initial Gaussian noise step, shown in Table 6. Gradually increasing the initial noise step og from to 500 consistently improves performance across all evaluation categories, suggest- ing that moderate noise promotes exploration and enriches the training signal. However, exceeding this threshold leads to performance degradation, as overly distorted images (see Figure 7) yield noisy rollouts with average near-zero rewards. These excessively noisy samples introduce harmful distribution shifts during policy updates, ultimately destabilizing the learning process. Additional ablation results on the MMK12 dataset are deferred to Appendix A.?
Original
Noise Step: 100
Noise Step: 300
15x
15
H
Naise Step:
Noise Step: 500
Noise Step: 600
Figure 7: Illustration of visual degradation under increasing Gaussian noise steps.
GRPO variant. Recently, several variants have been proposed to enhance the original GRPO im- plementation. Specifically, Liu et al. [45] identified question-level difficulty bias and proposed removing the standard deviation normalization (std(r)) address this issue. In addition, Yu et al. [87] increased the upper clipping threshold (etigh) to mitigate entropy collapse. As shown in Table applying NoisyRollout consistently improves performance not only on the original GRPO imple- mentation but also across these variants. This highlights that NoisyRollout provides complementary benefits alongside aprimization-focusee modifications, underscoring its broad applicability.
We also include additional ablations (e.g., number of rollouts and data seed variations) in Appendix A.

---

Gradient Projaction Ratio (%)
10
1332
Clean
Table Performance comparison when using GRPO variants for policy optimization.
Method
Geometry3K OOD Avg.
Qw Instrrucc
39.4
53.3
GRPO (w.o. std(r))
51.3
57.0
NoisyRollout (w.o. std(r))
56.1
58.9
Training
GRPO (Chigh 0.28)
52.6
58.2
Figure 8: Comparison of gradient projection
NoisyRollout (Chigh 0.28)
53.9
59.6
ratio over training.
3.4 Further Analysis: Quantitative Contribution of Noisy Rollouts on RL Optimization
Setup. For each training sample, we partition the collected rollouts into Clean and Noisy sub- groups, containing n and n2 rollouts, respectively. We measure each subgroup's contribution by projecting its specific effective gradients onto an anchor gradient g' Br+ 0', which represents the overall model update over At optimization steps, beginning at training step The subgroup effective gradients, Belean and Sioisyy are derived from actual optimization steps starting from 6', using only rollouts from the respective subgroup (by masking losses from the other subgroup). The projection ratios are then calculated rakann (gelaa g'g and yut noisy (goossy g')/llg' I|?.
These ratios provide a quantitative estimate of each subgroup's 's contribution to the overall model update g'. More details are included in Appendix I.
Result. Figure 8 shows that the Noisy subgroup consistently contributes more significantly to policy optimization compared to Clean, especially during early training phases when distortion strength a is high and the policy Te still struggles with visual understanding. This trend gradually diminishes towards the final stages of training as the learning is gradually "on- con-policy". These findings quantitatively confirm that our method effectively leverages noisy rollouts enhance training signals.
4 Related Work
VLMs have rapidly advanced through integrating vision encoders [59, 91] with large language mod- els [2, 36, 41, 42, 24, 17], with specialized efforts in reasoning tasks [66, 94]. Reinforcement learning training in LLMs and VLMs. initially employed for alignment via human feedback (RLHF) [54, 1, 88]. has evolved to incorporate rule-based rewards and advanced optimization methods like GRPO [64]. as exemplified by DeepSeek-RP [21] and Kimi- 1.5 [28]. Emerging RI. approaches in multimodal do- mains include LMM-R1 [56], Vision-RI [23], R1-V [7]. Open Vl.nnnnThinnke [13], and MMM-ureea [53], which extend RL to visual reasoning tasks. However, existing studies on training VLMs via RL have not adequately explored techniques that can enhance the explorative capabilities of models. Our method addresses this gap by proposing a data augmentation technique with visual-oriented inductive biases. A detailed discussion of related work is deferred to the Appendix D due to space limit.
5 Conclusion
In this paper, we investigate scaling test-time compute in VLMs via RL. We introduce NoisyRollout, a simple yet effective data augmentation technique that promotes diversity by mixing trajectories from both clean and distorted inputs with vision-oriented inductive biases. This approach enhances policy exploration during RL training without incurring additional training costs. Empirically, NoisyRollout demonstrates improved generalization and robustness, achieving ofate-of-the-ar performance across multiple visual reasoning and perception benchmarks with high sample efficiency.
Acknowledgement
This project was partially supported by the Singapore Ministry of Education Academic Research Fund Tier (Award Number: T1 251RES2514) and TPU Research Cloud. Additional computational support provided by Sea AI Lab.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
[2] Jean-Baptin Baan-Bapistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:
23716-23736, 2022.
[3] Anthropic. Claude 3.5 sonnet. http::://www..athrrr
com, 2024.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint MxXi::250211922. 2025.
[5] Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training rl rl-like reasoning large vision-language models, https: /// /gi thub com/ UCSC VLAA/VLAA Thinking, 2025.
[6] Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang. Kai Wang. Yubo Wang, Yuan- sheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410. 2024.
[7] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. RI-v: Reinforeing super gen- eralization ability in vision-language models with less than $3. https: / / tru...u....... com/ Deep-Agent/ R1-V,, 2025. Accessed: 2025-02-02.
[8] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. /78/1, 2025.
[9] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
[10] Chris, Yichen Wei, Yi Peng, Xiaokun Wang. Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork rlv2: Multimodal hybrid reinforcement learning for reasoning, 2025. URL https://arxix: orq/ abg//bs// 2504. 1144..666 16655.
[11] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URL https: //arxiv. .rg/ abs 22022 01456.
[12] Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint arXiv:2503.0700
2025.
[13] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vlthinker: early exploration complex vasion-language reasoning via iterative self- improvement, 2025. URL https: /aaxivv... org/abs/ 2
[14) Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint vrXiv:2411.144.. 2024.
[15] Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn, 2023. URL https: //taaiivvv.//////////////////aaa org.oo////ao.. abw/abaaaae 22006 08640.

---

[16] Yuying Ge, Sijie Zhao. Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi muati-granullrity comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
[17] Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[18] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https: / / /aaxivrvvvvaa/aa//eeaaaaaaa htrgg//b//2a0 /2403 05530.
[19] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14375-14385, 2024.
[20] Lujun Gui and Qingnan Ren. Training reasoning model with dynamic advantage estimation on reinforcement learning. https://githai: com/ http://aacccccaaaaaaaee. 2025. Notion Blog.
[21] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948. 2025.
[22] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. arXiv preprini 1. 2020.
[23] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06744 2025.
[24] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[25] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai of system card. arXiv preprint arXiv:2412.16770 2024.
[26] Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo Shilin Yan, Pheng- Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.0077
2025.
[27] Dongzii Jiang, Renrui Zhang. Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking thain-of-thouglee in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint 25X2:22502.062.1 2025.
[28] Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL https:
//axxiv. org/obgga/.......oooo... /etp//25...... 12599.
[29] Kimi Team. Kimi-Ve VL technical report, 2025. URL https: UtL.ttpss///aaxivccrr////// 07491.
[30] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[31] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[32] Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforce- ment learning for generalizable medical reasoning in llsion-language models. arXiv preprint 1533, 2025.
12

---

[33] Misha Laskin. Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systeess 33:19884-19892 2020.
[34] Bo Li, Kaichen Zhang. Hao Zhang, Dong Guo, Renrui Zhang. Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal ca- pabilities in the wild, May 2024. URL peatps://llaaaaaa weseaavvllgitttiii github. 1o/blog/ 2024-05- 10- 10100-100 ----laav--exx-strrrgeeee-- stt.sstr-oog/s////////
[35] Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
[36] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19742. PMLR, 2023.
[37] Weiqi Li, Xuanyu Zhang, Shijie Zhao, Yabin Zhang, Junlin Li, Li Zhang, and Jian Zhang. Q-insight: Understanding image quality via visual reinforcement learning. arXiv preprint vrXiv:2503.22679 2025.
[38] Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal llms with mixture of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[39] Yuting Li, Lai Wei, Kaipeng Zheng, Jingyuan Huang, Guilin Li, Bo Wang. Linghe Kong, Lichao Sun, and Weiran Huang. Revisiting visual understanding in multimodal reasoning through lens of image perturbation, 2025. URL https: : / waaaaiivvv....... org. brg//bs// 25000 09736.
[40] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint 2xXi:22402.00255: 2024.
[41] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[42] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 36296-263066 2024.
[43] Xiangyan Liu, Rongxue Li, Wei Ji, and Tao Lin. Towards robust multi-modal reasoning via model selection, 2024. URL pettss//arrxe org/ htt//bb//2311 2310. 0844.
[44] Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint 2:Xi:25033005200 2025.
[45] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20788 2025.
[46] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
[47] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2 100.04465, 2021.
[48] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
13

---

[49] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprini 2:Xiv:2503.21220) 2025.
[50] Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics. arXiv preprinl urXiv:2501.04665 2025.
[51] Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, and Pengfei Liu. Rethinking rl scaling for vision language models: A transparent, from- framework and comprehensive evaluation scheme. arXiv preprint
2025.
[52] Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang. Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, and Chong Ruan. Janustlow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation, 2024.
[53] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rulc-based large-scale reinforcement learning. arXiv preprint arXiv:2503.0736. 2025.
[54] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[55] Yi Peng, Chris, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, Rongxian Zhuang, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork rlv: Pioneering multimodal reasoning with chain-of-thought, 2025. URL https: / //t.x/xxvvv////////v/////////vvv org/ abs/ /2004. 05599.
[56] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprini arXivv2503.0755: 2025.
[57] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GungQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01288 2024.
[58] Qwen. Qvq: To see the world with wisdom. https:/ http://w/wemmmmmmmm hwwpeellm.ithu..i..... qvq 72b preview/ 2024.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 7748-8765. PMLR, 2021.
[60] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402-5415 2021.
[61] Fereshteh Sadeghi and Sergey Levine. CAD'RL: Real single-image flight without a single real image. In Robotics: Science and Systems, 2017.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: / wtwpiaivvvvo//////aaaaaaaaaa..aa wwwp:iivorr//..oo abs/ /ttp://7770044444
[63] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.055929, 2020.
[64] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 4rX.v:2402.03332 03300, 2024.
14

---

[65] Guangming Sheng. Chi Zhang, Zilingfeng Ye. Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[66] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- Kiong Ng, Lidong Bing, and Roy Ka-Wei Lec. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406. .00...77999 77244, 2024.
[67] Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: comprehensive survey, Artificial Intelligence Review, pp. 945-990, 2022.
[68] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020.
[69] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 240508840,, 2024.
[70] Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, and Salman Khan. Llamav-ol: Rethinking step-by-step visual reasoning in Ilms, 2025. URL https///axxvvvorreee 06186. 66.
[71] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
[72] Haonan Wang, Chao Du, and Tianyu Pang. V1: Toward multimodal reasoning by designing auxiliary tasks, 2025. URL https:ttt//// www.-liiidooorrooiin.... notiiooooonnnnninniiii.oio.ooooo notion. s site.
[73] Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504. 11455, 2025.
[74] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoming with math-vision dataset. Advances Neural Information Processing Systems, 37:95095-995 95155995669111 2024.
[75] Weiyun Wang. Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization, 2025. URL https:
/ //arxiv. orggabs//////eeooooooooooaoeaaar /22//24111 2211.1111 10442.
[76] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning, arXiv preprint arXiv:2503.1029) 2025.
[77] Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li. Chung Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mets-guided sample selection for data-efficient visual reasoning self-improvement, 2025. URL https: :/ //tp://aaav./////////eee/aaaa.la wwwxiv.org/// org abb//
2504.00 07934.
[78] Haoran Wei, Youyang Yin, Yumeng Li, Jia Wang, Liang Zhao, Jianjian Sun, Zheng Ge, Xiangyu Zhang, and Daxin Jiang. Slow perception: Let's perceive geometric figures step-by-step. arXiv preprint arXiv:2412.206311 2024.
[79] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint 2xXiv:2410.13845 2024.
15

---

[80] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixture-of efxeersss vision- language models for advanced multimodal understanding. arXiv preprint 24. 2024.
[81] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single trans- former to unity multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[82] Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning reason under off-policy guidance, 2025. URL https: http///axvvvvo///e org/ abs/ 2504. 14945.
[83] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503 10615, 2025.
[84] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang. Shunyu Liu, Yingjic Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering milm with like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319 2024.
[85] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.08: 09645, 2021.
[86] En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, and Wenbing Tao. Perception-rl:: Pioneering perception policy with reinforcement learning, 2025. URL abss/504 07954.
Setps://arxiv org./..o.rr..ooooooooooooooooeo.
[87] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[88] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng. Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- mmniniaeeed correctional human feedback, 2024. URL https: / //arxiv. org/ abs /2122.222222 00849.
[89] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B-star: Monitoring and balancing exploration and exploitation in self-taught reasoners. arXiv preprint MxXi::2412.17256 2024.
[90] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URLhttge : / aaaiiviiivv.. wwtg/////////////// 18892.
[91] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986. 2023.
[92] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937 2025.
[93] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp- 169-186. Springer, 2024.
16

---

[94] Renrui Zhang. Xinyu Wei, Dongzii Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu. Aojun Zhou, Bin Wei, Shanghang Zhang, al. Mavis: Mathematical visual instruction tuning. arXiv e-prints, pp. arXiv-2407, 2024.
[95] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv preprint
2tth:22502.109911, 2025.
[96] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality training framework. https:// github. com/coo/////oo///o wtom/hiyyee youga / EasyR1, 2025.
[97] Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive up- ward multimodal alignment to enhance mathematical reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 26183-261949 2025.
[98] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li. and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404. 13046, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The introduction (Section 1) clearly outlines the motivation, problem setting, proposed method (NoisyRollous and the key contributions. These are consistent with the experimental results and scope discussed throughout the paper (Section 3).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of NoisyRollout are detailed Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The training details of NollyRo are included in Appendix J. The details of used datasets are included in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and running instructions are provided the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https:/ //ss:://iiiii cc/
pub / gu des Codeeubmm ss onPol cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cc/ public cibbicc/a /e/guiides des CodeS uumi i cesioooollii (cy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of training and evaluation are included in Section 3 and Appendix J.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: NoisyRollout requires large-scale reinforcement learning (RL) training. Simi- lar to previous related works [53, 13, 23, 83), we do not report error bars due to the high computational cost. However, in certain ablation and analytical experiments-whie putational demands are more manageable- perform multiple independent runs to ensure statistical significance (e.g.. Table 10 and Figure 8).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of compute resources are included Table 13.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /neuriipp iss.iip..oo httppplpcccccccccieeeeeeeepppeee
Answer:
[Yes]
Justification:
We have reviewed the Code of Ethics and it conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The details of broader impacts are included in Appendix E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model and data focus vision-language understanding and reasoning, with minimal risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The licenses are mentioned in Appendix K.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw swr thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and model are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components. LLMs are used solely for editing (e.g.. grammar, spelling, word choice) and data processingffiteringg.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: //neuriiippp// ipp wwt.onneeeence///eeccccccccacaee 2025/ LLM) for what should should not be described.
24

---

A Additional Ablation Studies
Noise annealing strategy. On the Geometry3K training dataset, further examine the impact of different noise annealing schedules on NoisyRollout's performance by comparing our default sigmoid strategy with power (a,, = ag -1-1/ (1-l/lmax) ). === 3.0) and exponential (a, = ag : / 0.98) decay functions. As shown in Table 8, all three strategies enable NoisyRollout to outperform vanilla GRPO baseline on benchmarks. Among them, the sigmoid schedule achieves the highest average score (58.5%), surpassing both power and exponential decays (57.1% and 57.0%). The superior performance the sigmoid schedule likely results from its characteristic "slow-fast-slow" decay, which balances exploration and stability more effectively by maintaining sufficient early exploration and promoting rapid convergence afterward.
Table Ablation study on the strategies of noise annealing. "Avg." denotes the average accuracy across the six benchmarks. Best value per column is bold, second best is underlined.
Method
Geo3K Math Verse Math Vision
Math Vista
WeMath HalluBench
Avg.
Qwen2.5-VL + GRPO
39.4
46.2
25.0
67.5
63.1
64.6
51.0
52.0
50.8
27.3
70.5
67.4
69.8
56.3
+ NoisyRollout (Pow.)
52.2
52.1
26.4
72.0
68.7
71.0
57.1
NoisyRollout (Exp.)
51.9
52.6
27.7
70.5
70.1
69.1
57.0
+ NoisyRollout (Sig.)
54.9
53.2
28.5
72.6
69.6
72.1
58.5
Total rollout number. We analyze the impact of total rollout number by comparing vanilla GRPO and NoisyRollout under varying rollout budgets. As shown in Table 9, increasing the number of rollouts vanilla GRPO from n to n = improves in-domain performance (from 49.6% to 54.7%), but only marginally benefits out-of-domain generalization (from 56.8% to 57.5%). NoisyRollout consistently outperforms vanilla GRPO even when the total number of rollouts is held constant. Notably, NoisyRollout with nı n2 6 (total 12) achieves both higher in-domain (54.9%) and out-of-domain (59.2%) accuracy than vanilla GRPO with 16 rollouts.
Table 9: Comparision of NoisyRollout and GRPO on 1reen2255VL-7B-Imeee across different rollout configurations. "OOD Avg." denotes the average accuracy across all five out-of-domain benchmarks.
Rollout
Geo3K
Math Verse
Math Vision
Math Vista
WeMath HalluBench OOD Avg.
49.6
50.2
27.3
68.8
69.7
56.8
12
52.0
50.8
27.3
70.5
67.4
69.8
57.2
54.7
51.3
27.5
71.4
68.0
69.3
57.5
20
53.6
50.8
27.2
70.0
68.5
69.9
57.3
112 4
51.7
51.8
27.4
72.6
69.5
70.3
58.3
M1 122 6
54.9
53.2
28.5
72.6
69.6
72.1
59.2
11 n2 = 8
54.7
52.6
28.7
72.0
69.1
72.0
58.9
Unsuccessful image data augmentation. In experiments, we explored two classic im- age augmentation techniques: Gaussian noise and rotation with expansion (expand=True) Both proved effective as they introduce percep- tual diversity while preserving all essential vi- sual information. In contrast, we also inves- tigated two augmentation strategies that were ultimately unsuccessful: cropping and rotation without expansion. The latter refers to rotating an image and then cropping it back to its orig- inal dimensions, which cuts off the corners of the rotated content. Both of these unsuccessful methods frequently resulted eritical informa-
0.6
Robarton
Emoppng
Gaussian Nose
Training Reward
0.5
0.4
10
30
40
50
60
Training Steps
Figure 9: Performance of Rotation without expan- sion and cropping.
tion loss, as key parts of a problem diagram were removed from the image. This led rollouts with
25

---

Table 10: Comparision NoisyRollout and GRPO on Qwen2.5-VL-7B Insstrucctt across data seeds.
Method
Seed
Geo3K
Math VathVerse
MathVista WeMath HalluBench OOD Avg.
GRPO
52.0
50.8
27.3
70.5
67.4
57.2
NoisyRollout
54.9
53.2
28.5
72.6
69.6
72.1
59.2
GRPO
51.9
51.5
27.8
70.7
68.1
69.4
57.5
2
NoisyRollout
54.1
53.6
26.6
72.9
69.7
71.4
58.8
GRPO
51.1
50.8
27.1
69.7
67.8
69.5
57.0
3
NoisyRollout
53.7
52.4
27.8
72.3
70.5
70.9
58.8
GRPO
51.9
50.5
27.2
71.3
68.6
68.6
57.2
42
NoisyRollout
54.1
52.7
27.3
72.2
69.6
70.3
58.4
Table 11: Performance of NoisyRollout on the MMK12 dataset (2.1K) across different noise steps.
Method
Initial Noise Step
MathVerse
MathVision
Math Vista WeMath HalluBench OOD Avg.
GRPO
0
50.7
28.5
71.7
68.6
69.8
57.9
300
51.4
28.4
72.5
69.5
70.5
58.5
400
50.3
28.2
71.8
70.3
70.8
58.3
NoisyRollout
450
52.8
28.9
72.9
71.9
70.7
59.4
500
50.2
27.4
73.2
71.4
71.3
58.7
consistently near-zero rewards, providing unreliable policy gradient estimates that caused training instability and eventual divergence. This distinction underscores key finding: for NoisyRollout to be effective, the augmentation must introduce meaningful perceptual variance without fundamentally compromising the integrity of the input data.
Data seed. To evaluate the robustness of our approach to data sampling variations, we examine the performance consistency of NoisyRollout when using different random seeds for data sampling during training on the Geometry3K dataset. We compare these results against vanilla GRPO trained with the same seeds verify that the improvements offered by NoisyRollout consistent across different data orderings and not merely an artifact specific seed (Table 10).
Initial noise step (MMK12). Based on our experience, the initial noise step og is critical hyperpa- rameter. In addition to the Geometry3K dataset, we also evaluate the impact of different initial noise steps on the MMK12 dataset using Dwen2.5-VL--7B-Inse
Table 11 shows NoisyRollout outper- forms standard GRPO on MMK12 when configured with an appropriate initial noise step (ag = 450).
Proportion of noisy rollouts. Table 12 demonstrates that incorporating noisy rollouts during training significantly enhances model performance across both reasoning and perception benchmarks. balanced 50/50 distribution of clean and noisy rollouts achieves optimal results (58.5% average accuracy), outperforming both the no-noise baseline (56.3%) and higher noise proportions. This finding aligns with our earlier observations about the effectiveness of noise regularizer, confirming that the ideal approach combines clean rollouts exploitation of current state with noisy rollouts for exploration, rather than relying exclusively either strategy.
Table 12: Comparison across different proportions of noisy rollouts n2/(nu, (m n22 during rollout collection, with a fixed total of 12 rollouts.
Proportion
Geo3K
Math Verse
Math Vision
Math Vista
WeMath HalluBench
Avg.
0/12
52.0
50.8
27.3
70.5
67.4
69.8
56.3
3/12
51.1
52.8
28.5
71.8
69.8
70.6
57.4
6/12
54.9
53.2
28.5
72.6
69.6
72.1
58.5
9/12
54.2
52.
27.3
72.3
71.1
72.3
57.9
12/12
53.1
52.8
28.5
72.3
69.1
71.1
57.8
26

---

B Unsuccessful Attempts
During the early development NoisyRollout, we encountered several unsuccessful trials that are worth documenting. Due to limited computational resources, we could only explore a limited set of hyperparameter combinations heuristically. Some of these approaches might prove effective with further hyperparameter optimization and better design.
Optimizing noisy and clean trajectories corresponding inputs. We explored alternative design in which policy updates were conditioned on the same input used to generate each rollout-i.e. using clean inputs for clean rollouts and distorted inputs for noisy rollouts. However, this approach did not yield meaningful improvements over the original GRPO baseline. We hypothesize that this design reduces the benefits group-based advantage estimation. Specifically, by decoupling clean and noisy rollouts during optimization, the method effectively degrades into a form of sample-level data augmentation. This fragmentation weakens the shared reward signal across rollouts, thereby diminishing the informativeness of group-level statistics such as the normalized advantage. As result, the exploration benefits introduced by noisy rollouts are not fully leveraged during policy updates.
In contrast, our proposed approach treats clean and noisy rollouts as a unified group for advantage calculation, while anchoring all policy optimization to the clean inputs. This design retains the distributional diversity introduced by noise, but preserves a consistent input distribution for policy strkins-striking a balance between exploration and stable learning.
Reward penalty on noisy subgroup. We experimented with applying explicit reward penalties - to all noisy rollouts, aiming to encourage the model to better capture contrastive learning signals. However, this approach quickly led to training divergence. Rather than improving its core reasoning and perception abilities, the policy model learned to distinguish between clean and noisy rollouts. As a result, the noisy rollouts rapidly became highly "off-policy", since the model could easily identify. This distributional mismatch destabilized training and undermined the learning.
27

---

C Evaluating the Perception Quality of Reasoning Traces
Visual Information Extraction Prompt
Extract all visual perception and information recognition components from the following reasoning
trace.
{question Reasoning trace: (reasoning]
Original question:
Your task is to extract and summarize ONLY the parts that relate visual perception, information extraction, and understanding of visual elements from the image.
This includes:
1. Any measurements, dimensions, or numerical values extracted from the image
2. Description of visual elements like shapes, objects, positions, spatial relationships
3. Recognition of text, symbols, diagrams, or graphs from the image
4. Any visual features mentioned or used in the reasoning
Format your response with the tag:
<eisuul..rrccption>>> [Extracted visual information here] <<visual..prrcetion>>
Include ONLY visual perception elements, not mathematical reasoning that happens after the information is extracted. If there are clear visual perception elements, respond with "No clear visual perception elements identified."
Visual Perception Comparison Prompt
Compare the quality of visual perception between two models based on the image and the original question.
Original question: [question]
Visual perception from Model A: (visual A Visual perception from Model B: (visual B}
Your task is determine which model better captures and correctly extracts visual information from the image. Compare their visual perception quality based on:
Accuracy of visual information extraction (measurements, shapes, relationships)
Complete identification all relevant visual elements
Proper recognition of visual information required solve the problem
Score both models and determine the winner: Model A demonstrates significantly better perception than Model B, respond: <result>> if Model B demonstrates significantly better perception than Model A, respond: <result>- if both models show similar quality of visual perception, respond: <result><ie<</esult>>>>
Now:
1. identify what visual information required to solve this problem.
2. analyze how each model perceives this information.
3. provide your comparative judgment with specific reasons.
4. provide your <result>> tag with exactly A, B, or tie.
To further evaluate the perception quality of models trained with NoisyRollout and vanilla GRPO during reasoning, we perform paired comparison using strong VLM. We sample 300 reasoning traces from the evaluation logs the models performing visual reasoning on the Math Verse and Vista benchmarks, forming paired comparisons between NoisyRollout and vanilla GRPO.
To isolate visual perception, we extract only the visual components from each reasoning trace using specialized prompt, removing any influence from mathematical reasoning or final answers. To reduce potential position bias in the comparisons, each pair of traces is evaluated twice: once with the
*Specifically, Gemini- Flash- in this experiment.
28

---

NoisyRollout trace shown first and the vanilla GRPO trace second, and once with the order reversed. We combine the results using the Bradley-Terry model to compute win rates. This methodology offers a reliable measure focused specifically on visual perception quality during reasoning. The results are presented in Figure 3 (the 8th subfigure). The extraction and evaluation prompts are shown above.
D Detailed Related Work
Large Vision-Languan Models. VLMs have rapidly evolved to understand and reason with both visual and textual information [70]. These models combine visual encoders with large language models to enable comprehension and inference across modalities. Early VLMs like Flamingo [2] and BLIP-2 [36] established foundational integration techniques between vision and language components. The LLaVA series [41, 42, 35, 34] introduced effective visual instruction tuning methodologies that significantly advanced multimodal capabilities. For mathematical reasoning, specialized approaches [66, 94] have employed mathematical visual instruction tuning to enhance VLMs' abilities to interpret and solve mathematical problems in multimodal contexts.
Advanced VLMs including GPT-4o [24] and Gemini [17] have demonstrated unprecedented general visual understanding through massive pretraining. Mixture-of- Experts approaches in DeepSeek- VLL2 [80], Uni-MoE [38], and MoVA [98] improved computational efficiency by selectively activating specialized components based on input characteristics. Meanwhile, unified models like SEED-X [16], Chameleon [69], Show-o [81], and Janus series [79, 52, 8] integrated visual understanding and generation capabilities within single architectures. However, most existing VLMs still lack robust visual reasoning capabilities [14], especially for tasks requiring sophisticated analysis of visual information combined with complex reasoning [43, 75].
Reinforcement Baarning-Enhancee Visual Reasoning. RL has emerged as key methodology for enhancing the capabilities of LLMs and VLMs. Early research primarily focused on Reinforcement Learning from Human Feedback (RLHF) [54]. which aligned model outputs with human prefer- ences [1]. Recent advancements have further demonstrated that RL-based techniques can significantly enhance reasoning abilities. For instance, DeepSeek-RI [21] utilizes rule-based rewards combined with Group Relative Policy Optimization (GRPO) [64]. whereas Kimi- 1.5 [28] employs a variant of online policy mirror descent, both methods showing notable improvements in reasoning performance.
In the multimodal domain, research on leveraging RL to enhance VLMs' reasoning capabilities remains in early stages. Some approaches explore using generative reward models [95, 76] to enhance VLMs' general capability, but these typically require powerful closed-source models for training data generation. Recent work including LMM-RI [56]. Vision-RI R1 [23]. R1-V [7] and Open VLThinker [13] has applied RI-type RL to VLMs diverse specific subdomains like geometry problems and object counting tasks [55, 10, 12, 39, 37, 73, 26). Further more, pilot studies [53, 51] further extend large-scale rule-based RL. to broader multimodal mathematical reasoning, demonstrating significant performance gains without relying on in-domain training data.
Data Augmentation in Visual Reinforcement Learning. Data augmentation has played a central role in visual reinforcement learning (RL) from pixels by improving sample efficiency and gen- eralization without architectural changes. RAD (Reinforcement Learning with Augmented Data) first demonstrated that simple image transformations. random crop, translation, color jitter, and cutout- can substantially enhance policy learning, particularly in DMControl and ProcGen bench- marks [33]. CURL combined off-policy RL with contrastive learning on augmented views, improving representation quality and sample efficiency [68]. DrQ regularized Q-value targets and policy updates via random shifts, achieving stable gains with minimal computational overhead [30]; DrQ-v2 further refined training schedules, exploration, and multi-step targets to master high-dimensional control from pixels [85].
Beyond these basclines, several extensions explored stronger invariance and generalization. DrAC introduced actor -iitiii regularizers that promote consistency under augmentation and automated augmentation search for diverse environments [60]. SVEA mitigated instability under heavy aug- mentations by mixing augmented and clean views to reduce target variance [22]. Self-Predictive Representations (SPR) encouraged temporal consistency between augmented latent states to improve representation quality and low-data efficiency [63]. In the sim-to-real context, domain randomization perturbs textures, lighting, and geometry to improve transferability from simulation to real-world environments [71, 61].
29

---

While these studies have established augmentation key ingredient for robust visual control, most methods optimize control policies rather than multimodal reasoning. Current augmentation pipelines are designed to stabilize perception and dynamics modeling but rarely connect augmented visual representations higher-level symbolic or linguistic reasoning. Bridging this gap will likely require augmentations that preserve both control-relevant and semantic information, enabling joint optimization for perception, reasoning, and alignment in future multimodal agents.
Broader Impacts
Our proposed method, NoisyRollout, introduces simple yet powerful data augmentation approach designed to improve visual reasoning and perceptual robustness in VLMs. Given its effectiveness, especially noted through strong out-of-domain performance and high sample efficiency, this approach has broad applicability within resource-constrained training scenarios. This is particularly beneficial in domains where acquiring annotating large-scale datasets is costly practically challenging. such as medical imaging [32], robotic perception [67], and assistive technologies [15].
Moreover, by enhancing model robustness visual conditions, method can also facilitate safer and more reliable deployment of VLMs real-world applications, potentially leading to more trust- method involves relatively simple augmentation
worthy human-AI interactions. Furthermore, steps without additional computational overhead or complex training protocols, along with strong performance scaling experiments shown in Table 2 and Figure 6, it is suitable for integration into existing large-scale scale training pipelines, supporting broader adoption both academia and industry.
30

---

Limitations
Despite easy-to-adopt designs and promising empirical results, our study has several limitations. First, due to computational constraints, our experiments are limited in scale: we primarily explore model sizes up 32B parameters and training dataset scales in the order of few thousand sam- ples. Future work should validate and extend our findings using significantly larger- segge--call training scenarios-such as models with 72B parameters or training datasets the range of hundreds of thousands of sample. Second, NoisyRollout is applied during the RL fine-tuning phase of an already pre-trained VLM. A more fundamental, how the principles of NoisyRollout (i.c., learning from noisy signals in RL) could be integrated into the large-scale pre-training phase the VLM itself. Finally, while empirically effective, our study lacks formal theoretical analysis of how NoisyRollout, with its specific hybrid trajectory mixing and noise annealing, affects the exploration-exploiatii trade-off and the convergence properties of the RL algorithm. It's unclear if the introduced noise guarantees broader state-space coverage in principled way or if certain noise characteristics could inadvertently hinder convergence.
vastly more complex, direction would to explore
G Case Study
We present two case studies to demonstrate the improved perception and reasoning capabilities of our NoisyRollout compared vanilla GRPO, illustrated in Figure 10 and Figure respectively.
Question: Is the line longer than the red line in the image?
Source: HallusionBench, set :d:2i figure_idss id:0 question_ids id:1
Perception Issue
NeisyRollout
<thinlo
To determine if the green line longer than the red line, let's analyze the image step step:
1. The green line consists of two segments that meet of point, forming V-shape with the segments extending outward.
2. The red line single streight segment with arrows both ends, indicating it extends both directions but does not any additional segments like green line.
By visually comparing the two lines: green
extended, appeors
some length as
considering length the
segments.
The red line single, continuous segment without any additionnal segments, and appears the green when considering total length.
Since both lines appear be the langth, the green line not longer than the ned line. </thinlo
\boxed(no)
Vonilla
<thinko
To determine if the green line longer than the red line, I need to their lengths visually. The green line oppears straight line segment
branches stroight segment with arrows at both ends, indicating it is bidinectional arrow but does not change its length.
from the middle, forming Y-shope. The line single
green extends further from which is a straight segment without any branches extending from it.
By companing
point, suggesting longer
Therefore, green longer </thinio
\boxedlyes)
Figure 10: Case study showing improved perception capability of NoisyRollout over vanilla GRPO.
31

---

As showe figure, bisects angle CAB.
Cholces, 8.6cm B:Tom C.8cm D.9cm
Question:
According question shown the image, please first conduct reasoning, and then answer the question and provide the correct option letter, eg. A, B, the end.
D
Source: MathVerse- Saatieers-VVsionOnlly id: 45
A
B
Reasoning Issue
NoisyRollout
<thinko
angle triangle congruent friangle This meons CD and AD.
Since
(Side-Angle-Side) congruence criterion.
Given that AD bisects ongle CAB. and since AC BC, triongle ACD is isoscales with AC BC, Thee perpendicular from to AB at implies that DE the aititude and alse the median triangle AbB, making CE.
The perimeter triangle DBE is the sum D8, BE, and DE Since BD and DE CE, we con express the penimeter as:
DB + BE DE BD + BE CE BC + BE
Given that AB cm and AC BC, and since AC BC, we can infer that the segments are symmetrically divided, Therefore, the perimeter of triangle DBE is the sum of the segments that be derived from the given information.
Since and the segments are symmetric, the perimeter of triangle DBE is:
DB +
cm
Thus, the perimeter of triongle DBE is cm.
</thinko
\boxed(A)
Vanilla GRPO
<thinko
Siven that AC and AD bisects angle CAB, triangle ACD This that AD (comman side), angle CAD angle BAD (bisacted angle). and AC AB (given). Therefore, triongle congruent trinngle ABD.
triongle ABD the SAS (Side-Argle-Side) congruence
Since AD bidects angle CAB, it clso means that D the foot of the perpendiculor from A to BC, and E is the foot of the perpendicuka from to AB. This implies that DE parallel AC and BC, and thus DE the harmanic of AC and BC, but importantly. it altitude triongle
median
The perimeter of triangle DBE is the sum of the lengths of its sides: DB BE DE. Since is the foot of the perpendiculor from A to BC, and E the foot of the perpendiculor from D to AB, we use the properties of the right triangles to find the lengthss
right angle right triangle splits the hypoteruse into segments that are proportional to the squares of the legs. Howeven, a simpler approach is to recognize that since D is the foot of the perpendizulor from to BC, and is the foot of the perpendicularr from D to AB, the segments DB and BE equal holf the hypoteruse the right triongle ABD, which each.
Given
angle
Therefore, the penimetor of triongle
So, the perimeter of triongle DBE </thinko
\boxed(D)
Figure 11: Case study illustrating enhanced reasoning capability of NoisyRollout vanilla GRPO.
H Templates
Reasoning Template from EasyR1
SYSTEM:
You FIRST think about the reasoning process internal monologue and then provide the final answer, The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in {boxed().
USER: {question}
Direct-Answer Template
SYSTEM: You are helpful assistant
USER: {question]. Answer yes or no directly.
32

---

I Detailed Methodology for Gradient Contribution Analysis
To quantitatively assess the impact of noisy rollouts the reinforcement learning (RL) optimization process, we partition the rollouts associated with each training sample into two distinct subgroups based on their input type: Clean and Noisy. Specifically, Clean rollouts are generated from original inputs (I, q). whereas Noisy rollouts originate from distorted inputs (I, In this ex- perimental setup, each training sample comprises
= 6 C1 ean rollouts and n2 6 No sy rollouts. For computational efficiency, all gradient calculations and parameter differences discussed below (g', 8elean B are performed using only the parameters from specific model modules. The exact modules used are "Im-head. weight", "model layers 27, self selffattn... o-prof", "model. layers
"model layers. 27 wtt.sss...sel...aee.c....ee self.attn.k.prom and "model. layers 27 wtw..atttt.... v.roj""
self.atta.q.pro
To quantify the contribution of each subgroup to the optimization, we first define anchor gradient, denoted g". This quantity represents the effective overall update to the selected model parameters given training stage t. It is calculated the difference in these parameters between checkpoints at training steps and + At:
g = g
where H represents the selected model parameters at training step f, and we use At steps. This g' reflects the actual change in these parameters resulting from the standard training procedure which utilizes losses derived from both Clean and Noisy rollouts from each training sample.
Subsequently, starting from the same parameter state 6 we isolate the influence of each subgroup. This involves performing At actual optimization steps under two modified conditions, using the same batch of training samples that contributed to the standard update from 0 to t
1. To obtain &ttt' Starting from 8', we performed At optimization steps. During these steps, the loss components arising from the Noisy rollouts were masked (i.e., their corresponding loss was set to zero). Thus, the gradients and subsequent parameter updates were derived solely from the Clean rollouts. This procedure yielded the updated parameters olean
#*+Ar
2. To obtain Ar Similarly, starting from 0*, we performed At optimization steps. this case, the loss components arising from the Clean rollouts were masked. The gradients and subsequent parameter updates were therefore derived solely from the Noisy rollouts. This yielded updated parameters 0++A/
noisy""
"aiisy"
From these updates, we define the subgroup-specific effective gradients (or parameter deltas) with respect the selected modules:
-g++At gr
Selean
and
0++ noisy
Enoisy
We then quantify the contribution of each subgroup by projecting its effective gradient onto the anchor gradient. The projection ratios are computed follows:
Belean Ig
g
and
r noisy
Eooisy
clean
lg
These ratios, r and ret "oisy> represent the estimated proportion of the anchor gradient g¹ that can be attributed to the Can and No sy subgroups, respectively, at training stage f. To ensure robust estimates, all effective gradient quantities (g' getem and giunsy) are determined by averaging results from 5 independent runs of the At-step update processes described above. Each run starts with the same parameters 0*.
33

---

Supplementary Implementation Details
This section provides the detailed hyperparameter configurations for our experiments that were omitted from Section 3. In Table 13, we summarize our experimental settings across different model sizes and datasets, with specific focus on image distortion parameters and noise annealing schedules.
Table 13: Summary of hyperparameter configurations.
Parameter
Configuration
General Settings (All Experiments)
Model Base
Qwen2.5- -V--strrrtt
Vision Encoder
Frozen
Global Batch Size
128
Rollout Batch Size
512
Rollout Temperature
1.0
Learning Rate
le -
Optimizer
Adam
Policy Loss Aggregation
token-mean Gaussian noise Sigmoid-shaped
Image Distortion Strategy
Noise Annealing Schedule
CPU Memory
1TB
GPU
A100-SXM4440080GBB
1102:---BBVL-Instruue on Geometry3K (2.1K samples)
Initial Noise (oo)
500
Training Episodes
15
Total Optimization Steps (fmax)
60
Sigmoid Midpoint (7)
40
Sigmoid Steepness (A)
30
Rollout Number
7h = T12 = 6
Time Cost per Step
about 1100s
Instrructt on MMK12 (6.4K samples)
Initial Noise (ap)
450
Training Episodes
12
Total Optimization Steps (fmax)
120
Sigmoid Midpoint (7)
40
Sigmoid Steepness (A)
60
Rollout Number
11
72
6
Time Cost per Step
about 1500s
Swen2.5-32B-VL--nnste on Geometry3K (2.1K samples)
Initial Noise (oo)
450
Training Episodes
10
Total Optimization Steps (4max)
40
Sigmoid Midpoint (7)
35
Sigmoid Steepness (A)
30
Rollout Number
71 = T22 =
Time Cost per Step
about 3300s
Qwen2.5-32B-Vl. 3-- MMK12 (6.4K samples)
Initial Noise (ag)
450
Training Episodes
7
Total Optimization Steps (fmax)
70
Sigmoid Midpoint (7)
35
Sigmoid Steepness (A)
30
Rollout Number
n1
n22 = 4
Time Cost per Step
about 3300s
34

---

K
Licenses
We use standard licenses from the community. We include the following licenses for the codes, datasets and models we used in this paper.
Datasets & Benchmarks:
Geometry3K [47]: MIT
MMK12 [53]: Apache License 2.0
Math Verse [93]: MIT
Math Vision [74]: MIT
Math Vista [48]: Creative Commons Attribution Share Alike 4.0 International
WeMath [57]: CC 4.0
Codes:
verl [65]: Apache License 2.0
EasyRI [96]: Apache License 2.0
Models:
Owen2.5-VL-7B-Instue [4]: Apache License 2.0
Owen2.5-VV-S2D-nnttnuc [4]: Apache License 2.0
Gemini API [18]: Gemini API Additional Terms of Service
35


Reviews:
[{"review_id": "IVsC1R364K", "reviewer": "Reviewer_aFBH", "summary": "This paper introduces a simple strategy of augmenting image modality with gaussian noise during the VLM finetuning with RL. The authors show that current VLM methods are sensitive to visually OOD data -- and propose this simple pipeline that is scalable to any RL training.\nThey conduct experiments on 2 datasets and show significant, non-trivial improvements on OOD perceptual reasoning tasks.", "questions": "2. What is the intuition behind starting with a very noisy image and slowly annealing the noise to effectively get a clean image in the later iterations? Intuitively, I'd have assumed that slowly increasing the noise would act as a curriculum.\n\n3. It is indeed very interesting to see that only gaussian noise worked and other data augmentations like cropping or rotation did not. I can see why cropping may hamper the learning process, especially when crucial parts of the image are cropped out. However, it's interesting to see rotation fail as well. Have authors tried another curriculum where they train with gaussian noise, then with rotation / cropping (after letting the model be stable). One potential risk with this could be the ovbious forgetting problem in ML. But I'm curious if the authors had tried this in their experiments.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n\n- Practical and scalable approach with minimal changes required to the existing RL training framework for VLMs.\n\n- Significant improvement on OOD tasks.\n\n**Weaknesses**\n\n1. The use of data augmentation is a very common choice in RL and has been extensively studied in Visual RL and using them is not really novel (which is fine according to me). Works such as Drq, DrqV2, RAD, CURL [1, 2, 3, 4] should be cited and discusses in the related work section.\n---\n**References**\n\n[1] Yarats, D., Kostrikov, I., & Fergus, R. (2021). Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. In 9th International Conference on Learning Representations (ICLR 2021)\n\n[2] Yarats, D., Fergus, R., Lazaric, A., Pinto, L., & Kostrikov, I. (2022). Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning. In International Conference on Learning Representations (ICLR 2022)\n\n[3] Laskin, M., Srinivas, A., & Abbeel, P. (2020). Reinforcement Learning with Augmented Data. In Advances in Neural Information Processing Systems (NeurIPS 2020).\n\n[4] Srinivas, A., Laskin, M., & Abbeel, P. (2020). CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In International Conference on Machine Learning (ICML 2020).", "was_revised": true, "final_justification": "Thanks to the authors for their comprehensive rebuttal and for performing the additional experiments. I'm glad that the rotation augmentation worked reasonably well after the correct implementation for the augmentation.\n\nOverall, all of my concerns were adequately addressed and I will choose to keep my original rating of **Accept**.", "rebuttal": "Thank you for your supportive review and suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n> ***W1: Data augmentation in RL is common and non-novel; discuss and cite DrQ, DrQv2, RAD, and CURL in related work.***\n> \n\nThank you for emphasizing the importance of discussing existing RL works that use data augmentation. We recognize that data augmentation is well-established in both visual tasks and reinforcement learning. We will include foundational works like DrQ, DrQ-v2, RAD, and CURL in our revised related work section.\n\nNevertheless, we would like to further clarify what makes our approach novel:\n\n1. **Data augmentation designed for RLVR+VLM**. While reinforcement learning with verifiable rewards (RLVR) has shown great success in LLMs (like Tulu3 and DeepSeek-R1), it hadn't been thoroughly tested in VLMs when we started our study. We found that VLMs' weak visual perception prevents them from achieving the same impressive RLVR results as LLMs, which motivated us to use image augmentation to make the model's visual reasoning more robust to the imperfect perception characteristics of VLMs. **To the best of our knowledge, we are the first work to utilize data augmentation in RLVR + VLM.**\n    \n2. **Rollout-level vs. sample-level augmentation**. Despite direct adaptation from traditional data augmentation implementation, our method differs fundamentally:\n    - **Traditional sample-level augmentation** (DrQ, RAD, etc.): Rollouts are collected from both original and augmented images, with each trajectory optimized according to its actual source image during policy updates.\n    - **Our rollout-level augmentation** (NoisyRollout): We collect trajectories from both original and augmented images, but during policy optimization, we treat all rollouts as if they originated from clean images, regardless of their actual source, as indicated in Equation (2).\n    \n    In the Appendix (lines 616-628) of our original manuscript, we have previously noted that applying augmentation at the sample level is less effective than our proposed rollout-level augmentation, and as part of our rebuttal, we provide an additional quantitative experiment to substantiate this claim:\n    \n    - **Setup**: Trained on Geometry3K dataset (2.1K samples) with Qwen2.5-7B-VL-Instruct; `rollout=6+6` denotes 6 rollouts from clean images and 6 rollouts from noisy images\n    - **Observation**: Rollout-level augmentation consistently outperforms both baseline and sample-level augmentation across all benchmarks\n    - **Why it works**: Sample-level augmentation restricts exploration since augmented rollouts stay tied to their original image-question pairs (\"on-policy\"). Rollout-level augmentation enables diverse exploration through visual-oriented inductive bias, helping models benefit from more effective supervision signals brought by noisy rollouts. A recent study in the LLM domain [1] shares a similar observation regarding the effectiveness of \"off-policy\" rollouts.\n    \n    | Method | Augmentation Type | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n    | --- | --- | --- | --- | --- | --- | --- | --- |\n    | GRPO | None (rollout=12) | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n    | NoisyRollout | Sample-level (rollout=6+6) | 49.5 | 28.4 | 69.3 | 67.4 | 68.6 | 56.6 |\n    | NoisyRollout | Sample-level (rollout=12+12) | 50.7 | 26.5 | 70.1 | 67.9 | 70.0 | 57.0 |\n    | NoisyRollout | Rollout-level (rollout=6+6) | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n    \n    ***Note**: The above table shows sample-level augmentation underperforms vanilla GRPO. Unlike standard sample-level methods that randomly sample augmented data, we intentionally included both original and augmented versions of the same image within each batch to maintain consistency with our rollout-level approach (which includes all rollouts from a given sample in one batch). This design choice may slightly hurt sample-level performance, though optimizing the sampling strategy is left for future work.*\n    \n\nIn summary, NoisyRollout introduces a simple, effective rollout-level augmentation strategy that specifically addresses visual reasoning challenges through vision-oriented inductive biases in the emerging RLVR+VLM domain.\n\n[1]. Yan, Jianhao, et al. \"Learning to reason under off-policy guidance.\" (arXiv 2025)\n\n---\n\n> ***Q2: Why start with maximum noise and gradually reduce it rather than slowly increase noise as curriculum?***\n> \n\nThank you for this insightful question. Our annealing strategy (starting with high noise and gradually reducing it) is motivated by two key considerations:\n\n1. **Why annealing is necessary:** Our rollout-level augmentation creates beneficial \"off-policy\" exploration but can introduce distributional shift. Early in training, when the model has poor visual understanding, it naturally struggles with clean images anyway, so noisy rollouts provide valuable exploration without significant harm.\n    \n    However, as training progresses and the model's visual capabilities improve, continued use of highly noisy rollouts becomes counterproductive—the model no longer needs aggressive exploration and the noise primarily introduces unwanted distribution mismatch. Thus, we need to anneal the noise strength to avoid this.\n    \n2. **Why decreasing rather than increasing noise:** We start with high noise when exploration matters most (when the model is weakest), then gradually reduce it to align with the clean image distribution encountered at inference and exploit rewards more effectively at the later training stage.\n    \n    Increasing noise would provide minimal exploration benefit early on but cause maximum distributional harm later, leading to training divergence. Since Figure 5 shows that maintaining consistent noise strength causes divergence in later stages, an \"increasing noise\" approach would likely suffer from the same problem but to a greater extent.\n    \n    However, we think slowly increasing noise could be implemented in sample-level augmentation, since there is no distributional shift issue as it's fully \"on-policy\".\n    \n\n**In essence**, we use high noise when exploration matters most, then reduce it as exploitation becomes more important.\n\n---\n\n> ***Q3: Interestingly only Gaussian noise worked; tried curriculum training with Gaussian noise first, then rotation/cropping?***\n> \n\nWe further investigated this phenomenon and here are our responses to the points you proposed on image augmentation:\n\n1. **Rotation now works with proper implementation.** Our initial rotation experiments used suboptimal parameters (`expand=False`) that cropped rotated images and lost critical visual information. With corrected parameters (`image.rotate(max_angle, resample=Image.BILINEAR, expand=True)`), rotation now shows meaningful improvements:\n    \n    \n    | Method | Image Augmentation | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n    | --- | --- | --- | --- | --- | --- | --- | --- |\n    | GRPO | None | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n    | NoisyRollout | Gaussian noise | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n    | NoisyRollout | Rotation | 52.5 | 28.1 | 71.9 | 68.1 | 70.2 | 58.2 |\n    \n    Both augmentation types now outperform vanilla GRPO, confirming that NoisyRollout benefits from various augmentations when properly tuned.\n    \n2. **Sequential curriculum training faces entropy collapse challenges.** Your proposed approach of starting with Gaussian noise then transitioning to rotation/cropping is compelling, and we had similar considerations early in our research. However, we identified a fundamental concern: potential entropy collapse after the initial Gaussian noise stage could make the model's policy too deterministic, limiting subsequent rotation-based training effectiveness. This would require non-trivial technical solutions, preventing us from pursuing this direction.\n    \n3. **Random augmentation selection yields modest gains over rotation but falls short of Gaussian noise.** Instead, we explored randomly selecting between Gaussian noise and rotation for each sample during training (that is, the training compute still keeps the same among the three experimental settings):\n    \n    \n    | Image Augmentation | Average |\n    | --- | --- |\n    | Gaussian noise | **59.2** |\n    | Rotation | 58.2 |\n    | Gaussian noise  + Rotation | 58.5 |\n    \n    While the combination performs better than rotation alone, it doesn't exceed single Gaussian noise augmentation, suggesting that combining different augmentations effectively is non-trivial and needs further investigation.\n \nDue to computational constraints, we reserve the systematic exploration of curriculum-based augmentation strategies for future work."}, {"review_id": "BxvLhy2Dop", "reviewer": "Reviewer_mrqL", "summary": "This paper studies fine-tuning VLMs via reinforcement learning with visual domain shifts. It introduces NoisyRollout, which interleaves on-policy rollouts under additive Gaussian noise with clean rollouts in GRPO update, while annealing the noise strength over training. By always computing policy updates on the clean input, NoisyRollout drives exploration without corrupting the learned input distribution. Experiments shows empirical performance gain across 5 out-of-domain reasoning and perception benchmarks.", "questions": "1. On Proportion of noisy rollouts, I wonder if the authors explored adaptive ratios that shift over time or with model confidence?\n2. Gradient Contribution Analysis: the paper only report norm magnitudes, but not whether noisy gradients actually improve or sometimes misdirect the update. dominance in magnitude doesn’t definitively prove that noisy gradients *cause* better exploration, it could also be a byproduct of higher reward variance under noise. More in-depth discussions on this will be appreciated.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**: \n\nThe paper proposes a simple yet broadly applicable idea of interleaving noisy and clean rollouts without changing the core RL formulation, making it easy to integrate into existing VLM policy fine-tuning pipelines. The experiments and ablations are very thorough and show strong empirical performance gains.\n\n\n**Weaknesses**\n\n1. Limited novelty: Injecting synthetic distortions to improve robustness is not entirely a new idea, e.g., “domain randomization” in computer vision and robotics. the paper lacks a strong argument why its specific rollout-mixing strategy is meaningfully better  than existing perturbation-based exploration methods.\n2. The synthetic noise might not reflect real world situations: The proposed method uses only additive Gaussian noise for all noisy rollouts. This is a very narrow and idealistic corruption model, real-world domain shifts (e.g. occlusion, blur, viewpoint changes, etc.) aren’t covered. If the policy never sees those distortions in rollout, there’s no guarantee it will generalize to them. \n3. No sequential task: All experiments are on single-step vision-reasoning. It’s unclear whether hybrid rollouts would benefit longer horizon policies where perception errors compound. A good example could be robotic navigation or manipulation under changing visual conditions.\n4. No results on test-time noise: Because updates are always conditioned on the clean image, the policy never actually learns to interpret noisy inputs, so it may still fail at inference when images are added with noise. The only robustness induced is by selecting actions that succeed under Gaussian noise during rollout (more exploration), but we don’t see direct evaluation of the final policy on noisy test images.", "was_revised": true, "final_justification": "Post-rebuttal final justification: Thank you for your in-depth rebuttal. Your responses have addressed most my questions and concerns. However, I still think that Gaussian noise has limited novelty and is not broadly applicable to other more complicated domains. Therefore, I will maintain my score as 4: borderline accept.", "rebuttal": "Thank you for your supportive review and suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n> ***W1: Injecting synthetic distortions to improve robustness is not entirely a new idea; the paper lacks a strong argument why its specific rollout-mixing strategy is meaningfully better than existing perturbation-based exploration methods.***\n\nThank you for highlighting the importance of clarifying how our rollout-mixing strategy differs from existing perturbation-based methods. We acknowledge that injecting synthetic distortions is an established approach, however, we wish to clarify the specific novelty and unique considerations of our NoisyRollout method:\n\n1. **Rollout-level vs. sample-level augmentation.** Our method introduces a distinct augmentation strategy tailored specifically for the RLVR and VLM setting:\n    - **Traditional sample-level augmentation** (DrQ [1], RAD [2], etc.): Rollouts are collected from both original and augmented images, with each trajectory optimized according to its actual source image during policy updates.\n    - **Our rollout-level augmentation** (our NoisyRollout): We collect trajectories from both original and augmented images, but during policy optimization, we treat all rollouts as if they originated from clean images, regardless of their actual source, as indicated in Eq. (2).\n2. **Specific considerations in choosing rollout-level augmentation.** Our rollout-level strategy addresses key characteristics of RLVR with VLMs:\n    - We use noisy rollouts to **increase rollout diversity** and **enhance exploration**, targeting visual perception (a well-known VLM weakness) to provide more diverse and meaningful supervision signals during RL training.\n    - While we focus on rollout-level augmentation for these advantages, sample-level augmentation remains valuable for future exploration, potentially offering complementary strengths.\n3. **Empirical support.** In the Appendix (lines 616-628) of our original manuscript, we have previously noted that applying augmentation at the sample level is less effective than our proposed rollout-level augmentation, and as part of our rebuttal, we provide an additional quantitative experiment to substantiate this claim:\n    - **Setup**: Trained on the Geometry3K dataset with Qwen2.5-7B-VL-Instruct; `rollout=6+6` denotes 6 rollouts from clean images and 6 rollouts from noisy images\n    - **Observation**: Rollout-level augmentation consistently outperforms both baseline and sample-level augmentation\n    - **Why it works**: Sample-level augmentation restricts exploration since augmented rollouts stay tied to their original image-question pairs (\"on-policy\"). Rollout-level augmentation enables diverse exploration through visual-oriented inductive bias, helping models benefit from more effective supervision signals brought by noisy rollouts. A recent study in the LLM domain [1] shares a similar observation regarding the effectiveness of \"off-policy\" rollouts.\n        | Method | Augmentation Type | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n        | --- | --- | --- | --- | --- | --- | --- | --- |\n        | GRPO | none (rollout=12) | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n        | NoisyRollout | sample-level (rollout=6+6) | 49.5 | 28.4 | 69.3 | 67.4 | 68.6 | 56.6 |\n        | NoisyRollout | sample-level (rollout=12+12) | 50.7 | 26.5 | 70.1 | 67.9 | 70.0 | 57.0 |\n        | NoisyRollout | rollout-level (rollout=6+6) | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n        \n        ***Note**: Sample-level augmentation underperforms vanilla GRPO in our experiments. This is partly because we maintained batch consistency with our rollout-level approach by including both original and augmented versions of each image into one batch, rather than treating them like independent samples.*\n\nReference:\n\n[1]. Yarats, Denis, et al. \"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.\" (ICLR 2021)\n\n[2]. Laskin, Misha, et al. \"Reinforcement learning with augmented data.\" (NeurIPS 2020)\n\n---\n> ***W2: Only additive Gaussian noise for all noisy rollouts; real-world domain shifts (e.g. occlusion) aren’t covered.***\n\nThank you for raising this important limitation regarding the generalizability of our augmentation approach. We address this concern through domain-specific justification and empirical validation:\n\n1. **Gaussian noise is well-suited for our domain.** Mathematical diagrams and equations typically maintain consistent viewpoints and clear presentation. Gaussian noise naturally models sensor noise and compression artifacts that are more common when digitizing mathematical content, unlike occlusion or viewpoint changes which occur less frequently.\n2. **Empirical validation with additional augmentation types.** Furthermore, we conducted experiments with rotation augmentation as a representative geometric transformation. Our initial rotation experiments showed limited improvement due to implementation issues—specifically using `expand=False`, which cropped rotated images and removed critical visual information. After correcting this with proper parameters (`image.rotate(max_angle, resample=Image.BILINEAR, expand=True)`), rotation augmentation now demonstrates meaningful gains:\n    | Method | Augmentation | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n    | --- | --- | --- | --- | --- | --- | --- | --- |\n    | GRPO | None | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n    | NoisyRollout | Gaussian noise | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n    | NoisyRollout | Rotation | 52.5 | 28.1 | 71.9 | 68.1 | 70.2 | 58.2 |\n3. **Combining augmentations show mixed results and require further investigation.** We also explored combining two augmentation types by randomly selecting between Gaussian noise and rotation for each training sample. However, the results reveal that effective augmentation combination is non-trivial:\n    \n    \n    | Augmentation | Average |\n    | --- | --- |\n    | Gaussian noise | **59.2** |\n    | Rotation | 58.2 |\n    | Gaussian noise  + Rotation | 58.5 |\n    \n    The combination approach outperforms rotation alone but falls short of pure Gaussian noise augmentation. This suggests that simply combining augmentation techniques does not guarantee additive benefits. Understanding how to optimally combine multiple augmentation strategies remains an important direction for future work.\n    \n\n---\n\n> ***W3: No sequential task; Unclear whether hybrid rollouts would benefit longer horizon policies where perception errors compound.***\n\nThank you for this important observation. We address this concern from two perspectives:\n\n1. **Our preliminary evidence suggests combining augmentation strategies is non-trivial.** We tested mixed augmentation (Gaussian noise + rotation) and found it underperformed pure Gaussian noise (58.5 vs 59.2 average), indicating that effective combination requires more sophisticated approaches than random selection (see Weakness 2, point 3).\n2. **Sequential tasks are beyond our current scope.** They would require addressing catastrophic forgetting, entropy collapse, and continual learning complexities—substantial methodological challenges that need dedicated future investigation.\n\n---\n\n> ***W4: No results on test-time noise; it may still fail at inference when images are added with noise.***\n\nThank you for this important concern. We conducted additional experiments evaluating robustness under noisy test conditions on MathVerse:\n\n| Method | Clean | Noise Step (300) | Noise Step (500) |\n| --- | --- | --- | --- |\n| GRPO | 50.8 | 50.2 | 44.7 |\n| ThinkLite | 50.2 | 49.6 | 44.3 |\n| **NoisyRollout** | **53.2** | **52.3** | **46.2** |\n\nNoisyRollout consistently outperforms baselines across all noise levels, demonstrating that our training-time augmentation successfully improves test-time robustness to noisy inputs.\n\n---\n\n> ***Q1: Adaptive ratios that shift over time or with model confidence.***\n\nThank you for this thoughtful question. That's an good idea that aligns well with our current noise annealing strategy in terms of the training dynamics. \n\nWe didn't explore adaptive ratios empirically, but we did consider this approach during early development. Here's our reasoning for the current design choice:\n\n1. **Fundamental limitation of proportion-based approaches.** Even when adaptively decreasing noisy rollout proportions during training, each individual noisy rollout still introduces the same degree of distributional mismatch. The core problem—distribution shift at the `<input, rollout>` level—remains unresolved regardless of how many noisy rollouts we include.\n2. **Noise annealing addresses the root cause.** Our approach directly reduces augmentation intensity over time, allowing us to maintain exploration benefits early in training while progressively aligning with the clean image distribution encountered at inference. This manages the exploration-exploitation trade-off more systematically.\n\n---\n\n> ***Q2: Dominance in magnitude doesn’t definitively prove that noisy gradients cause better exploration, it could also be a byproduct of higher reward variance under noise.***\n\nThank you for this important methodological critique. You're right that our gradient norm analysis cannot definitively distinguish between beneficial exploration and misdirection from reward variance. While higher magnitudes correlate with improved performance across benchmarks, this doesn't establish causality. Our current analysis provides preliminary insights into the relationship between noisy rollouts and gradient behavior. \n\nNevertheless, we empirically believe that noisy gradients lead to better exploration, supported by our ablation study on rollout diversity in Section 3.2. More sophisticated rollout attribution analysis would provide deeper insights, which we leave as important future work."}, {"review_id": "bvfK0WJ6Ot", "reviewer": "Reviewer_HhCS", "summary": "This paper proposes NoisyRollout, a data augmentation strategy for RL on VLMs, aimed at enhancing visual reasoning robustness. The key idea is to augment rollout diversity by mixing inputs from both clean and moderately distorted images during training. The approach is only a data augmentation with GRPO, requiring minimal extra computation. Extensive experiments across five out-of-domain visual reasoning and perception demonstrate significant performance gains over various RL baselines. The approach also works well across model sizes and data scales. Overall, this is a solid paper. While I do have some detailed design questions hoping authors to justify, the overall contribution is novel and sound.", "questions": "See weaknesses. \n\nOne more question: \n1. The claim that NoisyRollout “scales test-time compute” is potentially confusing—it is not test-time computation per se that is scaled, but policy exploration during training.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper identifies an underexplored yet important aspect of RL in VLMs—exploration under noisy visual inputs (which itself is a very common computer vision training data augmentation technique).\n\n2. NoisyRollout integrates well with existing GRPO with minimal extra computation just to add noises to the visual inputs. The proposed algorithm is easy to understand and implement.\n\n3. Despite the simplicity, NoisyRollout consistently outperforms other vanilla RL methods. Ablations on noise strength, rollout temperature, annealing, and model size are thorough. The performance gain is general across datasets, architectures, and varying training setups.\n\nWeaknesses\n1. It is unclear to me why the authors choose only utilizing the rewards for noisy inputs in calculating the mean in GRPO objective. I would have thought having the “contrastive signal” between clean and noisy rollouts (by using both normal and noisy trajectories) could offer stronger supervision. I hope the authors could provide justification and/or results to support the chosen design option is better. \n\n2. Another design question: why is only Gaussian noise chosen with NoisyRollout? Technically I think many types of noise could work with NoisyRollout and this design option seems not justified and explored. \n\n3. Continuing point 2, while the method clearly improves performance, it would be helpful to include or reference failure cases where the current NoisyRollout hurts where Gaussian noise is not meaningful.", "was_revised": true, "final_justification": "I have read all reviews and authors' responses. The authors' response addressed my weaknesses 1-2 very well, and the authors mentioned they will include examples in the final version to address my weakness 3. As such, I increase my rating to 5: accept.", "rebuttal": "Thank you for your supportive review and suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n> ***W1: Why only utilize the rewards for noisy inputs when calculating the mean in the GRPO objective, since having a \"contrastive signal\" between clean and noisy rollouts could offer stronger supervision?***\n> \n\nYou raise a good point about contrastive signals that we agree with: when clean and distorted inputs yield divergent outcomes for the same query, these reward differences serve as **implicit contrastive signals** that refine the model's perceptual behaviors during reasoning (we have provided this discussion in our main text, lines 123-125), which is also evidenced in our perception quality experiments (Figure 3, fourth column).\n\nHowever, there appears to be a misunderstanding about our implementation regarding the roles of noisy rollouts. Our actual implementation:\n\n- We combine rewards from both clean and noisy rollouts: $r = \\lbrace r_i\\rbrace_{i=1}^{n_1+n_2}$ includes all rewards from both trajectory types ($n_1$ for clean rollouts, $n_2$ for noisy rollouts)\n- The baseline calculation uses this combined set: $\\text{mean}(r)$\n- Advantage computation: $\\hat{A}_i = (r_i - \\text{mean}(r))/\\text{std}(r)$ uses the group baseline\n- Both clean and noisy rollouts contribute to policy updates (**conditioned on clean images**)\n\n> ***W2: why is only Gaussian noise chosen with NoisyRollout?***\n> \n\nThank you for this important question. First, in Appendix A, we indeed discussed different image augmentation strategies. We previously noted that other augmentation strategies (e.g., rotation, cropping) can easily lead to critical information loss, thereby making the rollouts meaningless or even harmful for policy optimization.\n\nHowever, after carefully reviewing the image augmentation relevant code in our codebase, we found that our rotation implementation used suboptimal parameters (`expand=False`) that caused information loss. After adopting more appropriate parameters (`image.rotate(max_angle, resample=Image.BILINEAR, expand=True)`), rotation now demonstrates meaningful improvements:\n\n| Method | Image Augmentation | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| GRPO | None | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n| GRPO | Gaussian noise | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n| GRPO | Rotation | 52.5 | 28.1 | 71.9 | 68.1 | 70.2 | 58.2 |\n\nThis indicates that our NoisyRollout framework can work with various augmentation strategies which would not cause visual information loss. While Gaussian noise remains most effective, likely because it preserves all visual content while adding controlled perturbations, we argue other augmentations can also provide benefits when properly implemented.\n\n> ***W3: Include or reference failure cases where the current NoisyRollout hurts and where Gaussian noise is not meaningful.***\n> \nThank you for your kind suggestions. We would like to address this concern from two aspects:\n\n**Regarding failure cases:** Due to rebuttal policy limitations, unfortunately we cannot include visual examples here, but will add qualitative analysis of failure cases to our final submission as we believe it would indeed be helpful.\n\n**NoisyRollout degrades to vanilla GRPO performance (rather than hurting it) when visual dependency is minimal**—that is, when reasoning relies primarily on text rather than visual understanding. In such cases, visual augmentation provides no exploration benefit since clean and noisy rollouts yield similar outcomes, causing performance to converge to the baseline (vanilla GRPO). We will include this observation in our Limitations section, as it indeed represents a scenario where NoisyRollout might offer no additional benefit.\n\n> ***Q1: The claim that NoisyRollout “scales test-time compute” is potentially confusing; test-time computation per se is not scaled.***\n> \n\nThank you for raising this clarification question. In our context, \"scaling test-time compute\" actually refers to performing reasoning with chain-of-thought (CoT) at test-time. We would like to clarify that our claim is that **NoisyRollout enables more effective CoT reasoning during inference through our noisy RL training instead of using more tokens at test time**. Specifically, NoisyRollout enhances the model's intrinsic reasoning capabilities, allowing it to achieve superior visual reasoning performance when performing CoT at test-time through the improved problem-solving abilities developed via our noisy RL training."}, {"review_id": "HI5BAe4a5D", "reviewer": "Reviewer_5RUf", "summary": "The paper proposes NoisyRollout, a reinforcement learning (RL) framework that improves vision-language models (VLMs) on reasoning tasks.\nThe key idea is to augment rollout diversity by injecting noise into training images.\n\nSpecifically, for each image–text query pair, the algorithm performs the following steps:\n- roll out policies on both clean and distorted images;\n- compute normalized advantages under Group Relative Policy Optimization (GRPO);\n- perform policy updates only on clean inputs.\n\nExperiments on five benchmarks (MathVerse, MathVision, MathVista, WeMath, HallusionBench) show that NoisyRollout improves over vanilla GRPO and other RL-tuned VLMs.", "questions": "I have questions about novelty and contributions in the section above.\n\nRegarding the algorithm, could the authors clarify that if the updates are only performed on clean inputs, how does the model learn to reason on imperfect images?\nIf I understand correctly, the rewards on the distorted inputs only affect the baseline in the advantage computation.", "limitations": "The limitations are discussed in the appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nThe proposed hybrid rollout scheme is simple. It's compatible with GRPO and requires no extra training cost (other than augmenting the training set with noisy images).\n\nEmpirically, the proposed method outperforms several baselines.\n\n### Weaknesses\n\n**Novelty and contributions.**\nThe contribution focuses on the robustness of a reasoning VLM on imperfect images. The key idea is to inject noise into training images, which is a standard data augmentation technique.\nAlthough the paper claims to improve visual reasoning, the improvements appear to benefit more from robust perception than from improving the reasoning ability.", "was_revised": true, "final_justification": "The authors' clarifications have addressed my initial misunderstandings and concerns. While the proposed algorithm is not fundamentally novel compared with existing sample-level augmentation approaches, it is simple and shows clear empirical improvements.", "rebuttal": "Thank you for your valuable review and suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n> ***W1.1: The key idea is to inject noise into training images, which is a standard data augmentation technique.***\n> \n\n### **Generalizability Beyond Gaussian Noise**\n\nWe previously noted that other augmentation strategies (e.g., rotation, cropping) can easily lead to critical information loss, thereby making the rollouts meaningless or even harmful for policy optimization. However, after carefully reviewing the image augmentation relevant code in our codebase, we found that our rotation implementation used suboptimal parameters (`expand=False`) that caused information loss. After adopting more appropriate parameters (`image.rotate(max_angle, resample=Image.BILINEAR, expand=True)`), rotation now demonstrates meaningful improvements:\n\n| Method | Image Augmentation | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| GRPO | None | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n| GRPO | Gaussian noise | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n| GRPO | Rotation | 52.5 | 28.1 | 71.9 | 68.1 | 70.2 | 58.2 |\n\nThis indicates that our NoisyRollout framework can work with various augmentation strategies which would not cause visual information loss. While Gaussian noise remains most effective, likely because it preserves all visual content while adding controlled perturbations, we argue other augmentations can also provide benefits when properly implemented.\n\n### **Rollout-Level vs. Sample-Level Augmentation**\n\nWhile noise injection is a common data augmentation technique, we respectfully argue that our NoisyRollout is not a straightforward adaptation of standard data augmentation in terms of its specific implementation. Below we show the fundamental differences between traditional implementation and our specific design for reinforcement learning with verifiable rewards (RLVR) + VLM:\n\n- **Traditional sample-level augmentation (DrQ [2], RAD [3])**: Rollouts are collected from both original and augmented images, with each trajectory optimized according to its actual source image during policy updates.\n- **Our rollout-level augmentation (NoisyRollout)**: We collect trajectories from both original and augmented images, but during policy optimization, we treat all rollouts as if they originated from clean images, regardless of their actual source, as indicated in Equation (2).\n\nIn our appendix (lines 616-624), we have previously noted that applying augmentation at the sample level is less effective than our proposed rollout-level augmentation, and as part of our rebuttal, we provide additional quantitative experiments to substantiate this claim:\n\n- **Setup**: Trained on the Geometry3K dataset (2.1K samples) with Qwen2.5-7B-VL-Instruct; `rollout=6+6` denotes 6 rollouts from clean images and 6 rollouts from noisy images\n- **Observation**: Rollout-level augmentation consistently outperforms both baseline and sample-level augmentation across all benchmarks\n- **Why it works**: Sample-level augmentation restricts exploration since augmented samples stay tied to their original image-question pairs (\"on-policy\"). Rollout-level augmentation enables diverse exploration through visual-oriented inductive bias, helping models benefit from more effective supervision signals brought by noisy rollouts. A recent study in the LLM domain [1] shares a similar observation regarding the effectiveness of \"off-policy\" rollouts.\n\n| Method | Augmentation Type | MathVerse | MathVision | MathVista | WeMath | HallusionBench | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| GRPO | None (rollout=12) | 50.8 | 27.3 | 70.5 | 67.4 | 69.8 | 57.2 |\n| GRPO | Sample-level (rollout=6+6) | 49.5 | 28.4 | 69.3 | 67.4 | 68.6 | 56.6 |\n| GRPO | Sample-level (rollout=12+12) | 50.7 | 26.5 | 70.1 | 67.9 | 70.0 | 57.0 |\n| GRPO | Rollout-level (rollout=6+6) | **53.2** | **28.5** | **72.6** | **69.6** | **72.1** | **59.2** |\n\n***Note**: The above table shows sample-level augmentation underperforms vanilla GRPO. Unlike standard sample-level methods that randomly sample augmented data, we intentionally included both original and augmented versions of the same image within each batch to maintain consistency with our rollout-level approach (which includes all rollouts from a given sample in one batch). This design choice may hurt sample-level performance, though optimizing the sampling strategy is left for future work.*\n\nIn summary, NoisyRollout introduces a simple, effective rollout-level augmentation strategy that specifically addresses visual reasoning challenges through vision-oriented inductive biases in the emerging RLVR+VLM domain.\n\n[1]. Yan, Jianhao, et al. \"Learning to reason under off-policy guidance.\" (arXiv 2025)\n\n[2]. Yarats, Denis, Ilya Kostrikov, and Rob Fergus. \"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.\" (ICLR 2021)\n\n[3]. Laskin, Misha, et al. \"Reinforcement learning with augmented data.\" (NeurIPS 2020)\n\n---\n\n> ***W1.2: Although the paper claims to improve visual reasoning, the improvements appear to benefit more from robust perception than from improving the reasoning ability.***\n> \n\nGRPO explicitly optimizes accuracy, making it inherently challenging to separate gains in reasoning from perception enhancements.  Moreover, within VLM contexts, robust perception naturally supports better reasoning, and improved reasoning, in turn, guides the model's visual focus. Hence, distinguishing strictly between perception and reasoning improvements is less meaningful than recognizing their combined contribution to overall performance in our this context.\n\nHowever, we acknowledge that clearly defining and quantifying the interaction between perception and reasoning merits further exploration in the domain of Explainable AI (XAI). We leave this as our future work.\n\n---\n\n> ***Q1.1: Could the authors clarify that if the updates are only performed on clean inputs, how does the model learn to reason on imperfect images?***\n> \n\nThank you for this clarification question. You're correct that policy updates are conditioned on clean inputs. Regarding how the model learns robust visual reasoning on imperfect images: \n\n- Noisy rollouts sampled from distorted images simulate challenging training conditions where the model must solve tasks despite imperfect visual perception;\n- Successful noisy rollouts usually offer more diverse solutions than traditional rollouts, while failed attempts (rollouts with negative rewards) teach the model to avoid perceptual reasoning patterns that are difficult to learn through the normal exploration.\n\n---\n\n> ***Q1.2: If I understand correctly, the rewards on the distorted inputs only affect the baseline in the advantage computation.***\n> \n\nWe want to clarify that the rewards on the distorted inputs do **NOT** only affect the baseline in the advantage computation. Specifically, we treat noisy rollouts equivalently to those generated from clean images during policy optimization. The noisy rollouts not only contribute to reward baseline and advantage computation, **but are also rewarded and used for policy updates (conditioned on clean images).** This provides a special form of \"off-policy\" exploration that increases rollout diversity, enabling the policy to explore more effectively in the solution space (or environment)."}]

Author rebuttal:

Meta-review: This paper addresses visual reasoning in vision-language models (VLMs), a currently important area where recent models struggle with imperfect visual perception that affects reasoning processes. The proposed NoisyRollout method introduces a rollout-level data augmentation strategy that mixes trajectories from clean and noisy images during reinforcement learning training, representing a novel adaptation for the emerging RLVR + VLM setting.

Reviewers consistently recognized the method's simplicity and broad applicability with minimal computational overhead, strong empirical performance gains across multiple benchmarks, and thorough experimental evaluation including comprehensive ablations across model sizes and data scales.

The primary concerns centered on limited novelty since data augmentation in RL is well-established, questions about restricting to Gaussian noise rather than diverse augmentation strategies, and lack of evaluation on sequential tasks or complex real-world domain shifts. Some reviewers also sought clarification on specific design choices like rollout-level versus sample-level augmentation.

The authors provided a comprehensive rebuttal that effectively addressed these concerns by demonstrating quantitatively that rollout-level augmentation outperforms sample-level approaches, showing generalizability beyond Gaussian noise, providing test-time robustness experiments, and clarifying technical distinctions from traditional data augmentation.

While novelty is incremental, the paper makes a solid contribution to the currently high-profile field of visual reasoning in VLMs. Reasons for acceptance include: practical effectiveness and ease of adoption, consistent empirical improvements across diverse benchmarks, thorough experimental validation, and timely contribution to addressing known VLM visual perception weaknesses that limit reasoning performance.
Decision: Accept (poster)