Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
Authors: Linghao Zhu
Yiran Guan
Dingkang Liang
Jianzhong Ju
Zhenbo Luo
Bin Qin
Jian Luan
Yuliang Liu
Xiang Bai
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal large language model
reinforcement learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose Shuffle-R1, a simple and effective RL post-training framework that significantly improves RL training efficiency and model performance.

Abstract: Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM). However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing, where most advantages in a batch concentrate near zero, and Rollout Silencing, where the proportion of rollouts contributing non-zero gradients diminishes over time. These issues lead to suboptimal gradient updates and hinder long-term learning efficiency. To address these issues, we propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition. It introduces (1) Pairwise Trajectory Sampling, which selects high-contrast trajectories with large advantages to improve gradient signal quality, and (2) Advantage-based Trajectory Shuffle, which increases exposure of valuable rollouts through informed batch reshuffling. Experiments across multiple reasoning benchmarks show that our framework consistently outperforms strong RL baselines with minimal overhead. These results highlight the importance of data-centric adaptations for more efficient RL training in MLLM.

Published as a conference paper at ICLR 2026
EFFICIENT RL FRAMEWORK FOR MULTIMODAL LARGE LANGUAGE MODELS VIA DYNAMIC SHUFFLE
SHUFFLE-RI;
Linghao Zhu'
Yiran Guan Dingkang Liang'
Jianzhong Ju²
Zhenbo Luo²
Bin Qin Jian Luan²
Yuliang Liu' Xiang Bai
HHuzhong University of Science and Technology. MiLL Plus Xiaomi Inc.
{ih-zhu, yiranguan, dkl skliang, ylliu, xbai]ehust. wwweeustsuu....... edu. cn (jujianzhong, luozhenbo, qinbin, (aanjian)exiaoann com https://xenoz: github. io/iii///// Shuffle -RR
iouuuffee
ABSTRACT
Reinforcement learning (RI.) has emerged as effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM). However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing, where most advan- tages in a batch concentrate near zero, and Rollout Silencing, where the propor- tion of rollouts contributing non-zero gradients diminishes over time. These issues lead to suboptimal gradient updates and hinder long-term learning efficiency. To address these issues, we propose Shuffle-R1, simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition. It introduces (1) Pairwise Trajectory Sampling, which selects high-contrast trajectories with large advantages to improve gradient signal quality, and (2) Advantage-bann based Batch Shuffle, which increases exposure valuable rollouts through strategic batch reshuffling. Experiments across mul- tiple reasoning benchmarks show that our framework consistently outperforms strong RL baselines with minimal computational overhead. These results support the potential of data-centric adaptations for more efficient RL training for MLLM.
INTRODUCTION
Reinforcement learning (RL) has emerged as a powerful tool to enhance large language models (LLMs) to plan, reflect, and generalize, enabling stronger performance in complex reasoning do- mains such as mathematical problem solving and code generation (Guo et al., 2025; Deepmind, 2025; OpenAI, 2024; Seed, 2025). Notably, DeepSeek-RI (Guo et al., 2025) leverages reward sig- nals derived exclusively from verifiable outcomes to yield impressive performance gains. Beyond textual tasks, RL has also seen increasing application in various multimodal domains (Li al., 2025; Liu et al., 2025c;b; Wang et al., 2025c), highlighting its potential to support generalizable reasoning across modalities.
To better incorporate RL in LLM and Multimodal LLM (MLLM), recent studies have proposed var- ious improvements, including better framework optimization (Yu al., 2025; Wang al., 2025a; Chu et al., 2025b) and more sophisticated reward optimization (Liu et al., 2025a; Ma et al., 2025). However, most approaches remain confined to static sampling paradigm, where trajectories are sam- pled and treated uniformly. Such strategy overlooks crucial insight that not all learning signals are created equal and their informativeness varies and evolves during training. Ignoring variation in signal quality risks the training process being overwhelmed by noisy trajectories and underusing the truly useful signals. An ideal framework should instead concentrate updates on golden signals while filtering out noisy ones. Consequently, it leads to consider a fundamental question: Can dynamically prioritizing trajectories provide richer gradient information and lead to more effective training?

---

Published as a conference paper at ICLR 2026
5 de
Owanz 54L-18
Norizero Gradient Rello
Traiiin Data
Multimddal LLM
Nert adse nao zore!
Very Fee longe magnitaae
Tre or derimn
ea.
Traiing
6.-
Rotto ralluut drap!
cysceii aigl Treaena
0.5
Troning
Dynomic Selection
20
40
Advantage
Sheos
Figure 1: (a) Advantage Collapsing, where most advantages concentrate near zero. (b) Rollout Silencing, where the ratio of rollouts with non- zero gradient consistently drops.
Figure Pipeline comparison. (a) Static paradigm. (b) Rule-based pre-filter paradigm. (c) Dynamic paradigm can 'interact' with model during training.
Motivated by this question, we investigate into current RL training practices and reveal two critical yet might underexplored limitations. First, Advantage Collapsing emerges when most computed advantages cluster excessively near zero, drowning out informative signals from trajectories with large- sage-magnitudh advantages, resulting in extremely weak negligible gradient updates and wiping out decisive ones (Fig. (a)). Second, Rollout Silencing arises as the fraction of rollouts contribut- ing non-zero gradients steadily declines during training (Fig. (b)), leading to catastrophic waste of computation without fully utilizing informative signals. These two findings demonstrate a pressing need for adaptive mechanisms to prioritize, reuse, and reallocate gradient exposure toward informa- tive samples.
In this paper, we present Shuffle-R1, a framework that dynamically prioritizes and amplifies crit- ical gradient signals during RL fine-tuning. Guided by the philosophy that what data the model updates on is as important as how updates, Shuffle-RI introduces two effective modules: (1) Pair- wise Trajectory Sampling, which selects high-contrast trajectory pairs with large advantages gaps from an extended rollout pool, concentrating on the most discriminative learning signals to mitigate Advantage Collapsing; and (2) Advantage-based Batch Shuffle, which adaptively reshapes train- ing batches to emphasize informative trajectories while down- wewnnwwiggttin unhelpful ones, alleviating Rollout Silencing by removing noisy signals and improving computation utilization. Together, framework embodies the principle dynamic data prioritization (Fig. 2), enabling adaptive interac- tion between model and data. Experiments show that Shuffle-R1 substantially improves multimodal reasoning performance, surpassing GPT-4o (Achiam et al., 2024) and Claude-3.7 (Anthropic, 2025) on Math Verse (Zhang et al., 2024b) and Math Math (Lu et al., 2023), and matches GRPO (Shao al., 2024) with only half the training steps. In summary, our contributions are three-fold:
We reveal two critical yet underexplored limitations that undermine training efficiency in RL fine- tuning for MLLM, i.c., Advantage Collapsing and Rollout Silencing.
We propose Shuffle-RI, a novel and adaptive RL framework that dynamically selects high-contrast trajectories and reshapes training batches to emphasize informative samples. Extensive experiments across model scales and both in-domain and oft-o--domain benchmarks demonstrate the effectiveness and generalizability of our framework.
These results underscore the importance of rethinking which data to update on in RL post-training, moving beyond reward design toward dynamic and adaptive data structuring for more effective reasoning enhancement.
RELATED WORK
2.1 LARGE REASONING MODELS
Researchers have explored various approaches to equip LLM with reasoning ability. Some early studies performed SFT on complex long chain-of-thought data, leading to performance gains on reasoning tasks (Muennighoff et al., 2025; Ye et al., 2025; Guo et al., 2024). However, it has been argued that SFT merely enables the model to memorize the format of reasoning steps and long chains of thought, without fully grasping the ability reason independently (Chu et al., 2025a; Kang al.,
2

---

Published as a conference paper at ICLR 2026
2024). Some researchers control the model to generate structured chain-of-thouu tfthouht instead of free generation, achieving systematic step-by-step reasoning output (Xu et al., 2025; Wu et al., 2025; Thawakar et al., 2025) Other works attempted to test-time scaling like Monte Carlo Tree Search (MCTS) (Yao et al., 2023; Zhang et al., 2024a; Yao et al., 2024; Guan et al., 2026) to facilitate complex reasoning by actively extending the output of the model.
Recently, models such as OpenAl o1/o3 (OpenAI, 2024), DeepSeek-R1 (Guo et al., 2025), Seed- Thinking (Seed, 2025), and Kimi-k1.5 (Kimi, 2025) utilized RL enable the model explore directly conducted
independently, stimulating reasoning ability. In particular, DeepSeek-R1-Zen RL on pre-trained model without instruction fine-tuning, with verifiable outcome reward functions to replace reward models, achieving remarkable reasoning ability. The training algorithms for RL are also constantly being optimized (Yu et al., 2025; Chu et al., 2025b). Our work focuses on deeper investigation of the efficiency of RL training and proposes effective solution to improve both the efficiency and performance of RL training.
2.2 REINFORCEMENT LEARNING FOR MLLM
Following the success of DeepSeek-RI, series of studies have transplanted RL into the training of MLLM and downstream visual tasks, such as Open Vocabulary Object Detection (Liu et al., 2025c), Reasoning Segmentation (Liu et al., 2025b), Video Understanding (Li et al., 2025), Video Localization (Wang al., 2025c), etc. These works mainly focus the applicability of RL to downstream tasks. Some other works focusing on improving the general reasoning ability of MLLM have achieved performance improvements quality data (Meng et al., 2025; Huang et al., 2025; Yang et al., 2025; Zhang et al., 2025; Peng et al., 2025: Tan et al., 2025). These works mainly focus on the organization of high-quality reasoning data and the balance between SFT and RL in the training process.
reasoning tasks by collecting large amount of high-
Some researchers who conduct in-depth research on the RL mechanism have optimized the RL train- ing process from various aspects, including adding contrastive reward mechanism (Li et al., 2025), actively introducing reflection tokens during rollouts (Wang et al., 2025a), optimizing the RL ob- jective function and gradient update mechanisms (Chu et al., 2025b). and introducing more diverse rollouts (Liu al., 2025a; Yao al., 2025). The core objective of these works to optimize the RI. training process. In our work, we propose novel training framework that introduces dynamic and adaptive selection and resampling of queries and rollouts, reshaping the data distribution for better training efficiency and model performance.
METHOD
In this section, we begin by further analyzing the existing drawbacks in training. Then, as illus- trated in Fig. 4, we introduce Shuffle-R1, which optimizes the training process through two crucial modules: (1) Pairwise Trajectory Sampling and (2) Advantage-basee Batch Shuffle.
3.1 PRELIMINARIES
Policy gradient algorithm widely used RL method for MLLM, aiming to maximize the ex- pectation of return from the environment. For given query 9 N independent responses O (0r, o2.... ON) are sampled from the old policy model o ~ (g) Each response receives cor- responding reward R {r1, TNN computed via verifiable reward functions. Advantages Á (A, Âz. AN} are then estimated to guide policy updates. The core objective is defined as:
los
[(0) = B
min
(,,(0)A, clip((()))))))))) (110 (1)
ΣM
1m
where
To(ore oc)
A. P2 mcan
(0)
(2)
o (or, 1g. o1,c))
std(
and is clipping hyperparameter to prevent training collapse.
3

---

Published as a conference paper at ICLR 2026
Despite its practicality, this static RL paradigm has notable drawbacks: advantages often concen- trate near zero, yielding weak gradient signals, and the fraction of rollouts with non-trivial updates diminishes training progresses. These issues highlight the need for more flexible, dynamic and efficient training framework.
3.2 PROBLEM ANALYSIS
Advantage Collapsing. Our probe analysis reveals that, contrary to the ideal scenario, most roll- outs exhibit advantages sharply concentrated around zero in standard RL paradigm, leading to the Advantage Collapsing phenomenon. This concentrated distribution weakens gradient signals, as only a few rollouts with high-magnitude advantages drive meaningful updates. While simply in- creasing the number rollouts can partially mitigate this issue by increasing the chance of sampling valuable trajectories (Fig. 3(a)), it substantially increases computational overhead without address- ing the root cause. These findings highlight the need for a dynamic mechanism that adaptively selects valuable rollouts to improve training efficiency.
Rollout Silencing. We further observe a no- table Rollout Silencing phenomenon, where the fraction of rollouts contributing non-zero gradi- ents notably declines during training. It arises from factors such as zero advantages, gradient clipping, and excessive truncation, exposing the limits of static sampling paradigm. Tracking accuracy of queries in different difficulty re- veals that simple queries converge early while difficult ones remain inaccurate (Fig. 3(b)), both failing to generate informative rollouts. Moreover, standard pipelines use each rollout only once, preventing full exploitation of valu- able data. Consequently, it is crucial to design dynamic strategy that discards ineffective roll- outs while reusing informative ones.
GAPO
Madel Accuracy (%)
Curr
cotimal
Query Accuracy (%)
Update Rollout Size
Steps
(4)
[b)
Figure 3: (a) Model accuracy improves with
larger rollout sizes. (b) Queries with different
difficulties have varying accuracy during training.
yielding rollouts with different diversity and qual- ities.
3.3 PAIRWISE TRAJECTORY SAMPLING
To mitigate Advantage Collapsing, we seek to select trajectories that offer stronger learning sig- nals. Inspired by the observation that larger rollout pool increases the probability of capturing high-advantage samples, we propose Pairwise Trajectory Sampling (PTS), a data-centric module to selectively amplify valuable learning signals. Rather than evaluating trajectories in isolation, PTS organizes candidate rollouts into structured contrastive pairs. This pairing mechanism captures both high and low advantage signals jointly, forming informative "positive-negative" pairs. Only pairs with the largest advantage contrast are then retained for training. This process ensures that limited update bandwidth is focused on trajectories that are both diverse and gradient-rich.
Given query q and rollout size of 2N, the rollout trajectories group is denoted as O = [o:)22
The corresponding reward and advantage sets R {r. J and A {Á,) ) respectively. To identify informative trajectory pairs, our proposed pairing mechanism follows a straightforward 'max-min' pairing principle by matching the trajectory with the highest advantage to that with the lowest, the second highest to the second lowest, and so on. We denote the sorted advantage values in descending order as:
A. = {M where Aai 2 Aez 2 ... A
(3)
Based this ordering, we construct the pairing set as:
P ((ot)) (()N
(4)
In this scheme, the original 2N rollouts are easily sorted and reorganized into N pairs. The top- ranked pairs typically consist of trajectories with high-magnitude but opposite-sinn advantages, forming contrastive pairs akin to involve trajectories with advantages closer to zero
samples. In contrast, the nattom-ranker pairs
4

---

Published as a conference paper at ICLR 2026
Policy Optimization
Paining by
Ag
Original Batch
Shuffle by Weight S
Shuffled Batch
Policy Model
Reward
discord
Query
Adv.
sub-50 sompoe
A
Batck
Aze-1
Aze
1.PTS
2,ABS
Figure 4: Overview of our proposed Shuffle-RI. After advantage calculation, we first conduct Pairwise Trajectory Sampling to obtain valuable trajectory pairs from original rollout pool, then perform Advantage-based Batch Shuffle to reshape the distribution of valid trajectories in a batch.
As implied by Eq. I, trajectories with higher absolute advantages contribute more significantly to gradient update, while those with near-zero advantages have negligible impact. We apply a simple top-k sampling strategy to select a subset of valid pairs:
Pe = {(o()) o = a E (0, 1).
(5)
where a is hyperparameter controlling the sampling ratio from the pairing set. Only the trajectories within this valid set Pa are used for the subsequent gradient update.
By introducing structured contrastive sampling scheme, PTS enables more effective trajectory selection from a broader exploration space without increasing the gradient computation cost. The contrastive structure not only filters out low-signal trajectories. but also sharpen the model's policy gradient through direct comparison. PTS shifts the focus of RI. fine-tuning from uniform exploration to gradient-infonen selection, representing a principled step toward more efficient data usage in RI. training.
3.4 ADVANTAGE-BASEE BATCH SHUFFLE
While PTS mitigates Advantage Collapsing and improve RI. training performance, the Rollout Si- lencing issue remains unresolved. To overcome this issue, we propose Advantage-based Batch Shuf- fle (ABS) module that dynamically reshapes training batches to prioritize and reinforce high-value samples. Rather than relying static data flow, ABS adaptively redistributes trajectories within each training batch, enabling more frequent updates to trajectories with high learning utility. Built on top of PTS. serves to magnify the gradient exposure of informative samples, reshaping the training data distribution to achieve better data utilization and training efficiency.
Denote data batch provided by PTS:
B = {pf (o) AV,. ) )=-
(6)
with batch size of M In the standard gradient update process, B is sequentially divided into K mini-ban mini-batchess, each contains MG/K K samples.
In our ABS module, we first assign importance weight to each pair Pj E B based on the sum of the absolute advantages:
(p;)) ادررر + [
These weights are then normalized form a sampling distribution & over the entire batch B:
W(p)
(p
(8)
Σ1 W ((w)
Based on sampling distribution, we perform S sub-sampling from original batch B, each sub- sampling has a capacity of pairs (2T trajectories):
B. (Pa.)-m s.t. Pa, # Pa, PtN,Vtt vt.
All the sub-sampling batches are sequentially combined to form the reshuffled batch B' B U B., During the ABS process, we |B |B], i.e., S x T MG ensure the reshuffled batch
5

---

Published as a conference paper at ICLR 2026
Table Performance of Shuffle-RI trained on Geometry3K dataset.
Table 2: Performance of Shuffle-RI trained on
K12 dataset.
Method
Geo3K
Math Avg. HallBench ChartQA
Method
K12
Math Avg. HallBench ChartQA
Qwen-3B 25.79
41.71
59.83
73.08
Qwen-3B
42.42
41.71
59.83
73.08
GRPO
42.64
46,74
63.09
76.20
GRPO
59.19
48.71
64.14
77.12
+ DAPO
45.09
48.08
63.24
76.70
DAPO
61.42
49.75
65.08
77.00
GSPO
43,22
47.26
63.67
75.12
GSPO
60.44
48.76
64,14
77.56
+ Ours
47.88(+22.09)
48.70(-6.99)
63,
19(-3.6)
77.04:3306)
Ours
62.22(-19.80)
(0.05(-8.34)
65.72(+5.89)
78.28(+5.20)
Qwen-7B 38.12
49.82
65,
79.84
Qwen-7B
52.13
49.82
65.19
79.84
+ GRPO
52.60
53.13
68.56
80.84
GRPO
66.
54.47
67.75
82.48
DAPO $4.43
54.19
69.29
81.20
DAPO
68.35
54.52
68.66
82.52
+ GSPO $2.83
54.27
69.48
80.96
GSPO
67.44
54.77
69.13
82.04
Ours 65.89
54.63
69.51 81.64;-180)
Ours
68.78(-16.65 55.02/+5.20)
(9.87(--468) 82.60
matches the same size the original batch. The reshuffled batch will maintain the gradient update paradigm of the original method.
The ABS module optimizes the learning process through Advantage- Shuffling and Sub-batch Resampling. It increases the update frequency of trajectories with higher advantages, maintains diversity while reinforcing high-value samples through repeated exposure. Together, these designs transform each batch into soft-prioritized structure that better reflects training signal utility.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Benchmarks. We first conduct our experiments on Geometry3K dataset (Lu al., 2021) (2.1k training samples, 'Geo3K' for short) and subset of MMK12 dataset (Meng et al., 2025) containing the same amount of data ('K12' for short), to investigate model performance on limited training resources. To assess the scalability and effectiveness conduct experiments with MM-Euurcke dataset (Meng et al., 2025). Specifically, we construct a 30k- sample training set by combining the full Geo3K dataset with 27k randomly selected samples from MM-Eureka. All training samples are in free-form format.
larger corpus, we then
We first perform evaluation in-domain test of Geometry3K and MMK12. Further, RL is famous for its strong generalizability, we evaluate our model's performance on the following representative visual reasoning benchmarks: Math (Zhang et al., 2024b), MathVision (Wang et al., 2024), WeMath (Qiao et al., 2025), MathVista (Lu et al., 2023), HallusionBench (Guan et al., 2024) and ChartQA (Masry al., 2022). These benchmarks span across math reasoning, visual perception, and chart understanding. We use MathRuler to evaluate questions with free-form ground truths and Gemini-2.0- 2.--a----- (Deepmind, 2025) to evaluate questions with multi-choice ground truths. More information in Appendix E.
Implementation Details. We use EasyRI (Yaowei et al., 2025) as our training codebase. We employ Qwen2.5-VL - (Bai et al., 2025) as base model to verify method's gener- alizability on model scales. Parameters of vision encoder are kept frozen. We set update batch size to 128 and rollout batch size (G) to 512. Rollout temperature is set to 1.0 and learning rate is set to le 6. All experiments are conducted on 8x 80G GPUs. For each query, we generate 2N = 16 rollouts. For PTS, construct N 8 pairs and select the top M 4 pairs, corresponding retention ratio of a = 0.5, striking a balance between training cost and exploration space. For ABS, we set 256 pairs (512 query-respons. trajectories) for each sub-sampling batch. We construct shuffled batch by performing S = 8 rounds of shuffle. Decoding temperature for evaluation is set to 0.5, and we report average pass accuracy of tests to reduce randomness.
4.2 MAIN RESULTS
Comparison with Representative Algorithms. We compare method with GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025). On Geometry3K (Tab. 1), our 3B model reaches 47.88% accu- racy, outperforming GRPO by 5.2% and DAPO by 2.7%; the 7B model achieves 55.89%, with

---

Published as a conference paper at ICLR 2026
Table 3: Model performance on representative visual reasoning benchmarks. Models marked with <** are evaluated using our own evaluation scripts with vLLM. -Vision-R1-7B used WeMath and Math Vision as training data, its performance on these benchmarks are omitted. Best performance of RL-only models marked with Bold, second best with underline.
Model
Math MathVision MathVista WeMath HallBench ChartQA Avg.
Close-source
GPT-4o (Achiam et al., 2024)
50.8
30.4
63.8
68.8
55.0
(OpenAl, 2024)
57.0
73.9
Gemini-2.0 2.0 pro (Deepmind, 2025)
67.3
48.1 1
71.3
49.8
Siuude-37--Sometl (Anthropic, 2025)
52.0
72.6
55.4
Open-Source Source SFT
IntemVL-22.-88 (Chen ct al., 2025b)
39.5
17.0
64.5
50.1
79.1
IntemVVI-3-88 (Zhu et al., 2025)
29.3
71.6
49.9
86.6
Owen2.5-VL-33B (Bai et al., 2025)
34.8
21.9
58.4
51.7
59.8
73.1
49.9
Qwen2.5-VI.-777 (Bai al., 2025)
42.6
25.8
67.4
63.5
65.2
79.8
57.4
Cold-Start RL
RI-VL-7B' (Zhang al., 2025)
40.1
24.3
62.3
59.8
60.9
76.1
53.9
Vision-R1-7B" (Huang et al., 2025)
46.1
70.8
57.8
83.1
RI-OneVision-7B: (Yang al., 2025)
43.0
24.8
61.2
60.6
66.4
77.8
55.2
OpenV..Thinker-78 (Deng et al., 2025)
46.4
24.8
69.7
67.2
59.1
78.4
57.6
VLAA LLAA-Thinke--BB (Chen et al., 2025a)
48.9
26.3
69.9
67.7
67.5
80.1
60.1
Zero RL
MM-Eureka - (Meng
al., 2025)
49.6
27.4
70.6
67.4
66.7
79.0
60.1
MMR1-Math-7B" (Leng et al., 2025)
39.2
31.9
71.5
70.7
69.6
82.0
60.8
ThinkLite-VL-T VL-7B" (Wang et al., 2025b)
45.2
28.0
72.4
69.3
70.2
82.0
61.2
VI.:Rethinke-78 (Wang et al, 2025a)
$1.7
29.7
72.0
70.1
69.9
79.0
62.1
NoisyRollout7B-KII
al., 2025a)
50.1
28.0
70.9
20.8
70.1
81.4
62.1
Couffle-RI-Qwen--e (Ours)
44.2
26.8
70.4
66.5
69.2
79.9
59.5
Shuffle-R Bhuflle-I-Qwenn- RI-Qwen-7B (Ours)
53.9
30.0
77.0
72.3
71.0
84.1
64.7
gains of 3.3% and 1.4%, respectively. On out-of-domain math reasoning tasks, our method further improves average accuracy by 1.96% (3B) and 1.5% (7B) over GRPO, also surpassing DAPO. Con- sistent gains are observed on HallusionBench and CharlQA, where data distribution diverges more from training. Similar trends hold on the K12 experiments (Tab. 2), with improvements in both in- domain and out-of-domain settings, underscoring the robustness of our framework across tasks and distributions. Compared with the latest GSPO (Zheng al., 2025), which replaces token-level im- portance sampling with sequence-level importance sampling, our method also demonstrates superior performance on both in-domain and out-of-domain tasks. All the performance results above indicate our framework's generalizability different data distributions and model scales, highlighting the effectiveness of dynamic RL training paradigm. More results in Appendix C.
Comparison with RL-based models. We conduct larger scale experiments MM- Eureka dataset. As shown in Tab. 3, trained with 30k selected data from diverse sources for 150 steps, our 7B model exhibits a substantial accuracy gain
Orgginl
W: PTS
PTS AB5
0.6
Density
the base model (Qwen2.5- VL-7B). Moreover, it outperforms a series of open-source 7B competitors who also adopt RL training strategies, c.g. MM-Eureka with direct RL and VLAA-Thinker with RI. after cold-start SFT. Notably, our model achieves competitive or superior performance on several benchmarks compared to leading close-source models, for instance, Saaude337--oomee setting, our 3B variant also demonstrates strong performance, even outperforming several 7B models on certain benchmarks. These results highlight the superiority of our proposed approach in boosting the training efficiency in reinforcement learning.
0.0
-2.5
0.0
2.5
0.0
2.5
(b)
(c)
Figure 5: Advantage distribution in training batch GRPO and our framework.
(Anthropic, 2025) and GPT-4o (Achiam et al., 2024). Under the same

---

Published as a conference paper at ICLR 2026
CARd Das A83
GaRd
Nordter Rabo
Thain Accurscy (50
Val Accurayy
140 GPUa
Duri Dorn
Oas my
179
Oura
Stop
Stop
Stap
Rollour Size
Figure 6: (a) Training accuracy of Shuffle-RI on Geo3K. (b) Validation accuracy of Shuffle-RI on Geo3K. (c) Token utilization rate Shuffle-RI on Geo3K. (d) Shuffle-RI achieves better perfor- mance with minimal extra time cost.
Update=4
Update=
Update=12 12
Train Accuracy (%)
60
50
Ours
Ours
Ours
50
Val Accuracy (%)
45
40
35
4.0%
A 7.7%
4 - 7.2%
30
GRPO
GRPO
GRPO
0
10
20
30
40
50
0
50
100
50
100
150
200
GPU-Hour
GPU-Hour
GPU-Hour
Figure 7: Wall-Cloc Training Curve of Shuffle-RI compared with GRPO.
Efficiency Analysis. The improvement in model performance mainly stems from better training efficiency. Advantage distribution analysis in Fig. 5 confirms that, PTS effectively mitigates Advan- tage Collapsing by increasing the proportion the batch composition, enabling the model to focus
barge-magnitude advantages. ABS further optimizes
informative trajectories.
Fig. 6(a) and (b) further probe into training dynamics and demonstrate that our framework consis- tently achieves higher training and validation accuracy, reaching comparable performance GRPO with as little as half the training steps. Moreover, our framework effectively mitigates the issue of "Rollout Silencing" shown Fig. 6(c), maintaining high token utilization rate across all training stages. Fig. 6(d) further illustrates the favorable tradc-off between training scale and computational cost of our approach, uplifting training performance by a large margin.
Fig. 7 illustrates the actual wall-clock GPU time between GRPO and Shuffle-RI. Under the same update size and total training steps, Shuffle-R1 achieves substantially higher train/val accuracy than GRPO in the early training stage. More importantly, the total GPU time of Shuffle- R1 only increases by 4% 7.7% relative to GRPO. When targeting the same accuracy as GRPO, Shuffle-RI requires roughly half the number of training steps and approximately 60% of the total wall-clock time.
4.3 ABLATION STUDY
We conduct ablation experiments 2--n2.55VVL-BB Insttructttt using Geometry3K, focusing two objectives: (1) assessing the contribution of each component in our framework, and (2) validating the impact of key designs. More detailed results are reported in Appendix C.
8

---

Published as a conference paper at ICLR 2026
Component-wir wiipe Contribution. We evaluate the effectiveness of PTS and ABS by incrementally adding them to the baseline. As shown Tab. 4, On in-domain Geometry3K test set, PIS lifts accuracy from 42.64% to 46.21% % (+3.57). Adding ABS yields further +1.67, reaching 47.88%. Similar improvement trends appear on out-of-domain benchmarks: on the math reasoning set, the full setting (PTS ABS) attains 48.70% vs. 46.74% with GRPO (41.71% before RL training); ChartQA, we also observe effective performance gain (77.04% vs. 76.20%, with 73.08% before RL training) despite larger distribution shift. Our method also demonstrates improved performance in HallusionBench. To isolate the contribution of ABS, we conducted an experiment where ABS is applied to trajectory pairs formed by uniform random sampling instead of PTS. This setting (Tab. line 4) also demonstrates notable improvement over the baseline, confirming its effectiveness.
Analysis of Pairwise Trajectory Sam- pling. One core mechanism of the PTS lies in the structured contrastive sampling scheme. To validate the effectiveness of our bidirectional, contrastive sampling scheme, compare PTS against three alternative strategies: (1) One-way Posi- tive Sampling (only select trajectories with highest advantage); (2) One-way Nega-
Table 4: Ablation on effectiveness of PTS and ABS.
GRPO PTS ABS
Geo3k
Math Avg.
HallBench
ChartQA
25.79
41.71
59.83
73.08
42.64
46.74
63.09
76.20
46.21
47.64
63.40
76.52
46.82
47.79
62.77
75.12
63.19
77.04
tive Sampling (only select trajectories with lowest advantage): and (3) Unbiased Ran- dom Sampling. All settings maintained a consistent sampling ratio (8 valid trajecto- ries from rollouts) to ensure fairness. We disable ABS since it relies the pair- ing result of PTS. As shown Tab. model trained with PTS receives consis- tent performance gain on both in-domain and out-of-domain tasks, while both one- way positive and negative sampling result a performance decline even below the GRPO baseline, demonstrating the effec- tiveness and rationality of our design. Un- biased random sampling only receives mi- nor improvement over baseline, far behind the effectiveness PTS.
Table Ablation on rationality of PTS and ABS.
Setting
Geo3k
Math Avg.
HallBench
ChariQA
QW
25.79
41.71 46.74
$9.83 63.09
73.08
GRPO
42.64
76.20
Ablation PTS
only only min
41.26
44.77
63.30
75.64
23.36
41.52
60,98
74.36
random pick
43.53
46.62
63.19
76.00
PTS
46.21
47.64
63.40
76.52
Ablation on ABS
random shufile
46.05
47.40
63.19
76.60
reorder
46.28
47.64
63.09
76.64
47.88
48.80
63.19
77.04
Analysis of Advantage-based Batch Shuf- fle. ABS introduces Advantage-aware shuf- fling and Inter-sub-batch resampling reshape
225
1a M000 Coennn (96)
the training batch. To validate their effec- tiveness, designed two contrastive experi- ments: (1) Unbiased Shuffle: using uniformly distributed sampling weights perform shuffle strategy. and (2) Static Reorder: randomly re- order the training batch without sub-sampling, maintaining the original data distribution. We enable PTS during training as ABS relies on its pairing result. As shown in Tab. 5, model trained with ABS significantly outperforms the contrastive settings. The unbiased shuffle set- ting even performs worse compared to the PTS- only setting, demonstrating significance of weiggatn.. The static reorder setting has no improvement compared to PTS-only setting, as they have the same data distribution.
40
Step
Step
Figure 8: Ablation key hyper parameters. (a) Effect different sampling ratio a. (b) Effect of different shuffle times S.
Hyperparrameters. We investigate the impact of two key hyperparameters in framework: (1) the sampling ratio (a) in PTS, and (2) the shuffle times (S) in ABS. For a, we fix S and test values of 0.25, 0.5, and 0.75 (i.e., selecting 4, 8, and 12 samples from 16 rollouts). As shown in

---

Published as a conference paper at ICLR 2026
Table Extension experiments on Qwen2.5-VL-32B.
Setting
Math Verse
Math Vision
Math Vista
WeMath
HallBench
ChartQA
Avg.
Qwen2.5-328
57.0
38.2
75.4
72.9
71.3
80.7
65.9
GRPO
58.4
39.3
77.3
75.9
70.3
83.8
67.4
Ours
59.0
41.2
79.5
77.9
72.2
84.9
69.1
Table 7: Extension experiments on Referring Expression Comprehension task.
Setting
RefCOCOustt) RefCOCOnsd) Re(COCO+(aaa) RefCOCO+uun) RefCOCOgaen
Qwen2.5-3B
86.09
75.64
81.71
66.93
72.39
+GRPO
89.90
81.33
85.94
70.97
81.45
+Ours
91.83
84.31
87.84
76.27
86.07
Fig. 8(a), both a = 0.75 and a = 0.5 yield strong performance, while a = 0.25 lags behind. We attribute this over- veer-prniing where filtering out rollouts aggressively may reduce data diversity. We choose a = 0.5 for a balance between signal quality and computational efficiency. For S, we fix 0.5 and vary the shuffle times 4, 8, and 16. Fig. 8(b) shows that performance improves with increasing S, but saturates beyond S = 8. This suggests that moderate resampling enhances data exposure, but too many shuffles may offer diminishing returns, More details Appendix C.
4.4 EXTENSION EXPERIMENTAL RESULTS
Scaling 32B Model. We conduct experiments on Siven..5-LL32BB to inveritage the scalability and generalizability of Shuffle-RI on larger models. We train the 32B variant the selected 30k: data for 50 steps (to save training resources cost). Results in Tab. 6 demonstrate that Shuffle-RI also performs well on 32B scale.
Referring Expression Comprehension. For tasks beyond math/visual reasoning, we conduct an experiment Referring Expression Comprehension (REC). We train Qwen2.5-VL-3B 60K data randomly selected from RefCocoOReCCCCCOgReCCCCCCC in Tab. 7 show that Shuffle-RI be well adapted to soft-reward tasks and beyond.
using a loU-based soft reward. Results
Language-only Reasoning. While our initial analysis focused on the train- ing dynamics of MILMMs we conduct
Table 8: Extension experiments LLM.
Setting Math12K AIME24 MATH500 GSMSK GPQA Olymp.
an carly extension experiment val-
Qwen2.5- Mathh-MMhh--.5 I.58 1.5B-Base
idate the fcasibility of Shuffle-RI on LLMs. We train 0wen2.5-Math-11550
GRPO
67.6
10.0
66.0
74.0
25.7
33.3
Ours
70.4
16.6
71.0
79.6
36.8
Base (Yang al., 2024) Open-S1 dataset (Dang & Ngo, 2025) for 150
Qwen2.5 5-tn2.5--ath--- Math--B-BBse
steps and evaluate model performance on Math12K (Hendrycks et al., 2021). al..
+ GRPO 74.6
20.0
76.4
84,8
36.3
39.7
Ours
78.2
23.3
79.4
89.5
37.3
41.4
AIME24, MATH500 (Lightman
2023), GSM8K (Cobbe et al., 2021), GPQA-Diamond (Rein et al., 2024) and OlympiadBench (He al., 2024). As shown in Tab. 8, our framework delivers significant improvements all the bench- marks compared to GRPO, demonstrating its potential effectiveness on text-only LLMs.
5 CONCLUSION
In this paper, we introduce Shuffle-RI, a simple but effective framework designed to improve the training efficiency of reinforcement learning of multimodal large language models. Through Pair- wise Trajectory Sampling and Advantage-bate based Batch Shuffle, our framework significantly outper- forms representative algorithms and models in both in-domain and out-of- tasks, demonstrat- ing the value of data-centric adaptive design. We hope that our motivations, method, and findings helpful for further research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was done during the research internship of Linghao Zhu, Yiran Guan, and Dingkang Liang at Xiaomi.
This work was supported by the National Natural Science Foundation of China (NSFC) under Grants No. 62225603, 62576147 and 623B2038.
ETHICS STATEMENT
All training and evaluation data used in this project collected from public academic datasets. All the pre-trained model checkpoints and code framework employed are forked from open-source research projects. During the research process, we strictly adhered to the open-source licenses and usage requirements of all relevant parties. No non-public / private data was used.
Furthermore, all authors strictly complied with the ICLR Code of Ethics throughout the research process and affirm that no activities violating these ethical standards were involved.
REPRODUCIBILITY STATEMENT
We have reported the base model checkpoints, code framework, training data, evaluation bench- marks and metrics, and core hyperparameter settings used in the project in the main paper. A detailed pseudo code explaining our framework flow is provided in Appendix A. More detailed hy- perparameter settings and information about the training infrastructure are provided in Appendix D. In addition, we have also included the prompts used during training and inference, examples of the training and evaluation data, as well as examples of the model inference results in Appendix E, and G.
LLM USAGE STATEMENT
During this research process, the usage of LLM strictly restricted and was used in limited ways: (1) GitHub Copilot was employed during code development for code checking and bug fixes; (2) LL.M-based tools were used during paper writing for language refinement and grammar correction, and (3) LLM judges were applied as a part of evaluation metrics, including key answer extraction and correctness assessment.
All core components of this work, including research motivation, algorithm design, core code imple- mentation, data collection, and experiment analysis, and experimental conclusions, were conducted entirely and independently by the authors without reliance on LLMs.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report, 2024. URL hww.h:ttp//.tc...tt httpepp::///axvv../e www.aaaivvvoo...... abs 220033332000 08774.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in 1lms. In Annual Meeting of the Association for Computational Linguistics, pp. 12248-12267, 2024.
Anthropic. Claude 3.7 Sonnet, anthropic.com, 2025. URL claude/ sonnet. 2025-02-24.
com/
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report, 2025. URL https://arxiv. org/abs 2502. 13923.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training rl-like reasoning large Aision-language models. In Transactions Machine Learning Research, 2025a.

---

Published as a conference paper at ICLR 2026
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025b. URL https: / arxiv. org/abs/ 2412. 05271.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. In Proc. of Intl. Conf. on Machine Learning, pp. 10818-10838, 2025a.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: simple and strong reinforcement learning baseline for model reasoning, 2025b. URL atttss///axiv..oo abs/2504 02546.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.
org/abs/ 22100.2211.. 14168.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ilms: What works and what doesn't 1, 2025. URL setps://arrxor wrg/abs///aeeeoooeeeeeooeee /2503. 16219.
Google Deepmind. 2025.
Gemini 2.5: Our most intelligent AI model - blog.google,
URL https: http///gg..gggg.eeeee/oooooooo/e gemini mooeell-modll--- weweelltttiniiggg-tte updatesss
derem-deeeennddd
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-impprvement, 2025.
URL https: /arxiv..vvvvvv////ia/eee///aaav/
org/ abs/ 17352.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- lsnguala models. In Proc. of 2024.
IEEE Intl. Conf. on Computer Vision and Pattern Recognition, pp.
Yiran Guan, Sifan Tu, Dingkang Liang, Linghao Zhu, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, and Xiang Bai. Thinkomni: Lifting textual reasoning omni-modal scenarios via guidance decoding. In Proc. of Intl. Conf. on Learning Representations, 2026.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence, 2024. URL https: /awpivaiivvv//aa////a//.a.aaaaaa org/ abs 2200244111.00 14196.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in 1lms through reinforce- ment learning. Nature, 645(808) 1):633)6333 638, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 8828-3850, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks. 2021.
Jian Hu. Reinforce++:: A simple and efficient approach for aligning large language models, 2025.
URL /tttps://arxa
or.oorgggae...... abs 2501 03262.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models, 2025. https:// wwtps/wwwwwwww./.r//...oese/ 02500.6699. 06749.
12

---

Published as a conference paper at ICLR 2026
Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, and Aviral Kumar. What do learning dynamics reveal about generalization in Ilm reasoning?, 2024. URL hwtpt:tt////attttt
org/ abs/ 07681.
Team Kimi. Kimi kl. 5: Scaling reinforcement learning with 1lms, 2025. URL https: //rxxv..avvvvvv
org/abs/
Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang. Hang Zhang, Yuming Jiang. Xin Li, Fan Wang, Yu Rong, Aixin Sun, and Shijian Lu. Mmrl: Advancing the frontiers of multimodal reasoning, 2025. URL https: /ttp:::///itthb.../ com wttpe//oo//onciooog/iioooa
Xinhao Li, Ziang Yan, Desen Meng, Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforce- ment fine-tuning, 2025. URL :// arxiv, org. hbtg 2504. 06958.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In Proc. of Intl. Conf. on Learning Representations, 2023.
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. In Proc. of Advances in Neural Information Processing Systems, 2025a.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning- chain guided segmentation via cognitive reinforcement, 2025b. URL https:// Aaxiv.orge abs/2503 06520.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2034 2044, 2025c.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Annual Meeting of the Association for Computational Linguistics, pp. 6774 6786. 2021.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-
Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The 3rd Workshop on Mathematical Reasoning and Al at NeurlPS'23, 2023.
Yan Ma, Linge Du, Xuyang Shen, Shaoxiang Chen, Pengfei Li, Qibing Ren, Lizhuang Ma, Yuchao Dai, Pengfei Liu, and Junjie Yan. One rl see them all: Visual triple unified reinforcement learning, 2025. URL https: aaxiv.iiii.../... org/ abs 2505. 18129.
Ahmed Masry, Xuan Long Do. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. In Annual Meeting of the Association for Computational Linguistics, pp. 2263-2279, 2022.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, al. Mm-eureka: Exploring the frontiers of multi- modal reasoning with rule-based reinforcement learning, 2025. URL https: http://axxvvvorro abs/2503. 07365.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In Proc. of Conf. on Empirical Methods in Natural Language Processing, pp. 20286- 20332, 2025.
OpenAl. OpenAl ol System Card - openai.com, 2024. URL httttpps//cd...oeeee com/ o1 o1-system-ce ca .df.
13

---

Published as a conference paper at ICLR 2026
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xim Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl, 2025. URL https:/ www.v.org.abs//20000.. 07536.
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multi- modal model achieve human- mathematical reasoning? In Annual Meeting of the Association for Computational Linguistics, pp. 20023-20070, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In Proc. of Conference on Language Modeling, 2024.
ByteDance Seed. Seed-thinking-vl. 5: Advancing superb reasoning models with reinforcement learning, 2025. URL hattps://arx:t www.vvorg//aaae httg/abs/2500 13914.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL
/0rrg///b//22...... 03300.
Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning, 2025. URL
https:// /arxivivvvv. org/abs/////oooooooooo..o.o.oo.r 2503 20752.
Omkar Thawakar, Dinura Dissanayake, Ketan Pravin More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Ilmuz Zaman Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-ol: Rethinking step-by-step visual reasoning in llms. In Annual Meeting of the Associa- tion for Computational Linguistics, pp. 24290-24315, 2025.
Haozhe Wang, Chao Qu, Zuming Huang. Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. In Proc. of Advances in Neural Information Processing Systems, 2025a.
Ke Wang. Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In Proc. of Advances in Neural Information Processing Systems, volume 37, pp.
95169, 2024.
Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. In Proc. of Advances in Neural Information Processing Sys- tems, 2025b.
Ye Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenx- uan Wang, and Qin Jin. Time-rl: Post-training large vision language model for temporal video grounding. In Proc. of Advances in Neural Information Processing Systems, 2025c.
Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Ron- grong Ji. Grounded tfain-of-thouuht for multimodal large language models, 2025. URL.
https: //arxavvvvvv//////////eee/eaaav/ org/ abs/ 220225533111 127999.
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song. Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. In Proc. of IEFE Inil. Conf. on Computer Vision and Patlern Recognition, pp. 2087-2098, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmnt, 2024. URL https: //wp///avvvvv///////////// wwwpxivv.orr//ooooooe abs//b///////// 2499 121 22.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal rea- soning through cross cross-modal modal formalization, 2025. URL https: //aaiiv////ivv////// org/ 10611.
14

---

Published as a conference paper at ICLR 2026
Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng. Li Shen, et al. Mulbenr: Empowering milm with ol-like reasoning and reflection via collective monte carlo tree search, 2024. URL httttt:s/aaxiiiii org/ /bs/ 2412. 9.19.
Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, al. RI-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo. In Proc. of Advances in Neural Information Processing Systems, 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Proc. of Advances in Neural Information Processing Systems, volume 36, pp. 11809-11822, 2023.
Zheng Yaowei, Lu Junting, Wang Shenzhi, Feng Zhangchi, Kuang Dongdong, and Xiong Yuwen. Easyrl: An efficient, scalable, multi-modality training framework, 2025. URL https:// github. com SoayyuuggaEaa......
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025. URL http:: httpaaaxivvi.i org/ abss/2511. 19393.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL https: arxiv. org/ abs 22003.53..222 14476.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; Llm self-training via process reward guided search. In Proc. of Advances in Neural Information Processing Systems, volume 37, pp. 64735 64772, 2024a.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025. URL https: / arxii.iiiivv.aaaia wttg//bb///533.1223377
Renrui Zhang. Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? In Proc. of European Conference on Computer Vision, pp. 169-186. Springer, 2024b.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization, 2025. URL https: // /awxxvivvv.../////v//aa////aaava org/ 2b5//50../22111111 18071.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu. Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https: / //arx//i///////////////// crrg/bbs/2500 10479.
15

---

Published as a conference paper at ICLR 2026
A PSEUDO CODE
Here, we provide the pseudo code of Shuffle-R1 for readers to better understand the pipeline flow, and for better transparency in algorithm understanding and reproducibility.
Algorithm Shuffle-RI Workflow
1: Input: (queries), (policy), 2.V (rollouts per query), (sampling ratio), S (shuftte rounds)
2: Output: Optimized batch for gradient update
3: Initialize global batch <
4: for cach E Q do
PTS Phase
5: Generate [o:
To (4)
6: Compute {Ar} via R = uen
7: Sort pairs: {((ss) o2 + MaxMinPair(({A.))
8: Retain top- [N pairs: B, + (o O22 -4)
9: Aggregate: B <- B B,
10: end for
Compute W,, = |Apl + |ågl V(op. 0o) E
Phase
12: Calculate P, = / Wi for weighted sampling
13: B' + ShufficSample(B, P,, S) with S x T = [
14: return 8
Jointly optimized training data
B PROMPT DESIGN
We use "Thinking prompt" to explicitly control the output format of the model, which requires the model output its thinking process within special tokens <thi nk> and </thi nk>, and mark the final answer with \boxed J. In practice, we keep the system prompt of Qwen2.5-VL (Bai al., 2025), and insert the "Thinking prompt" at the beginning of user message. We keep the training and evaluation prompt in the same format. The full structure of instruction prompt is as follows:
Prompt Example
SYSTEM:
You are a helpful assistant.
USER:
You FIRST think about the reasoning process as an internal monologue then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put \boxed (). <QUESTION>
C MORE EXPERIMENTAL RESULTS
Detailed Model Performance. We provide a more detailed performance of models trained on Ge- ometry3K and K12 dataset in Tab. 9, reporting model performance on each out-of-domain bench- marks as supplement to "Math Avg." columns in main paper. Trained with only 2. 2.1k data, both 3B and 7B model demonstrate significant performance gains.
Full Setting Comparison. Improved training algorithm leads to improved performance. To better reveal the model performance beyond the limited data scale of Geo3K, we conduct full 30k-sample 150-step experiment on Owee2.5-VL-77. As shown in Tab. 10, Shuffle-RI achieves clear and con- sistent gains over strong baselines across all six evaluation benchmarks. This result further proven the robustness and scalability of Shuffle-RI.
Comparison with more RL algorithms. We provide additional comparisons with more repre- sentative RL algorithms, ie. RLOO (Ahmadian al., 2024) and REINFORCE++ (Hu, 2025). We conduct experiments on Qwen2.5- Dwen2.5-VL--BB Instructtt with Geometry3K dataset. All the experiment settings are kept the same GRPO/DAPO/GSPO in main paper. The final model performance is
16

---

Published as a conference paper at ICLR 2026
Table 9: Detailed performance on out-of-domain benchmarks of models trained on Geometry3K and K12 data. Highest accuracy marked in Bold.
Method
Math Verse Math Vision
Math Vista
WeMath
HallBench ChartQA
Total Avg.
Qwen2.5.VLL VL-BB
34.77
21.94
58,40
51.72
59.83
73.08
+ Ours (Geo3K)
43.55
25.30
61.80
64.14
63.19
77.04
55.84
+ Ours (K
44.06
26.48
64.90
64.77
65.72
78.28
57.37
Qwen2.5-VI.n VL
42.59
25.76
67.40
63.51
65.19
79.84
57.38
+ Ours (Geo3K)
50.96
27.47
70.90
69.19
69.51
81.64
61.61
+ Ours (K12)
48.59
28.61
73.20
69.71
69.87
82.60
62.09
Table 10: Full 30k experiment Qwen2.5-VI-7B. Highest accuracy marked in Bold.
Method
MathVerse
MathVision MathVista WeMath
HallBench CharlQA
Total Avg.
Qwen2.5-VL VL-7B
42.6
25.8
67.4
63.5
65.2
79.8
57.4
GRPO
50.6
28.3
74.5
69.7
70.7
81.4
62.5
DAAPO
51.4
28.8
75.3
71.5
71.0
82.8
63.4
+ GSPO
28.2
75.3
70.1
69.7
82.9
62.8
+ Ours
53.9
30.0
77.0
72.3
71.0
84.1
64.7
shown in Tab. 11. Our framework outperform these algorithms by large margin in both in-domain and out-of-domain tasks.
Comparison with Prioritize Experience Replay. We conduct a comparative study by replacing ABS with prioritized experience replay mechanism. Experience replay maintains decoupled buffer of past samples, whereas ABS adopts an online, in-place shuffle strategy to dynamically reshape the data distribution. As shown in Fig. 9()) and (b), the later stages, the experience replay setting exhibits plateau in training accuracy and even drop in validation accuracy, indicating potential overfitting to stale samples. This suggests that prioritized experience replay may overly emphasize historical trajectories, leading suboptimal convergence. Moreover, ABS proves to be more effective in mitigating the Rollout Silencing.
Detailed Impact of Hyperparameters. Here, we provide detailed model performance compari- son under different hyperparameter settings in Fig. 12, as supplementary information to correspond- ing ablation experiments main paper. For sampling ratio (ar), both the in-domain and out-of- domain accuracy improves as we increase from 0.25 to 0.5. The model performance shows a mild decrease when a is further raised to 0.75. This result demonstrates again that not all the rollouts contribute equally positive to RL training. For shuffle times (S), the model performance receives consistent gain when is increased from 4 to 8, but declines when S is set to 16. This result sug-
Durs
60
Exp
45
Nonzero Grad drat Ratto
0.95
50
Val Accuracy (%)
Accuracy
40
20
40
60
80
0
20
40
60
80
20
40
60
80
Step [a]
Step (b)
Step
(c)
Figure 9: Performance of Shuffle-RI compared with directly applying Prioritized Experience Re- play.
17

---

Published as a conference paper at ICLR 2026
Table 11: Performance Shuffle-RI on Geometry3K dataset compared with RLOO and REIN-
FORCE++. Highest accuracy marked in Bold.
Method
Geo3K
Math Verse
Math Vision
Math Vista
WeMath HallBench ChariQA
Qwen2.5-VL-3B
25.79
34.77
21.94
58.40
$1.72
59.83
73.08
RLOO
42.09
39.94
22.96
58.90
59.48
64.24
76.68
REINFORCE++
41.76
41.70
24.86
60.90
63.51
62.99
76.20
Ours
47.88
43.55
25.30
61.80
64.14
63.19
77.04
Table 12:
Performance of Shuffle-RI under different hyperparameter settings.
Highest accuracy
marked in Bold.
Setting
Geo3K
Verse
Math Vision
Math
WeMath
HallBench
ChartQA
Ablation sampling ratio (S 8)
0.25
45.75
42.43
23.05
60.20
63.62
63.09
75.36
0.50
47.88
43.55
25.30
61.80
64.14 14
63.19
77.04
a 0.75
47.41
41.70
25.62
60.90
64.54
62.99
73.36
Ablation shuffle times S (o 0.5)
S =
45.92
42.21
24.78
62.30
64.08
62.25
75.60
47.88
43.55
25.30
61.80
64.14
63.19
77.04
16
47.23
43.70
24.86
60.90
62.64
62.20
74.84
gests that appropriate repeated training on high-quality samples enhance the model's reasoning ability without affecting data diversity.
D EXPERIMENT SETTINGS
We report details of our training and evaluation settings here, including reward function design, main hyperparameters and computing resources.
Reward Calculation. mat reward and accuracy reward as the final reward reinforcement learning. The format reward and accuracy reward are calculated as follows:
adopt combination of for-
Table 13: Hyperparameter settings.
Hyperparameters
Value
Frocmat
1, if format is correct
(10)
max pixels min pixels
1000000
0, if format is incorrect
262144
max prompt length
2048
max response length
2048
1. if answer ground truth
(11)
rollout batch size global batch size learning
512
0. if answer स ground truth
128
le-6
The final reward is the weighted sum of above rewards:
optimizer
Adam W
rollout temperature
1.0
Tovemal = 0.1 Ploraa + 0.9 x Tace
(12)
rollout top
0.99
evaluation temperature
0.5
Format reward is assigned to smaller weight since re- sponse formatting easy to learn.
rollout group number
16
PTS sampling ratio (a)
0.5
ABS shuffle times (S)
Hyperparameters. We use EasyRI (Yaowei et al., 2025) as our training framework. Full hyperparameter settings during training shown in Tab. 13. For experi-
KL coefficient
0.0
vision encoder
frozen
ments on Geometry3K and K12 with ~ 2.1k training samples, we set the training steps to 80. For the joint training experiments (~ 30k training samples), we increase the training steps to 150 due to extended data size. Other hyperparameters that are not mentioned are kept to default values of Ea
Computing Resources. All experiments conducted 8x 80G GPUs.
18

---

Published as a conference paper at ICLR 2026
E DATASET COLLECTION
For training dataset, we choose Geometry3K (Lu al., 2021) and MMK12 dataset. Geometry3K dataset is high quality real-world geometry prob- lem solving dataset, containing 2.1K training sam- ples and 601 test samples. It contains wide range of geometry problems with varying levels of dif- ficulty. The text problems very compact with most of the geometric conditions represented in im- ages, making it suitable for training. MMK12 dataset is introduced by MM-Eureka (Meng et al., 2025), which contains 16k math reasoning training samples. The training samples have both geomet- ric and non non-geometric problems and have been care- fully examined and manually filtered to ensure qual- ity. The test set of MMK12 further includes other STEM problems such physics, biology and chem- istry. During our training. we used a randomly se- lected subset of MMK12 provided by NoisyRoll- out (Liu et al., 2025a) (referred to as 'K12' in the pa- per distinguish it from full set of MMK12), which has 2. k samples with the same size as Geometry3K, All the training samples in free-form format. Ex- amples of training data shown in Fig. 10.
Queeiux:
Asswer:
Question
Firs
Answer:
Question:
Wiech
relationalip and 21% C.y. P.D. D logut.
Questtoo::
graph fixt the aogment conndrates points (0-4), (6,4), respectivlly 808220
Figure 10: Top: Examples of Geometry3K. Bottom: Examples of MMK12.
Qucition:
Aseame are identical size. cylinder
beight Find surfice arza
To fully evaluate model's reasoning ability, apart from in-domain test set from training dataset, we select several representative benchmarks to exam- ine model's performance sual perception, and chart understanding tasks. For math reasoning task, we choose Math Verse (Zhang al., 2024b), Math Vision (Wang et al., 2024), We- Math (Qiao et al., 2025) and Math Vista (Lu et al., 2023). We choose HallusionBench (Guan et al., 2024) and ChartQA (Masry et al., 2022) for vi- sual perception and chart understanding tasks re- spectively. We believe that good RL algorithm not only improve in-domain performance, but also have the potential to promote robust generalization to out-of out-of-domain tasks. Examples of evaluation data shown in Fig. 11.
Question:
Wette the equation of
Aaswer
math reasoning, vi-
af-) (a 2)*
Quatiuu:
mag, the Democrate
Questivn:
For image, the colome tiles and the samee? Aaswer. Yeu
Quastion:
What was the 4th most populaz cmaction?
Aaswer
Inspired
F EVALUATION METRICS
Qurelan:
what the sam Beoommble vali: in the your 2014 and 2015?
We adopt pass@ accuracy as evaluation metric. During evaluation, set the decoding temperature to 0.5 and perform independent runs, reporting the average pass@ 1 accuracy as final metric. The choice of temperature doesn't 't degrade model perfor- mance, and the averaged accuracy reduces the ran- domness, resulting in a more stable and reliable eval- uation. Prompt format for evaluation is kept identi- cal to training prompt. Evaluation settings of our model and all the reproduced results in main paper are kept the same to ensure a fair comparison.
Answer: 95
Figure 11: Top: Examples of math reason- ing tasks. Middle: Examples of visual per- ception tasks. Bottom: Examples of chart understanding tasks.
For evaluation of Math Verse, Math Vision, WeMath, Math Vista and ChartQA. employ Gemini- 2.0-Flash-001 (Deepmind, 2025) to first extract the predicted answer from model response then compare it with ground truth. Fig. 12 and Fig. 13 demonstrate the extraction and verification prompt
19

---

Published as a conference paper at ICLR 2026
Pleas maa neorgle
Thes supu the mnon corndd fhem the madd in spae diccaty tattncdd an servece
Dolre two ssscess so wath qustkesitart undertrading quodits Quartaa is (Qantiee)
(Suaded Anrwot the mastad aanwer to quardias and (Modd sarvaf the sarwar coracte fore modil coeuut the qanaioe Deternar whether tiese non eocerr
reogunsion rog en eange eeswe and pooiie
Pless waitha only when the [Micee] proces] complesly rucleeste (fracled scod jawar consiect For Terneeeuiippee plcccpitee quoriiss monnnn cxpie inthcaum smny Riaadu coodleedd cas leset dhe exumple and Steus. trig cass oues Addgement a lfitey ar ditteras, dudgeee
Qpcatee Which numberr measngt
ceissing todta regrace
theopeaioo reg eeprring tearingg pairt wonder decinal posile
(Quatket: teak thead namben reprovataa cumberr teurcal ructitoon Panded Ameott)/e211
(Madd soeet) Tansaed Aurwor 1) 1M Indgreent
Qpootee femsico lecing he coment
smpense ofuttional Eacing
Po dove figare click Oiaes radia stangic BAC 60.8, dhzs lougl C
Chezse
Pleae enveer oepo requiing theinggeent sumterr dociaal places provice
AG2 A cosp waata
Bianded Amewotic
(Modk survett 042111911 Jndgrawet
money bey spple woly coudy??
Dirntont: domen and nungout hentive lawing interaal relidtoos Acrnet)
[Mald] moee)] Reage 40-4 1g Judgremen
pravide
Kragle lageh
Qeeakie: Whar Fracios art stope Mour?
diresty vidraocag
QUESTION
Auceet:
Extastdd
Figure 12: Extraction prompt for Gemini.
Figure 13: Score prompt for Gemini.
for Gemini. Specifically, we report accuracy on WeMath under loose mode, and overall accuracy on Math (including all sub categories: Text Dominant, Text Lite, Vision Intensive, Vision Dominant and Vision Only).
G CASE STUDY
Here we provide some qualitative case study of Shuffle-Rl's reasoning outputs.
Fig. 14 and Fig. 15 show two improved cases on math related tasks. In case 1, the base model has an error in visual information parsing, referring to angles not existed in the figure. resulting in reasoning fault. The RI. model correctly parsed and solved the geometry problem. In case 2, the base model misused geometric theorem, leading to wrong answer, while the RL model correctly solved the problem with accurate theorem.
Fig. 16 demonstrates improved reasoning ability in RL model, where the base model has accurate perception about the chart but has reasoning error in COT. The RL model can not only accurately parse the visual information, but also perform correct reasoning to finally reach the right answer.
Case 4 in Fig. 17 further demonstrates that RL model also has better visual perception ability com- pared to base model. The figure from HallusionBench has been artificially inserted with an image of a hen, which only occupies very small region of the original image. This modification has resulted in a perception error and in base model, but the RL model can accurately identify the inserted image.
H THEORETICAL ANALYSIS OF SHUFFLE-R1
In this section, we provide an intuitive analysis on how the adaptive selection & resampling improve the training dynamics, and discuss the bias introduced by this process. We begin by assuming the Advantage(A) A) follows a Gaussian distribution, A \((0,")) which matches the observation of our probe analysis.
We define the original expectation of A as:
Fagri = E[[
(13)
20

---

Published as a conference paper at ICLR 2026
We simplify the PTS as truncate process with threshold O, and define the kept set SPTS { > 7h the corresponding conditional expectation is:
EPTS = E||A|| |A] 2 T].
(14)
On the truncated set, we sample with weights w(a) x |a]. and the resulting expectation is denoted as:
Enow Elu((a) al |a| STT)).
(15)
H.1 INTUITIVE ANALYSIS ON TRAINING DYNAMICS
Proposition 1 (Amplification on expectation of Advantage).
Proof. Define X |A| on the SPTS, the weighted sampling probability density can be expressed as:
p(p S p((2 S
Prew
E[X
SPIS]
EPIS
(16)
Thus the weighted sampling expectation be calculated as:
/
p(æ SPT)
||X S
Enew
x.
de
(17)
EpTs
X SPIST
Epts
Based on Cauchy Schwarz inequality, derive that:
[B[x)] 2 (B[X))² = Enew EPTs
(18)
Further, the original expectation can be decomposed as:
Furigg (1 P-) ||A| < T] E|||| | 7]. where p+ P(IA) 7).
(19)
Since
E[[A|
|A| < <<<<<TT < B4
(20)
have EPTS Foorig Finew > Pariig
Proposition 2 (Amplification on gradient norm).
Proof. In the policy gradient training, the per-sample gradient can be expressed as:
A.u. u.
Valogma (o).
(21)
|oll| |A|] ||u|].
(22)
Derived from the standard assumptions of policy gradient that ||u|| is approximately independent from AA,, we have:
E Ball
Fnow [||[] ||] > B E|]l B(llg
(23)
which turn yields larger update magnitudes per optimization step.
H.2 BIAS UNDER SELECTIVE SAMPLING
Proposition 3 (Adaptive sampling introduce positive bias). Proof. Under the Gaussian distribution assumption, the expectation can be calculated
1 7 / alalpla)da + f." alalp(a)da)
Enew [A] =
2 S.
a p(padda > 0
where the normalized factor Z is defined as:
lalp(a)I((a] > 7)da
2 a p(a)da
The positive bias indicates that the gradient update is systematically tend to trajectories that has higher advantage. 0
21

---

Published as a conference paper at ICLR 2026
USER Input
Base Model
<hink- To sobee for the movem of LADC we need fae propeties of angles Sa und by taaseessal isto secting paallee lines.
-thak Stace laas and paralie,. ZABC can found by canideeris eegles foroed the intensctioon of trasvesel DC with the pasliee lines
In the figure below, LABC is intersected by parallel lines What the LABC! Express your answer degrees.
Menniyy
réforationn: 28A0 58
<ACB 33" liees and posilat the asgles
Spocificilly -ABC ieclded angle
calcolaaee
mtangle always
adilig edjazon angles ether
oqaaton
angles
LABC
mangle ADC
asale betwoen two
ZBAC 1 ZADC LACY 380" Sabochite the givce values ino equation:
calculased aokding adjocrt anglo ether side the vertes B Since LABC
as + ZABC + 33 180 simplity the
included ingle betweee two lires,
expetion: 71" a LADC 189" Salve for LABIC: ZAD0 1897
can bu coloutatted by viding the sijoore asgles or eiller side of vatee This, the LADC the angles ZABC 35" [21
Thes.
the momore of ZABC's [109..
Figure 14: A case study on math related task.
USER Input
Base Model
RL Model
the vakas of
fact
stienic- san intriie anglex
that the of intrrir angles of peraagon 540 pertagoo is divided iaso daze triauges, we can use foxt that the suci of ongles each trangle 150 Fist, lefs latel the angles in the pantipoon anale 1) fe-+
quaditaseen 160 degrees We can cuuation using the pivan experssoos for ti: axgles.
Find the of
(a 4) (r-+ 110 (e (9)
Simpliiynn the lest aide
Anotie
10 900,
(x 4)
ta
flon
36
regle 4j sun
segles parager
frac2054
x-+ (e++ 4) +
10) (it - 6)
the value T4laettiikk
Ux 101
Stagliy the loft side.
+ a-- 4-+
10
54t
ca + 500. Subtract 64 from both andos
Ge 476, Devide 4.3 119. So, the value io (19)
Figure 15: Another case study on math related task.
USER Input
Base Model
RL Model
From 2001 0. 2005
exported the most value of bread, biscuits, wafers, cakes and pastries UK? only be yes no.
expart value of Trcad, baculs, wales, cakes.
and postiles foo the UK from 2001 2010.
<think-TTn To deaemin 1005 lod the hights
-dieal--Foon the s-ank. the expert value in
aprcas
2001
need
answer
perkot
groph The
2001, pask
export steullly
The pesk
souma around titii---iin--
but export
compared to perkod
beduee 2005. Siace expart "alae is continonusly increasing fron 2007 2005 and then contines to incoss bur at slower rate, it ressonable to conclade that the highou moschal 2005 firaal yes-ithino- You
Figure 16: A case study chart understanding task.
USER Input
Base Model
"Shiic-Thee imaze is diggran of vusioas diressur, inchuting hetbivones and
<tinnk- The image is diapam of varicus dincwaure and Bick eviluiieaay edlateedigg digg.. ineludes taneliae showing the timtline of dinowaer evolution visal coprovetation also
any figure?
You final answer can only he yes
"Archoooprry"
betwon dinosur binks. There are depinedd irrqpe
dapisted
imags, oott-ttikk--- Yeil
Demostiatal chickens, whiki are not shown this disaan--ittiile Ne.
Figure 17: case study on visual perception task.
22


Reviews:
[{"review_id": "bffaTuUFvE", "reviewer": "Reviewer_5QmF", "summary": "The paper proposes Shuffle‑R1, a data-centric RL training wrapper for multimodal LLMs that addresses advantage collapsing and rollout silencing by pairing high-contrast trajectories and shuffling batches based on advantage magnitude. The method is simple, practical, and empirically effective, showing improved accuracy and efficiency over strong RL baselines on math and vision-language benchmarks.", "questions": "1. While steps and batch sizes are reported, it is not fully clear that all baselines (GRPO/DAPO/GSPO/RLOO/Reinforce++) were run with identical token budgets, rollout counts (2N=16), and decode temperatures. Small differences can swing math benchmarks materially. Is it possible to add a compute-matched table and a training curve wall-clock plot to substantiate the \"7% overhead\" claim across settings?\n\n2. HallusionBench results improve, but do PTS/ABS reduce refusal or increase over‑assertion? Any calibration metrics or abstention analysis?\n\n3. How does the pair selection affect solution diversity? Any evidence of collapse in reasoning styles (e.g., fewer distinct CoT patterns)?\n\n4. Are there tasks where advantage distributions are already well‑spread (e.g., dense/step rewards), making PTS/ABS less helpful? Negative results would help practitioners choose whether to use the proposed method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The strengths of the paper are:\n\n- Simple, practical method that is easy to implement on top of GRPO, which addresses a real pain point in RL fine-tuning: many trajectories are statistically uninformative.\n- Results are strong and it shows effectiveness on the good coverage of in-domain and out-of-domain benchmarks.\n- Detailed experiments with ablation studies.", "weaknesses": "The weaknesses of the paper are:\n\n- Theoretical analysis of bias/variance under selective sampling is limited; unbiasedness is not proven.\n- Missing some clarifications and ablation study", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7V4QDGe7g", "reviewer": "Reviewer_Codo", "summary": "Shuffle-R1 is a data-centric RL fine-tuning method for multimodal LLMs that combats Advantage Collapsing and Rollout Silencing. It introduces Pairwise Trajectory Sampling to extract high-contrast advantage pairs and Advantage-based Batch Shuffle to over-sample valuable pairs during mini-batch construction. On 2–30 k training samples, 3/7 B models outperform GRPO, DAPO, GSPO and match GPT-4o/Claude-3.7 on MathVerse, MathVista, etc.", "questions": "- Have you empolyed Shuffle-R1 on larger models (e.g., 30B+ parameters) or other domains beyond math and vision reasoning? What are the anticipated challenges?\n- Can you provide examples where Shuffle-R1 underperforms or fails to improve over baselines? What are the limitations of your method in its current form?\n- Can you provide a more formal or intuitive explanation of how PTS improves gradient estimation? For example, how does selecting high-contrast pairs reduce gradient variance or improve signal-to-noise ratio?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Diagnose Advantage Collapsing & Rollout Silencing in MLLM-RL; proposes contrastive pairing + advantage-weighted reshuffle instead of larger rollouts or reward re-design.\n- Extensive ablations (α, S, PTS variants), 8 datasets, 2 model scales; statistical gains significant; extend to LLMs; code & pseudo-code provided.", "weaknesses": "- While the empirical results are strong, the paper lacks formal analysis or theoretical justification for why PTS and ABS improve training dynamics. For example, it would be helpful to show (even intuitively) how contrastive sampling improves gradient variance or convergence rates.\n- While Shuffle-R1 outperforms GRPO, DAPO, and GSPO, it does not compare with other data-centric RL methods such as curriculum-based sampling, which are relevant to the idea of reusing or reweighting data. A short discussion or comparison would strengthen the positioning of the work.\n- While ablations on α and S are provided, other design choices (e.g., max-min pairing, absolute advantage weighting) are not thoroughly explored. For example, would cosine similarity or entropy-based weighting perform better?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pNYVYqAHSb", "reviewer": "Reviewer_gi6w", "summary": "This paper proposes Shuffle-R1, a data-centric reinforcement learning framework that targets two major issues in current RL fine-tuning pipelines: Advantage Collapsing (most advantages cluster near zero) and Rollout Silencing (the share of rollouts with non-zero gradients keep dropping).\n\nThe method introduces two simple modules: Pairwise Trajectory Sampling, which selects high-contrast trajectory pairs to strengthen gradient signals, and Advantage-based Batch Shuffle, which dynamically reshuffles batches to reuse more informative samples.\n\nExperiments across multimodal reasoning benchmarks show consistent gains over GRPO, DAPO, and GSPO with almost no extra computational cost.", "questions": "Overall, this paper provides a well-motivated and practically useful data-centric perspective on improving RL efficiency for multimodal LLMs, there are several questions:\n1. Since ABS repeatedly exposes high-advantage trajectories, how does the framework prevent over-exploitation of a narrow subset of rollouts? \n\n2. The experiments are convincing but domain specific. Does the author expect the same efficiency gains for RL tasks with sparse, delayed, or non-verifiable rewards (e.g., open-ended QA)? If not, what modifications would be necessary?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper pinpoints two concrete and observable issues, \"Advantage Collapsing and Rollout Silencing\", which intuitively explain why current RL pipelines waste computation and fail to leverage informative signals. This diagnostic perspective is well-motivated.\n\n2. Instead of modifying the reward model or policy objective, Shuffle-R1 improves RL efficiency purely from the data side through Pairwise Trajectory Sampling (PTS) and Advantage-based Batch Shuffle (ABS). Both modules are lightweight, easy to implement, and can plug into existing frameworks without architectural changes. And the proposed framework shows consistent improvements across different datasets, model scales, and reasoning benchmarks with less training steps and GPUs.\n\n3. The paper presents clear ablation studies and analyses showing how PTS mitigates advantage collapse and how ABS maintains token utilization over time. This makes the method’s effectiveness both transparent and reproducible.", "weaknesses": "1. **Overfitting to high-advantage samples**\nSince ABS repeatedly exposes high-value trajectories, the framework might bias the model toward a narrower distribution of “reward-dense” samples, reducing exploration and long-term diversity. \n\n2. **Scope of benchmarks**\nMost experiments are on math or visual reasoning tasks; while results are strong, these domains already have dense reward signals. It remains unclear whether Shuffle-R1 would bring similar benefits on tasks with sparse or noisy rewards (e.g. open-ended QA, safety...)\n\n3. **Lack of comparison to recent adaptive-sampling paradigms**\nRecent works like LIMO also tackle signal efficiency through adaptive data selection and contrastive training. Including these would further enhance impact within the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w8yUc6ecJM", "reviewer": "Reviewer_u5Bq", "summary": "This paper identifies two key inefficiencies in reinforcement learning (RL) fine-tuning for multimodal large language models (MLLMs): Advantage Collapsing—where most advantage estimates cluster near zero, weakening gradient signals—and Rollout Silencing—where fewer rollouts contribute useful gradients as training progresses. To address these, the authors propose Shuffle-R1, a data-centric RL framework featuring: (1) Pairwise Trajectory Sampling, which forms high-contrast trajectory pairs by matching high- and low-advantage rollouts to amplify informative signals, and (2) Advantage-based Batch Shuffle, which dynamically reshuffles training batches to prioritize high-value samples based on their advantage magnitudes. Experiments show Shuffle-R1 consistently outperforms strong baselines (e.g., GRPO, DAPO) across multiple multimodal reasoning benchmarks, achieves competitive results with leading closed-source models like GPT-4o and Claude-3.7, and even matches prior methods using only half the training steps—all with minimal computational overhead. The framework also generalizes to text-only LLMs, highlighting its broad applicability.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It demonstrates high quality through rigorous experiments across model scales, datasets, and multimodal benchmarks, supported by thorough ablations and efficiency analyses.\nThe writing is clear: concepts are intuitively explained, methods are well-structured, and figures effectively illustrate key ideas.", "weaknesses": "Unfair rollout budget: The method uses 16 rollouts per query but only trains on 8, while baselines like DAPO likely use only 8 total. The gains may come from more exploration, not smarter sampling. A fair comparison—using the same total number of rollouts (e.g., 16 for both)—is missing. Also, as N grows large, discarding (1−α) samples wastes potentially useful signals.\n\nResampling vs. reweighting: The paper uses advantage-based resampling (ABS), but reweighting the loss by advantage magnitude would achieve nearly the same effect with zero extra overhead and less variance. No justification or comparison is provided for choosing the more complex resampling approach.\n\nMarginal gains vs. added cost: Improvements over strong baselines (e.g., DAPO, GSPO) are small—often <1% on average—yet the method adds rollout generation cost (2× more rollouts) and pipeline complexity. If the real benefit is faster convergence (e.g., 2× fewer steps), that should be the highlighted advantage, not final accuracy. As-is, the practical value is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)