Scaling Large Vision-Language Model RL Training via Efficient Load Balancing
Authors: Zerui Wang
Qinghao Hu
Chang Chen
Jiecheng Zhou
Haojie Duanmu
Xingcheng Zhang
Peng Sun
Dahua Lin
Venue: ICLR 2026 Poster (2026)
Keywords: RL Training;Load Balancing;Seuqence Parallelism;Distributed Training
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: FlexRL removes data and computation bottlenecks in RL for Vision-Language Models by finely sharding sequences and decentralizing data loading, achieving up to 8.47× faster training on large clusters.

Abstract: Reinforcement learning (RL) is increasingly used to align vision--language models (VLMs), yet scaling RL for VLMs is bottlenecked by multimodal data handling and extreme workload skew. In typical RL pipelines, visual data loading and preprocessing are centralized, creating severe I/O and CPU/memory stragglers, while batches that mix short image-text prompts with long video contexts lead to large cross-GPU imbalance during rollouts, inference, and training. We present FlexRL, an end-to-end system that removes these bottlenecks. FlexRL introduces: (1) ShadowLoader, a distributed, metadata-driven pipeline that keeps only lightweight visual metadata on the controller, pushes decoding and preprocessing to worker-side preprocessors, and asynchronously materializes tensors to overlap I/O with GPU computation; (2) FlexUlysses, a cost-aware sub-sequence sharding and execution engine that adaptively splits sequences to balance compute and memory. Our evaluation shows that across multiple VLM scales and multimodal datasets on 128-GPU clusters, FlexRL improves end-to-end throughput by up to 8.47$\times$ over state-of-the-art RL systems.

Published as a conference paper at ICLR 2026
SCALING LARGE VISION-LANGUAGE MODEL RL TRAINING VIA EFFICIENT LOAD BALANCING
Zerui Wang : Qinghao Hu?, Chang Chen Jiecheng Zhou
Haojie Duanmu', Xingcheng Zhang', Peng Sun", Dahua Lin
I "Shanghai Jiao Tong University 2Shangaai Artificial Intelligence Laboratory 3 Massachussetss Institute of Technology UUniversity of Science and Technology of China $ *Peking University "The Chinese University of Hong Kong 'Unafiliiated
ABSTRACT
Reinforcement learning (RL) is increasingly used to align vasion-language mod- els (VLMs), yet scaling RL for VLMs is bottlenecked by multimodal data han- dling and extreme workload skew. In typical RL pipelines, visual data loading and preprocessing are centralized, creating severe I/O and CPU/memory strag- glers, while batches that mix short image-text prompts with long video contexts lead to large cross-GPP imbalance during rollouts, inference, and training. We present FlexRL, an end-to-end system that removes these bottlenecks. FlexRL in- troduces: (1) ShadowLooade a distributed, metadata-driven pipeline that keeps only lightweight visual metadata the controller, pushes decoding and prepro- cessing to worker-side preprocessors, and asynchronously materializes tensors to overlap I/O with GPU computation: (2) FlexUlysses, a cost-aware sub-sequence sharding and execution engine that adaptively splits sequences to balance compute and memory. Our evaluation shows that across multiple VLM scales and multi- modal datasets on 128-GB 228GGPU clusters, FlexRL improves end-to-end throughput by up to 8.47 over nate-of-the-art RL systems.
INTRODUCTION
Reinforcement learning (RI.) has proven to be powerful paradigm for aligning vision-language models (VLMS)) with human preferences and enhancing their instruction-following capabili-
ties (Team et al. 2025a bld d Wang et al. 2025b Cha 2023 Hu et al.) 2024b Zhou et al.) (2025). A
typical RL workflow for VLMs involves several distinct stages: (I) loading diverse multimodal data
and prompts, (2) generating responses via policy model inference (i.e., rollouts), and (3) updating the model parameters, etc. While conceptually straightforward, scaling this process for VLMs exposes
system-level bottlenecks across the entire pipeline, hindering efficiency and scalability.
RL training of VLMs suffers from two types of load imbalance: (1) Vision data imbalance. Firstly, the data loading phase becomes significant bottleneck. VLMS trained heteroge- neous datasets containing text, high-resolution images, and video clips. In many RI frameworks (e.g., VeRL (Sheng et 2025)), data loading and preprocessing are centralized, causing the master node to become a straggler, limited by its memory and compute capacity, especially when handling massive vision data. (2) Model execution imbalance. Secondly, the model execution phase suffers from critical load imbalance problem. A single batch of sequences can contain mix of short image-text queries, long text-only reasoning tasks, and video inputs with tens of thousands of kens. This extreme variation in sequence length and modality leads to highly skewed distribution of computational and memory loads across GPUs. Consequently, some devices are overwhelmed while others remain underutilized, creating bottleneck that stalls the entire distributed system.
Existing systems fail to provide a holistic solution for the VLM RL pipeline. For the vision data imbalance, most RL frameworks (Sheng et al. 2025: Hu al. 2024a) are optimized for text-only models and lack sophisticated mechanisms to handle the load imbalance inherent in multimodal data. These frameworks typically employ naive sequence bucketing and packing strategies, which fail to tackle highly imbalanced workloads. On the other hand, general-purpos. large model train-
ing systems (Wang al. 2025c d:
Li al. 2024
Ge et al. 2024 2025) propose heterogeneous

---

Published as a conference paper at ICLR 2026
Traditional RL Data Pipeline
Ours: Balance Workload
GPU Imbalance
Sing Controude
Vision Data P21BB
Rollout Workers
Sequences
High Latency
breakdown
Latency: 10s
the computtation
Workers
BI
CPU OOM
Ours: Decentralized Data Pipeline
Metadata
ShadowLoader
Sing Controle
FlexUlysses
Metadata
Rollout
Workers
Balanced
Workload
Actioe
Warkers
Only lightweight metadata
Latency:
Figure 1: FlexRL Overview. (Left) Traditional VLM RL pipelines centralize vision-data fetching and preprocessing with single controller, leading to high latency and CPU out-of-memory (OOM). ShadowLoader decentralizes the pipeline by parallelizing heavy I/O and preprocessing across roll- out/actor workers while the controller exchanges only lightweight metadata. (Right) Highly variable multimodal sequence lengths cause severe cross-GPU imbalance when sequences are scheduled as indivisible units. Flex Ulysses shards sequences into fine- grained chunks to balance computation for both rollout inference and model updates, reducing step latency (10s s in the example).
parallelism methods over DP instances and gradient steps, which require large batch sizes and gra- dient accumulation to be effective. This assumption breaks down in the context of RL, which often utilizes small batch sizes to maximize the utility of dynamically generated samples and maintain training stability. These methods are thus inefficient for the dynamic and iterative nature of the RL rollout-update loop.
We present FlexRI., system designed provide comprehensive, end-to-end optimization for the VLM RL pipeline. FlexRI. deconstructs the performance bottlenecks in each stage of the RL pro- cess and introduces targeted solutions. For the data-loading bottleneck, design ShadowLoader a data pipeline that parallelizes expensive data fetching and preprocessing tasks across all worker nodes and handles only lightweight sample metadata on the single controller. This decentralizes the workload, eliminating the master-node bottleneck and significantly accelerating data throughput for large media files. For inference and update imbalance, we propose Flex a sequence -asuuenne-shrding mechanism. Instead of treating sequences as indivisible units, we partition them into fine- ine-grained chunks. This allows FlexRL balance the load at a sub-sequence level, effectively mitigating the imbalance caused by extreme length variations during both the inference and training steps. This complemented by a specialized balancing strategy for the vision tower. FlexRL incorporates an efficient decision algorithm to determine the optimal sharding strategy for each sequence and dynamic execution engine to orchestrate complex computation and communication patterns, maxi- mizing hardware utilization.
We implement FlexRL on top of the veRL framework and conduct extensive experiments on two 128-GPU clusters (H800 and H200). Our updated evaluation demonstrates that by optimizing the entire RL workflow, FlexRI. achieves up to an 8.47 x improvement in end-to-end throughput over the baseline. End-to-end ablations confirm that ShadowLoader and Flex Flysesssses provide complementary gains when combined.
In summary, our contributions are:
We provide an end-to-end diagnosis of VLM RL training and identify two primary bottle- necks: centralized multimodal data loading and severe cross-GPU workload imbalance.
2

---

Published as a conference paper at ICLR 2026
We propose LhaderLoader, distributed, metadata-driven data pipeline that parallelizes vision-data fetching and preprocessing across worker nodes and enables metadata-on scheduling the controller. With prefetching and asynchronous materialization, Shad- ooleo nearly breaks the data bottleneck by removing data processing and loading la- tency from the training critical path.
We propose FlexUlysses, decision model and a deadlock free dynamic execution engine that overlaps communica- tion and computation, achieving fine-grate -nn-iia load balancing for both rollout inference and model updates under extreme multimodal length skew.
adaptive sub-sequence sharding mechanism with cost-aware
Our extensive evaluation on large-scale GPU clusters demonstrates that FlexRL signifi- cantly outperforms existing baselines, achieving up to an 8.47 improvement in end-to-end throughput.
LOAD BALANCING CHALLENGES IN VLM RL TRAINING
2.1 DATA PREPARATION BOTTLENECK
Unlike text-only RL, multimodal RL must repeatedly fetch and pre- process large visual payloads (e.g., decoding video clips and sampling frames) the critical path. Many RL frameworks, such as VeRL (Sheng et al. 2025), adopt a hybrid-controller architecture that centralizes these CPU- and 1/O-heavy operations on single master node, which then dispatches materialized tensors to workers. As the global batch size grows, the master's preprocessing time and host-memory footprint scale with the amount of visual data, quickly turning it into the pipeline strag- gler and frequently triggering CPU out-of- -o-ooorrymmryy (OOM) errors. Fig- ure [ illustrates this effect: in VeRL, data loading accounts for 57.1% of the step time, exceeding rollout (18.0%), inference (15.5%), and up- date (9.3%) combined. This motivates distributed, metadata-driven data pipeline SShadoWLoader, Sec. 4 that parallelizes preprocessing and hides data movement behind GPU computation.
1.0
Rollout
Update
Normalized Time
0.75
Inference
Data Load
0.50
7.66x
15.5%
0.25
18.0%
VeRL
Ours
Time Breakdown
Figure 2: Iteration time breakdown of GRPO.
2.2 COMPUTE AND MEMORY IMBALANCE MODEL EXECUTION
Beyond data loading, the execution phase suffers from severe compute and memory imbalances. First, attention computation scales quadratically with sequence length (O(L2)), while activation memory scales linearly (O())). A placement strategy that equalizes memory across GPUs can still result in severe compute imbalance, and vice versa. Second, multimodality introduces heteroge- neous costs that are poorly correlated with text length. The computational cost of the vision tower depends heavily on the number of images or video frames. Mixing video-heavy and image-heavy samples within the same batch creates massive variance in per-sample compute and memory quirements, making length-only bucketing objectives ineffective.
2.3 WHY EXISTING LOAD BALANCING FALLS SHORT IN VLM RL
Fixed-pan paxallparallelism bucketing/packing Classic bucketing methods (Team al. 2025b Wang
c
et al. 2025b Team et al. 2025d) sort sequences by length and pack them into Per-GPU GPU buckets under fixed parallelism configuration. While simple, this strategy fails when a single long outlier dominates the step. For instance, shown in Figure Middle,, with sequences of varying lengths (e.g., 32K, 16K, and 4K) on four GPUs, no matter how sequences are bucketed, the GPÚ processing the 32K sample becomes the bottleneck, leading to unavoidable slowdowns. This issue persists in 3D/4D parallelism settings that assume homogeneous configurations across DP ranks.
Heterogeneous DP across buckets. Methods such FlexSP (Wang et al. 2025c), HotSPa (Ge et al. 2024), Hydraulis (Li al. 2024), and ByteScale (Ge et al. 2025) assign different paral- lelism configurations accumulation to create sufficient scheduling space for sequence reordering, In RL, frequent rollout- update alternations and small batch sizes make re-sharding, extra synchronization, and optimizer-
different buckets. However, they rely large batch sizes and gradient
3

---

Published as a conference paper at ICLR 2026
(Capecity: 32K)
Seq Packing
ESmmies
Bucketing
HeteroDP
2%
3PK
64K
feads
#Tokons (Viceo + Tert)
20000 50000 75003
HeteroDP= Bucketing.
Iibbalance
T0k
164
Bucketing: Capacity
Imbalance
I0k
I6K
400
gea3k
4Smmplss
Proposed Method
/Ballanced
Balanced Comp.
350
(Imaga Tex))
400
sequercee
(b) Longer sequence
Figure 3: Motivation and idea of FlexUlysses. Left: Distribution of token counts in typical video- text (NExTQA (Xiao et al. 2021)) and image-text (Geo3K(Lu et al., 2021) datasets, showing treme variation in sequence lengths. Middle: An imbalance with shorter sequences, where conven- tional bucketing leads load imbalance. Right: For longer sequences, bucketing can cause out- of-memory (OOM) errors and further imbalance, while our approach enables fine-grained sharding and balanced attention computation across GPUs.
state movement prohibitively expensive. Furthermore, when sequence parallelism (SP) enabled, short sequences suffer from redundant communication overhead, and assigning a single SP degree per bucket fails accommodate mixed-length samples.
3 FLEXULYSSES
3.1 PRELIMINARY: ULYSSES SEQUENCE PARALLELISM
In the Ulysses approach, a sequence x of length L is split along the sequence dimension across N devices. Each device i E (O..... N' -0-1] 1} receives an equal-sized chunk x, of length L/N. During the forward pass of layer, each device computes its local Query (Q₂), Key (K,), and Value (V,) tensors from its chunk x,. To compute the full attention scores, the K; and V; tensors must be shared among all N devices. This is achieved via all-to-al communication operation. each device possesses the complete Key and Value tensors for a subset
After the all-to-all of attention heads, allowing it to compute its shard of the attention output. Another a all-to-all operation is then performed to gather the output, which is passed to subsequent layers.
3.2 SEQUENCE SHARDING FOR LOAD BALANCING
Bucketing fails in scenarios like Figure Middle), where the maximum sequence length is not large enough to require sequence parallelism (SP) for memory, yet the batch exhibits severe length skew. In this regime, the iteration time is dominated by the single longest sequence: regardless of how we bucket or pack samples, the DP rank that receives this outlier becomes the straggler.
We note that Ulysses sequence parallelism implements an equal-cost partition of sequence. In the MLP module, each GPU receives a shard of the same length, resulting in identical computation and memory footprints. In the attention module, Ulysses computes attention at the head level, and the computation for each head is identical. Therefore, using Ulysses naturally equalizes the computa- tion and memory footprint across GPUs, and it is applicable to any attention pattern. Consequently, applying Ulysses to packed sequences can also achieve perfect load balancing, as sequence pack- ing is essentially a specific attention pattern (concatenating multiple causal attentions). This differs from Ring which, despite partitioning the sequence, requires meticulously designed parti- tioning decisions and scheduling for different attention patterns. This inspired us to repurpose this mechanism load-balance had-balancing primitive: using Ulysses chunks scheduling units and allocating
4

---

Published as a conference paper at ICLR 2026
them across different devices to balance the workload on each GPU, even when original sequence lengths are highly variable.
Strawman Solution: Greedy Sharding. The most straightforward approach is to pack all quences in the batch and then apply Ulysses, which would force load balancing across all GPUs. However, it is impractical for two reasons. First, Ulysses' scalability is capped by the number of attention heads, limiting the effective degree of parallefism. Second, incurs substantial com- munication overhead. While the attention computation for packed sequences scales roughly as O L2), the all- tot-ttollll communication volume grows with the total token count, increas- ing the communication-tor cetpiiioonoo-oomuatttiioooo ratio and leading a higher GPU idle ratio.
Our Solution: FlexUlysses, FlexUlysses addresses this by adaptively sharding sequences: instead of sharding every sequence into the same (maximal) number of chunks, it assigns each sequence sharding degree based on its length and then buckets sequences accordingly. This brings two advantages. First, most sequences do not need to be sharded or only require light sharding to achieve near-optimal load balancing, without incurring communication overhead for the entire batch. Second, since sequences with different SP degrees are processed by different device groups, the all-to-all communication and attention computation are independent across sequences (and across different device groups), which enables pipelining and overlapping communication with com- putation to reduce idle time (bubbles).
Key Challenges. However, this hybrid approach introduces two significant implementation chal- lenges. First, the search space is prohibitively large. Finding an optimal configuration requires solving a two-level combinatorial problem: (a) Sharding Degree Selection, which chooses a shard- ing degree for each of the sequences, yielding search space that grows exponentially with batch size; (b) Device Group Placement, which assigns a concrete GPU group to each sharded sequence. The latter is constrained optimization problem akin to an NP-hard bin-packing problem, because placements across sequences are coupled and must jointly satisfy per-GPU resource limits.
Second, the resulting configuration creates a complex co-scheduling problem for computation and communication, which breaks the conventional SPMD paradigm. Each sequence is partitioned into multiple chunks and mapped to different device groups. Chunks on the same GPU may be- long to different sequences and thus require different all-to-all groups for communication. These groups may overlap, and an incorrect execution order can lead to deadlocks. Moreover, since all-to-all operations are collective and require synchronization, naive implementation that serializes the communication for each group would introduce significant GPU idle time (bubbles), diminishing the benefits of FlexUlysses Scheduling the computation and communication of these chunks to maximize GPU utilization is a non-trivial problem. Therefore, a sophisticated scheduling mechanism is required to manage these diverse computation and communication patterns efficiently.
3.3 PLANNING AND SCHEDULING FOR LOAD BALANCING
We design a two-level planning and scheduling mechanism to address these challenges. It first deter- mines per-sequence sharding degrees and device groups. Then it orchestrates the Ulysses execution order for the assigned sharded sequences on each GPU.
Shard Planning. Given global batch, the single controller collects lightweight metadata and outputs& a sharding plan. For each sequence i with length hús we choose a sharding degree P: E (1, 4, ... ,max) and assign it device group G, with |G,l = Pt. We pre-instantiate a hierarchical set of candidate device groups for all admissible degrees: for each degree p, the groups form a partition of GPUs (disjoint and covering). and partitions are nested across degrees (a p-group is the union of two p/2- grrousss. For example, on 8 GPUs, the admissible groups include [0-7] for p=8, [0-3) and [4-7] for p=4, and [0, 1). [2, 3). [4, 5). [6. 7] for p=2. This hierarchy ensures that any two groups are either disjoint or contains the other, which enables simple deadlock- free schedule. In practice, we decouple grouping from placement by first instantiating all admissible groups, then assigning sequences to groups using lightweight greedy heuristic (Appendix A
Bighest-Sharding-Fiss Execution. Collective primitives are oeder-sensitive: for a given commu- nicator, all ranks must issue collectives in the same order; otherwise, execution multiple sharding degrees, a GPU can participate in multiple (nested) device groups, and a naive, per-rank execution order can deadlock even when groups are hierarchical. For example, on 4 GPUs,
hang, With
5

---

Published as a conference paper at ICLR 2026
Algorithm FlexUlysses Forward Attention Execution.
Require: packed sequence groups (p((( SSEe global HSF-ordered list O over {SG E Q p(SG) > }} Worker rank k (execution)
Require:
k E G(SG) 1: O₂ - ISG E
O
D ordered subsequence no collectives
2: Us t [SG e & G(SG) ([SG))A(SS)) (PSG)
3: if' Or * 0 then
4: launch a112a111(O+ asynchronously
P comm stream
5: (UNUNSHARDED((4)
b hide a 1 12a 11
6: fort= 10 Ox x do
7:
SG + Ox[t]
8: wait a112a111(SG)
9:
ift < [Os] then
launch al12al11(O,[tt1) ssynhhhoossyyyynoooooooosss asynchronously
issued at compute start
11: compute attention(SC))
launch a152)11((SG) asychronouuly
b comm stream
consider a 4-way group G,- 1, and nested 2-way group G2= 13. If rank 0 enters an al12all on G2 while ranks 1-3 enter an al12all on G1, then rank O waits for rank 1 to join G2, while rank waits for rank to join G2, forming cyclic wait.
FlexUlysses avoids deadlocks by combining hierarchical device groups with nighest-sharding-fine first (HSF) schedule. HSF then enforces consistent global order across all ranks: each GPU always executes collectives for larger degrees before smaller ones (e.g., p-8 + 4 + 2), and within the same (p. G), follows consistent order so the collective call sequence matches across ranks. As result, GPU never interleaves collectives from multiple device groups the same time. With HSF, any rank that belongs to both G4 and G2 must enter G4 collectives before G2 collectives, preventing the cyclic wait.
Vision Tower Balancing. Unlike the LLM backbone, the vision tower is naturally parallelizable across images/frames. Vision encoders typically process images independently by stacking them along the batch dimension, and video frames are sampled and processed with intra-frame attention, making computation independent across frames. As result, vision-tower compute and memory scale near-linearly with the number of images or frames. We leverage this property by distributing images and video frames evenly across GPUs to balance both compute and memory loads.
3.4 DYNAMIC PACKING AND OVERLAPPING
Dynamic Shard Packing. Executing HSF at single-sequence granularity would repeatedly launch small attention kernels and two all2all collectives per sequence, which can incur non-trivial launch overhead and leave little room to hide communication. We therefore pack sequences assigned to the same device group into sequence groups and run Ulysses on each group unit. Concretely, within each (p. G) we greedily concatenate short sequences (e.g.. h4 S hpack) until the packed token count reaches cap pmcc: while keeping long sequences as singleton groups. This bounded packing trades off between overhead amortization and scheduling flexibility: larger groups reduce kernel/collcctive launch frequency and create longer compute windows the caps prevent a pack from becoming a new straggler and preserve enough groups for pipelining.
all2all while
Overlapping, For single sequence group, Ulysses induces strict dependency all2all1 - attn comp + all2all> Different sequence groups are independent, so we pipeline them to overlap communication with computation, illustrated in Fig. 4 When processing sequence groups under HSF, once SG, enters attention computation, we immediately launch a 2al3 of the next group SG,+1 the communication stream.

---

Published as a conference paper at ICLR 2026
For each sequence comp 2l122
all2al11
Before Overlap
atin
ali2all1
attn
all2al12
all2il1
attn
allaali2
attnn
RANK
After Overlap
Stream D:
attn
attn
attn
atin
Time Saved
Stream 1:
all2all1
aliaal11
alizali2
all2al122
Time
Figure 4: Sliced sequence computation pattern of FlexUlysses and communication overlapping strategies.
4 SHADOWLOADER FOR THE DATA-LOADING BOTTLENECK
To overcome the data loading bottleneck in the hybrid-controller architecture, we design Shad- owLoader, a distributed. metadata-driven data-loading system. ShadowLoader consists four key components: (1) Proxy Dataloader, which resides the single controller manage lightweight metadata and dispatch preprocessing tasks; (2) Local Preprocessor, deployed on each worker node to independently handle heavy preprocessing tasks such video decoding and frame sampling; (3) MetaStore, a registry that maintains the mapping between sample metadata and its physical storage location; and (4) Materializer, instantiated on each worker fetch the materialized visual tensors from the corresponding Local Preprocessor based on metadata.
On the single controller, the Proxy Dataloader replaces the standard dataloader. For multimodal data, it only retains lightweight metadata, such as video length. expected frame count, image size, resolution, and file path. To seamlessly integrate with the existing RL training pipeline without load- ing actual pixel values into the controller's memory, the Proxy Dataloader uses a Fake as placeholder for visual data. The TakeTenso contains shape information (e.g., wew..valees)) but no actual data, and it overrides the shard operation to support subsequent Ulysses partitioning decisions. During training, these Tenorss are distributed across different workers according to the batching and
strategies.
Overall Workflow. The end-to-end data loading process proceeds as follows. First, the Proxy Dat- aloader assigns unique ID to each data sample and dispatches it Local Preprocessor using a simple foadnbalan algorithm (c.g., routing to the node with the least load). Upon receiving the task, the Local Preprocessor registers the mapping between the data's metadata and its storage location in the MetaStore. It then executes the preprocessing and caches the materialized visual tensors in its local host memory. Meanwhile, the single controller schedules the RL training loop using the FakeTensors. When a worker receives a FakeTensor and requires the actual visual data for computation, it invokes its local Materializer. The Materializer queries the MetaStore using the metadata to locate the target preprocessor and fetches the materialized tensors directly from the corresponding Local Preprocessor.
To further remove data processing from the critical path and minimize overhead, we introduce two additional optimizations to reduce transfer volume and hide latency:
Prefetching and Asynchronous Materialization. ShadowLoader overlaps data movement with training by prefetching the metadata for the next step and materializing visual tensors asyn- chronously on the workers. By launching non-blocking materialization, CPU-side fetching, de- coding, and network transfers are completely hidden behind ongoing GPU computation, effectively climinating step-level bubbles caused by large visual payloads.
Ulasessaaaae Loading. We further co-design ShadowLoader with FlexUlysses by making sharding decisions early and loading only the required data slices. Because the FaneTensor place- holders carry slice-award metadata, workers can fetch precisely what they need necd-such as specific frame ranges for videos required image tensors for materialization with Flex FlexUlysser partition decisions significantly reduces inter-node transfer volume and improves end-to-end throughput.
inputs. Aligning this slice-aware

---

Published as a conference paper at ICLR 2026
Bucketing
Ulysses(5P=d)
Tokens, GPU /Second
600
tramen/A/BiC)) 2/))-258/118//4
600
400
4.39X
400
4.79x
200
1,74x
200
1.60x
P-0
0-0
OOW
0.00
Tokens GPU /econd
600
framestA/R/C) 12/12//64/44
328, GPUS
25
200
400
150
7.35x
1.5.x
200
1,54x
100
6-4.3
-000
000
OOW
50
00M
Image-Heavy
Video-Heavy
Only-Video
Image-Heavy
Video-Heavy
Only-Video
Figure 5: End-to-end throughput (tokens/s/GPU) across three dataset settings
Heavy/Only-Videe Video) under different model scales and cluster sizes. Geo3K, NEXTQA, and LongVILA-Rese mixtures with sampling weights 5:2:1, 1:2:5, and only LongVILA-Resd re- spectively. Speedup labels are computed against veRL Bucketing when available, otherwise against Ulysses (SP=4). Red OOM markers CPU or GPU out-of-memorp
EVALUATION
5.1 EVALUATION SETUP
Testbed. We evaluate FlexRL two 128-GUU clusters (each 16x8): with NVIDIA H800 GPUs and one with NVIDIA H200 GPUs Intra-node GPU interconnect bandwidth 400GB/s (H800) and 900GB/s (H200) via NVLink/NVSwitch Inter-node connectivity uses RoCEv2 RDMA, with aggregate per-node bandwidth of 1600Gb/s (H800 cluster) and 3200Gb/s (H200 cluster). Unless otherwise stated, end-to-end comparisons are reported on the H800 cluster.
Models and Datasets. We evaluate FlexRL on MiMo- VL-7B-RL (Tam et al. 2025a) and Qwen2.5-
VL-32B (Bai et al. (025). We benchmark on mixture of image-text (Geo3K (Lu et 2021 )
short-video (NExTQQ TQA (Xiao et al. 202T) and long-video V (Chen et al. 2025b
VILA-Reasoo
datasets under different sampling weights: (1) Image-Heavy, 5:2:1; (2) Video-Heavy: 1 1:2:5; (3) Only- Video: LongVILA-RResoe
only. We use the GRPO for all experiments, with maximum response length 1024 tokens. For the 7B model, we use TP=4 for rollout and FSDP=16 for model forward/update. For the 32B model, we use TP=16 for rollout and FSDP=64 for model for- ward/update. We set the maximum sharding budget Pmax to 8 for all runs unless otherwise specified.
5.2
MAIN RESULTS
End-to-end Results. We compare FlexRL with two baselines: (1) veRL+Bucketing, the original veRL system with bucketing; and (2) UlyssesSS,, conventional sequence parallelism with fixed sharding degree. Figure S reports end-to-end throughput (tokens/GPU/s) for GRPO under differ- ent dataset sampling weights, global batch sizes, and maximum numbers of video frames. Overall, FlexRLconsistently outperforms both veRL+Bucketing and fixed-degree Ulysses-SP SP. In particular, FlexRLimproves end-to-end throughput by up to 8.47x over the baselines. tings. Lacheeas to 7.35 speedup, while in video-heavy settings the speedup reaches up to 5.35x. In the only-video setting, where sequence lengths are longest and most heterogeneous, FaexeLachieves the largest gain of up to 8.47x. The improvements are more pronounced for larger models and heavy datasets, where load imbalance is more severe. Moreover, throughput scales well with the video volume, demonstrating FlexRL's effectiveness in handling long sequences. In contrast, the veRL baseline suffers from severe slowdowns and oet-rry-mmory (OOM) issues due ViT-induced load imbalance the number of video samples increases. Ulysses-SP mitigates the
meave set-
8

---

Published as a conference paper at ICLR 2026
VeRL
V Shadewlooder
VeRL+ FlexUlysss
FlexRL
6000
117.42x
200
7.67%
1.00x
Rollout
Throughput (token/secon)
5000
175
Inference
Update
04 09.0%x
Data Load
4000
Normalized Time
6.00x
125
3000
100
6.47x
2000
50-
1.00x
4.17%
1000
1000x
62x
1.00x
30x
2.93x
1.00x
L,2xx
1.00x
25-
Rollout
Inference
Train
E2E Throughput
VeRL +Shadow FlexRL
Loader (Both)
(a) Per-stage and E2E Throughput.
(b) Step time breakdown.
Figure 6: Ablation results on 128 H800 GPUs with VL----LLBB-RL. (a) Throughput (tokens/s) for each RL stage and the end-to-end throughput, along with the speedup against VeRL.. (b) Normalized iteration time breakdown.
veRL (Baseline)
+Ulysses (5P=2)
+Ulysses (5P=4)
+Ulyyses (SP=B)
FlexUllysees)
300
10
2.00
0.59x
1.50k
1.75
250
(veRL=1.0
0.76x
6,70
1.50-
200
1.05x
1.12x
1.13x
1.25
1.00x
1.0ZX
1.00x
1.01x
0,97x
1.11x
Normatzed
1.00-
1.36x
151x
150
3.22
L.2x
0.75
0.50
2-
1.00
1.00
0.25
0.00
Update Actor
Ref Log Prob
Old Log Prob
Throughput (tokens//GPL)) 1
Balance Ratio
Figure 7: Comparison with Ulysses-SP. Per-stage normalized step time with the speedup (Left).
End-to-end throughput (Middle). Balance ratio calculated using the attention flops (Right).
veRL baseline by balancing attention computation; however, its fixed sharding degree incurs sub- stantial communication overhead, and its performance gain diminishes the batch size increases.
Comparison with USy We evaluate our method and Ulysses-SP method using the image- heavy dataset and 7B model on 32 GPUs. We enable ShadowLoader for all methods to eliminate the influence of dataloading, and compare fixed-degree UlyssesSSP (SP=(2, 4, 8}) against ours. We set BuckeBug attention FLOPs / mean FLOPs. As shown in Figure 7 when using Ulysses-SP. increasing the SP degree can also achieve load balance. However, the throughput gain of Ulysses-SP limited due to the all2all communication dominating overhead. In contrast, FlexUlysses succeeds to reaches balance ratio of 1.0 while improving throughput by 1.50x. Figure 7 further breaks down the per- stage step time. The results show that our method achieves the highest speedup across all stages, demonstrating the effectiveness of our fine-grained sharding strategy improving load balance and reducing communication overhead.
the baseline. The attention balance ratio equals the maximum per-GPU
5.3 ABLATION STUDY.
We separately evaluate the contributions of ShadowLoader and Flex under the image-heavy setting using the 7B model. Figure reports both the throughput breakdown and the iteration time

---

Published as a conference paper at ICLR 2026
breakdown. Figure 6a shows that ShadowLoader primarily removes the data bottleneck, improving data-loading throughput by 8 and end-to-end throughput by 4.68x. x. Figure further confirms that this translates into 5.35x reduction in overall step time by largely eliminating the data-loading slice. In contrast, FlexUs mainly improves compute-heavy stages (e.g.. 2.01 x rollout and 1.28x training throughput) but yields limited end-to-end gains (1.17x) when data loading remains dominant. Without ShadowLoader, the single controller has to send the data to all workers, incurring high data transfer latency. Combining both components yields complementary benefits: after Shad- owLoader makes the workload compute-boune large gains inference (11.8, (11.83x) x) and training (6,47x), reaching 117.42x data-loading throughput, 7.67x end-to-end throughput, and 7.68x reduction in step time. The detailed per-stage runtimes and throughput are shown in T
Flex FlexUlysses' fine tne-grained sharding translates into
Table 1: Detailed step time breakdown of ablation study of FlexRL components on 128 H800 GPUs with MiMo-VL- VL-7B-RL.
Method
Update Actor (s)
RefLogProb (s)
OldLogProb (s)
Data Loading (s)
Generation (8)
Step Time (8)
veRL.
61.3 (1.00x)
51.3 (1.00x)
50.2 (1 (1.00x)
375.0 (1.00x)
118.4 (1.00x)
656.2 (1.00x)
ShadowLoader
24.3 (2.52x)
5.0 (10.26x)
25.3 (1.99))
3.9 196.15x)
64.1 (1.85x)
122.6 (5.35x)
+ FlexUlysses
47.8 (1.28x)
50.1 (1.02x)
27.8 (1.8 (1.81)
(1.00k)
58.9 (2.01x)
$59.6 (1.17))
FlexRL (Both)
9.5 (6.45x)
3.7 (13.86x)
4.9 (10.24x)
3.2 (117.199)
64.4 (1.84x)
85.7 (7.66x)
RELATED WORK
RL Training Frameworks. Reinforcement learning has become central paradigm for aligning and enhancing large language models (LLMs). Recent frameworks such as VeRL (Sheng et al. 2025), siRL (Wang et al. 2025e AReal (Fu et al. 2025). StreamRL (Zhong et al.] 2025). Mi TTeam
& Team, 2025). and ROLL [Wang et al.] 2025a) provide system-level support for distributed RL training. These frameworks focus on issues such as asynchronous rollout-update decoupling, scal-
able data pipelines, and integration with model model-parallel training backends.
VLM Training Frameworks. The emergence of multimodal LLMs has motivated training frame- works such as DistTrain (Zhang et al. 2205)). DistMM (Huang et al. (024). VILA (Chen et al.) 2025b), and VeOmni (Ma et al. 2025). These systems propose optimizations for heterogencous architectures combining vision towers and language backbones. For instance, they disaggregate model components, employ hybrid parallelism, or develop scheduling algorithms reduce commu- nication overheads. Despite these advances, existing VLM frameworks primarily target pretraining or supervised fine-tuning. They do not explicitly address the unique challenges of RL. training, such as small batch sizes, dynamically generated trajectories, and highly variable sequence lengths across modalities.
Load Balancing for Large Model Training. A line of work focuses on load balancing techniques for efficient large model training. Classic methods rely on sequence bucketing and packing, which sort samples by length and allocate them across GPUs to minimize padding (Team et al., 2025b c d Wang et al. (0255). Recent works such as FlexSP (Wang et al.) (025c). HotSPa (Ge et al. 2024)) Hydraulis (Li et al. 2024), ByteScale (Ge al. 2025), and WLBBLLM (Wang et al. 2025d) mit- igate load imbalance via heterogeneous parallelis lism and dynamic reconfiguration across DP ranks
or gradient steps. Zeppelin (Chen et al.) (202aa) addresses training imbalance by jointly optimizing attention-specine sequence partitioning and layout remapping between attention and linear layers.
7 CONCLUSION
In this work, we present FlexRL,, a holistic system that addresses the unique system-level challenges of reinforcement learning for large Vision-Language Models. By systematically analyzing the bot- tlenecks across the RL pipeline, we identify critical inefficiencies in both data loading and workload balancing that hinder scalability and hardware utilization. FlexRL introduces ShadowLoader to eliminate I/O and memory bottlenecks on the single controller, and FlexUsysses to achieve fine- grained, shard-level load balancing across GPUs. Our efficient scheduling algorithm and dynamic execution engine further maximize overlap between computation and communication. Combining both components achieves high throughput even under extreme data heterogeneity.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank the anonymous reviewers and the area chair (AC) for their insightful comments. This work supported by the Shanghai Municipal Science and Technology Major Project and Shanghai Artificial Intelligence Laboratory.
REFERENCES
Chatgpt.
2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://azxii org. abs/ 20..//5522..0 13923
Chang Chen, Tiancheng Chen, Jiangfei Duan, Qianchao Zhu, Zerui Wang, Qinghao Hu, Peng Sun, Xiuhong Li, Chao Yang. Dahua Lin, and Xin Jin. Zeppelin: Balancing variable-length work- loads in data parallel large model training, 2025a. URL http://axxvv.rr/////
/ org..rr///abb///s/
21841
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He. Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. LongVILA: Scaling long-context visual language models for long videos. In The Thirteenth International Conference on Learning Representations, 2025b. URL htt:ps://ppenrevieeee
net htte:/ffrummiddee wtwp://r/m///
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang. Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system language reasoning, 2025. URL htttps://aaxii..r..
fbs/2505.245
Hao Ge, Fangcheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Yujie Wang, Xiaonan Nie, Hailin Zhang, Xupeng Miao, and Bin Cui. Enabling parallelism hot switching for efficient training of large language models. In Proceedings of the ACM SIGOPS 30th Symposium Oper- ating Systems Principles, SOSP '24, pp. 178-194, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400712517, doi: 10.1145/36947 17.39449757339966 URL httpps://oii
http//...111555599111 3695969
Hao Ge, Junda Feng, Qi Huang. Fangcheng Fu, Xiaonan Nie, Lei Zuo, Haibin Lin, Bin Cui, and scaling of Ilm training with 2048k context length on 16384 gpus. In Proceedings of the ACM SIGCOMM 2025 Conference. SIGCOMM '25, pp. 963 978. ACM, August 2025. doi: (0.1145/3718958837555
Xin Liu. Bytescale: Communication--ffiicae
1145/ 3754352
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An casy-to-use, scalable and perforrrformecc rlhf framework. arXiv preprint arX1v:2405.11143, 2024a.
Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang. Yingwei Luo, Yonggang Wen, and Tianwei Zhang. Characterization of large language model development in the datacenter. 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 709-729, 2024b.
Jun Huang, Zhen Zhang, Shuai Zheng, Feng Qin, and Yida Wang. DISTMM: Accelerating dis- tributed multimodal model training. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 1157-1171, Santa Clara, CA. April 2024. USENIX As- sociation. ISBN 978- 1-939113--3-77 URL https: //www//ww..//// usenix. org/ cot.erence/
nsdi 2aaaa22//eeettt... on/ huang
11

---

Published as a conference paper at ICLR 2026
Haoyang Li, Fangcheng Fu, Sheng Lin, Hao Ge, Xuanyu Wang. Jiawen Niu, Jie Jiang, and Bin Cui. Demystifying workload imbalances in large transformer model training over variable-length sequences, 2024. (tth::::: //trxaaaaavv/aa///ao org/ abs/ 2atps//241..c. 07894
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang. Xiaodan Liang, and Song- Long-Chun Zhu. Inter mnter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference the 59th Annual Meeting of the Association for Computational Linguis- lies and the I Ith International Joint Conference on Natural Language Processing (ACI-IJCNLPP 202 1, 2021.
Qianli Ma, Yaowei Zheng, Zhelun Shi, Zhongkai Zhao, Bin Jia, Ziyue Huang, Zhiqi Lin, Youjie Li, Jiacheng Yang, Yanghua Peng. et al. Veomni: Scaling any modality model training with model-centric distributed recipe zoo. arXiv preprint 2rXiv:2508.02317, 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp- 1279- 1297, New York, NY. USA, 2025. Association for Computing Machinery. ISBN 979840071196/ doi: 10. 1145/368903 13696077. .3696075. URL BRRnhtps::. :/tttss//dci...... org/10.......oo 11..111111111 1145/ 36999933933666773338
Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li. Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang. Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang. Zihan Jiang, Zhixian Zheng, Zhichao Song. Zhenbo Luo, Yue Yu, Yudong Wang. Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang. Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong. Xin Zhang. Xiangwei Deng, Wenyu Yang. Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang. Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, and Bingquan Xia. Mimo-vl technical report, 2025a. URL htt.sttp::// //t::///rxxii /ariiviivvva org abs 2506. 03569
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu. Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang. Hongcheng Gao, Huabin Zheng. Jiaming Li. Jianlin Su, Jianzhou Wang, Jiaqi Deng. Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng. Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingeheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang. Zhiqi Huang. Zihao Huang, Zijia Zhao, Ziwei Chen, and Zongyu Lin. Kimi-vl technical report, 2025b. htt.ttt.:. a www. org/ abs absp/504b///55//.. 20w./2554.000 07491
Kwai Keye Team, Biao Yang, Bin Wen. Changyi Liu, Chenglong Chu. Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, Fan Yang, Guorui Zhou, Hao Peng, Haojic Ding, Jiaming Huang, Jiangxia Cao, Jiankang Chen, Jingyun Hua, Jin Ouyang, Kaibing Chen, Kaiyu Jiang, Kaiyu Tang, Kun Gai, Shengnan Zhang, Siyang Mao, Sui Huang, Tianke Zhang, Tingting Gao, Wei Chen, Wei Yuan, Xiangyu Wu, Xiao Hu, Xingyu Lu, Yang Zhou, Yi-Fan Zhang, Yiping Yang, Yulong Chen, Zhenhua Wu, Zhenyu Li, Zhixin Ling, Ziming Li, Dehua Ma. Di Xu, Haixuan Gao, Hang Li. Jiawei Guo, Jing Wang. Lejian Ren. Muhao Wei, Qianqian Wang, Qigen Hu, Shiyao Wang, Tao Yu, Xinchen Luo, Yan Li, Yiming Liang, Yuhang Hu, Zeyi Lu, Zhworan Yang, and Zixing Zhang, Kwai keye-vl technical report, 2025c. URL
wwg. /bs/
2507 01 949
MiroMind Foundation Model Team and MiroMind AI Infra Team. Mirorl: An mep-first reinforce- ment learning framework for deep research agent. ht https: //tpttghtbh/.tt///// cow/MirooindA... Mi ro 2025.
12

---

Published as a conference paper at ICLR 2026
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi. Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang. Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang. Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang. Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang. Shi Zhong, Shiyu Huang. Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang. Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang. Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and glm-4. Iv-thiki-- Towards versatile multimodal reasoning with scalable reinforcement learning, 2025d. URL htttp:://aaxi...ttttt
01006
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang. Jiaheng Liu, Zhendong Li, Xiaoyang Li. Zichen Liu, Haizhou Zhao, Dakai An, Lunxi Cao, Qiyang Cao, Wanxi Deng, Feilei Du, Yiliang Gu, Jiahe Li, Xiang Li, Mingjie Liu, Yijia Luo, Zihe Liu, Yadao Wang, Pei Wang, Tianyuan Wu, Yanan Wu, Yuheng Zhao, Shuaibing Zhao, Jin Yang. Siran Yang. Yingshui Tan, Huimin Yi, Yuchi Xu, Yujin Yuan, Xingyao Zhang, Lin Qu, Wenbo Su, Wei Wang, Jiamang Wang, and Bo Zheng. Reinforcement learning opti- mization for large-scale learning: An efficient and user-friendly scaling library, 2025a. URL https: wrwaa.vvvvv..... org/ abs/ 2506. 06122
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglnn Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang. Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang. Huipeng Deng. Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang. Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu. Huanze Tang, Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internv13.5: Advancing open-soure multimodal models in versatil- ity, reasoning, and efficiency, 2025b. URL URL //trx//rxii/aa org/ / abs / 2500 18265
Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xuefeng Xiao, Huixia Li, Ji- ashi Li, Faming Wu, and Bin Cui. Flexsp: Accelerating large language model training via flex- ible sequence parallelism. In Proceedings of the 30th ACM International Conference on Ar- chitectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS '25. pp. 421-436, New York, NY, USA, 2025c. Association for Computing Machinery. ISBN 9798400710799 107977 doi: 10.1145/3676641.371555 3677641..71558
URL
1145/
Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, and Yufei Ding. WIb-llm: Workload-balanced 4d paral- lelism for large language model training, 2025d. htttttt://aaxiiii obtp:///50rr/aa/2550333 17924
Zhixin Wang, Tianyi Zhou, Liming Liu, Ao Li, Jiarui Hu, Dian Yang. Yinhui Lu, Jinlong Hou, Siyuan Feng, Yuan Cheng, and Yuan Qi. Distflow: A fully distributed rl framework for scalable and efficient IIm post-training, 2025e. URL
13833
Junbin Xiao, Xindi Shang, Angela Yao. and Tat-Seng Chua. Next-qamext phase of question- wttp:i/ww..rrabb///..5.....
answering to explaining temporal actions, 2021. URL
08276
Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang. and Xin Jin. Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. In Proceedings of the ACM SIGCOMM 2025
13

---

Published as a conference paper at ICLR 2026
Conference, SIGCOMM '25, pp. 24-38. New York, NY. USA, 2025, Association for Com- puting Machinery. ISBN 9798400715242. doi: 10.114523718968.7554422 URL hetps:
//doi.org/10. 10. 11t..1111177195.833333 37550472
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, and Daxin Jiang. Streamrl: Scalable, heterogeneous, and elastic for Ilms with disaggregated stream generation, 2025. URL
htttps:/aaxxv.com org/ abs/ 2504. 15930
Jiecheng Zhou, Qinghao Hu, Yuyang Jin, Zerui Wang, Peng Sun, Yuzhe Gu, Wenwei Zhang, Ming- shu Zhai, Xingcheng Zhang, and Weiming Zhang. in the wild: Characterizing rlvr training in Ilm deployment, 2025. URL BRLhttps:: /htttss///ariiil org/ abs abs.abas....../...... 25279
14

---

Published as a conference paper at ICLR 2026
A SEQUENCE PLACEMENT ALGORITHM
Problem formulation. We formalize planning over D devices for global batch of N se- quences. Let each sequence E [I.....N} .N) have length h;- We choose a sharding degree | and an assignment Vi,d E {0, 1} indicating whether device d holds a shard of sequence such that Σ d= 4:,d = Ps. For executability, we restrict sharded sequences (p: 1) to hierarchical family of admissible device groups: for each degree the groups form a partition of GPUs (disjoint and covering), and partitions are nested across degrees.
E
Cost model. On each device d, we define (i) memory load M2 E Vi.a with a hard threshold Ma S Ma aM* (we use 1.1), where M is the average per-device token count; and (ii) compute load Ca x Σ (1 (p) vi
attention-cost proxy. Sharding also intro- duces communication overhead; following the standard Seq model, the per-device traffic volume scales vol(h, p) h (p 1)/p", optionally weighted by intra-/inter-node bandwidth. In Algorithm we use SplitCost(h, p) as a practical proxy for this overhead.
2
Objective. We solve lexicographic optimization:
min (p..tia)
max Ca.
Σ Comm(hs: pi)
(I)
where we primarily minimize the maximum compute imbalance and use the communication cost to break ties.
The detailed sequence placement algorithm is formal- ized in Algorithm Our algorithm incorporates threshold multiplier (1.1 for memory constraint Ms and searches over power-of-two candidate degrees.
FlexUlyssee 3-Stage Scheduling Heuristic.
Algorithm FlexUlysses Placement Algorithm
Require: sequence heights H = ( number of devices D
Ensure: placement D' {(i, d, h Pt)} and per-item split degree P {p.}
1: M + D Σm= h4, Mih + a."
Da= 1.1
2: Pmax < 2 PJ, Peand < {2 0 105 1: < loga Pmax
3: ()', m) <- BASELINESPLLPPLACCEEEC D, M Mih, Peand)
4: <- item states induced by y
Ds[ p= cuurrett split degree, sli).G device group
5: C -(2 1
loga Pmax) } in decreasing order
6: for all p C do
7: while 3 merge candidate at split level do
8: Cp FINDMERGECANNDDATEER ()), s. p, Meh, m)
SEEFAULTTMM.CANDIAATEE
9:
if C, then
10:
break
c* t arg mincec, (C)st(c)
12:
EXECUTEMERGE()) SHORTTITEMREORDERD) H. Mch)
14: return J/P
Split cost.
ps1 1
h(p-1) n2
SplitCost(hhp p)
2 T
with 8, 7 16.
h(p 1) p²
p>T
TotalCost(P))
(h..)Costthh..p)..
15

---

Published as a conference paper at ICLR 2026
Algorithm 3 Flexes Sub-procedures
1: procedure BASELNESPPLTLLAACCCMME... D, M Mth, Peand)
2: <- O, 4- 1.1M*
Davailable memory proxy
3: sort items by h2 descending
4: for i 1 to .N do
5:
+ GETSPSIZE(h;; Peand)
D-power-ofttwo
6:
while Pmax and VALIDGROUPEXISSS((. i, Ma.) false do - 2p
7:
8:
if VALLIGROUUEXIISSSSP.. m. Mi)) = false then
9:
p Pmax
10:
G + arg G. G|=p, vdEG:mat "j Saa Edea ad
11:
for all E do add tuple (i, d, hi bi to to
12:
13:
md - Mid + h
14:
ad 4 ad
Dproxy update
15:
return ), m
procedure FNDMEEGGCANDDDATSSSSS' S, Mth, m)
17: partition devices into consecutive groups Gp {G S [D] |G| =
C,
19:
for all G E Gp do La +- (i - =
21:
if Tal < 2 then
22:
continue
23:
for m' = 2 to do
24:
choose a multiset S S Ig with |S| = m' (greedy by increasing h2) (S1, S2) TWOWAYPACK(S)
25:
Dubbject to balance constraints
26:
S + 0 A S2 / A MERGEEEASIBLE(G, S1, S2, Maa) then
27:
Cp C, U (c= (G, S2,
28:
return Cp
29:
procedure EXECUTEMERGEC), s, S1,
30:
p' < p/2., split G into halves G()), ((2) with [G()] = | =
remove all shards of items S1 U S2 from devices in G for all d E (1)) do
32:
33:
for all
S do add (i, h//h////// 1//p/p') to
34:
for all d E (2) do
35:
for all S2 do add (i, h:/p', ,r' to y
36: for all E S₁ do s|i] + p/.
+ G()
37:
for all i S2 do / p/. s + ((2)
38: procedure SHORT|TEMMEORDERRI s, (hi),, Man
S {i: sli.pp .p S 2} sorted by decreasing h; 40: for all E S do
remove all tuples y with item s(i].p + s t-
42:
43:
for all do
d' <- arg mindm.tth,MM. (Eudd h -h))
44:
45:
if no feasible d' then d' + arg mina(ma + hz) add (i,d*, dr, h4, 1) to V: update m.g max hi
46:
47:
16

---

Published as a conference paper at ICLR 2026
B DETAILED END-TO-END RUNTIME BREAKDOWN
End-to-End Results.
CR
Model
GPUs
DS
Method
GBS
Frames
Throughput
DutaLooding(s) Rolloutts)
Oldis)
Refis)
Stepts)
Update(s)
CR-01
Image- Heavy
256
233.21
15.045
6.354
46,404
4.952
CR-01
7B
32
Image- Heavy
ulysses.sp
64
256
92.50
54.873
45.607
20 30.866
20.337
114,868
28.002
CR-01
7B
32
Image Heavy
verl bucketing
64
2556
53.18
55,572
46.447
50.763
49.881
200.155
53,006
CR-02
HidoyHHevy
64
128
313,99
11,838
29.641
18.318
8.108
63.934
7.777
CR-02
Video-Heavy
ulysses_sp
128
137.15
174811
28 197
27.751
33.
CR-02
7B
32
Heavy
berlbbcckttigg
61
128
58.65
179:157
71,829
97.575
95.036
363,035
98,510
CR-03
32
Onde-Video
64
525.93
6.732
19.384
3,979
3,908
32.307
4,909
CR-03
Only Video
ulysses sp
207.03
82,010
CR-03
78
32
Only-Video
verl bucketckt
64
63.96
181.610
52.853
70.287
69.811
265 265.406
72 386
CR-04
328
64
Imuge-eary
128
255
101.96
17.147
61.914
15.757
13,660
104, 506
13,053
32B
Image Heavy
bulkbucketing
CR-04
32B
Imnge-Hoa Heavy
war buccbuckoting
128
256
33.29
95.529
89.113
59 432
58. 244
272,536
63 640
CR-05
32B
64
Vidoo-Heavy
ouns
128
64
189.79
6.184
41.596
7.063
6,606
68.579
13.197
CR-05
338
Video-Heayy
ulysses_sp
67.29
230.772
91.813
32,929
52.871
199,
41.476
CR-05 328
Vido-HHavyy
verl bucketing
128
43.81
236.807
94.
68 385
67. 5711
305.111
74.634
CR-06 32B
64
Only-Vidkr
ours
128
96
221.07
9.856
63.503
17.763
9.567
113.518
22.712
CR-06 328
61
Only- Vidko
ulysses.sp
128
78.85
196,666
156 860
50. 50.508
49 09.079
318.196
61.579
32B
Only
bucketing
78.757
CR-10 78
64
Image-Heavy
ours
128
128
183.06
10.548
19,875
9.610
6.340
39.275
3,300
CR-10
64
Image- Heavy
ulysses.sp
128
128
61.09
97,044
50.916
24.396
24.136
126 604
27.065
CR-11
Heavy
ours
6.267
7,238
CR-11 7B
61
Vidco-Berry ulysses.sp .sp
128
128
74,93
369.151
123.124
58. 18.1335
58.773
303,776
63.573
CR-11 7B
64
Hedy
ulysses.sp
128
64
82.45
231,429
71.792
29.816
29.491
163.260
32.051
CR-12
Only-Video
443.34
6.761
23.774
4.438
4.329
38.909
5.635
CR-12 7B
Ouly-Vidoo
ulysses.sp
128
89.94
356.238
91.013
31.649
31,392
188.718
34.536
CR-12 7B
64
Only Video
verl_buckkttng
128
64
52.34
349.610
92. 12.
77.244
75.763
324.317
79.057
CR-13
128
Image Heavy
256
128
158.91
10.956
24.982
11.185
6.643
47.537
4.541
Image- Heavy
ulysses.gg sp
128
182.141
225.579
CR-13 78
128
Hewyy
verl bucketing
256
128
21.63
186.269
76.72
91.113
88.703
347.586
90
CR-15
128
Vidoo
256
405.44
6.973
24.783
5.229
4.921
41,885
6.692
CR-15
Ouly-Vidu
ulysses_ap
335.006
Table 2: Detailed E2E experiment results.
Sequence Packing Miero-Benchmark. To clarify how the runtime of FlashAttention variable- ing depends the sequence-length distribution, we length attention (flash attn.vvv len) on a single H800 GPU. We run GQA with q heads = 64, kv heads 8, and head dim 128. We keep the total number of tokens fixed 64K and vary how they are partitioned across sequences (Table 3).
pack-
Table 3:
runtime (ms) under different sequence-length distributions with the same total tokens (64K).
Input sequences
1x64K
1x 16K
8x8K
16x4K
32
64 IK
Runtime (ms)
182.9
46.6
24.0
12.7
7.0
4.3
C ADDITIONAL COMMUNICATION OVERHEAD RESULTS
SeqAlltoAll Minro-Benchmark. We micro-benchmark FlashAttention2 GQA SeqAlltAAl com- munication time two clusters with different bandwidth characteristics: H200 (900GB/s intra- node, 3200Gb/s inter-node) and H800 (400GB/s intra-node, 1600Gb/s inter-node). We use GQA with head q=32, head kv=8, and head. diiddddim.....m and report the average AlltoAll latency (ms) under different sharding degrees (sp_size) and sequence lengths (seq_len). Table 4 shows a representative case with seq_len=262 144 tokens, while Table 5 the full sweep.
17

---

Published as a conference paper at ICLR 2026
Table 4: SeqAlltoAll micro-henchmark communication latency (ms) in FlashAttention2 GQA.
H200 Cluster
H800 Cluster
sp_size
seq_ien
Comm_avg.nm
sp_size
seq_een
comm_avg.m
2621
23.401
262144
18.245
2621
16.107
2621
16.480
8
2621 144
8,513
262
12.543
26214 144
14.639
16
262144
19.644
32
262144
11.030
32
262144
16.558
Table 5: Full SeqAlltoAll micro-benchmark communication time (ms) on H200 and H800 clusters.
H1200 Cluster
11500 Cluster
seq len
seq len
262144
23.401
262144
18.245
524288
52.172
524288
60.709
131072
6.547
131072
6.943
524288
17.36]
524288
44.372
131072
1.501
131072
6.360
262144
8.513
262144
12.543
104K576
64.878
104K576
193
16
131072
7.871
16
131072
12.355
262144
14,639
262144
19.644
131072
6.155
131072
17.219
262144
11.000
262144
16,558
24.150
18


Reviews:
[{"review_id": "yQk9s5JhlK", "reviewer": "Reviewer_apGj", "summary": "FlexRL is an end-to-end optimization system built on the verl framework to improve the efficiency of RL training for MLLMs. It addresses two primary bottlenecks: (1) a Decentralized Data Pipeline that distributes multimodal data loading and preprocessing across worker nodes while the control node handles only lightweight metadata, eliminating centralized I/O bottlenecks; and (2) a Hybrid Sequence Sharding mechanism that partitions sequences into fine-grained chunks to achieve subsequence-level load balancing, mitigating uneven GPU utilization caused by extreme length disparities across modalities such as text, images, and video.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper systematically analyzes practical bottlenecks across the entire RL training pipeline for MLLMs rather than focusing on a single stage, and it demonstrates strong system-level completeness.", "weaknesses": "I must first note that I am not very familiar with mlsys, while I only offer a limited perspective on this paper.\n\n1. Why not compare against other verl-based optimized frameworks, such as [1], which also targets long-video scenarios?\n2. The paper lacks concrete ablations; for example, it does not separately quantify the contributions of the Decentralized Data Pipeline and the Hybrid Sequence Sharding components.\n3. Is the framework primarily intended for highly imbalanced workloads? The design appears to degenerate to conventional parallelism, but comparisons under balanced workloads (e.g., image-only or pure-text) are missing.\n4. Performance under different settings is not reported, e.g., varying batch size and tensor/pipeline/sequence parallelism (TP/PP/SP) sizes.\n5. As an mlsys work, the paper does not provide an engineering code release or community usage feedback, which I consider a weakness.\n\n[1] Scaling RL to Long Videos. NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vn3XbWRZcf", "reviewer": "Reviewer_cr9u", "summary": "This paper presents a distributed training pipeline for RL training stage of large-scale VLMs involving multimodal data (images, videos, texts). The key challenge in the multimodal RL training is the highly diverse data length (short text, long text, long image&video tokens), which is hard to be properly scheduled in a distributed training system. It mainly proposes a decentralized data pipeline to properly schedule the data with a single controller, and a hybrid sequence sharding technique to partition sequences into finegrained chunks to enable sub-sequence level load balancing. Existing technique Ulysses Sequence Parallelism is used to enable sequence parallel training.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed hybrid sharding technique is novel and alleviate the issue of imbalanced loading.\n-\tExperiments show the proposed approach outperform the speed of existing approach such as verl on video understanding tasks.", "weaknesses": "-\tThe speed of running a batch on a gpu should be clarified more. How does the gpu handles sequence with different length in a batch? From figure2, a gpu will pack samples with different lengths into groups and conduct their attention operation separately. While in some implementations using generic sequence packing and masked attention, the running time is irrelevant to the sequence length of each sample since a global masked attention of all tokens in a batch is conducted. How much speed up does the proposed approach have compared to the global-attention method?\n-\tThe paper only provides numbers for speed, while accuracy numbers are not provided. The accuracies of using the proposed approach is also required to show the method’s robustness for different applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n1vX2WC8C7", "reviewer": "Reviewer_uQAx", "summary": "This paper presents FlexRL, a system designed to address data loading bottlenecks and computational load imbalance in RL training for large multimodal models. Its core contributions are a decentralized data pipeline and a novel hybrid sequence sharding mechanism. The authors claim up to 4.2× speedup on a 128-GPU cluster.", "questions": "Please refer to Weaknesses. Btw, I have some optional questions:\n\n- [Optional] Dynamic sequence packing techniques, which concatenate short sequences into longer ones during training, have emerged recently. How does FlexRL fundamentally compare to such methods in terms of load balance efficiency and memory utilization? What are the relative advantages and disadvantages?\n\n- [Optional] The title claims \"Universal,\" but the method heavily relies on All-to-All communication within the attention mechanism. If future VLM architectures shift towards SSMs or other non-attention-based mechanisms, would FlexRL's core sharding mechanism remain effective? What is your view on this architectural dependency risk?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Accurately identifies key systemic bottlenecks in multimodal RL training.\n\n- The proposed hybrid sequence sharding mechanism is ingenious and offers a promising direction for handling workload skew from heterogeneous sequences.\n\n- The design of a dynamic execution engine to manage the scheduling complexity introduced by hybrid sharding is a significant engineering step.", "weaknesses": "- [Mandatory] There are minor typos, formatting inconsistencies, and grammatical errors. The authors should carefully proofread the manuscript.\n\n- [Mandatory] The paper does not separately evaluate the individual contributions of the two core components: the \"Decentralized Data Pipeline\" and the \"Hybrid Sequence Sharding.\" It is unclear which component drives the performance gains.\n\n- [Mandatory] The hybrid sharding introduces complex All-to-All communication. While overlap strategies are qualitatively mentioned, a quantitative analysis of the communication overhead's impact on end-to-end performance under different cluster scales and network topologies is missing.\n\n- [Mandatory] Key details regarding the scheduling heuristic and cost estimation model are missing, hindering reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)